Best way to specify a directory input?

Follow

Bryan Cosca

• October 30, 2020 20:54

Hi!

I was wondering what's the best way of specifying a directory input of files in a task. I see a pull request with it being implemented but I don't see it in the spec documentation. I can think of a couple ways, using structs or tarring the directory and unzipping it at runtime. Is there another way?

Thanks,

Bryan

0

• 

  Tiffany Miller

  • November 02, 2020 16:18

  Hi Bryan,

  Yes, those are definitely ways. Structs are used for bundling groups of variables into a single variable so that it's easier to pass them around as task inputs. Another option is creating a list of file paths and passing it to the task to be localized by gsutil as described here and in the spec here.
  Keep in mind that using these options will require the whole files to be localized instead of being streamed to the task. If your tool(s) are able to stream the files, that would be a good option as well. 

  Many thanks,
  Tiffany

  0

  Comment actions Permalink

Please sign in to leave a comment.