Is there a way to incorporate "gsutil -m" for the localization of the input files for a Terra method?
Hi, I was reviewing a log from a method I ran which takes in array file input, and I saw a following message in the log
- NOTE: You are performing a sequence of gsutil operations that may run significantly faster if you instead use gsutil -m cp ... Please see the -m section under "gsutil help options" for further information about when gsutil -m can be advantageous
So I was wondering if there is a way to enable the use of gsutil -m feature during the localization of the input files when running Terra method. I think it would greatly save time and cost especially in case when dealing with hundreds of files in large size
Hi Seunghun Han,
Thank you for writing in about this issue. Can you share the workspace where you are seeing this issue with Terra-Support@firecloud.org by clicking the Share button in your workspace? The Share option is in the three-dots menu at the top-right.
Please provide us with
Please provide us with a link to your workspace. We’ll be happy to take a closer look as soon as we can!
I see this in my WDL workflows, as well, any time I have an Array[File] with several large files.
The -m flag to gsutil just allows parallel operation of copies (https://cloud.google.com/storage/docs/gsutil/addlhelp/GlobalCommandLineOptions) and I think could be made the default for all localization/delocalization calls.
Please sign in to leave a comment.