Is there a way to incorporate "gsutil -m" for the localization of the input files for a Terra method?
Hi, I was reviewing a log from a method I ran which takes in array file input, and I saw a following message in the log
- NOTE: You are performing a sequence of gsutil operations that may run significantly faster if you instead use gsutil -m cp ... Please see the -m section under "gsutil help options" for further information about when gsutil -m can be advantageous
So I was wondering if there is a way to enable the use of gsutil -m feature during the localization of the input files when running Terra method. I think it would greatly save time and cost especially in case when dealing with hundreds of files in large size
Comments
2 comments
Hi Seunghun Han,
Thank you for writing in about this issue. Can you share the workspace where you are seeing this issue with Terra-Support@firecloud.org by clicking the Share button in your workspace? The Share option is in the three-dots menu at the top-right.
Please provide us with
Please provide us with a link to your workspace. We’ll be happy to take a closer look as soon as we can!
Kind regards,
Samantha
I see this in my WDL workflows, as well, any time I have an Array[File] with several large files.
The -m flag to gsutil just allows parallel operation of copies (https://cloud.google.com/storage/docs/gsutil/addlhelp/GlobalCommandLineOptions) and I think could be made the default for all localization/delocalization calls.
Please sign in to leave a comment.