Is there a way to incorporate "gsutil -m" for the localization of the input files for a Terra method?

Post author
Seunghun Han

Hi, I was reviewing a log from a method I ran which takes in array file input, and I saw a following message in the log

- NOTE: You are performing a sequence of gsutil operations that may run significantly faster if you instead use gsutil -m cp ... Please see the -m section under "gsutil help options" for further information about when gsutil -m can be advantageous

So I was wondering if there is a way to enable the use of gsutil -m feature during the localization of the input files when running Terra method. I think it would greatly save time and cost especially in case when dealing with hundreds of files in large size

 

 

 

 

Comments

2 comments

  • Comment author
    Samantha (she/her)

    Hi Seunghun Han,

    Thank you for writing in about this issue. Can you share the workspace where you are seeing this issue with Terra-Support@firecloud.org by clicking the Share button in your workspace? The Share option is in the three-dots menu at the top-right.

    1. Add Terra-Support@firecloud.org to the User email field and press Enter or click the Add button
    2. Click Save.

    Please provide us with

    1. A link to your workspace
    2. The relevant submission ID
    3. The relevant workflow ID

    Please provide us with a link to your workspace. We’ll be happy to take a closer look as soon as we can!

    Kind regards,

    Samantha​ 

    0
  • Comment author
    Ray Jones

    I see this in my WDL workflows, as well, any time I have an Array[File] with several large files.  

    The -m flag to gsutil just allows parallel operation of copies (https://cloud.google.com/storage/docs/gsutil/addlhelp/GlobalCommandLineOptions) and I think could be made the default for all localization/delocalization calls.

    0

Please sign in to leave a comment.