Job scheduling at 4 days, total submission time limit of 7 days

Post author
tmajarian

Hi-

 

I have some very long running jobs (~5 days) that I have been trying to run for a few weeks now. I am seeing some odd behavior in Terra where there is a delay of 3+ days from submission to start. The real issue is the 7 day built-in time limit on Terra jobs; it seems to count from submission, rather than job start, leaving me with <4 days for a given job. This ends up wasting a lot of money as the jobs all fail due to this time limit, even though the GCS imposed limit is much longer than a week.

 

Is there any way to get around this? Either the submission-to-start lag or the time limit?

 

I really don't understand why it only takes ~5 minutes to provision a GCS VM through Terra, but 3 days for a batch job. I get that they may be using different google services, but 3 days just seems astounding for cloud computing.

 

Any advice would be great! Thanks!

-Tim

Comments

2 comments

  • Comment author
    Samantha (she/her)

    Hi tmajarian,

     

    Thank you for writing in about this issue. Can you share the workspace where you are seeing this issue with GROUP_FireCloud-Support@firecloud.org by clicking the Share button in your workspace? The Share option is in the three-dots menu at the top-right.

    1. Add GROUP_FireCloud-Support@firecloud.org to the User email field and press enter on your keyboard.
    2. Click Save.

     

    Please provide us with

    1. A link to your workspace
    2. The relevant submission ID
    3. The relevant workflow ID

     

    We’ll be happy to take a closer look as soon as we can!

     

    Kind regards,

    Samantha​ 

    0
  • Comment author
    Adam Nichols

    Hi tmajarian,

    Terra engineer here. How long do you estimate these tasks would take, if allowed to run beyond 7 days?

    Thanks,

    Adam

    0

Please sign in to leave a comment.