Workflows not changing from "Submitted" Completed
Hi Terra Support Team,
Members of our team have noticed that several of our workflows in the last few days remain at the "Submitted" state (with the clock icon) for an unusually long time.
Upon re-launching the exact same workflow (with identical parameters), these second workflows (some even submitted hours later) begin running within minutes.
Is there a reason for this odd behavior? Prior posts on this forum from 2 years ago suggest the cause might have been a service incident. They appear to have been quickly resolved. Is this a similar situation?
Thanks,
Vrishab
Comments
6 comments
We have resolved the underlying issue, and workflows should now be submitting more quickly.
Hi Vrishab,
Thank you for writing in about this issue. Can you share the workspace where you are seeing this issue with Terra Support by clicking the Share button in your workspace? The Share option is in the three-dots menu at the top-right.
Please provide us with
We’ll be happy to take a closer look as soon as we can!
Kind regards,
Samantha
Hello Samantha,
This issue has also affected my workflows since the recent March Maintenance (came to my notice after this). The issue seems to happen at random because only a subset of jobs running through the same workflow get halted. For example, if I am processing 10 samples individually, then maybe 3-5 sample analyses get prolonged. The aborting also isn't finished when triggered. We have to re-run samples separately a few times for normal execution. Also, this has happened with all workflows that were running smoothly before the maintenance.
I would greatly appreciate your support and any updates regarding the resolution.
Hi Terra Team,
I am also experiencing this exact issue.
Workflow ID: 87277fbd-c0e3-46c3-9f16-b5c928d7e75b
Submission ID: d46ab1ae-cdf6-469b-a11b-ac36d008775b
Workspace: TyMillerLab
Thank you!
Hi all,
We are investigating a probable root cause for the long submission time issue, and we will update this thread once we believe the issue is resolved.
In the meantime, workflow submissions should reach a running state eventually if left untouched. You can also try resubmitting the job if you urgently need it to start running, as this has reportedly worked for some users.
Kind regards,
Jason
I have hit this bug almost fifty different times in the last week. I can now reliably recreate it.
It only happens if I update the settings for a workflow. For a period of about two to five minutes, new workflows launched immediately after saving the workflow settings will fail to properly schedule and become wedged.
After updating a workflow, the first and possibly second launch will likely wedge. Once a relaunch succeeds, any subsequent launch is successful assuming the underlying workflow parameters did not change. It's almost as if the scheduling system knows the parameter cache is dirty, but the changes have not yet propagated. In most of these wedged workflows, call caching is not enabled. Once I hand verify a wedged workflow, I abort it by hand. I have never seen a wedged workflow come back to life.
Conversely, the longer I wait after saving new workflow settings, the more likely my workflow will launch successfully upon the first try.
Please sign in to leave a comment.