Job status is slow to update
I am posting here since this is an issue I see in both Terra and FireCloud. After yesterday's maintenance window, I am seeing instances when the job completes hours before the status in FireCloud/Terra changes to "succeeded". For example, in the first two snapshots, you can see that the job finished on April 11, but it did not change to "Done" until 7:04 the next morning. This is particularly frustrating as I am launching sample sets of about 1000 at a time, and I cannot go back to each individual task to check if it actually completed ahead of the UI update. It would be good if this could be fixed soon.
Best,
Matt
Comments
10 comments
I'm also experiencing this issue.
Hello Justin and Matt -
I am going to pass this information to the team and get back to you with an update.
Update:
We are investigating an issue that could delay workflow status updates. Based on the nature of the issue, we believe it should resolve itself and are working on an ETA for when that is likely to be. I will continue to update this thread with more information as I receive it!
Thanks for reporting!
Hello -
The metadata backlog issue seems to be resolved and all the details should now be updated. Apologies for any inconvenience as the team worked to resolve the problem!
Jobs are updating properly now, thank you for your help!
Hello,
I am running into the same issue described above where my jobs seem to have succeeded when I navigate to the specific job, but the status still shows as "running". The workflow this is happening on is STAR (https://portal.firecloud.org/?return=terra#methods/broadinstitute_gtex/star_v1-0_BETA/7). Any ideas/suggestions on how to fix this? Thank you!
Hi Sonia Moreno,
Thank you for reporting this issue. Can you share the workspace where you are seeing this issue with GROUP_FireCloud-Support@firecloud.org by clicking the Share button in your workspace (see the icon with the three dots at the top-right)?
1. Add GROUP_FireCloud-Support@firecloud.org to the User email field and press enter.
3. Click Save
Let us know the workspace name, as well as the relevant submission ID where you are seeing this issue. We'll be happy to take a closer look.
Kind regards,
Jason
Hi Jason,
I just shared the workspace named ccRCC_scRNAseq_WESandRNA_ALL-SAMPLES. The submission ID is d4cc5353-95e3-4146-8055-e3def92bd6bc.
Thanks!
Sonia
Hi Sonia,
Thanks for that! I'll take a look and get back to you as soon as I can.
Kind regards,
Jason
Hi Sonia,
We were able to track down this mismatch being due to a bug in the service that updates the status for the overall submission. A fix for the bug is currently being developed.
Please note that your submission did indeed complete—the bug is associated with the displayed status. No extra costs are being accrued due to this error.
Many thanks for reporting this! If you have any questions, please let me know.
Kind regards,
Jason
Please sign in to leave a comment.