unable to view status of running job
I am unable to retrieve job status of a recently submitted workflow.
When I attempt to view the workflow's status, I receive the following message:
Job Manager is running but encountered a problem getting data from its workflow server.
500: Internal Server Error
Note there is no problem viewing the status of previously submitted workflows, regardless of whether is was successful or not.
Also, I am able to view workflow status from the old FireCloud UI.
https://app.terra.bio/#workspaces/broad-firecloud-cptac/CBB_20191122_hg38_wes_char_pipeline_LUAD
I have granted read access to GROUP_FireCloud-Support@firecloud.org
-Chet
Comments
13 comments
Hi Chet,
Thanks for writing in about this. This may be related to the issues currently being on worked on: https://support.terra.bio/hc/en-us/community/posts/360058314871-Queued-submissions-and-inaccurate-job-status
I will be happy to follow-up with you once the hotfix goes out to see if you are still experiencing the error. If so, we will take a closer look.
Kind regards,
Jason
Thanks Jason. Not a high-priority issue as I am able to obtain job status through the FireCloud GUI.
-Chet
Hi Chet,
Are you still seeing this issue at this time? If so, please share the workspace with GROUP_FireCloud-Support@firecloud.org along with the relevant submission/workflow IDs and we'll be happy to take a look.
Kind regards,
Jason
I am able to see job status now. Thanks for following up!
-Chet
Hi Chet,
No problem! If you run into any further issues, please let me know.
Kind regards,
Jason
I am having the same issue:
Submission ID: 84b157be-a26e-40e6-9f6e-5af12b39976f
Workflow ID: 5029f6c6-2069-449d-9a70-38afd5450366
I was able to monitor the job until it failed. I know it failed because I was able to find the failure in one of the stderr files. However, I am unable to retrieve the status of the tasks through the Job Manager.
Hi Giulio Genovese,
Thanks for writing in. This error is often symptomatic of the job's metadata being very large. At this time, FireCloud's job monitor is a bit more robust. Can you see the details of your job by going to your workspace in the FireCloud interface? You can get there by either navigating to https://portal.firecloud.org/ and searching for your workspace or by changing the URL of your workspace.
For example if your Terra workspace URL is https://app.terra.bio/#workspaces/example-billing/example_workspace
you would change it to http://portal.firecloud.org/#workspaces/example-billing/example_workspace
Let me know if you are able to view the job details in the Monitor tab. A more robust replacement for the Terra Job Manager is currently in discussion.
Kind regards,
Jason
I don't understand. I went to this page:
Then had to switch to the legacy interface (which is so much better than Terra interface, holy cow!).
I then found my job:
I clicked on View and I got this:
Now, it is indeed the case that the job has very large metadata. This was a job run on 27,778 samples at once and it does generate a few vectors that long ... But I actually am trying to scale this to even larger sample sizes, like ~500,000. What am I supposed to do? How do I learn about all these limitations?
Giulio
Can I have some more information about what job metadata I should try to minimize? Is it the metadata passed between workflow and tasks that is overwhelming the job manager? I can take some steps to minimize that. However, if this is the case, then the issue is compounded by the fact that some objects cannot be serialized by Cromwell and I am forced to input them into tasks as such rather than as files. Again, what are the limitations of the job manager?
Hi Giulio,
Job metadata is constructed through a combination of the overall number of jobs/tasks, the number of inputs/outputs, and more. Workflows with many calls inside the scatters generate a lot of metadata. One good way to reduce the number of metadata rows would be to redesign the workflow to reduce number of scatters and/or calls (e.g., split it in several independent workflows, rather than running as subworkflows of a single workflow). You can also look to separate the tasks to be separate workflows that feed into one another, where the output of one becomes the input of another.
Here are a couple of featured workspaces that break up the work between multiple workflows you can use as reference:
https://app.terra.bio/#workspaces/fc-product-demo/Terra-Workflows-Quickstart
https://app.terra.bio/#workspaces/help-gatk/GATK4-Germline-Preprocessing-VariantCalling-JointCalling
By the way, this metadata limit is a known issue currently being worked on, and once the configuration changes are put in place to allow for larger amounts of metadata, it's possible that you'll no longer face difficulty getting these job details from Terra or FireCloud. I'm happy to follow-up with you once I get word of these configuration changes.
I'll also reach out to some colleagues to see what additional advice and support we can offer in the meantime!
Kind regards,
Jason
Hi Giulio,
I've discussed your idea for reducing the amount of metadata with my colleagues.
Right now I send lots of very long Array[String] arrays with 1000s+ of elements to different tasks ... if I change that to sending files with 1000s+ of lines, would this possibly solve the issue with the Job Manager?
They believe this would show certain improvement in the amount of metadata generated. I think this is definitely worth a shot to see if it gets you under the threshold so you can view your details in Terra or FireCloud.
Kind regards,
Jason
Just wanted to bring some closure. Rewriting the WDL to minimize the amount of metadata passed to tasks by passing files rather then Array[String] objects whenever possible did the trick and I was able to monitor the job until its completion:
My workflow does require to pass long arrays of files from one task to another though and I assume that there is no way to avoid those long Array[File] vectors to come to the attention of the Job Manager. I could very well tell that, as my workflow progressed, it became slower and slower for the Job Manager to retrieve the page. I am therefore a little worried that my solution including rewriting the WDL to minimize the metadata might be only a temporary solution and once I scale the WDL to 100k+ samples I might hit the same issue again. It would be great if a workaround on the Job Manager side could be found in the long term. In the meanwhile thank you for the help! :-)
Hi Giulio,
Thank you for following up on this! I'm happy to hear about the improvements in performance. The Cromwell team is continuing to investigate solutions to the large metadata issue, so stay tuned!
Kind regards,
Jason
Please sign in to leave a comment.