job manager fails Answered

Post author
Ilya Shlyakhter

Job manager seems to always fail when trying to view this job:

https://job-manager.dsde-prod.broadinstitute.org/jobs/fb0ae9e2-8968-4514-8b95-de28faceb213

saying "

Job Manager is running but encountered a problem getting data from its workflow server.

Click here to start over.

500: Internal Server Error"

 

This might be due to the size of a scatter -- is a scatter of size 640 too large?

 

Thanks,

 

ilya

 

 

Comments

1 comment

  • Comment author
    Jason Cerrato

    Hi Ilya,

    This is a known issue with workflows that produce a particularly large amount of metadata. The scatter size may not inherently result in this outcome, but the amount of metadata produced by large scatters can definitely lead to Job Manager having trouble returning the results. You have a couple of options for getting information for this job:

    • Use the Workflow Dashboard, aka the middle icon in the Links section of the Job History page for this submission. This will give you information about the success and failure of the workflow tasks, and which WDL was used to submit the workflow.
    • Navigate to your workspace in https://portal.firecloud.org/# and find your job of interest under the Monitor tab. The old FireCloud UI is a little more robust in its ability to serving up large amounts of metadata.

    Our engineers are making improvements to the manner in which metadata is served, so that Job Manager can process the metadata for large jobs such as these.

    Let me know if either of those two options gives you the visibility you need.

    Kind regards,

    Jason

    0

Please sign in to leave a comment.