Workflow hung writing to data model
I have a workflow that finished a couple days ago based on the existence of outputs in the execution directory, but it still hasn't added those outputs to the data model. This is submission 419778fb-30f9-4507-b39e-45b6406e3731 in broad-firecloud-dsde-methods/gCNV-CMG-test, which is already shared with the support account. I could probably update the data model manually with a TSV, but I don't want to have to do that every time. Is a sample set of 1500 samples too big? It's certainly too big for Job Manager to retrieve the metadata.
-
Hi Laura,
The issues you're experiencing with workflows
- d71cb42e-599d-47c0-a4f5-13574b780817 (https://support.terra.bio/hc/en-us/community/posts/360073751431-Terra-API-workaround-)
- 60785a70-2aac-4985-ae11-af380cf1143e (https://support.terra.bio/hc/en-us/community/posts/360073877991-Workflow-aborting-for-6-days)
- 486197c8-6d1a-4574-acfe-64d6815a2556 (this thread)
are all due to the same Cromwell carboniting issue. Cromwell has carbonited some metadata and uploaded it to GCS, but it is now refusing to download the metadata which it carbonited due to its size. A member of the Batch team is making an adjustment on all of these workflows to force Cromwell to read from the database again instead of failing to read from GCS.
I'll let you know once they make the change and of any other details you should know about any of these situations.
Kind regards,
Jason
-
Hey Laura,
I've been informed that the adjustments have been made, and I see that one of your submissions have successfully finished aborting, and the other is now Done. I also see that you are now able to see the job details for 3fb30759-583c-493f-88fb-4273b487a4a4.
Let us know if everything is looking good on your end! If you have any other troubles or concerns, please let us know and we'll be happy to look into it.
Kind regards,
Jason
-
Hey Laura,
This is an oversight currently being worked on by the Cromwell team. Even though they are already working on resolving this, we recommend that you definitely continue to let us know if any other workflows end up in this state both so we can resolve the error state and so that the team has more data to work with for crafting the fix.
Kind regards,
Jason
Please sign in to leave a comment.
Comments
5 comments