Workflow hung writing to data model

Follow

Laura Gauthier

• October 19, 2020 13:50

I have a workflow that finished a couple days ago based on the existence of outputs in the execution directory, but it still hasn't added those outputs to the data model.  This is submission 419778fb-30f9-4507-b39e-45b6406e3731 in broad-firecloud-dsde-methods/gCNV-CMG-test, which is already shared with the support account.  I could probably update the data model manually with a TSV, but I don't want to have to do that every time.  Is a sample set of 1500 samples too big?  It's certainly too big for Job Manager to retrieve the metadata.

 

 

0

• 

  Jason Cerrato

  • October 19, 2020 16:42

  Hi Laura,

  I'll check on the status of this and get back to you as soon as I can.

  Kind regards,

  Jason

  0

  Comment actions Permalink
• 

  Jason Cerrato

  • October 19, 2020 21:11

  Hi Laura,

  The issues you're experiencing with workflows

  • d71cb42e-599d-47c0-a4f5-13574b780817 (https://support.terra.bio/hc/en-us/community/posts/360073751431-Terra-API-workaround-)
  • 60785a70-2aac-4985-ae11-af380cf1143e (https://support.terra.bio/hc/en-us/community/posts/360073877991-Workflow-aborting-for-6-days)
  • 486197c8-6d1a-4574-acfe-64d6815a2556 (this thread)

  are all due to the same Cromwell carboniting issue. Cromwell has carbonited some metadata and uploaded it to GCS, but it is now refusing to download the metadata which it carbonited due to its size. A member of the Batch team is making an adjustment on all of these workflows to force Cromwell to read from the database again instead of failing to read from GCS.

  I'll let you know once they make the change and of any other details you should know about any of these situations.

  Kind regards,

  Jason

  0

  Comment actions Permalink
• 

  Jason Cerrato

  • October 20, 2020 13:39

  Hey Laura,

  I've been informed that the adjustments have been made, and I see that one of your submissions have successfully finished aborting, and the other is now Done. I also see that you are now able to see the job details for 3fb30759-583c-493f-88fb-4273b487a4a4. 

  Let us know if everything is looking good on your end! If you have any other troubles or concerns, please let us know and we'll be happy to look into it.

  Kind regards,

  Jason

  0

  Comment actions Permalink
• 

  Laura Gauthier

  • October 20, 2020 14:16

  Yes, everything looks great!  Data model has data in it and I was able to access it.  Thanks for all your help.

  Is there anything I need to know for the future about this "carboniting" process or was this just an oversight in a new process?

  0

  Comment actions Permalink
• 

  Jason Cerrato

  • October 20, 2020 19:39

  Hey Laura,

  This is an oversight currently being worked on by the Cromwell team. Even though they are already working on resolving this, we recommend that you definitely continue to let us know if any other workflows end up in this state both so we can resolve the error state and so that the team has more data to work with for crafting the fix.

  Kind regards,

  Jason

  0

  Comment actions Permalink

Please sign in to leave a comment.