Workflow submissions not finishing

Post author
Daniel Boiarsky

Hi,

 

I have this workflow https://portal.firecloud.org/?return=firecloud#methods/neoantigen_prediction/run_prediction/27 in vanallen-firecloud-dfci/CR_NV_WES_IO_Meta and it seem like some of my submissions stop processing but when I abort the workflow and then try again they run. Can you please let me know why some of these workflows seem to just hang?

Here is a current submission: https://firecloud.terra.bio/#workspaces/vanallen-firecloud-dfci/CR_NV_WES_IO_Meta/job_history/c298a9fd-c7ec-47f4-a347-6e7c883d9f5a

and here is a prior submission https://firecloud.terra.bio/#workspaces/vanallen-firecloud-dfci/CR_NV_WES_IO_Meta/job_history/b9e3adce-f4aa-44fa-8ba0-b790b3460f0e

Note, for example, how for data entity P290_T_N, it didn't finish running after ~6 days in the prior submission but finished within a day in the current submission.

Please let me know if theres anything I can do to resolve this.

Thank you!

Comments

2 comments

  • Comment author
    Beri

    Hi Daniel,

    We'll have a look on our end and get back to you.

    0
  • Comment author
    Beri
    • Edited

    Hi Daniel, 

    I asked the team and here is their suggestion: 

    According to the log files it looks like the neoantigenPredictionType1 task froze in the middle of the script execution, somewhere at the moment where it should have completed  Making binding predictions and switch to Parsing binding predictions. It doesn’t seem like the problem is on Terra side. Try ensuring the script or tool being used runs to completion, for instance check if there are any hidden infinite loops, waiting on conditions which never happen or anything similar. 

    0

Please sign in to leave a comment.