Whole-Genome-Analysis-Pipeline failing in ImportGVCFs step
Hi,
I am running the Whole-Genome-Analysis-Pipeline, and the workflow keeps failing at the JointGenotyping.ImportGVCFs step (PAPI error code 10 or 9). I've ran this workflow before on 531 samples, and after I added 211 more (in the same bucket), I can't get joint genotyping to work successfully (note: all these gvcfs were preprocessed using GATK locally/outside of terra and I just uploaded the reblocked gvcfs to the cloud with intent of joint genotyping). Using the same workspace, I increased the disk space multiple times, but it has not resolved the problem. Unfortunately the log files aren't too informative (either just PAPI code 10 or code 9 with a truncated log output for ImportGVCFs). I also saw that there was a new version of the workspace released recently, so I cloned that workspace and then restarted the job, but still the problem reoccurs. Interestingly, the JointGenotyping.ImportGVCFs step seems to fail in shard 200 everytime.
Here are the relevant Submission IDs/Workflow IDs on workspace https://app.terra.bio/#workspaces/mgb-UO1_Ozelius/Whole-Genome-Analysis-Pipeline%20copy
b4849176-86c3-4a2a-bf30-19f80fbd0c7a Workflow ID90580e00-92f2-4537-85fc-b0032b2a2594
71081897-76de-4355-81c7-bd0674fc2a2b Workflow ID ef5779cf-8a71-4516-a8a9-ae4f4ab0a465
d512969d-a33c-457a-953c-f660c6e58d47 Workflow ID 52affaf4-4ba5-4ee6-aaf5-74173c0d5431
New workspace: https://app.terra.bio/#workspaces/mgb-UO1_Ozelius/Whole-Genome-Analysis-Pipeline%20copy%205-6-2023
Submission ID 8b17ac66-6ed9-44bb-acaf-0d9dcef7140e Workflow ID 69c94265-64fe-46e9-8057-462db612d32f
Thank you. Please let me know if you have insights on why this workflow keeps failing.
Comments
6 comments
Hi Aloysius,
Thank you for writing in about this issue. Can you share the workspaces where you are seeing this issue with Terra Support by clicking the Share button in your workspace? The Share option is in the three-dots menu at the top-right.
We’ll be happy to take a closer look as soon as we can!
Kind regards,
Samantha
Thank you! I've just shared both workspaces to support!
Hi Samantha (she/her), do you already have some insights on why this workflow has been failing? Thank you!
Hi Aloysius Domingo,
Sorry for the delay. Unfortunately, I do not have access to the workspace yet. It looks like it is protected by an authorization domain named "mgb-UO1_Ozelius." Would you be able to add my account (svelasqu@broadinstitute.org) to that group so I can access the workspace?
Best,
Samantha
Hi Samantha (she/her), I've just added you to the group. Please let me know if there is anything else I need to do!
Hi Aloysius,
Based on the errors you are receiving, it looks like your workflow is failing due to insufficient memory. Both PAPI error code 9 and PAPI error code 10 are usually indicative of this - you can read more about the errors in these articles:
https://support.terra.bio/hc/en-us/articles/360039299991
https://support.terra.bio/hc/en-us/articles/360039010292
Unfortunately, the featured workflow does not have an input option for a user to increase the runtime memory (the values are hardcoded into the WDL). I've let the pipeline developers know and they've created a ticket on their end to add the option in the next workflow version. As a workaround in the meantime, you can either create your own version of the workflow and hardcode higher memory value, or you can try resubmitting the workflow with a smaller sample set.
Please let me know if you have any other questions.
Best,
Samantha
Please sign in to leave a comment.