Troubleshooting Workflows: Fails for unknown reason (memory failure)
What this looks like
Usually out of memory failures looks like the workflow is working, working, working, working... until suddenly it just stops. There isn't any message in the stdout or stderr that says “completed successfully. starting delocalization.” Sometimes there aren’t even any error logs printed because there isn't enough memory to create them.
How to solve/test the problem
Increase memory in the runtime block for the failing task. You can bump it high to see if this is the issue, then lower to a reasonable amount later.
Comments
2 comments
Allie Hajian How do I "Increase memory in the runtime block for the failing task."? I'm running the https://github.com/gatk-workflows/gatk4-exome-analysis-pipeline workflow, and it keeps failing at the same place each time I run it. The log file reports: "Job Failed with Error Code 10 for a machine where Preemptible is set to false." I suspect this is caused by low memory allocation or low disk space, but I do not know how to increase either of those. Can you please point me to some documentation so I can learn how to control those parameters.Thanks in advance for you help.
Hi Roger Dettloff,
As mentioned in the Error message: PAPI error code 10 comments, you can post your concern to this forum board if the values are hard-coded in and we'll be happy to investigate: https://support.terra.bio/hc/en-us/community/topics/360001603491-Featured-Workspaces
If you wanted to try editing WDLs yourself, you can download or copy-paste a WDL's contents to a text/code editor of choice, and manipulate it per the WDL spec. We also have intro WDL documentation you can look through, though please note that some of the command examples are outdated. Replacement documentation is being produced, and the concepts are still valid.
Kind regards,
Jason
Please sign in to leave a comment.