PAPI error code 9 Answered
Hi Terra,
A method that worked previously on FireCloud/Terra and currently works locally now produced the error code "PAPI error code 9". The error code is non-descriptive and the logs do not produce much insight. Could you provide any details?
The job seems to just die without additional text, as the task.log states
/cromwell_root/script: line 32: 14 Killed
Stderr reiterates this statement
/cromwell_root/script: line 32: 14 Killed python /common_variant_filter.py --id Sample-A-pair --maf /cromwell_root/fc-secure-37246767-4e2a-42e3-a380-a246c4a164d5/c75b72ab-f7af-4c1b-8e1d-51fa31102f7c/captureWorkflowSNVs/e61fd742-bbf1-44ae-b344-da6e30534dd5/call-ponFilter_Task/Sample-A-pair.pon_annotated.pass.maf
While stdout produces requested print statements. I attempted to attach these files but it seems that I can only attach images to forum posts.
Any thoughts? Thank you,
Comments
11 comments
HI Brendan,
Are you able to share the workspace with GROUP_FireCloud-Support@firecloud.org?
I will follow up with you in a ticket so that you can attach files. We can circle back to this thread and post what we find.
Many thanks,
Tiff
Great, thank you Tiffany.
I've shared the workspace with that support e-mail address: https://portal.firecloud.org/#workspaces/vanallen-firecloud-dfci/terra-help-workspace
Sorry, I read that there is a request to share workspaces with that e-mail ahead of time but wasn't sure if it would be necessary for this inquiry.
Let me know what you think.
Tiffany and I were chatting but weirdly the comments were not populating this thread. Is that expected? None the less, it seems that PAPI error code 9 indicates that the job has insufficient memory.
Hi Brendan,
I started an additional ticket so that you could be able to attach the files and share them with me. Though, we didn't need to do that in the end.
Reporting back for others: If you see 14 killed python error in your task's log, try increasing the memory in your task's runtime.
Has this been investigated further? Is there any other diagnosis or thoughts as to why this has happened or how to prevent it other than providing more memory through trial and error?
Hi Brendan,
PAPI error code 9 is a true error when you do not have enough memory. Depending on inputs or any other variable that may change, the amount of memory may need to be increased to facilitate a successful workflow. Any Workflows that we share as "Featured" have been tested to run on sample inputs but we do not have any standard suggestions on the amount of memory, disk, etc since they are dynamic based on the work each person is doing and how the WDL has been written/optimized.
Sushma
Thank you, Sushma!
This afternoon one of our colleagues was just set up on a fresh Terra billing project and is now running into many PAPI v2 error code 9 codes using the same parameters on the same underlying data that worked successfully under the previous billing project. Has this behavior been observed before? Do you know why this could be happening?
Hi Brendan,
I am not sure if this is behavior that has been seen before but I cannot be sure without taking a look at the workspace and the logs. Would you be able to get us shared with that workspace? The user to share with is GROUP_FireCloud-Support@firecloud.org
It may be worth sharing a workspace where it was successful so we can compare the two to see if there is something clear that is causing the issue.
Sushma
Ah drats, thank you for replying so quickly. It actually turns out and I misunderstood, the colleague is receiving PAPI v2 error code 2, out of disk space with the same parameters. I'm going to look into this more and will write back.
Ah yes, the dreaded out of disk space. I would increase memory first to test if that solves the issue.
Please sign in to leave a comment.