PAPI Error Code 2 Completed
The number of reports for PAPI error code 2 have greatly spiked over the course of the past month, and especially within this past week. We have reached out to Google to investigate—they are aware of the issue and are currently working on a fix to be released this week.
I will update this thread as we hear more from them. Please click "Follow" at the top right to receive emails for updates to this post.
Google has reported that the fix for this has finished rolling out. It should no longer occur for new pipelines going forward but those in flight might still encounter it.
Root cause: Jobs are failing due to preemption and should be returning with error code 10 or 14 but are incorrectly returning 2.
Workaround: Add maxRetries option to your job's runtime to retry jobs in case of preemption failure. More info here: https://cromwell.readthedocs.io/en/stable/RuntimeAttributes/#maxretries
Users can alternatively use non-preemptibles.
Thank you for the update, I'll relaunch my jobs.
Please sign in to leave a comment.