On the morning of May 6, 2020, we reported massive spikes of jobs receiving PAPI error code 2 to Google. They have concluded this to be an error on their end, and are working to put out a fix within the week.
See the Timeline section for the latest troubleshooting and resolution updates and the Impact section to understand how this could impact your use of the system.
May 7, 2020 9:33 AM ET - Issue remediation - Google has reported that the fix for this issue has finished rolling out. It should no longer occur for new pipelines going forward but those in flight might still encounter it.
May 6, 2020 3:37 PM ET - Root cause - Google has reported that jobs are failing due to preemption and should be returning with error code 10 or 14 but are incorrectly returning 2. See workaround in Impact section.
May 6, 2020 1:41 PM ET - Issue identification - Google has informed us that this is an issue on their end and that they are working on a fix.
May 6, 2020 11:03 AM ET - Issue discovered and reported - Our engineers noticed a sharp increase in the number of PAPI error code 2 instances over the last 24 hours. They reported this to Google.
Users may see their jobs failing with PAPI error code 2.
Workaround: Add maxRetries option to your job's runtime to retry jobs in case of transient failure. More info here: https://cromwell.readthedocs.io/en/stable/RuntimeAttributes/#maxretries
Users can alternatively use non-preemptibles.
For more information
Please follow this article to get the most up to date information on this incident. If you would like to be notified of all service incidents or upcoming scheduled maintenance, click Follow on this page.