Increased frequency of task failures due to "PAPI error code 2...HTTP response code 503"

Some users may be seeing an increase in task failures with the following message:

The job was stopped before the command finished. PAPI error code 2. Execution failed: allocating: selecting resources: querying available resources: getting project quota: querying machine types: listing machine types: googleapi: got HTTP response code 503 with body

This is a transient Google error and has been escalated to Google. It seems to be happening most often when users are out of quota, and some call inside the Google pipeline manager is trying to find out what quota is missing so they can return a useful error message. Unfortunately, if that internal quota lookup fails, the entire pipeline is killed. 

As a workaround for now, our engineers are suggesting that users add a maxRetries value to the task runtime attributes. By default Cromwell won't retry this error, but by adding that attribute to tasks users can request retires up to 3 times for (almost) any transient error like this one.

Users can also contact Terra Support (support@terra.bio) to request a quota increase. Since the error happens when waiting for quota, increasing your quota limit will remove the chance for this issue to happen.