Workflows aborted due to exceeding quota
Hi Terra team,
In the last two days I observed many instances of workflows aborted with the following error:
Quota exceeded for quota metric 'Requests per project in the US multi-region' and limit 'Requests per project in the US multi-region per minute' of service 'artifactregistry.googleapis.com'
In all cases the workflows were executed in single region workspaces and required multiple submissions before completion. Do you know what is causing such behaviour?
Best regards,
Edyta Malolepsza
Comments
8 comments
Hi Edyta,
Thank you for writing in about this issue. Can you share the workspace where you are seeing this issue with Terra Support by clicking the Share button in your workspace? The Share option is in the three-dots menu at the top-right.
Please provide us with
We’ll be happy to take a closer look as soon as we can!
Kind regards,
Samantha
Hi Samantha,
I just shared with you some examples. Thank you for looking at it!
Best regards,
Edyta
I'm also running into the same issue.
Hi Edyta,
I'm not sure what workspace you shared so I haven't looked at the examples yet, but I'm assuming you are running into the issue described in this Known Issue I just posted: https://support.terra.bio/hc/en-us/community/posts/22194351987227--Quota-exceeded-error-for-GCR-pulls-Workflows. As mentioned in the post, the workaround is to retry the failed jobs, either manually or by adding maxRetries to the task runtime.
A case is open with Google and we will update the Known Issues post as soon as we have more information. I would suggest following the post for further updates.
If you'd like me to take a closer look at your workflow, please provide a link to your workspace.
Best,
Samantha
We are also running into exactly the same error message running some medium sized jobs in a cromwell server for another service that is not connected to terra. Have you folks made any progress running this to ground? We have tried raising our project quota and it doesn't seem to fix the problem,
Hi all,
Updates regarding this issue will be posted to the following Known Issues post: https://support.terra.bio/hc/en-us/community/posts/22194351987227--Quota-exceeded-error-for-GCR-pulls-Workflows.
As mentioned in the post, the quota limit is for a Google-owned project. Our engineers have opened a case with Google to investigate. Please follow the post to be notified of any new information.
Best,
Samantha
I am running to the same issue as well.
Hi Yueyao,
Thanks for letting us know. Are you still experiencing this error on any workflows launched after January 22nd?
Kind regards,
Pamela
Please sign in to leave a comment.