Increase quota requested for CryptoRequestsPerMinutePerProject for Terra projects Completed
When launching a large number of workflows concurrently, some operations are observed to fail with:
PAPI error code 2. Execution failed: pulling image: decrypting credentials: generic::unknown: retry budget exhausted (3 attempts): googleapi: Error 429: Quota exceeded for quota metric 'cloudkms.googleapis.com/crypto_requests' and limit 'CryptoRequestsPerMinutePerProject' of service 'cloudkms.googleapis.com' for consumer 'project_number:<project-id>'., rateLimitExceeded
Looking in the Cloud Console at the "All Quotas" list, can see that the quota:
Cloud Key Management Service (KMS) API
Cryptographic requests per minute
is set to 600 and is being exceeded.
This creates arbitrary failures that we should recover from, but today require an explicit manual relaunch.
The limit has been increased from 600 to 60,000. Please let us know if you continue to experience similar issues.
Please sign in to leave a comment.