Increase quota requested for CryptoRequestsPerMinutePerProject for Terra projects Completed

Post author
Matt Bookman

When launching a large number of workflows concurrently, some operations are observed to fail with:

PAPI error code 2. Execution failed: pulling image: decrypting credentials: generic::unknown: retry budget exhausted (3 attempts): googleapi: Error 429: Quota exceeded for quota metric 'cloudkms.googleapis.com/crypto_requests' and limit 'CryptoRequestsPerMinutePerProject' of service 'cloudkms.googleapis.com' for consumer 'project_number:<project-id>'., rateLimitExceeded

Looking in the Cloud Console at the "All Quotas" list, can see that the quota:

Cloud Key Management Service (KMS) API
Cryptographic requests per minute

is set to 600 and is being exceeded.

This creates arbitrary failures that we should recover from, but today require an explicit manual relaunch.

 

Comments

1 comment

  • Comment author
    Sushma Chaluvadi

    Hi Matt,

    The limit has been increased from 600 to 60,000. Please let us know if you continue to experience similar issues.

     

    Sushma

    0

Please sign in to leave a comment.