When launching a large number of workflows concurrently, some operations are observed to fail with:
PAPI error code 2. Execution failed: pulling image: decrypting credentials: generic::unknown: retry budget exhausted (3 attempts): googleapi: Error 429: Quota exceeded for quota metric 'cloudkms.googleapis.com/crypto_requests' and limit 'CryptoRequestsPerMinutePerProject' of service 'cloudkms.googleapis.com' for consumer 'project_number:<project-id>'., rateLimitExceeded
Looking in the Cloud Console at the "All Quotas" list, can see that the quota:
Cloud Key Management Service (KMS) API
Cryptographic requests per minute
is set to 600 and is being exceeded.
This creates arbitrary failures that we should recover from, but today require an explicit manual relaunch.