Delays in Google IAM permissions propagating
Due to Google’s recent changes to IAM and GCP services, temporary downtime or delays may occur when making access changes (workspaces/groups memberships). We are working diligently to resolve this matter and apologize for any inconveniences you may experience.
Users may notice permission errors in newly created workspaces or workspaces that were recently shared with them:
{
"error": {
"code": 403,
"message": "pet-1234567890@terra-12345678.iam.gserviceaccount.com does not have storage.objects.list access to the Google Cloud Storage bucket. Permission 'storage.objects.list' denied on resource (or it may not exist).",
"errors": [
{
"message": "pet-1234567890@terra-12345678.iam.gserviceaccount.com does not have storage.objects.list access to the Google Cloud Storage bucket. Permission 'storage.objects.list' denied on resource (or it may not exist).",
"domain": "global",
"reason": "forbidden"
}
]
}
}
Google has recently changed the way IAM permissions propagate, which has resulted in noticeable delays in access during workspace creation, sharing, and cloning. We've seen that these delays can take anywhere from a couple of minutes to a few hours in rare cases.
If you encounter this error in your newly created or shared workspace, please wait a few minutes and then refresh the page. The errors should resolve and you should be able to access your bucket without issues.
Comments
2 comments
Our engineers have released the following UI change to address this issue:
Terra will now display a banner message in your workspace if you are impacted by IAM propagation delays instead of throwing 403 errors throughout the UI. The 403 errors can still occur, but are much rarer now.
We have released a workaround for the Google IAM propagation delays. Upon workspace creation or clone, workspace owners now receive immediate access. This workaround also allows users added to a workspace to receive immediate access; however, the user’s email must be directly added to the workspace.
Users given workspace access via group membership will continue to experience IAM propagation delays at this time. We are diligently working on a long-term solution to minimize the impact to you, our valued user, and improve ease of workspace administration. Please contact Terra Support if you encounter errors or unexpected behavior.
Please sign in to leave a comment.