Summary
Issues with the Google Directory API are resulting in some users not gaining access to Google resources they should be able to access, such as Google buckets, VMs, and more.
For example, a user may create or clone a workspace and then receive an error that they don't have access to the workspace bucket.
See the Impact section to learn more about how you might be affected and how to contact support to resolve this issue.
Timeline
July 25, 2022 9:00 AM ET - Significant reduction in errors - Our engineers made an adjustment on July 22 that significantly reduced the frequency of this issue newly occurring for users. We are still in conversation with Google to fix the permissions for previously affected users/groups.
July 20, 2022 12:00 PM ET - Ongoing issues & contacting Google - We are seeing continued reports of this issue and have followed up with Google support.
July 18, 2022 12:15 PM ET - Report of ongoing issues by Google - Google has informed us that they are still seeing issues with their Google Directory API. Users of the Terra platform may continue to be affected until the issue is resolved.
July 15, 2022 04:55 PM ET - API Stability / Remediation - We have not seen any of the original errors in the past hour and believe the issue is now remediated. We will continue to monitor to make sure the errors don't start cropping up again.
July 15, 2022 01:43 PM ET - Google ticket raised -We've raised a ticket with Google for their awareness. Google has confirmed this is a known issue that is actively being investigated.
July 15, 2022 12:00 PM ET - Issue confirmation - We successfully identified an ongoing user issue that resulted from the ongoing Google Directory API issue.
July 15, 2022 10:35 AM ET - Issue identified - One of our engineers noticed a spike in errors from Google starting around 7AM ET, but impact to the platform was not obvious. Frontline Support worked with the engineer to investigate current reported issues to see if issues with the Google Directory API were the root cause.
Impact
Users may see error messages like
"pet-123456789123456789@terra-0000a11c.iam.gserviceaccount.com does not have storage.objects.list access to the Google Cloud Storage bucket."
or other varieties of error messages due to this issue. We are currently investigating workaround options.
If you are experiencing this issue, please contact support by emailing support@terra.bio with the following information:
- Screenshot of the error message
- Affected Terra account email address
- Affected pet service account address
- Workspace Google Project ID (found in Cloud Information section of the dashboard)
- Workspace Bucket Name (found in Cloud Information section of the dashboard)
For more information
Please follow this article to get the most up-to-date information on this incident. If you would like to be notified of all service incidents or upcoming scheduled maintenance, click Follow on this page.