Summary
We've identified an issue that impacts users of Cloud Environments. It appears that users trying to create or start environments may be unable to do so.
See the Timeline section for the latest troubleshooting and resolution updates and the Impact section to understand how this could impact your use of the system.
Timeline
October 5 2023 2:40 PM ET - Issue Remediated - Google was able to resolve the issue and it should be possible to create and pause Cloud Environments as normal.
October 5, 2023 1:20 PM ET - Incident reported by Google - Google has reported that multiple cloud products are experiencing networking issues, which is most likely impacting Terra cloud environments. Please see Google's status page for more information: https://status.cloud.google.com/incidents/U39RSGjaANJXtjHpRkdq
October 5, 2023 12:05 PM ET - Issue Escalated - Our engineers were able to determine that this issue was affecting our production environment. The issue has been escalated.
October 5, 2023 9:00 AM ET - Issue Discovered - Our engineers noticed there may be an issue with GCP providing virtual machines for our use as Cloud Environments.
Impact
Users trying to unpause or start a Cloud Environment may have the environment fail to create and return a Welder failed to start after 10 minutes error.
For more information
Please follow this article to get the most up to date information on this incident. If you would like to be notified of all service incidents or upcoming scheduled maintenance, click Follow on this page.