Summary
Our engineers have identified an issue resulting in stalled workflows for some May 31 workflow submissions.
Timeline
June 1, 2021 02:00 PM ET - Full resolution - The underlying issue has been resolved and all previously stalled workflows have been marked as Aborted.
June 1, 2021 02:00 PM ET - Second update on guidance - Users with stalled workflows can now resubmit them without concern. Newly submitted workflows are not affected by this stalling issue but may take a little longer than usual to dispatch.
June 1, 2021 11:20 AM ET - Update on guidance - If you have not yet aborted and resubmitted your stalled workflow, please leave it in a stalled state for the time being. If you have already done so, do not abort the resubmission. We will update this article as soon as we get the all-clear for aborting and resubmitting all stalled workflows.
Please 'Follow' this article to get email updates whenever this article is updated.
June 1, 2021 09:00 AM ET - Ongoing remediation - Our engineers have remediated the issue for newly submitted workflows. Older workflows may be in a stalled state, and users should abort and resubmit these workflows to get them to a completed state.
May 31, 2021 08:57 AM ET - Issue remediation - Our engineers started performing remediation steps.
May 31, 2021 07:55 AM ET - Issue discovered - One of our engineers was alerted to failures in our workflows tests, and identified an issue causing workflows to stall.
Impact
Users may see their workflows from May 31 have stalled. Users with stalled workflows should resubmit to get them to a completed state. The Workflows team will mark stalled workflows as Aborted.
Note about costs: Workflow tasks showing as stalled in Terra still run to completion on the Google side. As such, stalled workflows are not accruing additional costs even though they might be showing as "Running" in the Terra Job Manager.
For more information
Please follow this article to get the most up-to-date information on this incident. If you would like to be notified of all service incidents or upcoming scheduled maintenance, click Follow on this page.