Service Incident - May 31, 2021 (Stalled Workflows)

Jason Cerrato
  • Updated

Summary

Our engineers have identified an issue resulting in stalled workflows for some May 31 workflow submissions.

 

Timeline

June 1, 2021 02:00 PM ET - Full resolution - The underlying issue has been resolved and all previously stalled workflows have been marked as Aborted.

June 1, 2021 02:00 PM ET - Second update on guidance - Users with stalled workflows can now resubmit them without concern. Newly submitted workflows are not affected by this stalling issue but may take a little longer than usual to dispatch.

June 1, 2021 11:20 AM ET - Update on guidance - If you have not yet aborted and resubmitted your stalled workflow, please leave it in a stalled state for the time being. If you have already done so, do not abort the resubmission. We will update this article as soon as we get the all-clear for aborting and resubmitting all stalled workflows.

Please 'Follow' this article to get email updates whenever this article is updated.

June 1, 2021 09:00 AM ET - Ongoing remediation - Our engineers have remediated the issue for newly submitted workflows. Older workflows may be in a stalled state, and users should abort and resubmit these workflows to get them to a completed state.

May 31, 2021 08:57 AM ET - Issue remediation - Our engineers started performing remediation steps.

May 31, 2021 07:55 AM ET - Issue discovered - One of our engineers was alerted to failures in our workflows tests, and identified an issue causing workflows to stall.

 

Impact

Users may see their workflows from May 31 have stalled. Users with stalled workflows should resubmit to get them to a completed state. The Workflows team will mark stalled workflows as Aborted.

Note about costs: Workflow tasks showing as stalled in Terra still run to completion on the Google side. As such, stalled workflows are not accruing additional costs even though they might be showing as "Running" in the Terra Job Manager.

 

For more information

Please follow this article to get the most up-to-date information on this incident. If you would like to be notified of all service incidents or upcoming scheduled maintenance, click Follow on this page

Was this article helpful?

0 out of 0 found this helpful

Have more questions? Submit a request

Comments

0 comments

Please sign in to leave a comment.