Summary
11/30/22 UPDATE: We've identified an issue with submission monitoring which is causing a delay in workflow status updates or aborts.
We've identified an issue where aborting workflows takes longer than usual.
See the Timeline section for the latest troubleshooting and resolution updates and the Impact section to understand how this could impact your use of the system.
Timeline
December 5, 2022 5:00 PM ET - Issue Resolved - The fix for this issue went out with the weekly release. Our engineers expect this issue to be fully resolved.
December 1, 2022 9:07 AM ET - Root cause identified - Our engineers have identified the root cause for the issue and have a fix to address it. Unfortunately, the fix is currently blocked by separate issue, but the team is working on resolving it to unblock the release. In the meantime, the team will continue to restart Rawls periodically to reduce user impact.
November 29, 2022 2:19 PM ET - Issue Returns - We received reports from users seeing a significant delay in workflow status updates in Terra. Our engineers are currently investigating.
November 18, 2022 5:45 PM ET - Issue Remediated
November 11, 2022 10:53 AM ET- Issue Discovered - A user reported a delay when trying to abort their workflow.
November 14, 2022 4:16 PM ET - Investigation Started - Our engineers began looking into this issue to find the root cause.
Impact
11/30/22 UPDATE: Users may see a delay in the update of their workflow statuses and aborts.
Whenever a user tries to abort a workflow, it tends to take longer than usual. This could lead to increased costs when VMs don't terminate in a timely manner. The team is currently looking into the issue and recommends swiftly contacting support upon noticing any issues aborting workflows.
For more information
Please follow this article to get the most up-to-date information on this incident. If you would like to be notified of all service incidents or upcoming scheduled maintenance, click "Follow" on this page.