Summary
Cromwell runners experienced a very high load that caused degraded performance and IO timeout errors. The incident ended when the responsible submission was aborted by the user.
Timeline
January 4, 2023 3:28 PM ET - Investigation Continued - Terra engineers have logged a ticket to further investigate the root cause of the issue and opportunities to reduce the chance of this occurrence in the future.
December 27, 2022 4:30 PM ET - Issue Resolved - The issue resolved itself when the workflow submission was aborted by the user.
December 27, 2022 12:00 PM ET - Issue Reported - A user reported very slow workflow runtimes and IO timeout errors to Terra engineers.
December 25, 2022 6:30 PM ET- Issue Began - Cromwell began experiencing performance degradation due to unusually high load.
Impact
During the time of the incident, users experienced frequent Cromwell timeouts due to degraded performance. The issue is now resolved but Terra engineers are continuing to investigate the cause.
For more information
Please follow this article to get the most up to date information on this incident. If you would like to be notified of all service incidents or upcoming scheduled maintenance, click Follow onthis page.