Summary
Users may be experiencing a delay starting new workflows due to slowness of one of our internal services. The issue was found at approximately 10:00 AM Tuesday, May 21, 2019, by internal monitoring and impacts users attempting to start new workflows on Terra/FireCloud (existing workflows are finishing and running). See the Timeline section for the latest troubleshooting and resolution updates and the Impact section to understand how this could impact your use of the system. Existing ones are finishing and running
Timeline
1:07 PM EDT | May 21, 2019
Final Confirmation: The system is stable and workflows are progressing normally. A permanent fix will be implemented once root cause has been determined.
12:00 PM EDT | May 21, 2019
Issue Resolved: The temporary fix is being monitored for stability.
11:41 AM EDT | May 21, 2019
Fix Deployment: A temporary fix has been put in place and the team is observing the stability.
11:20 AM EDT | May 21, 2019
Issue Investigation: Team filtering through submissions potentially leading to slowness of internal service.
10:25 AM EDT | May 21, 2019
Issue Investigation: Team has restarted service to try and restart jobs that may be in delay.
10:00 AM EDT | May 21, 2019
Issue Discovered: Internal monitoring services alerting teams of slowness in the execution engine, Cromwell, resulting in a delay starting new workflows.
Impact
Users may experience a delay when starting new workflows in Terra/FireCloud.
For more information
Please follow this article to get the most up to date information on this incident. If you would like to be notified of all service incidents or upcoming scheduled maintenance, click Follow on this page.