Summary
Terra's workflows saw degraded performance due to a problematic workflow submission that impacted the workflow engine. See the Timeline section for troubleshooting and resolution updates, and the Impact section to understand how this affected use of the system.
Timeline
April 29, 2026 11:49 AM ET - Fix Released, Continuing to Monitor
We have released a fix that successfully resolves DRS URLs with NIH RAS Passports. We are asking users to test workflows using this data at a small scale as we monitor the system.
All other workflow users should see improved performance in the workflow queue.
April 28, 2026 11:24 AM ET - Issue identified; Workflow Failures Continue
Terra is currently experiencing issues accessing NIH data via DRS URL requests to the NIH Research Auth Service Passport. Due to shared workflow infrastructure, excessive DRS failures are impacting the workflow queue for all Terra users.
Please refrain from submitting workflows that process NIH data via DRS until further notice.
April 27, 2026 1:47 PM ET – Possible Recovery; Active Monitoring
We are seeing improvements in workflow performance, but are continuing to monitor for signs of degraded performance.
We are simultaneously investigating an issue with Terra Data Repository which may be causing downstream impacts to Terra's workflow engine. See Service Incident - April 27, 2026 (Terra Data Repository) for more details.
April 27, 2026 – Reemergent Issues
We are seeing signs of high load on the system, which is causing workflows to slow or fail in unexpected ways. Additional details to follow.
April 25, 2026 – Problematic Workflow Fixed / Issue Remediated
An engineer identified a workflow submission in the system that was causing the engine to stall, which resulted in slowdowns and unexpected behavior in other submitted workflows. The engineer has fixed the problematic workflow, and we believe the issue to now be remediated.
April 2026 – Significant Slowdowns and Unexpected Behaviors Observed
Users have observed significant slowdowns and unexpected behavior during recent workflow submissions.
Impact
Users may see slowdowns or unexpected behaviors in jobs until the issues are fully resolved.
If you experience any further unexpected behavior, please write to us at support@terra.bio or post to our community forum.
For more information
Please follow this article to get the most up-to-date information on this incident. If you would like to be notified of all service incidents or upcoming scheduled maintenance, click Follow on this page.
