Workflow delays and upcoming improvements

If you've noticed slower queue times or workflow delays recently, we have some context and a few improvements in the works.

What's going on

During periods of high platform activity, workflows can sit in the queue longer than usual, and status updates for completed jobs may lag. This is expected behavior under elevated load on a shared cluster; things are processing normally, just potentially slower than you're used to.

What we're doing about it

Improved queue visibility

The Job Submission UI will be updated to show whether your workflow is waiting behind others, so you'll always know where your submission stands.

Limits on extremely large workflows

To protect performance for everyone on shared infrastructure, we'll be introducing limits on very large workflow requests that can cause broader slowdowns. Our first limit will automatically abort workflows that exceed a metadata writing threshold, returning a descriptive error message with your workflow ID and relevant metadata size. This is expected to affect only a very small number of users; if Terra Support has not reached out to you about large workflow submissions in the past, we do not expect you to be impacted. If you are impacted and have questions, please write to us at support@terra.bio.

We'll share updates as these changes roll out, and appreciate your patience.