Make the 7 day hard limit for VMs configurable Completed
There is currently a hard limit of 7 days on the length of time a VM can run a task. We have several workflows that can exceed that limit when processing large data sets, and there is no good way to re-configure the workflow to avoid this.
It would be very helpful to remove the hard limit and allow the default timeout to be configurable. This does not appear to be a limitation of the underlying architecture.
https://cloud.google.com/life-sciences/docs/reference/rest/v2beta/projects.locations.pipelines/run
https://github.com/broadinstitute/cromwell/blob/develop/cromwell.example.backends/PAPIv2.conf
Comments
2 comments
Hi David Vismer,
Thank you for writing in. I've sent this request to our development team for consideration, and I'll be happy to follow up with you if this feature gets built.
Kind regards,
Samantha
In October 2023 we increased this limit from 7 days to 14 days, and then again in August 2024 from 14 days to 21 days.
Workflow tasks in Terra now support 21 days of run time on non-preemptible VMs. Tasks that previously timed out at the 7-day mark will now continue to run until they terminate on their own, or for 21 days, whichever is sooner. The maximum duration of preemptible VMs is unchanged at 24 hours.
Please sign in to leave a comment.