Task timeouts
A non-preemptible VM will run for at least a week (I'm not sure of the current maximum time), but more often than not we know that a task should finish within some amount of time that's much shorter than that. If it doesn't, it's a good indication that the tool being run is hanging for some reason, or that the VM is in some other pathological state.
Unfortunately, I don't know of any good way to avoid racking up potentially huge charges on "stuck" tasks, given that the VMs might live for a very long time without anyone noticing.
Most other workflow managers have some way of setting a maximum time per task/flow, after which the task will be forcibly failed. GCP Batch let's you set something like this, so it would be great if Cromwell+Terra could expose that option in the runtime block.
Comments
0 comments
Please sign in to leave a comment.