Trying to run joint-discovery-gatk4.wdl over 4000+ samples, the workflow errored out with:
2019-07-09 23:21:54,288 ERROR - WorkflowExecutionActor-54cf1196-53fa-48ab-8f96-b042abc85549 [UUID(54cf1196)]: Job BackendJobDescriptorKey_CommandCallNode_JointGenotyping.CollectMetricsSharded:671:1 failed to be created! Error: Root workflow tried creating 50043 jobs, which is more than 50000, the max cumulative jobs allowed per root workflow
I believe that this occurs because the intervals file contains 10,187 intervals and the workflow scatters 5 times over those 10,187 intervals.
I have worked around this problem by commenting out the collection and gathering of metrics (CollectMetricsSharded, GatherMetrics, and the *_metrics_files workflow outputs) such that the scatter over ApplyRecalibration can run. I can then re-run the workflow with call caching enabled, commenting out the ApplyRecalibration, in order to gather the metrics.
But it would be great if this maximum could be increased.