Support workflows with more than 50,000 nodes

Comments

2 comments

  • Matt Bookman

    Two additional notes:

    1- My statement above was incorrect:

    I can then re-run the workflow with call caching enabled, commenting out the ApplyRecalibration, in order to gather the metrics.

    The CollectMetricsSharded needs the input of ApplyRecalibration, so it isn't as simple as I indicated. We will need to craft a separate workflow that takes the ApplyRecalibration as input and does the metrics collection and gathering.

    2- I also noticed that the maximum number of jobs is configurable in Cromwell and the default is 1,000,000:

    https://github.com/broadinstitute/cromwell/blob/9d0cf9d964ef1328f73b69da7e21f51f3b604bc4/engine/src/main/scala/cromwell/engine/workflow/lifecycle/execution/WorkflowExecutionActor.scala

    private val DefaultTotalMaxJobsPerRootWf = 1000000
    private val DefaultMaxScatterSize = 1000000
    private val TotalMaxJobsPerRootWf = params.rootConfig.getOrElse("system.total-max-jobs-per-root-workflow", DefaultTotalMaxJobsPerRootWf)
    private val MaxScatterWidth = params.rootConfig.getOrElse("system.max-scatter-width-per-scatter", DefaultMaxScatterSize)

    If possible, please increase the Terra configuration to 60,000 so that the joint discovery workflow can run to completion.

  • Matt Bookman

    Note that I have added a github issue for the workflow itself:

    https://github.com/gatk-workflows/gatk4-germline-snps-indels/issues/40

    If this 50,000 limit is going to stay as a hard limit, there are options within the workflow to examine.

Please sign in to leave a comment.

Powered by Zendesk