1. Terra Support
  2. Terra on Google Cloud
  3. Workflows
  4. WDL Resources
  5. Frequently Asked WDLs Questions

Can runtime parameter values be generated dynamically?

Kate Herman

This article is now archived.

WDL articles now live in the openwdl/wdl-docs GitHub repository. Find WDL resources and documentation on the wdl-docs website. 

Say you have a task where the runtime requirements (like disk size and memory) can vary a lot depending on e.g. the size of the input files for a given run. To minimize costs, you want to be able to allocate those runtime parameters depending on what is needed for each job; and to minimize effort you want to do that dynamically, without having to change the WDL script or manually changing input values for each job.

This can be done by using the size() function to check the size of the files, then tacking on a bit of arithmetic to adjust as needed (eg if you want the task to have a bit of extra headroom). In principle, you could use size() directly in the runtime block, since it will be evaluated before localization. However, size() returns a float (i.e. a type of decimal number) which is not valid for any property you want to set that requires an integer.

Obviously, if you can convert the float to int, then it will work fine. The problem is we don't currently have a nice civilized conversion function or equivalent numerical type coercion mechanism. So as a workaround, you have to use an ugly hack:

  Float f = size(infile)
  String s = f 
  String string_before_decimal = sub(s, "\\..*", "") 
  Int final_int = string_before_decimal

  runtime {
    memory = final_int
  }

This will hopefully be improved in future evolutions of the WDL standard library so that we will no longer need the hack. But in the meantime at least this will work.

You should be able to add whatever headroom arithmetic you like (e.g. multiplying or adding to either the number produced by size() or the one that is produced by the hack) to calculate the final allocation value.

Was this article helpful?

0 out of 0 found this helpful

Comments

1 comment

  • Comment author
    Ash O'Farrell
    • Edited

    This does not appear to work in WDL 1.0, but there is still a way to do this!

    You can actually declare variables outside of the input or command sections, and those variables can be functions of variables in the input section, and they can be used by the runtime section. Using this, we can just place something like this between some section of your WDL task...

    version 1.0

    task vcftogds {
      inputs {
        File vcf
      }

      # Estimate disk size required
      Int vcf_size = ceil(size(vcf, "GB"))
      Int finalDiskSize = 4*vcf_size

      command {

    And then, in your runtime section...

    runtime {
       disks: "local-disk " + finalDiskSize + " HDD"
    }
    0

Please sign in to leave a comment.