Can runtime parameter values be generated dynamically?

Kate Herman
  • Updated

This documentation is in the process of being updated. In the meantime, you may find that some GATK commands are out of date, or that the WDL information is incomplete. If you encounter any issues you can't solve, please let us know.

Say you have a task where the runtime requirements (like disk size and memory) can vary a lot depending on e.g. the size of the input files for a given run. To minimize costs, you want to be able to allocate those runtime parameters depending on what is needed for each job; and to minimize effort you want to do that dynamically, without having to change the WDL script or manually changing input values for each job.

This can be done by using the size() function to check the size of the files, then tacking on a bit of arithmetic to adjust as needed (eg if you want the task to have a bit of extra headroom). In principle, you could use size() directly in the runtime block, since it will be evaluated before localization. Howeversize() returns a float (i.e. a type of decimal number) which is not valid for any property you want to set that requires an integer.

Obviously, if you can convert the float to int, then it will work fine. The problem is we don't currently have a nice civilized conversion function or equivalent numerical type coercion mechanism. So as a workaround, you have to use an ugly hack:

  Float f = size(infile)
  String s = f 
  String string_before_decimal = sub(s, "\\..*", "") 
  Int final_int = string_before_decimal

  runtime {
    memory = final_int

This will hopefully be improved in future evolutions of the WDL standard library so that we will no longer need the hack. But in the meantime at least this will work.

You should be able to add whatever headroom arithmetic you like (e.g. multiplying or adding to either the number produced by size() or the one that is produced by the hack) to calculate the final allocation value.

Was this article helpful?

0 out of 0 found this helpful

Have more questions? Submit a request



Please sign in to leave a comment.