Incorporating scripts into a workflow

Post author
Hannah Dewey

I have a WDL that will currently run well on an HPC and locally. In order for other collaborators to use it we are trying to make it available for use on Terra via FireCloud. 

Several of the tasks require use of perl scripts, or R scripts which I have available on github.

What is the best way to include these in the workflow, since there doesn't seem to be a way to upload them along side the workflow? Can I link directly to the script when calling the task using the github URL or is there some other method that I'm not seeing?

Thanks in advance!

Comments

1 comment

  • Comment author
    Josh Evans

    Hi Hannah,

    Thanks for writing in! When Terra runs a WDL, it spins up a runtime - or Docker Container - to do the actual compute outlined in the Command block. For your use case, you'll need to localize the files containing your scripts into the Docker container used by the WDL's runtime. 

    There are several ways to do this:

    • I've seen curl used in the command block of the WDL to import a script. This would make which script is imported unknown the users, and the log files can be difficult to read if curl fails for some reason.
    • You can create a file input in your WDL and input the scripts location either coded directly in the WDL or within a Data Table in Terra.  This allows the application to pull in and localize the script.  And if you ever need to change the link, you can simply update the Data Table instead of the workflow itself.
    • You can create a custom Docker image that already has all the scripts installed and call them as normal in the WDL command block.  The drawback to this is that every time the scripts need to be updated, a new docker image will need to be created.

    My suggestion would be to use a file input in the WDL, but I wanted to list out all the options here. If you'd like to learn more about how to do this, here is a link to our documentation: Wrapping R Scripts in WDL.

    Please let me know if this information was helpful or if you have any questions.

    Best,

    Josh

    0

Please sign in to leave a comment.