Terra Should Support Dockstore Tools
Currently, Dockstore has a "Launch with Terra" button that is available on WDL workflows.
This button is not available for Dockstore WDL tools. It should be, so that Dockstore WDL tools are also available in Terra.
Background
Dockstore and TRS treat tools and workflows separately. This stems in part from CWL, which Dockstore supported first, and which has a formal distinction between CommandLineTool and Workflow.
To fit WDL into this model, Dockstore allows users to register WDL tools. Since WDL does not formally distinguish the two, Dockstore imposes the restriction for tools that the WDL have only one task.
In the TRS API, which Dockstore implements, to distinguish the IDs of workflows and tools, workflow ids are preceded with #workflow/ (which becomes %23workflow%2f when URL-encoded).
When invoking Launch with Terra from Dockstore, Dockstore passes the ID of the workflow to Terra, e.g., https://app.terra.bio/#import-tool/dockstore/github.com/HumanCellAtlas/skylab/Optimus:terra-optimus
Terra then reads the URL, extracting the portion after import-tool/dockstore, then calls back to Dockstore to fetch the WDL contents. It uses the TRS API to fetch WDL content from Dockstore. It always prepends #workflow/ to the ID that Dockstore passes to Terra, assuming it's a workflow, and making it impossible to pass a tool. See https://github.com/DataBiosphere/terra-ui/blob/5c325fdcd362083cd01df0f39af7c357943f889d/src/libs/ajax.js#L115
Possible Solutions
The solution should not break the existing integration, as this change will require changes on both the Terra and Dockstore sides, and we will not be able to do a simultaneous release.
- Add a new path, e.g., change dockstore to to dockstore-tool: https://app.terra.bio/#import-tool/dockstore-tool/github.com/HumanCellAtlas/skylab/Optimus:terra-optimus
- Add a "query parameter" at the end, e.g., https://app.terra.bio/#import-tool/dockstore/github.com/HumanCellAtlas/skylab/Optimus:terra-optimus?type=tool. Not sure if this actually is a query parameter, since the query parameter should go before the # fragment character in an URL, but in any case the ? is not a valid Git tag character, so Terra would know the "?" is not part of the version and be able to handle it. In this case, if the type is not specified, Terra would continue to default to assuming it's a workflow.
Comments
9 comments
Hi Charles,
The team has conferred and at this time they have chosen to reallocate resources to other platform stability issues and will return to this compatibility with Dockstore feature in the future! We appreciate your feedback and will be sure to update you should any plans change!
Sushma, can I request that Terra reconsider this request to make Dockstore tools available in Terra? We have had BDCat several users get confused when registering their pipelines as tools and not being able to launch with Terra or BDC.
Hi Elizabeth Sheets,
I'll be happy to inform our product managers of this consideration. Can you provide some additional details about the process BDCat users follow to register their pipelines, and if there's a particular reason they are registering them as tools vs other options?
Many thanks,
Jason
Hi Jason,
Our BDC learning material asks users to consider publishing their tools and workflows on Dockstore and they use a lot of learning material from Dockstore to write their first descriptor (BDC users use both WDL and CWL). From the Dockstore website, the distinction between these terms for all descriptor languages is that tools represent a single task while workflows chain multiple tasks together: https://docs.dockstore.org/en/develop/getting-started/dockstore-workflows.html
So, users create tools for single tasks based on this learning material. After publishing, when they click on the WDL tool on Dockstore, they are then confused about why there is no option to launch tools into Terra/BDC/AnVIL. We then have to explain the issue to them and request that they go back and publish their single task as a workflow so that it is easily usable with Terra. This creates usability issues.
Thanks for considering this request!
Beth
Hi Beth,
Thanks so much for that context! I'll definitely be sure to let them know.
I've also flagged this up with Michael Baumann who will reach out to you later to discuss possible strategies for mitigating the issue in the meantime.
Kind regards,
Jason
Is this solved?
Hi Stephen Fleming,
I've heard from the relevant engineering team that this feature has not yet been built. Would building this feature resolve any problems you're currently facing?
Kind regards,
Jason
Hi Jason, nope this isn't blocking anything I'm doing, I was just wondering :)
Thanks,
Stephen
For what it's worth, Dockstore (I'm on the Dockstore team) has deprecated WDL tools, and it's no longer even possible to create a WDL tool via the new preferred mechanism of creating tools, GitHub App Tools: https://docs.dockstore.org/en/stable/advanced-topics/dockstore-tools-overhaul.html#github-app-tools
Please sign in to leave a comment.