Terra Should Support Dockstore Tools

Currently, Dockstore has a "Launch with Terra" button that is available on WDL workflows.

This button is not available for Dockstore WDL tools. It should be, so that Dockstore WDL tools are also available in Terra.

Background

Dockstore and TRS treat tools and workflows separately. This stems in part from CWL, which Dockstore supported first, and which has a formal distinction between CommandLineTool and Workflow.

To fit WDL into this model, Dockstore allows users to register WDL tools. Since WDL does not formally distinguish the two, Dockstore imposes the restriction for tools that the WDL have only one task.

In the TRS API, which Dockstore implements, to distinguish the IDs of workflows and tools, workflow ids are preceded with #workflow/ (which becomes %23workflow%2f when URL-encoded).

When invoking Launch with Terra from Dockstore, Dockstore passes the ID of the workflow to Terra, e.g., https://app.terra.bio/#import-tool/dockstore/github.com/HumanCellAtlas/skylab/Optimus:terra-optimus

Terra then reads the URL, extracting the portion after import-tool/dockstore, then calls back to Dockstore to fetch the WDL contents.  It uses the TRS API to fetch WDL content from Dockstore. It always prepends #workflow/ to the ID that Dockstore passes to Terra, assuming it's a workflow, and making it impossible to pass a tool. See https://github.com/DataBiosphere/terra-ui/blob/5c325fdcd362083cd01df0f39af7c357943f889d/src/libs/ajax.js#L115 

Possible Solutions

The solution should not break the existing integration, as this change will require changes on both the Terra and Dockstore sides, and we will not be able to do a simultaneous release.

  1. Add a new path, e.g., change dockstore to to dockstore-tool: https://app.terra.bio/#import-tool/dockstore-tool/github.com/HumanCellAtlas/skylab/Optimus:terra-optimus
  2. Add a "query parameter" at the end, e.g., https://app.terra.bio/#import-tool/dockstore/github.com/HumanCellAtlas/skylab/Optimus:terra-optimus?type=tool. Not sure if this actually is a query parameter, since the query parameter should go before the # fragment character in an URL, but in any case the ? is not a valid Git tag character, so Terra would know the "?" is not part of the version and be able to handle it. In this case, if the type is not specified, Terra would continue to default to assuming it's a workflow.

Comments

9 comments

  • Comment author
    Sushma Chaluvadi

    Hi Charles,

    The team has conferred and at this time they have chosen to reallocate resources to other platform stability issues and will return to this compatibility with Dockstore feature in the future! We appreciate your feedback and will be sure to update you should any plans change!

    0
  • Comment author
    Elizabeth Sheets

    Sushma, can I request that Terra reconsider this request to make Dockstore tools available in Terra? We have had BDCat several users get confused when registering their pipelines as tools and not being able to launch with Terra or BDC. 

    0
  • Comment author
    Jason Cerrato

    Hi Elizabeth Sheets,

    I'll be happy to inform our product managers of this consideration. Can you provide some additional details about the process BDCat users follow to register their pipelines, and if there's a particular reason they are registering them as tools vs other options?

    Many thanks,

    Jason

    0
  • Comment author
    Elizabeth Sheets

    Hi Jason, 

    Our BDC learning material asks users to consider publishing their tools and workflows on Dockstore and they use a lot of learning material from Dockstore to write their first descriptor (BDC users use both WDL and CWL). From the Dockstore website, the distinction between these terms for all descriptor languages is that tools represent a single task while workflows chain multiple tasks together: https://docs.dockstore.org/en/develop/getting-started/dockstore-workflows.html

    So, users create tools for single tasks based on this learning material. After publishing, when they click on the WDL tool on Dockstore, they are then confused about why there is no option to launch tools into Terra/BDC/AnVIL. We then have to explain the issue to them and request that they go back and publish their single task as a workflow so that it is easily usable with Terra. This creates usability issues. 

    Thanks for considering this request!

    Beth

     

    0
  • Comment author
    Jason Cerrato

    Hi Beth,

    Thanks so much for that context! I'll definitely be sure to let them know.

    I've also flagged this up with Michael Baumann who will reach out to you later to discuss possible strategies for mitigating the issue in the meantime.

    Kind regards,

    Jason

    0
  • Comment author
    Stephen Fleming

    Is this solved?

    0
  • Comment author
    Jason Cerrato

    Hi Stephen Fleming,

    I've heard from the relevant engineering team that this feature has not yet been built. Would building this feature resolve any problems you're currently facing?

    Kind regards,

    Jason

    0
  • Comment author
    Stephen Fleming

    Hi Jason, nope this isn't blocking anything I'm doing, I was just wondering  :)

    Thanks,
    Stephen

    0
  • Comment author
    Charles Overbeck

    For what it's worth, Dockstore (I'm on the Dockstore team) has deprecated WDL tools, and it's no longer even possible to create a WDL tool via the new preferred mechanism of creating tools, GitHub App Tools: https://docs.dockstore.org/en/stable/advanced-topics/dockstore-tools-overhaul.html#github-app-tools

    0

Please sign in to leave a comment.