Batch submit jobs through API (Swagger)

Post author
Steve Huang

On Terra, the job submission page allows one to select multiple samples in a batch (a sample set is formed automatically in that case), and the jobs history will show one submission, in which there's one entry for each sample being processed.

On FISS, this feature looks to be approximated by its API `create_submission`, which in turn makes API call to

https://api.firecloud.org/#!/Submissions/createSubmission

Consider the scenario where one wants to do advanced filtering on the samples to process, which isn't possible yet on Terra GUI but possible with FISS, and then submit a batch job with the selected samples, which is possible with Terra GUI, but unclear how with FISS.

As I study the API, the magic may be in `expression`, whose documentation reads

expression

string

Expression that resolves to one or more entities matching the entity type in the method configuration. Optional if the method configuration has no root entity.

I'd guess the Terra GUI's "batch submission" works through this magic `expression`. So I'm interested in seeing an example of that.

Or if not, I'm interested in learning however Terra cleverly resolves "batch submission" issue.

 

Thank you!

Steve

 

Comments

5 comments

  • Comment author
    Samantha (she/her)

    Hi Steve Huang,

    Thanks for writing in. We'll take a look at your question and get back to you as soon as we can.

    Best,

    Samantha

    0
  • Comment author
    Samantha (she/her)

    Hi Steve Huang,

    Sorry for the delayed response. I asked our engineers about this, and unfortunately, it sounds like the  expression parameter does not work in the way you are hoping. Expression is for traversing from the submitted entity/entities to the root entity/entities to launch workflows against.

    Say you have a method configuration that expects a sample and you want to run it on all of the samples in a set. If you submit "entityType": "sample", "entityName": "mySampleSet", that will fail because you gave the method configuration a sample_set instead of a sample. However, if you add "expression": "this.samples" (which is the property of sample_set that contains references to sample entities), then it will run the method against each of the referenced samples.

    Please let me know if you have any questions.

    Best,

    Samantha

    0
  • Comment author
    Steve Huang

    Hi Samantha, 

    Thanks for the reply!

    So it's not the 'magic' `expression`. Then how should be submit jobs in batch mode, through an API call? Currently we do so by looping over the entities that need to be analyzed, which results in one submission per entity. 

    In the end, we aim to achieve something similar to the following with the Terra GUI: select multiple entities from a table, and submit them as a `set` for analysis, but each entity will be processed through the WDL workflow independently.

    Thanks!

    Steve

    0
  • Comment author
    Chani

    Hi!

    I was wondering if I could get a response for this discussion.

    If you submit a create_submission using etype="sample_set", entity="MySampleSet", expression="this"

    This will run the workflow assuming the input submission is an array. If your workflow is configured to ONLY run on a single sample at a time. This will fail.

    As Steve asked, Is there a way to use expression to have the create_submission process loop through the sample_set/MySampleSet and run the workflow for each entity at a time or is that not possible?

    0
  • Comment author
    Chani

    For anyone in the future wondering,

    Yes it is possible.

    entityType = "sample_set"

    entity = "MySampleSet"

    expression = "this.samples"      < Note the extra s after sample

    This will perform what my question is asking.

    1

Please sign in to leave a comment.