Gather script fails in scatter gather.
Hi,
I am trying to use WDL on my local cluster. I was trying to use WDL scatter gather parallelism. But everytime I run it I get a weird error that I have not gotten before. What makes it wierd is that it is something related with docker image which seems unlikely since both WDL tasks use the same docker image.
Attached are my WDL scripts. `scsorter_run.wdl` is the wdl script for task1. `scatter_gather_scsorter.wdl` is the second script which scatters the `run_scsorter` task across different marker gene lists and then gather task `merge_scsorter_results` merges the different output into a single object.
The WDL pipeline runs fine till the scatter task and the necessary files are produced but then fails at the gather task and throws error in `scatter_gather_scsorter/53ff816a-8c88-40b9-be97-08470a2d5a0a/call-merge_scsorter_results/execution/stderr.submit` which looks like something related to docker image. There is no pull limit from dockerhub on my institution VPN so that seems unlikely.
`You must supply an 'application' with the -a or -app flag. See applications with 'bapp -l'
Request aborted by esub. Job not submitted.`
I am wondering what is causing the failure? I would appreciate any insights into the same. All WDL scripts I have used so far have worked fine with my pipeline so its not the problem with cromwell config I think.
Here is the scatter gather script : scatter_gather_scsorter.wdl
version 1.0
import "../tasks/scsorter_run.wdl" as scsorter
workflow scatter_gather_scsorter{
input {
String docker_image
String queue_name
Int mem_gb
File scsorter_script
String Sample_name
String rds_file_path
File inputSamplesFile
Array[Array[File]] inputSamples = read_tsv(inputSamplesFile)
}
scatter (sample in inputSamples) {
call scsorter.run_scsorter as run_scsorter {
input:
docker_image=docker_image,
queue_name=queue_name,
mem_gb=mem_gb,
Sample_name=Sample_name,
rds_file_path=rds_file_path,
scsorter_script=scsorter_script,
marker_list_rds=sample[1],
marker_list_name=sample[0],
}
}
call merge_scsorter_results {
input:
docker_image=docker_image,
queue_name=queue_name,
mem_gb=mem_gb,
Sample_name=Sample_name,
seurat_rds_path=rds_file_path,
scsorter_runs=run_scsorter.scsorter_preds_rds
}
}
task merge_scsorter_results {
input {
String docker_image
String queue_name
Int mem_gb
File merge_scsorter_script
String Sample_name
String seurat_rds_path
String type
Array[File] scsorter_runs
}
command <<<
Rscript ~{merge_scsorter_script} ~{seurat_rds_path} ~{Sample_name} ~{type} ~{sep=" " scsorter_runs}
>>>
output {
File Seurat_merged_scsorter = glob("*.seurat_scsorter_mergedpreds.*.rds")[0]
}
}
Here is the first task wdl : scsorter_run.wdl
version 1.0
task run_scsorter {
input {
String docker_image
String queue_name
Int mem_gb
File scsorter_script
String rds_file_path
String Sample_name
String marker_list_rds
String marker_list_name
}
command <<<
Rscript ~{scsorter_script} ~{rds_file_path} ~{Sample_name} ~{marker_list_rds} ~{marker_list_name}
>>>
runtime {
docker : docker_image
memory: mem_gb + " GB"
queue: queue_name
}
output {
File scsorter_preds_rds = glob("*_scsorter_preds_*.rds")[0]
}
}
workflow scsorter_markerprediction {
call run_scsorter
}
The Rscript called in `merge_scsorter_results` works fine on outside of WDL.
Comments
1 comment
Hi smk_84,
Thank you for writing in about this issue. Can you share the workspace where you are seeing this issue with Terra-Support@firecloud.org by clicking the Share button in your workspace? The Share option is in the three-dots menu at the top-right.
Please provide us with
We’ll be happy to take a closer look as soon as we can!
Kind regards,
Pamela
Please sign in to leave a comment.