Bug? Workflow shows as succeeded, but outputs were not delocalized
In the Job History and in Job Manager, my workflow shows as succeeded, but I had a typo in the task outputs, so my outputs were not localized. How can this be a success?
In the Job History and in Job Manager, my workflow shows as succeeded, but I had a typo in the task outputs, so my outputs were not localized. How can this be a success?
Comments
7 comments
Hey Laura,
Happy to take a closer look here! Is it possible to share the workspace? If so, can you share it with GROUP_FireCloud-Support@firecloud.org and provide the submission and workflow IDs of interest, as well as a link to the workspace?
If it's not possible to share the workspace can you share the WDL and describe the part that seems to be acting strangely.
Many thanks,
Jason
Submission ID: 862094cc-695f-4e3c-bd82-5b47ee67a780
JobID: 4f27a426-d95e-44ac-bdd9-739a375a5d3a
Workspace ID: 50b3997e-bf04-41e2-bfee-e4ff4df5cf33
Workspace link: https://app.terra.bio/#workspaces/broad-firecloud-dsde-methods/AssortedExomes/
The config is gone because I fixed the bug, but we could reproduce it pretty easily by taking the ~ out of the variable in the task output.
Thanks!
Laura
Hey Laura,
It looks like the workflow succeeded because it did indeed output an array, it just happens to be the case that the array.
My sense is that Cromwell/WDL isn't restricting users from outputting empty arrays, as there might be valid reasons for doing so. I'm happy to check with the engineers if you want confirmation though! Other users who've experienced this issue have typically edited their WDL to fail the task if the array is empty.
Let me know if you would like me to check on this with the Batch team - happy to do so!
Kind regards,
Jason
You'll have to take my work for it since I've been doing more developement, but even the empty array didn't end up in the data model.
I can imagine there could be cases when you want to output an empty array, but my concern is that there was an error in the Cromwell log that it couldn't delocalize the file I requested from the task. In my opinion, that should be a task failure and thus a workflow failure.
Gotcha! Let me check with the team about this and get back to you.
Hey Laura,
We were able to confirm that this behavior is expected because of the way glob() works in WDL. As explained in the WDL spec:
Because zero outputs is a valid number of outputs when using glob(), Cromwell did not interpret this as a failure.
You can work around this by defining Array[File]+ in your output block, which requires that there be at least one element. Link to relevant spec documentation. Below is an example of how you can define that output variable.
workflow testingGlob { call testGlob{ } } task testGlob { command { mkdir testdir touch ./testdir/file1 touch ./testdir/file2 } output { Array[File]+ outputfiles = glob("./testDir/*") } }I hope this helps!
Kind regards,
Jason
Array[File]+ for globs is a good tip/best practice -- thanks Jason!
Please sign in to leave a comment.