Outputting and inputting a directory

June 02, 2020 22:34
6 comments

I'm writing a wdl that uses both salmon index and salmon quant. It runs fine locally using:

task index {
...
output {
    File salmonindex
    }
...
}
=>
input {
    File index.salmonindex
}

But running on terra, I'm returning:

2020/06/02 22:26:50 Delocalizing output /cromwell_root/memory_retry_rc -> gs://fc-43d16d14-b13b-4a0e-9b7e-f88166b33b87/2af48365-8cfc-4e2f-8768-8832a16e8db0/analysis/755330a2-92af-4a1e-884e-69669b5e4973/call-salmon_index/memory_retry_rc 2020/06/02 22:26:50 Delocalizing output /cromwell_root/rc -> gs://fc-43d16d14-b13b-4a0e-9b7e-f88166b33b87/2af48365-8cfc-4e2f-8768-8832a16e8db0/analysis/755330a2-92af-4a1e-884e-69669b5e4973/call-salmon_index/rc 2020/06/02 22:26:52 Delocalizing output /cromwell_root/stdout -> gs://fc-43d16d14-b13b-4a0e-9b7e-f88166b33b87/2af48365-8cfc-4e2f-8768-8832a16e8db0/analysis/755330a2-92af-4a1e-884e-69669b5e4973/call-salmon_index/stdout 2020/06/02 22:26:53 Delocalizing output /cromwell_root/stderr -> gs://fc-43d16d14-b13b-4a0e-9b7e-f88166b33b87/2af48365-8cfc-4e2f-8768-8832a16e8db0/analysis/755330a2-92af-4a1e-884e-69669b5e4973/call-salmon_index/stderr 2020/06/02 22:26:54 Delocalizing output /cromwell_root/salmonindex/ -> gs://fc-43d16d14-b13b-4a0e-9b7e-f88166b33b87/2af48365-8cfc-4e2f-8768-8832a16e8db0/analysis/755330a2-92af-4a1e-884e-69669b5e4973/call-salmon_index/salmonindex/ Required file output '/cromwell_root/salmonindex/' does not exist.

I am trying to pass it as a directory, not a series of files via Array[File}. How would I fix this?

Thank you

Comments

6 comments

Jason Cerrato
- June 03, 2020 14:55
Hi Alex,

Can you share the workspace where you're running this with GROUP_FireCloud-Support@firecloud.org so we can take a closer look at the job? Please also share the workflow with jcerrato@broadinstitute.org if it's not publicly available.

Many thanks,

Jason

0
Alex O
- June 03, 2020 16:15
Shared

0
Jason Cerrato
- June 04, 2020 15:50
Hi Alex,

Can you also share the workflow with jcerrato@broadinstitute.org so I can take a closer look at the WDL itself?

Many thanks,

Jason

0
Alex O
- June 04, 2020 16:25
Added, but versions 3 and 5 were edits I made to try to get it to work. The one that gave me the error originally was v2. Thanks!

0
Jason Cerrato
- June 04, 2020 20:34
Hi Alex,

It looks like the root of the issue is the trying to delocalize a directory, which is not something you can do. However, you have a couple options.

You can’t delocalize a directory as a File, but an array of files can technically delocalize the directory contents with a glob():
```
output {
   Array[File] directoryContents = glob("myDir/*")
}
```
Then you can use the array of files elsewhere, but the array is technically not the same as a “directory”. See the docs here: https://github.com/openwdl/wdl/blob/master/versions/1.0/SPEC.md#globs

The other way to "delocalize directories” is to tar the contents using something like tar czf myDir.tgz myDir then
```
output {
  File myDirTgz = "myDir.tgz"
}
```
Then the next steps can untar the previous output.

I hope that makes sense. If you have any questions, please let us know.

Kind regards,

Jason
1
Alex O
- June 05, 2020 18:52
Awesome, thank you so much!

0

Please sign in to leave a comment.