Issue interpreting array inputs to workflow
I have a workflow which takes two Array[File] inputs and one Array[String] input, each with only one item. As recently as yesterday at 10:30 AM, it worked to use a File/String as an input to a variable of type Array[File]/Array[String] and it would be interpreted as an array of length 1. However, since I tried launching the workflow yesterday afternoon going forward, this no longer works.
A possible solution to this could be editing the sample metadata table to change the File to a file array and the method input string to be a string array, but there seems to be an issue interpreting arrays of length 1. This includes when I created a workflow which takes a file/string as input and outputs an array of just that input to the metadata table. Even using that output as the input to the workflow mentioned at the top results in the workflow interpreting the array as just a file/string.
Also, just to clarify why I need the input to stay as an array rather than be a single file/string, the inputs to the task will either be directly from the user (where there could be arrays of length one or more), or the outputs of a prior task (which will generally be arrays of length >1). I wanted to avoid being too specific to my workflow in this post, but feel free to ask more specifics if it would be helpful.
Comments
26 comments
Hello Micah,
Checking into this right now and will get back to you with an update.
Sushma
Micah,
Can you try putting `[ ]` around the element of your list?
Yes, I have done that
Would you be able to share your workspace that contains this Workflow with GROUP_FireCloud-Support@firecloud.org
as a Writer so that we can take a look directly - it would be helpful to troubleshoot. We can be removed right after :)
Sushma,
I have shared access with you. The workflow is `RNASeqPipeline_withRG`. Just let me know if there is an email I should send a description of the workflow to and/or if I need to share the WDL as well.
Thanks for your help,
-Micah
Micah,
I am unable to see the workspace - can you double check that it was shared successfully? A good way to check is if you see the word "Pending" next to the user email from the Share module. Does your workspace have an Authorization Domain on it?
The workspace doesn't have an authorization domain and there isn't a "Pending" message next to the FireCloud support email. Can I try sharing it with you directly?
Ah I misread workflow for workspace and was searching the wrong name - I am able to see it, sorry about that confusion. We will take a look and get back to you.
I believe however you may need to also share the Workflow with us independently. You can do so from the Methods Repository.
Micah,
Were you able to share the Method that you are working with? It shows as unavailable to us.
Hi,
In reference to
However, since I tried launching the workflow yesterday afternoon going forward, this no longer works.
can you please provide screenshots and/or exact error messages, please.
To show what I mean by the timing, here is a screenshot of the same configuration results before and after the issue:
As for the error message, here is that:
Based on a quick browse through the WDL spec, I think the coercion from File to Array[File] is not actually supported - rather, it happened to work by accident.
Such an accident is never guaranteed to keep working forever and indeed a recent maintenance update to WDL caused it to go away.
If you think I'm off-base about the spec - which is very possible! - please do speak up.
I just tried making the inputs required rather than optional and the single inputs were coerced into an array. (FirstEndFastq and SecondEndFastq are both File objects)
Interesting, thanks for the update.
I still believe you may be relying on undocumented functionality and recommend you implement the previously-suggested approach:
Can you try putting `[ ]` around the element of your list?
That's the other thing: that doesn't work unless I set the input as required and not optional
I set up a side workflow while I was trying that out to take a single input and output an array of length one with that input:
I then set the output arrays as inputs to the configuration where the arrays are optional:
However, the input is still interpreted as single files for some reason:
I then changed the configuration to the one with the required arrays:
And the inputs were correctly interpreted:
So I think an issue here is how optional arrays are being interpreted. Tomorrow I'll test out an optional array with length >1 to determine whether the issue is specifically about inputs with length 1.
Tomorrow I'll test out an optional array with length >1 to determine whether the issue is specifically about inputs with length 1.
That would be great to know, yes.
We have identified the issue that is preventing the application from correctly accepting arrays as the value of an optional array input. A fix is in progress.
At this time, we do not believe that coercing a single String or File in the inputs into an Array[String] or Array[File] is a supported behavior.
This may part of the issue you identified with optional arrays, but her is what happened when I tried optional arrays with length >1:
First, I modified my support workflow to output an array that is just the input values repeated twice:
I again used these as inputs to optional arrays:
And launching the workflow prompted this immediate error:
It seems to be expecting one value for an optional array.
For your second point about coercion from a single value to an array, I think it makes sense for that not to occur, so once the optional array bug is fixed, I will make sure my inputs are actually arrays.
Thank you for the updates, it's very helpful to see all the things you've tried.
We have a verified fix for the problem and I'll try to get you an estimate for when that would be hitting production.
Seeing as you're a very technical user, you may find the PR interesting - basically, the combination of "optional" and "array" was a corner case that had incorrect handling.
No problem, thanks for the quick response!
I'm having the same problem as Micah, trying to input an Array[File] as an optional argument fails to coerce the array and I'm getting the same "Expected single value for workflow input..." message. Tried setting the argument as required and it solved the issue.
The fix is scheduled to release on Wednesday July 31 at 4 PM.
I am pleased to report a schedule change in our favor, the fix has just been released.
Thank you for your help! As a follow-up question (and this may be pretty simple), how do I specify an array of length 1 in a .tsv file which I will upload as sample metadata? I have a column which I want to contain an array of files (with length 1 in this case) for each sample. Here are the patterns I have tried:
The input seems to be processed as a string, then coerced into an array of length 1 containing that string (e.g. the first input would result in an array ["[\"gs://path/to/file.fastq\"]"] )
It still works if I just upload the value as "gs://path/to/file.fastq", but as you said above, the coercion from this to an array is not intentional and should not be relied upon, so I wanted to figure out how to get the explicit array input working.
Very interested in this as well, I have experienced the same issue with optional Array inputs.
Stephen,
Here is the follow up thread!
https://support.terra.bio/hc/en-us/community/posts/360048411512-Uploading-metadata-TSV-file-including-arrays-of-length-1?page=1#community_comment_360006958812
Please sign in to leave a comment.