Issue interpreting array inputs to workflow

Comments

26 comments

  • Sushma Chaluvadi

    Hello Micah,

    Checking into this right now and will get back to you with an update.

    Sushma

  • Sushma Chaluvadi

    Micah,

    Can you try putting `[ ]` around the element of your list?

  • Micah Rickles-Young

    Yes, I have done that

  • Sushma Chaluvadi

    Would you be able to share your workspace that contains this Workflow with GROUP_FireCloud-Support@firecloud.org

    as a Writer so that we can take a look directly - it would be helpful to troubleshoot. We can be removed right after :)

  • Micah Rickles-Young

    Sushma,

    I have shared access with you. The workflow is `RNASeqPipeline_withRG`. Just let me know if there is an email I should send a description of the workflow to and/or if I need to share the WDL as well.

    Thanks for your help,

    -Micah

  • Sushma Chaluvadi

    Micah,

    I am unable to see the workspace - can you double check that it was shared successfully? A good way to check is if you see the word "Pending" next to the user email from the Share module. Does your workspace have an Authorization Domain on it? 

  • Micah Rickles-Young

    The workspace doesn't have an authorization domain and there isn't a "Pending" message next to the FireCloud support email. Can I try sharing it with you directly?

  • Sushma Chaluvadi

    Ah I misread workflow for workspace and was searching the wrong name - I am able to see it, sorry about that confusion. We will take a look and get back to you.

    I believe however you may need to also share the Workflow with us independently. You can do so from the Methods Repository.

  • Sushma Chaluvadi

    Micah,

    Were you able to share the Method that you are working with? It shows as unavailable to us.

  • Adam Nichols

    Hi,

    In reference to

    However, since I tried launching the workflow yesterday afternoon going forward, this no longer works.

    can you please provide screenshots and/or exact error messages, please.

  • Micah Rickles-Young

    To show what I mean by the timing, here is a screenshot of the same configuration results before and after the issue:

    As for the error message, here is that:

    For whatever reason (maybe a Cromwell update or the WDL update?), coercion from a single input to an array of length one no longer works. One thing I also may try is to make the array required and not optional, just to see if that could be an issue.

  • Adam Nichols

    Based on a quick browse through the WDL spec, I think the coercion from File to Array[File] is not actually supported - rather, it happened to work by accident.

    Such an accident is never guaranteed to keep working forever and indeed a recent maintenance update to WDL caused it to go away.

    If you think I'm off-base about the spec - which is very possible! - please do speak up.

  • Micah Rickles-Young

    I just tried making the inputs required rather than optional and the single inputs were coerced into an array. (FirstEndFastq and SecondEndFastq are both File objects)

  • Adam Nichols

    Interesting, thanks for the update.

    I still believe you may be relying on undocumented functionality and recommend you implement the previously-suggested approach:

    Can you try putting `[ ]` around the element of your list?

  • Micah Rickles-Young

    That's the other thing: that doesn't work unless I set the input as required and not optional

    I set up a side workflow while I was trying that out to take a single input and output an array of length one with that input:

    I then set the output arrays as inputs to the configuration where the arrays are optional:

    However, the input is still interpreted as single files for some reason:

    I then changed the configuration to the one with the required arrays:

    And the inputs were correctly interpreted:

     

    So I think an issue here is how optional arrays are being interpreted. Tomorrow I'll test out an optional array with length >1 to determine whether the issue is specifically about inputs with length 1.

  • Adam Nichols

    Tomorrow I'll test out an optional array with length >1 to determine whether the issue is specifically about inputs with length 1.

    That would be great to know, yes.

  • Adam Nichols

    We have identified the issue that is preventing the application from correctly accepting arrays as the value of an optional array input. A fix is in progress.

    At this time, we do not believe that coercing a single String or File in the inputs into an Array[String] or Array[File] is a supported behavior.

  • Micah Rickles-Young

    This may part of the issue you identified with optional arrays, but her is what happened when I tried optional arrays with length >1:

    First, I modified my support workflow to output an array that is just the input values repeated twice:

    I again used these as inputs to optional arrays:

    And launching the workflow prompted this immediate error:

    It seems to be expecting one value for an optional array.

     

    For your second point about coercion from a single value to an array, I think it makes sense for that not to occur, so once the optional array bug is fixed, I will make sure my inputs are actually arrays. 

  • Adam Nichols

    Thank you for the updates, it's very helpful to see all the things you've tried.

    We have a verified fix for the problem and I'll try to get you an estimate for when that would be hitting production.

    Seeing as you're a very technical user, you may find the PR interesting - basically, the combination of "optional" and "array" was a corner case that had incorrect handling.

  • Micah Rickles-Young

    No problem, thanks for the quick response!

  • Jorge Buendia-Buendia

    I'm having the same problem as Micah, trying to input an Array[File] as an optional argument fails to coerce the array and I'm getting the same "Expected single value for workflow input..." message. Tried setting the argument as required and it solved the issue.

  • Adam Nichols

    The fix is scheduled to release on Wednesday July 31 at 4 PM.

  • Adam Nichols

    I am pleased to report a schedule change in our favor, the fix has just been released.

  • Micah Rickles-Young

    Thank you for your help! As a follow-up question (and this may be pretty simple), how do I specify an array of length 1 in a .tsv file which I will upload as sample metadata? I have a column which I want to contain an array of files (with length 1 in this case) for each sample. Here are the patterns I have tried:

    • ["gs://path/to/file.fastq"]
    • [gs://path/to/file.fastq]
    • [ "gs://path/to/file.fastq" ]
    • [ gs://path/to/file.fastq ]

    The input seems to be processed as a string, then coerced into an array of length 1 containing that string (e.g. the first input would result in an array ["[\"gs://path/to/file.fastq\"]"] )

    It still works if I just upload the value as "gs://path/to/file.fastq", but as you said above, the coercion from this to an array is not intentional and should not be relied upon, so I wanted to figure out how to get the explicit array input working.

  • Stephen Fleming

    Very interested in this as well, I have experienced the same issue with optional Array inputs.

Please sign in to leave a comment.

Powered by Zendesk