Terra is a cloud-native platform for biomedical researchers to access data, run analysis tools, and collaborate. Terra powers important scientific projects like FireCloud, AnVIL, and DataSTAGE. Learn more

Issue interpreting array inputs to workflow

Comments

26 comments

  • Avatar
    Sushma Chaluvadi

    Hello Micah,

    Checking into this right now and will get back to you with an update.

    Sushma

    0
    Comment actions Permalink
  • Avatar
    Sushma Chaluvadi

    Micah,

    Can you try putting `[ ]` around the element of your list?

    0
    Comment actions Permalink
  • Avatar
    Micah Rickles-Young

    Yes, I have done that

    0
    Comment actions Permalink
  • Avatar
    Sushma Chaluvadi

    Would you be able to share your workspace that contains this Workflow with GROUP_FireCloud-Support@firecloud.org

    as a Writer so that we can take a look directly - it would be helpful to troubleshoot. We can be removed right after :)

    0
    Comment actions Permalink
  • Avatar
    Micah Rickles-Young

    Sushma,

    I have shared access with you. The workflow is `RNASeqPipeline_withRG`. Just let me know if there is an email I should send a description of the workflow to and/or if I need to share the WDL as well.

    Thanks for your help,

    -Micah

    0
    Comment actions Permalink
  • Avatar
    Sushma Chaluvadi

    Micah,

    I am unable to see the workspace - can you double check that it was shared successfully? A good way to check is if you see the word "Pending" next to the user email from the Share module. Does your workspace have an Authorization Domain on it? 

    0
    Comment actions Permalink
  • Avatar
    Micah Rickles-Young

    The workspace doesn't have an authorization domain and there isn't a "Pending" message next to the FireCloud support email. Can I try sharing it with you directly?

    0
    Comment actions Permalink
  • Avatar
    Sushma Chaluvadi

    Ah I misread workflow for workspace and was searching the wrong name - I am able to see it, sorry about that confusion. We will take a look and get back to you.

    I believe however you may need to also share the Workflow with us independently. You can do so from the Methods Repository.

    0
    Comment actions Permalink
  • Avatar
    Sushma Chaluvadi

    Micah,

    Were you able to share the Method that you are working with? It shows as unavailable to us.

    0
    Comment actions Permalink
  • Avatar
    Adam Nichols

    Hi,

    In reference to

    However, since I tried launching the workflow yesterday afternoon going forward, this no longer works.

    can you please provide screenshots and/or exact error messages, please.

    0
    Comment actions Permalink
  • Avatar
    Micah Rickles-Young

    To show what I mean by the timing, here is a screenshot of the same configuration results before and after the issue:

    As for the error message, here is that:

    For whatever reason (maybe a Cromwell update or the WDL update?), coercion from a single input to an array of length one no longer works. One thing I also may try is to make the array required and not optional, just to see if that could be an issue.

    0
    Comment actions Permalink
  • Avatar
    Adam Nichols

    Based on a quick browse through the WDL spec, I think the coercion from File to Array[File] is not actually supported - rather, it happened to work by accident.

    Such an accident is never guaranteed to keep working forever and indeed a recent maintenance update to WDL caused it to go away.

    If you think I'm off-base about the spec - which is very possible! - please do speak up.

    0
    Comment actions Permalink
  • Avatar
    Micah Rickles-Young

    I just tried making the inputs required rather than optional and the single inputs were coerced into an array. (FirstEndFastq and SecondEndFastq are both File objects)

    0
    Comment actions Permalink
  • Avatar
    Adam Nichols

    Interesting, thanks for the update.

    I still believe you may be relying on undocumented functionality and recommend you implement the previously-suggested approach:

    Can you try putting `[ ]` around the element of your list?

    0
    Comment actions Permalink
  • Avatar
    Micah Rickles-Young

    That's the other thing: that doesn't work unless I set the input as required and not optional

    I set up a side workflow while I was trying that out to take a single input and output an array of length one with that input:

    I then set the output arrays as inputs to the configuration where the arrays are optional:

    However, the input is still interpreted as single files for some reason:

    I then changed the configuration to the one with the required arrays:

    And the inputs were correctly interpreted:

     

    So I think an issue here is how optional arrays are being interpreted. Tomorrow I'll test out an optional array with length >1 to determine whether the issue is specifically about inputs with length 1.

    1
    Comment actions Permalink
  • Avatar
    Adam Nichols

    Tomorrow I'll test out an optional array with length >1 to determine whether the issue is specifically about inputs with length 1.

    That would be great to know, yes.

    0
    Comment actions Permalink
  • Avatar
    Adam Nichols

    We have identified the issue that is preventing the application from correctly accepting arrays as the value of an optional array input. A fix is in progress.

    At this time, we do not believe that coercing a single String or File in the inputs into an Array[String] or Array[File] is a supported behavior.

    1
    Comment actions Permalink
  • Avatar
    Micah Rickles-Young

    This may part of the issue you identified with optional arrays, but her is what happened when I tried optional arrays with length >1:

    First, I modified my support workflow to output an array that is just the input values repeated twice:

    I again used these as inputs to optional arrays:

    And launching the workflow prompted this immediate error:

    It seems to be expecting one value for an optional array.

     

    For your second point about coercion from a single value to an array, I think it makes sense for that not to occur, so once the optional array bug is fixed, I will make sure my inputs are actually arrays. 

    0
    Comment actions Permalink
  • Avatar
    Adam Nichols

    Thank you for the updates, it's very helpful to see all the things you've tried.

    We have a verified fix for the problem and I'll try to get you an estimate for when that would be hitting production.

    Seeing as you're a very technical user, you may find the PR interesting - basically, the combination of "optional" and "array" was a corner case that had incorrect handling.

    1
    Comment actions Permalink
  • Avatar
    Micah Rickles-Young

    No problem, thanks for the quick response!

    0
    Comment actions Permalink
  • Avatar
    Jorge Buendia-Buendia

    I'm having the same problem as Micah, trying to input an Array[File] as an optional argument fails to coerce the array and I'm getting the same "Expected single value for workflow input..." message. Tried setting the argument as required and it solved the issue.

    0
    Comment actions Permalink
  • Avatar
    Adam Nichols

    The fix is scheduled to release on Wednesday July 31 at 4 PM.

    0
    Comment actions Permalink
  • Avatar
    Adam Nichols

    I am pleased to report a schedule change in our favor, the fix has just been released.

    0
    Comment actions Permalink
  • Avatar
    Micah Rickles-Young

    Thank you for your help! As a follow-up question (and this may be pretty simple), how do I specify an array of length 1 in a .tsv file which I will upload as sample metadata? I have a column which I want to contain an array of files (with length 1 in this case) for each sample. Here are the patterns I have tried:

    • ["gs://path/to/file.fastq"]
    • [gs://path/to/file.fastq]
    • [ "gs://path/to/file.fastq" ]
    • [ gs://path/to/file.fastq ]

    The input seems to be processed as a string, then coerced into an array of length 1 containing that string (e.g. the first input would result in an array ["[\"gs://path/to/file.fastq\"]"] )

    It still works if I just upload the value as "gs://path/to/file.fastq", but as you said above, the coercion from this to an array is not intentional and should not be relied upon, so I wanted to figure out how to get the explicit array input working.

    0
    Comment actions Permalink
  • Avatar
    Stephen Fleming

    Very interested in this as well, I have experienced the same issue with optional Array inputs.

    0
    Comment actions Permalink

Please sign in to leave a comment.

Powered by Zendesk