Importing TSVs that contain arrays Completed

Post author
Gwen Miller

Hello,

It would be extremely helpful if Terra supported importing arrays into your data model directly with a TSV. This functionality would be useful in a variety of situations, and the lack of functionality has caused me issues in the past. Take the following example:

  1.  a workflow produces a column with array data
  2. the TSV is downloaded for some reason (e.g. systematic changes to metadata, etc.)
  3. the edited TSV is uploaded back to Terra
  4. the array data is not uploaded in the correct format
  5. downstream workflows depending on the array data as input fail

According to this link, this functionality was being worked on even 1.5 years ago. Are there any updates on this project? Is there an estimated timeline for this functionality to exist?

Thank you,

Gwen

Comments

13 comments

  • Comment author
    Samantha (she/her)
    • Official comment

    Hi all,

     

    This feature request was implemented as part of the release on September 13, 2021. Please see below for more information:

    Terra now supports the ability to upload arrays inside a TSV. If your data already contains arrays, you can now download the data as TSV, then re-upload that TSV without causing changes to your data.

    When creating a new TSV, any cell that you would like to be an array must be a valid JSON array - for instance, boolean values must be lower-case a la [true, false] instead of [True, FALSE].

    Inside the array, elements can be strings, numbers, booleans, or entity references. This feature does not yet support arrays of nulls or arrays of json objects (unless those json objects are entity references).

    Arrays must be of homogenous datatype; for instance, you cannot mix strings and numbers in the same array.

    If the TSV cell contains a syntactically invalid JSON array, the value will be saved as a string.

     

    Best,

    Samantha

  • Comment author
    Sabrina Camp

    I am running into the same issue as well. I would love to see this feature! 

    2
  • Comment author
    Alexander Hsieh

    +1 running into the same issue, would love to know if there's any updates on this as well.

     

    Thanks,

    Alex

    1
  • Comment author
    Megan Shand

    Hello, 

    Is there any update on this? I'm running into this as well and the workaround I found here of writing a WDL to output an array is fairly cumbersome. 

    Thanks!

    1
  • Comment author
    Brendan Reardon

    Me three! 

    0
  • Comment author
    Jason Cerrato

    Hi all,

    Thank you for your feature request. I've logged these requests in our database for the Product team to review. I'll also be bringing this up with the appropriate manager at our next meeting due to the amount of interest. We'll be happy to let you know of any progress made in building this feature!

    Kind regards,

    Jason

    0
  • Comment author
    Chris Whelan

    This fix would be extremely helpful for me and my team as well.

    0
  • Comment author
    Chris Whelan

    Just to expand on the reason why this is important for my team:

    We're trying to figure out ways to automatically create example workspaces in Terra when we deploy a new workflow or workflow version. This keeps us from being able to get the workspace set up the way we'd like it without manually adding or editing values in the UI. 

    We'd like to be able to upload arrays in TSV into both data tables and into the "workspace data" section of the workspace.

    Currently we can upload TSVs into workspace data but we still need to go into the UI to change the data type of array values from "String" to "String List".

    0
  • Comment author
    Samantha (she/her)

    Thanks for the additional context, Chris. I've shared this with our product team to review.

    0
  • Comment author
    Samantha (she/her)

    Hi Megan Shand,

    I'll get in touch with our product team and find out about any updates.

    Best,

    Samantha

    0
  • Comment author
    James Emery

    Hello, 

    I recently ran into this issue as well. I spent a bunch of time debugging my WDL because it was not parsing an array correctly and I didn't realize the issue. If there is no ETA on being able to parse TSVs with arrays can there at least be a warning put into the UI somewhere you the TSV is uploaded that alerts you there was a problem you might have to address? 

    (Also would it be possible to expand the textbox width in the "edit row" UI so its easier to expand these out correctly?)

    Thank You,
    James

    0
  • Comment author
    Samantha (she/her)

    Hi all,

    Our team is currently evaluating how we can fit in this feature request. We don't have a firm date yet, but we should know more on whether it is possible in the coming weeks.

    Best,

    Samantha

    0
  • Comment author
    Justin Rhoades

    Hi,

    Curious if there are any updates on this feature request.

    Best,

    Justin

    0

Please sign in to leave a comment.