Workflow outputs not copied to data model
Hi-
I seem to have an issue similar to: https://support.terra.bio/hc/en-us/community/posts/360043439671-Successful-workflows-not-copying-to-data-model
I have a workflow that completes successfully and should add a new column to my sample set with the results as an array of files. This column is created but filled with an empty array. I have verified that the desired output was produced and is stored in my workspace google bucket. Below is the configuration, job manager for the workflow, and data model after running.
Comments
7 comments
The workflow can be found here: https://dockstore.org/workflows/github.com/AnalysisCommons/genesis_wdl/genesis_tests:master?tab=info
Update: I ran the exact same workflow but specified a different output column in the data model. This time, the specified column was filled correctly. Could this be an issue with overwriting a column in the data model with an array of files? I'm not sure if that column existed prior to running the workflow.
Hi Timothy,
Apologies for the delay! When you first added this.raw_gwas_results in the shared screenshot, was that a result of clicking on "Use Defaults" or did you happen to pick that column name and type it in manually.
I see that you were not sure if the raw_gwas_results already existed in your Sample Set table so I am curious to determine if it is a matter of overwrite.
I had typed this in manually. Have you heard of this potential overwrite issue before?
I am checking with the team to be sure but I think that if you have a column that already exists in the Data Table, you cannot overwrite outputs from a Workflow. If you press "Use Defaults" what is the name of the column that is generated and does the workflow append outputs to that column successfully?
Sushma
Hi-
If I use the default, "this.result", a new column is created and populated with the correct array of files.
Is not overwriting an expected behavior? If so, is this stated in the docs? For our group, this would be important to know, especially when we're iterating through versions of workflows using the same input data.
-Tim
Hi Tim,
I did a test run to see what the status of Data Model columns are with different versions of a Tool. I used a simple Hello World WDL that writes a String Hello World to the Data Model as output (in the out column - the Default).
I repeated the process by modifying the WDL to output the strong "Overwrite Successful" to the same out column - again the Default:
You can see that the original Hello World! is being replaced by Overwrite successful!! - this tells me that column content is overwritten.
Finally to test one more item, I modified the WDL one more time to output a string Check new column! and changed the Default from this.out to this.out_new_col and you can see that the string is appended to a new column without over writing the previous output:
Based on this I would say that overwrite is a property of the Data Model if the output column is the same but if you were to change the Default output column, you will not get an overwrite.
Please sign in to leave a comment.