How can I get my files into the data model?
Apparently I have exceeded the maximum number of workflow outputs by 1.8%. I don't know of another way to get an array into the data model and I suspect if I try to read_lines from a FOFN (file of file names) Cromwell won't let me read a file that big. Any other ideas? If I have to combine exactly 364 of these shards before I can output them I'm going to be grumpy.
Comments
4 comments
Hi Laura,
Thanks for writing in. We'll take a closer look at this and get back to you as soon as we can.
Kind regards,
Jason
Hi Laura,
Would a FOFN be larger than 10 MB? I'm looking at our recommended solutions for particularly large files here, and I'm wondering if either of those workarounds would work for your needs.
Kind regards,
Jason
Looks like it's 1.5M for all the VCF paths with 6787 VCFs and the same number of indexes and the same number of intervals lists, so a config that outputs any two of those should be fine. The simplest thing might be to no output the scattered intervals to the data model, which will get me to 13574 "attributes", which should work. I didn't realize the limit is 10MB now. That should be fine for this project anyway. So I have a couple of options to try. Thanks Jason!
My pleasure, Laura! If you run into any further trouble please let us know and we'll be happy to help.
Kind regards,
Jason
Please sign in to leave a comment.