Too many samples to copy data table
Hello,
I'm working on some auxiliary data generation in Terra for All of Us and have been running into a bunch of issues due to the size of that cohort. In the upcoming release we'll have about 451,258 samples. We have a master read-only workspace for data generation and then I have a workspace on a different billing project that's linked to the appropriate cost object for this project. I want to copy some columns from the master data table to become the data table in my workspace so I don't have to manage any potential workflow failures manually. However, rawls thinks this is too many samples:
Other team members have suggested some workarounds using the firecloud API or rawls API, but would it be possible to double this arbitrary-seeming rawls limit?
Thanks,
Laura
Comments
1 comment
Hi Laura,
Thanks for writing in! I spoke with our developers and the 300,000 page size limit is there to protect the stability of the Terra Platform, and we will be unable to increase it at this time. Our suggestion would be to use those workarounds with the API's at this time.
That said, the team thought it would be a great idea to create an API call that would help with this directly. I've added your name to that feature request and I'll contact you if that feature is built one day.
Please let me know if you have any questions.
Best,
Josh
Please sign in to leave a comment.