trouble selecting data sub set
I am doing a lot of usability, performance, and scaling testing. So far my preliminary results look promising. I stumbled upon "an ease of use, usability issue"
My workspace has a 'samples' table with over 17,000 cases. I want to create a couple of small sets to do performance and scalability testing on. I want the results to populate the "master" sample table.
This turns out to be very hard to do. Workflows, step 2 does not provide an easy way to select subsets that are disjoint. What I want to do is define 3 sets before running anything. This would be easy if I could download the master sample files, slice and dice to select my 3 sets, and then upload the data set membership list. It would be really cool if I could do this in a notebook. (good platforms typically try to make some kinds of problems easily and provide an escape hatch to let you work around uses cases that do not quite fit in". I think it would be very hard to create a comprehensive data selection UI.
My current workaround is very tedious. I sort a column of data. You can not do anything sophisticated. I use the first column in the "master sample" table to select 20 samples that do not have a value in this column
I run the workflow. Next, I have to way a couple of hours for the workflow to complete, go back, and repeat. very error-prone and tedious
I suspect anyone working with a large data set will want to do something similar if for no other reason than to work out bugs in their task more quickly.
Kind regards
Andy
Comments
4 comments
Hi Andy,
Thanks for sending this in. Creating sets through a notebook should be possible! Let me think over this question and look at our tools and get back to you.
In the meantime, you could download your sample_set membership TSV and edit it as needed to manually define your sets, then upload that TSV back to your workspace. You could even create a sample_set membership TSV from scratch and upload it, as Terra will simply append those rows to the table that currently exists.
Here's an example sample set table (found here):
I'll get back to you as soon as I can regarding the notebook option.
Kind regards,
Jason
Hi Andrew Davidson,
You might be able to make the changes you're describing in a notebook by using FISS. FISS is a programmatic interface for Terra (also known as FireCloud). One of our featured workspaces, BioData Catalyst Collection has a notebook called Intro to FISS API in Python that can serve as a helpful starting point. To run through the notebook, first clone the BioData Catalyst Collection workspace, then start the notebook in Edit mode. You may need to create a runtime configuration if you do not yet have one set up for your billing project. The defaults should work just fine!
I'm also investigating another package that may work for this purpose. I'll follow up after I've investigated it a bit more!
Kind regards,
Jason
thanks
Andy
Hi Andy,
Just to close the loop here, I wanted to highlight another package that could possibly work for your purposes: the AnVIL R package.
This section of the documentation describes a function that allows you to define your sample sets
We also have a general article about the AnVIL package in our Documentation here: https://support.terra.bio/hc/en-us/articles/4403801203483-Accessing-workspace-data-tables-using-AnVIL-s-R-package
Kind regards,
Jason
Please sign in to leave a comment.