How can I get TCGA data into the notebooks for analysis?

• 

  Samantha (she/her)

  • September 17, 2020 14:34

  Hi Rebecca Eliscu,

   

  Thanks for reaching out. The hg38 workspaces require users to run a workflow in order to retrieve the data. The workflows for this purpose can be found in the 'Workflows' tab of the workspace. Once you have retrieved the data and copied it into your own workspace's bucket, you can follow the steps in this article to use that data within your notebook: https://support.terra.bio/hc/en-us/articles/360046617372-Analyzing-data-from-a-workspace-bucket-in-a-notebook

  Please let me know if you have any questions.

   

  Best,

  Samantha

  0
• 

  Rebecca Eliscu

  • September 17, 2020 20:41

  Hi Samantha, thanks for your response.

   

  Having run a workflow, I see each sample's output is buried deep within it's own sub directory path in my workspace bucket, so I have a couple questions about how to proceed:

  1) How do I actually point to these individual paths from the notebook? In other words, I see that you can set an environmental variable for the bucket, but how do I dig deeper into the bucket / mount the bucket to load each file?

  2) How can I efficiently "stitch" together all of the outputs into a single file/matrix?

  Thanks,

  Rebecca

   

  0
• 

  Samantha (she/her)

  • Edited September 18, 2020 20:38

  Hi Rebecca Eliscu

   

  1) Once you set the environment variable, you would need to do a recursive copy of the main bucket directory by doing running the command gsutil cp -R​. It would look something like this for a Python kernel: 

  

  You would just need to replace ac0bcc8b-3425-46bc-a747-dc1e373d35f2/​ with the bucket directory you want to copy the contents of. You can read more about the gsutil cp command here: https://cloud.google.com/storage/docs/gsutil/commands/cp 

   

  2) Could you provide more detail about what you are trying to do here? What are the files you are trying to "stitch" together?

   

  Best,

  Samantha

  0
• 

  Rebecca Eliscu

  • September 18, 2020 20:58

  Hi Samantha, 

  Thanks again.

  The output of the workflow (gdc_file_downloader__WXS__SomSnipAggrMask__masked_somatic_mutation__maf__cfg) is MAF files for 1015 samples. As such, to do any analysis on the measurements across samples, I would need to append the data per sample together, no?

  E.g. here's a snapshot of some of these files in my bucket: 

  

  I'm seeing that there are some R tools for processing MAF files, which I'll try out, including merge_maf (https://rdrr.io/github/PoisonAlien/maftools/man/merge_mafs.html).

  I'm curious though, is there a way to customize output of a workflow such that all the files are combined into one? Is there a better way to handle/process this data in general?

  Hopefully that made sense.

  Thanks again for the help!

   

   

  0
• 

  Samantha (she/her)

  • September 21, 2020 20:31

  Hi Rebecca Eliscu,

   

  You can customize the actual workflow gdc_file_downloader__WXS__SomSnipAggrMask__masked_somatic_mutation__maf__cfg​ to combine the MAF files for a sample into one file before writing that file as output. As far as a method configuration, there isn't a way to combine output files if the script is written to do that ahead of time. 

  You can also read the files into a notebook and use the R package to combine them too if you prefer that.

   

  Best,

  Samantha

  0
• 

  Rebecca Eliscu

  • September 21, 2020 20:52

  This has been very helpful -- thank you!

  0
• 

  Samantha (she/her)

  • September 21, 2020 21:10

  Happy to help! Please let us know if you need any further assistance.

  0

Please sign in to leave a comment.