Need Help?

Search our documentation and community forum

Terra is a cloud-native platform for biomedical researchers to access data, run analysis tools, and collaborate.
Terra powers important scientific projects like FireCloud, AnVIL, DataSTAGE. Learn more.

Accessing GTEx data when running workflow

Comments

11 comments

  • Avatar
    Q DI

    I am having the exact same problem.  It appears that I can download directly from the Google Cloud Platform console, but using the command line utility gsutil, I get the following:

    gsutil cp gs://fc-secure-ff8156a3-ddf3-42e4-9211-0fd89da62108/GTEx_Analysis_2017-06-05_v8_RNAseq_BAM_files/GTEX-1117F-0226-SM-5GZZ7.Aligned.sortedByCoord.out.patched.md.bam.bai ./
    BadRequestException: 400 Bucket is requester pays bucket but no user project provided.

    I set up my credentials with gcloud auth list, and I have my account set to the free AnVIL project.  I also had pass_credentials_to_gsutil = True.

     

    Thanks for your assistance

     

    1
    Comment actions Permalink
  • Avatar
    Sushma Chaluvadi

    Hello Tanya and Q,

    When using the command line, can you try passing in --gcs-project-for-requester-pays with the billing project to be billed to see if that works? 

    I will get back to you about options for when using the Workflow in Terra. 

     

    Sushma

     

    0
    Comment actions Permalink
  • Avatar
    Q DI

    Hi Sushma,

    Thanks for your response.  I think the option --gcs-project-for-requester-pays is only an option for gatk?  I am using the gsutil command directly as specified on the "File Details" of a bam file on the AnVIL_GTEx_V8_hg38 data page.

    Not sure if this is related, but I see this outstanding issue on the gatk github page that references the same error? https://github.com/broadinstitute/gatk/issues/6179

    Thanks for your help.

    0
    Comment actions Permalink
  • Avatar
    Tanya Phung

    Hi Sushma, 

    Thank you so much for looking into this. Any updates on how I can access GTEx data when using the workflow in Terra? 


    Thank you!

    Tanya

    0
    Comment actions Permalink
  • Avatar
    Sushma Chaluvadi

    Q DI,

    When you run your gsutil cp command can you try adding the -u parameter followed by your google billing project as follows:

    gsutil -u [billing-project] cp gs://fc-secure-ff8156a3-ddf3-42e4-9211-0fd89da62108/../GTEX-1117F-0226-SM-5GZZ7.Aligned.sortedByCoord.out.patched.md.bam.bai ./

    Passing in your billing project with the -u parameter should allow you to download files with gsutil when the bucket has requester-pays enabled.

    0
    Comment actions Permalink
  • Avatar
    Sushma Chaluvadi

    Tanya,

    Apologies for the delay! I spoke with some of our WDL developers and they said that the way to circumvent this is to manually modify the tasks in the WDL that are localizing files from the requester-pays bucket. You would need to pass in the --gcs-project-for-requester-pays parameter so that the command knows to accept the billing-project you want to bill for accessing files in the requester-pays bucket.

     

    I am currently working to modify one of the tasks in the haplotypecaller WDL for you to test out! I will be in touch shortly!

    0
    Comment actions Permalink
  • Avatar
    Tanya Phung

    Hi Sushma, 

    Thank you so much for the help. 

    Best,
    Tanya

    0
    Comment actions Permalink
  • Avatar
    Q DI

    Hi Sushma,

    Thanks much, the -u parameter was exactly what I needed!

    Just a suggestion, it may be good to indicate the existence of that parameter here: https://support.terra.bio/hc/en-us/articles/360029251091-Broad-Genomics-Downloading-data-from-a-Terra-workspace#gsutildownload

    Unfortunately, none of the Google gsutil documentation mentions this parameter (or at least if they do, I have not been able to find it :) )

    Thanks again!

    0
    Comment actions Permalink
  • Avatar
    Sushma Chaluvadi

    Q DI,

    I will let our team know to add this information! For reference on this thread, here is the Google documentation on the -u parameter: https://cloud.google.com/storage/docs/using-requester-pays#using

     

    Sushma

    0
    Comment actions Permalink
  • Avatar
    Q DI

    Thanks for the Google documentation link!

    0
    Comment actions Permalink
  • Avatar
    Tanya Phung

    Hi Sushma, 

    I just wanted to check in to see if there is a test version of the Haplotype caller WDL to allow me to specify the billing project. Thank you so much for your help. 

    Best,
    Tanya

    0
    Comment actions Permalink

Please sign in to leave a comment.

Powered by Zendesk