Unable to copy an output to a google bucket

Post author
Seunghun Han

Hi, 

I'm using Hail package on Terra Notebook, and when I make a VCF output using following line,

hl.export_vcf(filtered_mt,'1106_RCC_filtered_variants.vcf.bgz') 

2019-11-06 20:07:49 Hail: INFO: while writing:
    1106_RCC_filtered_variants.vcf.bgz
  merge time: 469.209ms

I can import the vcf without any problem by using below line 

test=hl.import_vcf('1106_RCC_filtered_variants.vcf.bgz')

However, when I try to copy the vcf output to a google bucket using 

!gsutil cp '1106_RCC_filtered_variants.vcf.bgz' 'my_google_bucket_path', 

It gives me this error, and I cannot copy it 

CommandException: No URLs matched: 1106_RCC_filtered_variants.vcf.bgz

All other outputs I have in the same notebook can be copied without any problem using gsutil, 
and when I do os.listdir("."), I see all the outputs except for '1106_RCC_filtered_variants.vcf.bgz' 

I would appreciate it if you could help me on this

Thanks! 


Comments

5 comments

  • Comment author
    Tiffany Miller

    Hi Seunghun, 

    That is odd. Maybe you could try this or get ideas from the Hail tutorial notebooks here:

    https://app.terra.bio/#workspaces/help-gatk/Hail-Notebook-Tutorials/notebooks

    0
  • Comment author
    Seunghun Han

    Hi Tiffany,

    Thank you for the suggestion, but I've already been using the ways described in the link you provided to copy files to and from google bucket
    Looks like I need to talk to Hail people. 

    -Seunghun 

    0
  • Comment author
    Tiffany Miller

    Hi Seunghun, ok, good luck. The Hail forum is here. If you want us to take a deeper look, let us know!

    0
  • Comment author
    Seunghun Han

    I've talked to the Hail people, and found an answer. Thanks!

    0
  • Comment author
    Yaron Einhorn

    I had the same issue. I'm still not sure where the data was saved, but in order to overcome this, I had to save it directly to the google path rather than the current folder and then copying it .

    (e.g

    hl.export_vcf(filtered_mt,'my_google_bucket_path/1106_RCC_filtered_variants.vcf.bgz') 

    In addition, if you already did this "mistake", for me it took it a really long time to with the original vcf  writing, 

    writing the "test" from the question was much much faster

    (test=hl.import_vcf('1106_RCC_filtered_variants.vcf.bgz')

    hl.export_vcf(test,'my_google_bucket_path/1106_RCC_filtered_variants.vcf.bgz') 

    )

    0

Please sign in to leave a comment.