Overview: Reference Disks in Terra

Anton Kovalsky

To save time localizing large reference inputs, Terra can attach a disk containing reference genomes to your Google Virtual Machine (VM). If the checkbox labeled ‘Use Reference Disks’ is selected, the execution engine examines the job inputs to see if any correspond to reference inputs available on a reference disk image.

Reference disk overview

Screenshot showing how to check for a relevant reference input when running a workflow. An orange arrow highlights the check-box labeled 'use reference disks'. Check this box to check when setting up the workflow to indicate that your workflow should check for available reference inputs.

 

What happens when selecting the reference disk option?

If you select this option, the engine mounts an additional disk containing those inputs onto the virtual machine (VM) so it can skip the delocalization process for the reference inputs. See the reference image manifests below.

Human References (image manifest)

HG 19 public image

  • gs://gcp-public-data--broad-references/hg19/v0/Homo_sapiens_assembly19.fasta.fai
  • gs://gcp-public-data--broad-references/hg19/v0/Homo_sapiens_assembly19.dict
  • gs://gcp-public-data--broad-references/hg19/v0/Homo_sapiens_assembly19.haplotype_database.txt
  • gs://gcp-public-data--broad-references/hg19/v0/dbsnp_138.b37.vcf.gz.tbi
  • gs://gcp-public-data--broad-references/hg19/v0/dbsnp_138.b37.vcf.gz
  • gs://gcp-public-data--broad-references/hg19/v0/Homo_sapiens_assembly19.fasta

HG 38 public images

Terra can now access reference files for human genome assembly 38, release 27 for single cell pipelines -- including hg38-sc-v27-mod, which includes both intron and exon annotations. You can learn more about the tool that marks intronic regions here.

  • gs://gcp-public-data--broad-references/hg38/v0/Mills_and_1000G_gold_standard.indels.hg38.vcf.gz
  • gs://gcp-public-data--broad-references/hg38/v0/exome_evaluation_regions.v1.interval_list
  • gs://gcp-public-data--broad-references/hg38/v0/Homo_sapiens_assembly38.fasta.64.ann
  • gs://gcp-public-data--broad-references/hg38/v0/Homo_sapiens_assembly38.dbsnp138.vcf.idx
  • gs://gcp-public-data--broad-references/hg38/v0/Homo_sapiens_assembly38.fasta.fai
  • gs://gcp-public-data--broad-references/hg38/v0/exome_calling_regions.v1.interval_list
  • gs://gcp-public-data--broad-references/hg38/v0/contamination-resources/1000g/1000g.phase3.100k.b38.vcf.gz.dat.UD
  • gs://gcp-public-data--broad-references/hg38/v0/Homo_sapiens_assembly38.known_indels.vcf.gz
  • gs://gcp-public-data--broad-references/hg38/v0/wgs_evaluation_regions.hg38.interval_list
  • gs://gcp-public-data--broad-references/hg38/v0/Homo_sapiens_assembly38.known_indels.vcf.gz.tbi
  • gs://gcp-public-data--broad-references/hg38/v0/Mills_and_1000G_gold_standard.indels.hg38.vcf.gz.tbi
  • gs://gcp-public-data--broad-references/hg38/v0/Homo_sapiens_assembly38.haplotype_database.txt
  • gs://gcp-public-data--broad-references/hg38/v0/wgs_calling_regions.hg38.interval_list
  • gs://gcp-public-data--broad-references/hg38/v0/Homo_sapiens_assembly38.fasta.64.pac
  • gs://gcp-public-data--broad-references/hg38/v0/wgs_coverage_regions.hg38.interval_list
  • gs://gcp-public-data--broad-references/hg38/v0/Homo_sapiens_assembly38.fasta.64.amb
  • gs://gcp-public-data--broad-references/hg38/v0/Homo_sapiens_assembly38.fasta.64.alt
  • gs://gcp-public-data--broad-references/hg38/v0/Homo_sapiens_assembly38.fasta.64.sa
  • gs://gcp-public-data--broad-references/hg38/v0/contamination-resources/1000g/1000g.phase3.100k.b38.vcf.gz.dat.mu
  • gs://gcp-public-data--broad-references/hg38/v0/Homo_sapiens_assembly38.fasta
  • gs://gcp-public-data--broad-references/hg38/v0/contamination-resources/1000g/1000g.phase3.100k.b38.vcf.gz.dat.bed
  • gs://gcp-public-data--broad-references/hg38/v0/Homo_sapiens_assembly38.fasta.64.bwt
  • gs://gcp-public-data--broad-references/hg38/v0/Homo_sapiens_assembly38.dict
  • gs://gcp-public-data--broad-references/hg38/v0/Homo_sapiens_assembly38.dbsnp138.vcf

HG38-sc-v27 Public Image

  • gs://gcp-public-data--broad-references/hg38/v0/gencode.v27.primary_assembly.annotation.gtf
  • gs://gcp-public-data--broad-references/hg38/v0/GRCh38.primary_assembly.genome.fa
  • gs://gcp-public-data--broad-references/hg38/v0/star/star_2.7.9a_primary_gencode_human_v27.tar

HG38-sc-v27-mod Public Image

  • gs://gcp-public-data--broad-references/hg38/v0/single_nucleus/modified_gencode.v27.primary_assembly.annotation.gtf
  • gs://gcp-public-data--broad-references/hg38/v0/single_nucleus/modified_GRCh38.primary_assembly.genome.fa
  • gs://gcp-public-data--broad-references/hg38/v0/single_nucleus/star/modified_star_2.7.9a_primary_gencode_human_v27.tar

Nirvana public image

Nirvana is a software that annotates genomic variants and their effects. The reference files available on Terra pull from datasets such as gnomAD, the 1000 Genomes Project, and the Clinical Genome Resource.  

  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/phyloP_hg38.npd
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/phyloP_hg38.npd.idx
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/gnomAD_gene_scores_2.1.nga
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/1000_Genomes_Project_Phase_3_v3_plus.nsa
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/1000_Genomes_Project_Phase_3_v3_plus_refMinor.rma.idx
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/References/Homo_sapiens.GRCh38.Nirvana.dat
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/Cache/GRCh38/Both.sift.ndb
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/ClinGen_Dosage_Sensitivity_Map_20220512.nsi
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/SpliceAi_1.3.nsa
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/dbSNP_151_globalMinor.nsa
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/TOPMed_freeze_5.nsa
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/Gerp_20110522.gsa
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/Cache/GRCh38/Both.transcripts.ndb
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/MultiZ100Way_20171006.pcs
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/dbSNP_155.nsa.idx
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/ClinVar_20220505.nsa
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/dbSNP_155.nsa
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/Cache/GRCh38/Both.polyphen.ndb
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/TOPMed_freeze_5.nsa.idx
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/gnomAD_SV_2.1.nsi
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/ClinGen_disease_validity_curations_20220512.nga
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/OMIM_20220516.nga
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/REVEL_20200205.nsa
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/GME_20160618.nsa.idx
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/DANN_20200205.gsa.idx
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/PrimateAI_0.2.nsa
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/ClinVar_20220505.nsi
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/GME_20160618.nsa
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/1000_Genomes_Project_Phase_3_v3_plus.nsa.idx
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/gnomAD_LCR_2.1.lcr
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/PrimateAI_0.2.nsa.idx
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/ClinGen_Dosage_Sensitivity_Map_20220512.nga
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/MITOMAP_SV_20200819.nsi
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/REVEL_20200205.nsa.idx
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/ClinGen_20160414.nsi
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/SpliceAi_1.3.nsa.idx
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/DECIPHER_201509.nsi
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/DANN_20200205.gsa
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/gnomAD_3.1.2.nsa.idx
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/COSMIC_GeneFusions_94.gfj
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/dbSNP_151_globalMinor.nsa.idx
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/FusionCatcher_1.33.gfs
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/1000_Genomes_Project_(SV)_Phase_3_v5a.nsi
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/MITOMAP_20200819.nsa.idx
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/ClinVar_20220505.nsa.idx
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/1000_Genomes_Project_Phase_3_v3_plus_refMinor.rma
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/Gerp_20110522.gsa.idx
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/MITOMAP_20200819.nsa
  • gs://broad-public-datasets/gvs/vat-annotations/Nirvana/3.18.1/SupplementaryAnnotation/GRCh38/gnomAD_3.1.2.nsa

Exomiser public image

Exomiser is a software that uses whole-exome and whole-genome sequencing data to identify variants that may cause disease.

  • gs://gcp-public-data--broad-references/hg38/v0/ReMM/v0.3.1/ReMM.v0.3.1.post1.hg38.tsv.gz
  • gs://gcp-public-data--broad-references/hg38/v0/ReMM/v0.3.1/ReMM.v0.3.1.post1.hg38.tsv.gz.tbi
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_phenotype/rw_string_10.mv
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_phenotype/phenix/out/9.out
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_phenotype/phenix/out/9_symmetric.out
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_phenotype/phenix/out/8.out
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_phenotype/phenix/out/8_symmetric.out
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_phenotype/phenix/out/7.out
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_phenotype/phenix/out/7_symmetric.out
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_phenotype/phenix/out/6.out
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_phenotype/phenix/out/6_symmetric.out
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_phenotype/phenix/out/5.out
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_phenotype/phenix/out/5_symmetric.out
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_phenotype/phenix/out/4.out
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_phenotype/phenix/out/4_symmetric.out
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_phenotype/phenix/out/3.out
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_phenotype/phenix/out/3_symmetric.out
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_phenotype/phenix/out/20.out
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_phenotype/phenix/out/20_symmetric.out
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_phenotype/phenix/out/2.out
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_phenotype/phenix/out/2_symmetric.out
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_phenotype/phenix/out/19.out
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_phenotype/phenix/out/19_symmetric.out
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_phenotype/phenix/out/18.out
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_phenotype/phenix/out/18_symmetric.out
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_phenotype/phenix/out/17.out
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_phenotype/phenix/out/17_symmetric.out
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_phenotype/phenix/out/16.out
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_phenotype/phenix/out/16_symmetric.out
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_phenotype/phenix/out/15.out
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_phenotype/phenix/out/15_symmetric.out
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_phenotype/phenix/out/14.out
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_phenotype/phenix/out/14_symmetric.out
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_phenotype/phenix/out/13.out
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_phenotype/phenix/out/13_symmetric.out
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_phenotype/phenix/out/12.out
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_phenotype/phenix/out/12_symmetric.out
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_phenotype/phenix/out/11.out
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_phenotype/phenix/out/11_symmetric.out
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_phenotype/phenix/out/10.out
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_phenotype/phenix/out/10_symmetric.out
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_phenotype/phenix/out/1.out
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_phenotype/phenix/out/1_symmetric.out
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_phenotype/phenix/hp.obo
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_phenotype/phenix/ALL_SOURCES_ALL_FREQUENCIES_genes_to_phenotype.txt
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_phenotype/hp.obo
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_phenotype/2302_phenotype.h2.db
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_hg38/2302_hg38_variants.mv.db
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_hg38/2302_hg38_transcripts_ucsc.ser
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_hg38/2302_hg38_transcripts_refseq.ser
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_hg38/2302_hg38_transcripts_ensembl.ser
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_hg38/2302_hg38_genome.h2.db
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_hg38/2302_hg38_clinvar_whitelist.tsv.gz
  • gs://gcp-public-data--broad-references/hg38/v0/exomiser/2302_hg38/2302_hg38_clinvar_whitelist.tsv.gz.tbi
  • gs://gcp-public-data--broad-references/hg38/v0/CADD/1.6/whole_genome_SNVs.tsv.gz
  • gs://gcp-public-data--broad-references/hg38/v0/CADD/1.6/whole_genome_SNVs.tsv.gz.tbi
  • gs://gcp-public-data--broad-references/hg38/v0/CADD/1.6/gnomad.genomes.r3.0.indel.tsv.gz
  • gs://gcp-public-data--broad-references/hg38/v0/CADD/1.6/gnomad.genomes.r3.0.indel.tsv.gz.tbi
  • gs://gcp-public-data--broad-references/hg38/v0/CADD/1.5/whole_genome_SNVs.tsv.gz
  • gs://gcp-public-data--broad-references/hg38/v0/CADD/1.5/whole_genome_SNVs.tsv.gz.tbi
  • gs://gcp-public-data--broad-references/hg38/v0/CADD/1.5/InDels.tsv.gz
  • gs://gcp-public-data--broad-references/hg38/v0/CADD/1.5/InDels.tsv.gz.tbi
  • gs://gcp-public-data--broad-references/hg38/v0/CADD/1.4/whole_genome_SNVs.tsv.gz
  • gs://gcp-public-data--broad-references/hg38/v0/CADD/1.4/whole_genome_SNVs.tsv.gz.tbi
  • gs://gcp-public-data--broad-references/hg38/v0/CADD/1.4/InDels.tsv.gz
  • gs://gcp-public-data--broad-references/hg38/v0/CADD/1.4/InDels.tsv.gz.tbi

Mouse References (image manifest)

The following M21 and M23 mouse reference files are used as inputs to WARP single-cell pipelines. See the MM10 overview of mouse reference images for more information. For more information regarding the use of the references, please see the overviews of Optimus and Smartseq2.

MM10 Public Image

  • gs://gcp-public-data--broad-references/mm10/v0/GRCm38.primary_assembly.genome.fa

M21 public image

  • gs://gcp-public-data--broad-references/mm10/v0/gencode.vM21.primary_assembly.annotation.gtf
  • gs://gcp-public-data--broad-references/mm10/v0/star/star_2.7.9a_primary_gencode_mouse_vM21.tar

M23 public image

  • gs://gcp-public-data--broad-references/mm10/v0/gencode.vM23.primary_assembly.annotation.gtf
  • gs://gcp-public-data--broad-references/mm10/v0/star/star_2.7.9a_primary_gencode_mouse_vM23.tar
  • gs://gcp-public-data--broad-references/mm10/v0/single_nucleus/modified_gencode.vM23.primary_assembly.annotation.gtf
  • gs://gcp-public-data--broad-references/mm10/v0/single_nucleus/modified_mm10.primary_assembly.genome.fa
  • gs://gcp-public-data--broad-references/mm10/v0/single_nucleus/star/modified_star_2.7.9a_primary_gencode_mouse_vM23.tar

Cost tradeoffs when using reference disks

While the reference disk feature should reduce the total time the VM runs, it does increase the hourly rate of the VM since an additional disk is mounted to hold the reference image. For this reason, ‘Use reference disks’ is unselected by default.

How to decide what's best for you

Your decision about whether to use a reference disk will depend on your workflow. If a job spends a large proportion of its time localizing reference files, it could be worthwhile to use a reference disk. The best way to determine this is to try running your workflow on a single sample both with and without a reference disk. If the test run is cheaper using a reference disk, the same will be true when running your workflow on your full dataset.

Was this article helpful?

Comments

2 comments

Please sign in to leave a comment.