There are several options for adding data to your workspace Google bucket.This article outlines three options for uploading data to your bucket. Your use case and familiarity will likely determine which you will want to use.
- Upload through the Terra interface
- Upload from a terminal with gsutil
- Engage BITS to help upload large amounts of data (Broad Institute community members only)
Option 1: Upload and download through the Terra interface
To upload a file to your workspace bucket, go the to Data tab of the workspace and use the file navigation panel on the left side of the screen:
Then click on the "+" button in the lower right to upload files:
To download files from the workspace bucket, use the navigation panel shown above to find the file of interest, and then click it. A pop-up window will appear with a download button, an autogenerated bash command for use in a terminal if desired, and if applicable a cost for the egress of the data.
Option 2: Upload and download in a terminal using gsutil
First, install gsutil to your local computer. The Google Cloud SDK installation includes gsutil. To install Google Cloud SDK
You can run the following command using bash shells in your Terminal:
curl https://sdk.cloud.google.com | bashOr download google-cloud-sdk.zip or google-cloud-sdk.tar.gz and unpack it. Note: The command is only supported in bash shells.
Restart your shell:
exec -l $SHELLor open a new bash shell in your Terminal.
gcloud initto authenticate.
Before uploading/downloading data using gsutil, you can use the ls command to look at the buckets you have access to:
gsutil lsto see all of the Cloud Storage buckets under your default project ID
gsutil ls -p [project name]to list buckets for a specific project
To upload data to a bucket run
gsutil cp [local file path] [bucket URL] (you must be an Owner or Writer of the workspace to upload)
The bucket URL is the path to your file or folder in the Google Cloud SDK. It will look like gs://[bucket name] or for folders within a bucket, gs://[bucket name]/[folder name]
For example, to upload a file into folder in a bucket:
gsutil cp /Users/Documents/Example.bam gs://WorkspaceBucket/gene_files
To download data from a bucket, run the command this way instead:
gsutil cp [bucket URL]/[file name] [local file path]
Make sure to leave a space between the the bucket URL and the file path:
gsutil cp gs://WorkspaceBucket/gene_files/example.bam /Users/Documents
Option 3 (Broad Institute community members only): Contact BITS for help with moving large amounts of data in and out of the cloud
If you are a member of the Broad Institute community, BITS is a great resource to help migrate large amounts of on-prem data to the cloud in a cost-effective way!