Moving data between local storage and workspace bucket

Allie Hajian
  • Updated

Learn how to move files between your local system and your workspace bucket, you using gsutil command-line tools in a terminal running on your local system.

Contents

What is "local storage" system and why would you want to move data files?
Overview: How to move/copy data
    1. Install gsutil locally (step-by-step instructions)
    2. Run gsutil (step-by-step instructions)
Example 1: Copy (upload) file from local storage to the workspace bucket
Example 2: Copy (download) from the workspace bucket to local storage

Data-transfer_Bucket-PD-local-storage_Diagram.png

What is "local storage"?

Local storage can be your institutional HPS or your personal computer. It is connected to the cloud, but cannot be accessed from outside the local system.

Reasons to copy/move from local storage to workspace bucket

  •  To analyze private data stored on a local system in Terra
     (copy/move data from local storage to your workspace bucket)

  • To back up generated data from an analysis in Terra
    (copy/move from workspace bucket to local storage)

To learn more about the Terra ecosystem (infrastructure) and where your data live in it, see this article.

Overview: How to move/copy data

To move (or copy) the data files, you will use gsutil in a terminal running on your local storage system. See the step-by-step instructions below for installing gsutil locally and running commands to copy/move files. Note that you can move files in either direction (from local storage to the workspace bucket or from the workspace bucket to local storage). 

G0_warning-icon.png


You must have permission to upload to/download from a Google bucket!

  This includes the workspace bucket (you must be a workspace owner or writer).

G0_tip-icon.png


For additional details on the gsutil cp command, click here.

1. Install gsutil locally (click for instructions)

First, open a terminal running on your local system.

Then, follow Google’s installation instructions for Cloud SDK - or the directions below - to install Google Cloud SDK, which includes gsutil:

1.1. Run the following command using bash shells in your Terminal:
      curl https://sdk.cloud.google.com | bash 
       Note: the command is only supported in bash shells. 
      Alternative: download google-cloud-sdk.zip or google-cloud-sdk.tar.gz and unpack it. 

1.2. Open a new bash shell in your terminal, or restart your shell:
       exec -l $SHELL 

1.3. Authenticate by running gcloud init 

Click here for Google’s installation instructions for Cloud SDK.

G0_tip-icon.png


Verify gsutil installation

  Before downloading data using gsutil, you can use the ls command to look at the buckets you can access:
  • Run gsutil ls to see all of the Cloud Storage buckets under the workspace's project ID
  • Run gsutil ls -p [project name] to list buckets for a specific project

 

2. Run gsutil to copy/move files 

Once gsutil is installed, you can perform command-line gsutil tasks to move/copy files between local storage and your workspace bucket (or any external bucket).

The basic structure for the gsutil cp command looks like this (note that you need to use the full path to the bucket (unstructured storage):

gsutil cp <where_to_copy_data_from/> <where_to_topy_data_to>  

Finding the full path to workspace bucket
To find the full path to the workspace bucket, click the Clipboard icon in the right ride of the workspace Dashboard: 

Moving-data_Google-bucket_Screen_Shot.png


Example 1: Copy (upload) from local storage to the workspace bucket

To copy the file "Example.bam" from your local storage system to the workspace bucket "gs://fc-7ac2cfe6-4ac5-4a00-add1-c9b3c84a36b7":

gsutil cp <local_folder>/Example.bam cp gs://fc-7ac2cfe6-4ac5-4a00-add1-c9b3c84a36b7

Where will the copied files be?
This will copy the files into the default directory. You will be able to see the files by clicking the "Files" icon at the bottom left in the Data tab, or the "Open in browser" link on the right in the Dashboard tab:

Move-data-local-storage_Files-icon-Data-tab_Screen_Shot.png Move-data-local-storage_Open-in-browser-Dashboard-tab_Screen_Shot.png

  

 

Example 2: Copy (download) from the workspace bucket to local storage

To download the file "Example.bam" from the workspace bucket "gs://fc-7ac2cfe6-4ac5-4a00-add1-c9b3c84a36b7" into your local system:

gsutil cp gs://fc-7ac2cfe6-4ac5-4a00-add1-c9b3c84a36b7/Example.bam .

The ' . ' at the end of the command means "here" (i.e. the home directory of the local system running the terminal). 

Was this article helpful?

0 out of 0 found this helpful

Have more questions? Submit a request

Comments

0 comments

Please sign in to leave a comment.