How to move data between local and workspace storage

Learn how to move files between local storage and your workspace bucket using gsutil command-line tools in a terminal running on your local system. Note: For small numbers of small files, you can do this in Terra. See How to move data to/from a Google bucket (workspace or external).

What is "local storage"?

Local storage can be your institutional HPS or your personal computer. It is connected to the cloud, but cannot (usually) be accessed from outside the local system.

Reasons to copy/move from local storage to workspace bucketTo analyze private data stored on a local system in Terra
 (copy/move data from local storage to your workspace bucket)

To back up generated data from an analysis in Terra
(copy/move from workspace bucket to local storage)

To learn more about the Terra ecosystem, see Terra architecture and where your files live in it.

How to move/copy small numbers of small files (in Terra) 

Uploading in Terra Recommended for small numbers (one to ten) of small files

Only for transfers between workspace bucket and local storage (i.e., laptop)

Note: This is the sort of transfer you often see where you upload or download a file from the internet. Because your local storage has no cloud-native  "path", you can transfer only files stored on the system running your browser. 

• Upload files (local storage to workspace bucket)

  1. Start from the workspace Data page.

  2. Select the "Files" icon on the left side of the screen:
  

  3. In the "files" section, click on UPLOAD icon.
  

• Download files (workspace bucket to local storage)

  1. Start from the workspace Data page.

  2. Select the "Files" icon on the bottom of the left column (underneath "Other Data").

  3. Find the file you want to download (Note: you may have to navigate down many levels of file folders) to access the file you want.
  

  4. Click on the file to download; this will open a popup window (screenshot below). The popup window offers multiple choices for copying the data, as well as the cost.
  

• Download files (workspace data table to local storage)

  1. Click on one of the table tabs on the left side of the screen. Example below uses a tab labeled"sample".

  Any files available for download will be shown as a link in the relevant sample row.
  

  3. Click on a file link to open a pop-up window describing the size and cost of the download.
  

  4. Click on the “Download for $0.56” link to initiate the download. Note: This button starts the download immediately. You don't have another opportunity to verify before the download starts. However, you can cancel the download at any time during the process. 

  5. Repeat for any additional files you would like to download.

How to move/copy large numbers of large data files (gsutil)

To move (or copy) large numbers of large data files between local storage and your Workspace bucket, use the command-line Python application gsutil in a terminal running on your local storage system.

See step-by-step instructions below for installing gsutil locally and running commands to copy/move files. Note: You can move files in either direction (from local storage to the workspace bucket or from the workspace bucket to local storage).

You must have permission to upload to/download from a Google bucket!This includes the workspace bucket (you must be a workspace Owner or Writer).

For additional details on the gsutil cp command, see the Google documentation.

Step 1. Install gsutil locally 

First, open a terminal running on your local system. Then, follow Google’s installation instructions for Cloud SDK - or the directions below - to install Google Cloud SDK, which includes gsutil.

1.1. Run the following command using bash shells in your Terminal.

curl https://sdk.cloud.google.com | bash

Note: The command is only supported in bash shells.      

 Alternative: download google-cloud-sdk.zip or google-cloud-sdk.tar.gz and unpack it. 

1.2. Open a new bash shell in your terminal, or restart your shell using this command:

exec -l $SHELL

1.3.Authenticate by running the following:

gcloud init

Verify gsutil installationBefore downloading data using gsutil; use the ls command to look at the buckets you can access.

Run gsutil ls to see all of the Cloud Storage buckets under the workspace project ID.

Run gsutil ls -p [project name] to list buckets for a specific project.

Once gsutil is installed, you can perform command-line gsutil tasks to move/copy files between local storage and your workspace bucket (or any external bucket).

Step 2. Run gsutil to copy/move files 

The basic structure for the gsutil cp command looks like this (Note: You need to use the full path to the Workspace or external bucket, i.e. gs://<bucket-path>):

gsutil cp <where_to_copy_data_from>/<file_name> <where_to_topy_data_to>  

• 1. Click the Clipboard icon in the right side of the Dashboard. 

  

  2. Add gs:// to the front of the clipboard content to use in the gsutil command.

• 1. Go to Cloud Storage browser in Google Cloud console. 

  2. Click on the bucket link.

  

  3.Click the clipboard icon (to the right of the bucket name) to copy the bucket path. 

  

  4. Add gs:// to the front of the clipboard content to use in the gsutil command.

Example code

Select the use-case below for the exact code you will run to download or upload.

• Copy a single file
  To copy the file "Example.bam" from your local storage system to the Workspace bucket "gs://fc-7ac2cfe6-4ac5-4a00-add1-c9b3c84a36b7":

  gsutil cp <local_folder>/Example.bam cp gs://fc-7ac2cfe6-4ac5-4a00-add1-c9b3c84a36b7

  If you are in the directory/folder where the data are stored
  Note: You can find the local directory you are in by running the command pwd - which stands for "print working directory".

  gsutil cp example.bam gs://fc-7ac2cfe6-4ac5-4a00-add1-c9b3c84a36b7 

  To copy all the files in the directory 
  Use the wild card *.

  gsutil cp * gs://fc-7ac2cfe6-4ac5-4a00-add1-c9b3c84a36b7

  Where will the copied files be?
  This will copy the files into the default directory. You can see the files by clicking the "Files" icon at the bottom left in the Data page, or the "Open in browser" link on the right in the Dashboard.

  In the Data page

  In the Dashboard

• To copy a single file
  To download the file "Example.bam" from the workspace bucket "gs://fc-7ac2cfe6-4ac5-4a00-add1-c9b3c84a36b7" into your local system:

  gsutil cp gs://fc-7ac2cfe6-4ac5-4a00-add1-c9b3c84a36b7/Example.bam .

  The ' . ' at the end of the command means "here" (i.e. the home directory of the local system running the terminal). So the command is saying "copy example.bam from Google bucket fc-7ac2cfe6-4ac5-4a00-add1-c9b3c84a36b7 to the directory I am in in my terminal."

  To copy all the files in the directory
  Use the wild card *.

  gsutil cp * gs://fc-7ac2cfe6-4ac5-4a00-add1-c9b3c84a36b7

  Where will the files be?
  You can find the local directory by running the command pwd - which stands for "print working directory".