Learn how to move files between local storage and your workspace bucket using gcloud storage command-line tools in a terminal running on your local system. Note: For small numbers of small files, you can do this in Terra. See How to move data to/from a Google bucket (workspace or external).
What is "local storage"?
Local storage can be your institutional High-Performance Substrate (HPS) or your personal computer. It is connected to the cloud, but cannot (usually) be accessed from outside the local system.
Reasons to copy/move from local storage to workspace bucketTo analyze private data stored on a local system in Terra
(copy/move data from local storage to your workspace bucket)
To back up generated data from an analysis in Terra
(copy/move from workspace bucket to local storage)
To learn more about the Terra ecosystem, see Terra architecture and where your files live in it.
How to move/copy small numbers of small files (in Terra)
If you're moving small numbers (1-10) of small files, you can do so directly in Terra. This involves uploading a file from your local system (e.g., a laptop) to a Terra workspace, or downloading a file from a Terra workspace to your local system.
-
Upload files (local storage to workspace bucket)
1. Start from the workspace Data page.
2. Select the "Files" icon on the left side of the screen:
3. In the "files" section, click on UPLOAD button.
-
Download files (workspace bucket to local storage)
1. Start from the workspace Data page.
2. Select the "Files" icon on the bottom of the left panel (underneath "Other Data").
3. Find the file you want to download (Note: you may have to navigate down many levels of file folders) to access the file you want.
4. Click on the file name; this will open a popup window, where you can click to download the file or copy a gcloud storage command to download it from your workspace's Terminal. You'll also see the cost associated with downloading the file.
-
Download files (workspace data table to local storage)
1. Navigate to the Data section of your workspace.
2. Under the Tables section on the left-hand panel, click on the table that contains links to a file that you want to download.
Any files available for download will be underlined and shown in blue.
3. Click on the file link; this will open a popup window, where you can click to download the file or copy a gcloud storage command to download it from your workspace's Terminal. You'll also see the cost associated with downloading the file.
How to move/copy large numbers of large data files (gcloud storage CLI)
To move (or copy) large numbers of large data files between local storage and your Workspace bucket, use the command-line Python application gcloud storage cp in a terminal running on your local system.
See step-by-step instructions below for installing SDK (includes the gcloud storage library) locally and running commands to copy/move files.
You must have permission to upload to/download from a Google bucketThis includes the workspace bucket (you must be a workspace Owner or Writer).
For additional details on the gcloud storage cp command, see the Google documentation.
Step 1. Install SDK locally
The instructions in this section rely on bash shells, which are most straightforward to run on Mac or Linux systems. See Google's documentation to learn how to install gcloud storage on other systems.
1.1. Open a terminal running on your local system. Then, follow Google’s installation instructions for Cloud SDK - or the directions below - to install Google Cloud SDK, which includes gcloud storage.
1.2. Run the following command using bash shells in your Terminal.
curl https://sdk.cloud.google.com | bash
1.3. Open a new bash shell in your terminal, or restart your shell using this command:
exec -l $SHELL
1.4.Authenticate by running the following:
gcloud init
Verify gcloud storage installationBefore downloading data using gcloud storage; use the ls command to look at the buckets you can access.
Run gcloud storage ls to see all of the Cloud Storage buckets under the workspace project ID.
Run gcloud storage ls -p [project name] to list buckets for a specific project.
Once gcloud storage is installed, you can perform command-line gcloud storage tasks to move/copy files between local storage and your workspace bucket (or any external bucket).
Step 2. Run gcloud storage to copy/move files
The basic structure for the gcloud storage cp command looks like this (Note: You need to use the full path to the Workspace or external bucket, i.e. gs://<bucket-path>):
gcloud storage cp <where_to_copy_data_from>/<file_name> <where_to_copy_data_to>
-
1. Locate the bucket name listed in the Cloud Information section on the right-hand side of your workspace's Dashboard.
2. Click the clipboard icon to the right of the bucket name to copy it to your clipboard.
3. Add
gs://to the front of the clipboard content to use in the gcloud storage command. -
1. Go to Cloud Storage browser in Google Cloud console.
2. Click on the bucket link.
3.Click the clipboard icon (to the right of the bucket name) to copy the bucket path.
4. Add
gs://to the front of the clipboard content to use in the gcloud storage command.
Example code
The examples below illustrate how this code might look if you're uploading a file from your local system to the Cloud, or downloading from the Cloud to your local system.
-
Copy a single file
To copy the file "Example.bam" from your local storage system to the Workspace bucket "gs://fc-7ac2cfe6-4ac5-4a00-add1-c9b3c84a36b7":
gcloud storage cp <local_folder>/Example.bam gs://fc-7ac2cfe6-4ac5-4a00-add1-c9b3c84a36b7
If you are in the directory/folder where the data are stored
Note: You can find the local directory you are in by running the command
pwd- which stands for "print working directory".gcloud storage cp example.bam gs://fc-7ac2cfe6-4ac5-4a00-add1-c9b3c84a36b7
To copy all the files in the directory
Use the wild card
*.gcloud storage cp <local_folder>/* gs://fc-7ac2cfe6-4ac5-4a00-add1-c9b3c84a36b7
Where will the copied files be?
This will copy the files into the default directory. You can see the files by clicking the "Files" icon at the bottom left in the Data page, or the "Open in browser" link on the right in the Dashboard.
In the Data page In the Dashboard -
To copy a single file
To download the file "Example.bam" from the workspace bucket "gs://fc-7ac2cfe6-4ac5-4a00-add1-c9b3c84a36b7" into your local system:
gcloud storage cp gs://fc-7ac2cfe6-4ac5-4a00-add1-c9b3c84a36b7/Example.bam .
The ' . ' at the end of the command means "here" (i.e. the home directory of the local system running the terminal). So the command is saying "copy example.bam from Google bucket fc-7ac2cfe6-4ac5-4a00-add1-c9b3c84a36b7 to the directory I am in in my terminal."
To copy all the files in the directory
Use the wild card
*.gcloud storage cp * gs://fc-7ac2cfe6-4ac5-4a00-add1-c9b3c84a36b7 .
Where will the files be?
You can find the local directory by running the command
pwd- which stands for "print working directory".