Your RStudio Cloud Environment includes a detachable persistent disk that maintains generated data even when you re-create the virtual machine that your analysis ran on. Sometimes, though, you will want to copy data files to more permanent cloud storage. This article describes how to move data from the RStudio persistent disk storage to a Google bucket (including your workspace storage) when working in RStudio in Terra.
Why copy RStudio-generated data to workspace or external storage?
The data that you generate when you run an analysis in RStudio are stored in the RStudio virtual machine's persistent disk (PD) by default. Here are the primary reasons why you may want to copy that data to your workspace's permanent storage (or an external Google bucket).
To analyze the data with another Terra tool
Files saved to the persistent disk are not accessible outside of your virtual RStudio Cloud Environment, even to other Terra analysis tools in your own workspace. Your files must be saved to your workspace storage bucket or another bucket outside of Terra in order to access them from a workflow, Jupyter notebook, or Galaxy analysis.
To share data with collaborators (including in a shared workspace)
You need to copy data to workspace storage if you want colleagues to have access to it. This is true even if you are working in a shared workspace, since each user has their own unique workspace Cloud Environment and Persistent Disk.
To archive data
If you want to save valuable data for the long term -- especially if you want to copy it to less expensive Nearline or Coldline storage -- you need to copy it to an external bucket first.
To safeguard data when re-creating or deleting the Persistent Disk (PD)
If you delete your PD or reconfigure your Cloud Environment in some ways (for example, decreasing your virtual machine's memory or the size of your persistent disk), you can lose all or some of the data on your PD, unless you explicitly save your output to workspace or external storage (i.e., Google bucket).
How to copy RStudio data to workspace storage
Step 1. Open the built-in RStudio terminal
You can access a bash terminal from the Terminal tab in the main RStudio pane:
Step 2. Set the variable "bucket" for the destination storage
To use the workspace bucket for storage, run the command bucket="$WORKSPACE_BUCKET".
To save data to an external Google bucket, run the command bucket="$gs://<your-bucket-name>".
Setting this as a variable allows you to copy/paste the commands from the documentation below.
WORKSPACE_BUCKET is an environment variable that is predefined when using the terminal in Terra. Using environment variables lets RStudio grab the workspace Google bucket directly. This helps avoid hardcoding these variables into the code to move the data.
Step 3. Save files to "bucket" with bash commands
Note: Workspace storage is a Google bucket, so basic bash commands in the RStudio terminal need to be preceded by "gcloud storage."
To copy all files generated in the notebook into the bucket, use the command:gcloud storage cp * "$bucket"
To make sure the files are in the bucket, run the following:gcloud storage ls "$bucket"
Be careful when copying all filesUsing `*` can mean copying a lot of large files, which can be expensive. Check the size of the files in the bucket before copying! If you want to copy individual files, you can replace `*` with the file name to copy.
Additional resources
To learn more about your workspace Cloud Environment storage, see Overview: Cloud environment storage (detachable persistent disks).
For additional bash capabilities, see Using the terminal and interactive shell in Terra.
For complete details on all gcloud storage options, see Google's documentation.
A deeper dive: Terra's Cloud Environment To understand what's under the hood and why RStudio and notebooks have these characteristics, see this article about key notebook components or this article about key notebook operations.