Your Jupyter Cloud Environment includes a detachable persistent disk (PD) that maintains generated data even when you re-create the virtual machine (VM). Sometimes, though, you will want to copy your data to more permanent cloud storage: e.g., when archiving data, or to allow collaborators or workflows access. This article describes how to move data from Jupyter VM storage to a Google bucket (including your workspace storage) when working in notebooks in Terra.
Interested in a deeper dive? Terra's Cloud Environment To understand what's happening in the backend and why notebooks and RStudio analyses have these characteristics, see this article about key notebook components or this article about key notebook operations.
Why copy generated data to workspace storage?
Below are the primary reasons you might want to copy data generated in a notebook analysis to workspace storage (or external Google bucket).
Use generated data as input for a workflow
Files generated by a notebook are not automatically saved in workspace storage (Google bucket) and are not accessible outside your personal virtual Jupyter Cloud Environment. In order for your workflows to access generated notebook files as input data, you will need to copy to workspace storage.
Share generated data with collaborators - even in a shared workspace
For the same reason, you need to copy data to workspace storage if you want colleagues to have access. This is true even if you are working in a shared Workspace, since each user has their own Cloud Environment and Persistent Disk (PD) that is inaccessible to anyone else.
Archive data
If you want to archive data, especially if you want to copy it to less expensive Nearline or Coldline storage, first copy it to an external bucket.
To safeguard data when re-creating or deleting the PD
Sometimes you may want to reconfigure your Cloud Environment (e.g., if you are moving between a notebook and RStudio analysis) or delete your PD. Be careful - in some case you can lose all or some generated data unless you explicitly save your output to workspace or external storage (i.e., Google bucket).
For example, if you want to decrease your PD (because you overestimated how much you would need and don't want to pay for unused space), you should back up data before decreasing the disk size, in case the part of the disk that is deleted includes some generated data.
Don't lose data when running both Jupyter and RStudio!You have to re-create the Cloud Environments when swapping between RStudio and Jupyter in the same workspace. To protect data (because Jupyter and RStudio have a shared persistent disk), it is important to limit changes to those maintaining your PD data integrity (i.e., only increase disk size and keep the same disk type).
How to copy data to workspace storage
To move generated data to permanent cloud storage, make sure to explicitly save your outputs in the workspace bucket by following the directions below.
Step 1. Set environment variables
Setting the environment variables lets the notebook grab variables such as the workspace name and Google bucket directly. This makes cleaner and more flexible notebooks that don't require you to hardcode these variables.
Run the commands below in a code cell.
-
import os
BILLING_PROJECT_ID = os.environ['WORKSPACE_NAMESPACE']
WORKSPACE = os.environ['WORKSPACE_NAME']
bucket = os.environ['WORKSPACE_BUCKET'] -
project <- Sys.getenv('WORKSPACE_NAMESPACE')
workspace <- Sys.getenv('WORKSPACE_NAME')
bucket <- Sys.getenv('WORKSPACE_BUCKET')
Step 2. Save output files to a bucket with bash commands
The workspace storage is a Google bucket, so basic bash commands in notebooks need to be preceded by "gsutil."
These commands will work only if you have run the commands above to set the environment variables. Once you execute these cells, the data files should be visible in the workspace bucket.
To save all generated files after the notebook runs, use the commands below. If you want to copy individual files, you can replace `*` with the file name to copy.
-
# Copy all files in the notebook into the bucket
!gsutil cp ./* $bucket
# Run list command to see if file is in the bucket
!gsutil ls $bucket -
# Copy all files generated in the notebook into the bucket
system(paste0("gsutil cp ./* ",bucket),intern=TRUE)
# Run list command to see if file is in the bucket
system(paste0("gsutil ls ",bucket),intern=TRUE)
What to do if you lose your notebook data
Your notebook (.ipynb) file is saved in workspace storage (i.e., Google bucket). This means you can rerun the notebook to regenerate any output data (you will pay for this, of course).