Terra attaches a persistent disk (PD) to your Cloud Environment virtual machine (VM) where you can store data (such as generated data and installed libraries) even if you delete or update your Cloud Environment.
PDs also act as a safeguard to protect your data in case something goes wrong with the VM. If you delete your cloud environment, but keep your PD, the PD will be reattached when creating the next Cloud Environment.
Note that there is a cost for maintaining the disk even when the Cloud Environment is paused or deleted. Their cost per month is shown in the user interface in the Cloud Environments section under your profile, as well as within the Cloud Environment Setup pane when you're customizing your environment. The default PD costs $2 per month.
Terra storage overview
Terra workspaces have two dedicated storage locations – the workspace bucket and the Cloud Environment virtual machine (VM) Persistent Disk. Like a USB drive, the PD can be detached from the VM before deleting or recreating the Cloud Environment and attached to a new one. The PD lets you keep your notebook code packages, input files necessary for your analysis, and outputs you’ve generated - without having to move anything to workspace storage (i.e., Google bucket).
Sharing data with colleaguesSince the PD is unique to your Cloud Environment, and your Cloud Environment is unique to you, you need to copy data to workspace storage to share with colleagues (even those working in a shared workspace) or to use as input for a workflow.
What is a “persistent disk” (PD) and how does it work?
|
The default Cloud Environment virtual machine automatically comes with 50G of storage (the persistent disk) attached. You can access files in the PD by launching a Jupyter Cloud Environment and opening a VM terminal (to launch a terminal, click on the terminal icon in the right sidebar of any workspace page when a Jupyter Cloud Environment is running). Note: The terminal will open in a new browser tab. |
 |
Types of persistent disksStandard: standard hard disk drives
Solid State Drive: solid state drives
Balanced: a combination of standard and solid-state drives (SSD). They are an alternative to SSD persistent disks that balance performance and cost.
Which disk type is right for you?
Solid state drives are more expensive, but run faster and are more power efficient than standard hard disk drives.
The PD file directory
In the terminal, you can navigate the PD’s file structure using bash commands, just as you would on a local machine. Any files you want to save, or persist, must be saved to the directory where the PD is mounted. Anything saved outside this directory is not saved to the persistent disk and will be lost when the PD is deleted.
- RStudio PDs are mounted to the directory
/home/rstudio - Jupyter PDs are mounted to
/home/jupyter
To find the name of the mount point for your Jupyter Cloud Environment PD, run !echo $HOME from within your notebook.
Persistent disks save time and reduce error
The persistent storage option saves time and reduces error because you don't have to reinitialize the Cloud Environment when using apps that require time-consuming package installation or input files. Data and other files stored on the persistent disk are safe even if you delete or re-create the Cloud Environment.
- Input data (e.g., genomics files, tabular data)
- Generated data files (from an interactive analysis)
- Figures (e.g., PDFs, PNGs, JPEGs, etc)
- Packages installed on the Cloud Environment
Maintaining important data and files when deleting or re-creating the Cloud Environment makes the Terra experience more like working on a local machine.
When is persistence especially useful?
- When running an analysis that requires a very lengthy initialization (i.e., package installation).
- When running a notebook that expects a certain input from the Cloud Environment. For example, this Encode tutorial downloads results created by one of its workflows into the Jupyter Cloud Environment for further analysis.
- Whenever you want to be able to save your outputs/results, either to keep them organized or because some outputs are used in other parts of your analysis.
Scenarios when you might need to delete or re-create the Cloud Environment
- To make certain types of changes to the environment. Some types of changes to the compute profile cannot be updated within an existing VM (e.g. adding GPUs), and if you want to use the data in your PD with such a configuration, you will need to delete the existing VM and create a new one.
- To run with the latest updates described in our release notes (Notebooks Best Practices) guidelines suggest recreating Cloud Environments regularly).
- If the Cloud Environment is automatically deleted to ensure it has the latest updates.
