Terra provides infrastructure for running interactive analyses with Jupyter Notebooks, which are files that contain analysis code and embedded documentation. This article is meant to help enhance your ability to do interactive analyses with a deeper understanding of key components (i.e. Terra Billing projects) in Terra's notebook environment.
See Terra's Jupyter Notebooks environment Part II: Key operations for a secondary article about key operations in Terra's notebooks environment, and how they impact your work, such as the amount of flexibility you have to customize the environment, how you will access data, and how to save your analysis results.
| Content for this article was contributed by Matt Bookman from Verily Life Sciences based on work done in Terra for AMP PD, a public/private partnership collaborating toward biomarker discovery to advance the development of Parkinson’s Disease therapies. |
This section defines terms and explains where notebook files and their output live -- both when you are working with the files and when you are not.
Jupyter kernel
The kernel is the computer program that runs while you have a Jupyter notebook open. The kernel process maintains the runtime state of the Jupyter notebook.
Terra supports R and Python kernels. When cells in the notebook are executed, they are interpreted by this language-specific kernel. Note that Terra generally selects the kernel automatically, though it is always a good idea to check.
Notebook Service (aka Leonardo)
The Notebook Service manages the cloud environment (Compute Engine instance) you use to edit and run your notebook. In Terra, the notebook service is called "Leonardo," and the two terms are often used interchangeably.
Cloud Environment (aka "Cluster"; aka "Compute Engine VM")
When you interact with your notebook in a web browser on your own computer, the characters you type and the code you execute are all sent to the Jupyter kernel process running on a Google Compute Engine virtual machine (VM), or Cloud Environment. Much of the discussion in this document involves understanding the Cloud Environment as a host for your notebooks.
In the rest of this article, your Cloud Environment refers to the Compute Engine VM running your notebooks as well as the associated VM boot disk and Persistent Disk.
When you create your Cloud Environment, by default you create a single VM. However, the Terra environment supports more powerful clusters of VMs using Google Cloud Dataproc. The use of a VM cluster is an advanced topic that is outside of the scope of this document.
VM boot disk
A virtual machine has a disk (the boot disk) for storing data files, the operating system, or other software. Information on the boot disk is lost if you delete or update the Cloud Environment since the boot disk will be deleted as well. Terra's Persistent Disk, however, will not be automatically deleted with the cloud environment unless explicitly requested by the user.
Detachable Persistent Disk
When you create a Cloud Environment, Terra automatically creates a Persistent Disk to store libraries and packages, input files, and generated outputs, even if you delete the Cloud Environment. This persistent disk is automatically detached from a VM prior to its deletion and reattached to a newly created VM.
Accessing and saving notebook analysis output Save any files you want to keep to your mounted Persistent Disk at /home/jupyter or /home/jupyter-user, depending on the age of the Persistent Disk. To determine the name of the mount point, run!echo $HOME from within your notebook.
Note that generated files are stored in this directory by default. You only need to actively check that files are saved there before you delete your Cloud Environment as well as the Persistent Disk, not when you pause and resume.
To learn how and when to copy data from a notebook to your workspace bucket, see Copying notebook output to a Google bucket.
Docker containers
Docker is a (branded) container technology for packaging software for rapid deployment and reproducibility. Like a sandboxed virtual machine, Docker containers exist wholly inside the Compute Engine Virtual Machine. Docker images include all the software and tools needed for an analysis and can be quickly deployed on a Cloud Environment. To learn more about using Dockers, including custom Docker images, in Terra, see Docker tutorial: Custom cloud environments for Jupyter notebooks.
Workspace bucket
Every Terra Workspace has an associated Google Cloud Storage bucket for long-term storage of notebooks and other files. In "edit" mode, notebook files (i.e. only ".ipynb" files) are automatically saved to your workspace bucket. You can save other generated files to your workspace bucket manually. See Terra's Jupyter Notebooks environment Part 1: Key components for more details on how to do this.