Interactive applications - such as Jupyter Notebooks - run on virtual machines or clusters of machines in your Cloud Environment. When running an interactive application in Terra, you can adjust the configuration of the your Cloud Environment to fit your computational needs.  This article gives an overview of the components that make up your cloud environment and step-by-step instructions of how to customize them.   

Contents

Cloud environment components  
   - Application configuration (packages and dependencies)
   - Compute power
   - Cost-saving recommendations
   - Persistent disk
Customizing your Cloud Environment Overview
    - How to customize your cloud environment
    - Customizing with a Docker image
    - Customizing with a startup script
    - Setting a custom compute power
What real-time updates can you make to Notebook compute resources without losing data?
How to save generated data to your workspace bucket

Cloud Environment Components Overview

The cloud environment is the components of the virtual machine or cluster of machines that run your interactive analysis application. Cloud environments consist of

    1) an application configuration
    2) cloud compute
    3) a persistent disk

To see your Cloud environment configuration, click the gear icon at the top right of any workspace page to reveal the form below. To customize your VM (including clusters!), you will select the "Customize" button at the bottom right of the Cloud environment form. 

Application configuration 

The application configuration includes the software and dependencies that are pre-installed in the cloud environment container. Terra includes several pre-configured environments for biomedical use-cases like Bioconductor or Hail analyses.

Pre-configured options

Terra has five varieties of pre-configured application configurations - plus a custom option - available in the drop down menu. What versions and what libraries are included in each pre-configured option is also listed in the dropdown.

• Terra-maintained Jupyter environments
• Community-maintained Jupyter environments (verified partners)
• Community-maintained RStudio environments (verified partners)
• Custom environments
• Project-specific environments 

If pre-configured application configurations don't meet your needs, you can customize with a Docker image or startup script.

Why use pre-installed software and dependencies?

If your analysis requires software packages that are not part of the default or pre-configured configurations, you could start your interactive application by installing the ones you need. This approach can turn into a maintenance headache if you have multiple notebooks that require the same configuration commands. Here are three reasons to move some of those software installation steps into the application configuration proper. Efficiency You don't have to spend time (and $$) running setup code to install what's needed for your analysis every time you spin up your application. Simplified setup Customizing the application configuration simplifies and standardizes the setup needed to run an application.  Reproducibility Custom Docker containers and startup scripts put you in control of exactly what version of programming languages and packages to include. Recording the image URL within the notebook you wish to share is a quick-and-easy way to spin up and share an identical environment, ensuring that colleagues get the same results.

Compute power 

The compute power is the CPU and RAM available to your application, which determines how much processing can be done at a time. Customizing the compute power allows you to balance cost and functionality. For example, if your analysis is running slow, it could mean the CPUs and memory allotted are insufficient for the computations you're doing. It may be worth the cost of increasing the compute power so your analysis will complete quicker in real time.

Compute power comes with a cost! Find the balance that's right for you.

Note that more compute power costs more, and you don't want to request (and pay for) significantly more than your computation needs. Running a high-powered notebook costs a certain amount per unit time no matter what computations are done. You don't need (or want to pay for) a high-performance parallel Spark cluster if you're running a simple, non-parallel computation. Featured and template workspaces and notebooks will include recommended and project-specific configurations, as well as estimated costs to run (where possible). Since it is fairly straightforward to adjust the compute power, you can estimate an initial power to try and then dial it up or down as needed. Just be sure to be careful to save any generated data you want to keep when recreating the cloud environment (see below). To learn more about controlling cloud costs, see this documentation.

Cost saving recommendations

Size your compute power appropriately You pay a fixed amount while a notebook is running, whether or not you are doing active calculations. The cost is based on the compute power of your virtual machine or cluster, not how much computation is being done. So you want to have enough power to do your computations in a reasonable amount of time, but not a lot of extra that you will be paying for and not using. Note that Terra automatically pauses a notebook after twenty minutes of inactivity.  To learn more about controlling cloud costs in a notebook, see this article.

Persistent Disk

Your cloud environment comes with Persistent Disk (PD) storage by default that lets you keep files stored in your Cloud Environment even after you delete the VM or cluster. The PD can be detached prior to deletion and reattached to a newly created VM. Using this as storage lets you keep the packages your notebook code is built upon, input files necessary for your analysis, and outputs you’ve generated (without having to move anything to permanent cloud storage).

Data in PD is not available outside the cloud environment
Note that because the PD is not accessible from outside the Cloud Environment, data generated in a notebook cannot be used as input for a workflow analysis, and it is not accessible by other users collaborating in a shared workspace.

To learn more about how to save data generated in a notebook to permanent cloud storage (including the workspace bucket), see this article. 

	Learn more about Terra's Jupyter notebook environment
	- Key components - Key operations - Best Practices

Customizing your Cloud Environment Overview

Cloud environment considerations

Changing the cloud environment can mean files generated or stored in the application memory will be lost when Terra recreates the cloud environment. To avoid this, make sure to include the persistent disk option (default), copy your files to the workspace bucket, and set the right environment and power before doing any work. If you don't have a persistent disk, see the section below to understand what changes you can make without losing generated data. If the work you're doing in your notebook includes mostly short-running commands that don't amount to much computation cost, this isn't a big problem: there is an option in the Jupyter Notebooks menu to re-run all code cells (or all up to a certain point) so you can simply regenerate the previous state. However, if some of your work involves massive computations that would not be trivial to re-run, you may need a better strategy. Remember that your cloud environment is unique to you, and works at the Billing Project level. This means that you use the same unique-to-you cloud environment in all workspaces under the same Billing Project. If you recreate the cloud environment in one of your workspaces, you will see the change reflected in the cloud environments of all workspace you have under the same Billing Project.

How to customize your cloud environment

If the default or project-specific environments don't fit your needs, you can use a custom Docker Image or include a startup script. Anyone using the same Docker image or startup script will have the exact same environment, which is critical for reproducibility.

To learn more about developing and using custom Docker images in Terra, see these articles. Note that you can also use a custom environment to revert back to a previous version of the pre-configured environments. 

To adjust the virtual environment and/or compute power of your application, first click on the gear icon in the widget at the top right of your workspace:

This will reveal the form at left, with the current values of your cloud environment (see default values in screenshot below). To make changes, click the "Customize" button at the bottom right. You can modify the cloud environment at any time, even if you've already started working in an application (i.e. notebook). You'll see Terra's Cloud Environment configuration panel (screenshot below). Note that it is much simpler than the equivalent Google Cloud Platform interface!

You'll pull up the configuration panel, specify what you want and let the system regenerate your cloud environment with the new specifications. It's not necessary to guess up front the resources you're going to need to do your work. You can start with minimal settings, then dial them up if you run into limitations.

1. Application configuration Includes preconfigured con configurations with popular packages. Custom option offers the ability to use a custom Docker. 2. Cloud compute Dropdown options include standard VM, Spark master node and Spark cluster. This is also where you can specify a custom startup script.  3. Detachable Persistent Disk Learn more about detachable persistent disks for notebook applications in Terra here. Don't forget to save the configuration, after changing any values. This will recreate the application compute with the new values, which can take up to five or ten minutes.

What real-time updates can you make to Notebook compute resources without losing data?

1. You can increase or decrease the # of CPUs or memory
   During this update, the Notebook will pause the cloud environment, update, and then restart. The update will take a couple of minutes to complete and you will not be able to continue editing or running the Notebook while it's completing.
2. You can increase the disk size or change the number of workers (when the number of workers is > 2)
   During this update, you can continue to work in your Notebook without pausing your cloud environment. When the update is finished, you will see a confirmation banner. 

 Note that if you want to simultaneously change both the workers and CPU/memory, we advise doing this sequentially by first updating the CPUs/memory, waiting for the Notebook cloud environment to restart, and then adjusting the workers.

Any other cloud environment changes (e.g. decreasing the disk size or changing the environment type) require deleting the existing cloud environment runtime and creating a new one, and any generated data files and installed packages will be lost. Please backup files as appropriate.

How to save interactive analysis outputs to workspace bucket

To avoid losing your data, make sure to explicitly save your outputs in the workspace bucket. You can find step by step instructions and exact code to do this within the notebook below.

import os
BILLING_PROJECT_ID = os.environ['WORKSPACE_NAMESPACE']
WORKSPACE = os.environ['WORKSPACE_NAME']
bucket = os.environ['WORKSPACE_BUCKET']

!gsutil cp ./* $bucket
# Run list command to verify file is in the bucket
!gsutil ls $bucket

project <- Sys.getenv('WORKSPACE_NAMESPACE')
workspace <- Sys.getenv('WORKSPACE_NAME')
bucket <- Sys.getenv('WORKSPACE_BUCKET')

#Copy all files generated in the notebook into the bucket
system(paste0("gsutil cp ./* ",bucket),intern=TRUE)
#Run list command to see if file is in the bucket
system(paste0("gsutil ls ",bucket),intern=TRUE)