This article summarizes the cloud components you'll use when working in Terra, and how working in the cloud differs from working locally.
This is a living document. Check back here to see the current state of Terra on Azure.
Terra on Azure is a public preview release intended to allow users to access tools and resources as they are brought online. Your candid feedback will help us improve the Terra experience as we develop and roll out additional functionality.
Preview disclaimersSince this is a preview environment, features may change without notice. Also note that we cannot guarantee that you will not lose data.
Interested in joining the Terra on Azure Preview Environment?
To try out Terra on Azure, you must be included in the Terra on Azure public preview allow list. Submit your request by filling out the form here.
Allow-list caveats for Terra on Azure
- Any user can be added to an Azure billing project or workspace
You do not have to be on an allow list to work in Terra on Azure if you are a Terra Billing project user or workspace reader. You can log into Terra with your Google ID and work in Terra on Azure if you use a shared workspace or Azure billing project. - If you are not on the allow list and you log in with a Microsoft/Azure ID
You will not be able to register/use Terra on Azure. - If you are not on the allow list and you log in with a Google ID
You will not be able to create an Azure billing project. However, you can work in ToA if you can access an Azure billing project. - If you do not have access to an Azure-funded Terra Billing Project
You cannot clone a Terra on Azure workspace.
Overview: Terra on Azure
Terra is a cloud-native platform for storing and analyzing biomedical data whose mission is “to help accelerate research by integrating data, analysis tools, and built-in security components to deliver frictionless research flows from data to results.” This release of Terra uses Microsoft Azure’s cloud infrastructure for data analyses and storage.
Project data and tools - together in a Terra workspace
Whether you're interested in running pipelines, a statistical analysis, or visualizing your data, you can access and manage the tools and data you need in a Terra workspace dedicated to your project.
Workspaces function like a (very powerful) desktop computer, except the working parts are all in the cloud, and you operate it from your browser.
Browser-based and cloud-native
- Streamline your work by consolidating resources
- Access data stored in different cloud locations in a single analysis
- Seamless collaboration with built-in security and access controls
The vision for Terra on Azure
Terra on Azure offers several major functional upgrades for Terra. Refactoring existing features
- Improves performance and scalability
- Makes it easier to integrate new analysis capabilities (such as upcoming support for additional workflow languages)
- Gives you - the user - maximum control of where all of your data in Terra is stored
Toward a unified Terra experience
Our vision is to iterate and improve these upgrades based on user feedback, starting with Terra on Azure Preview.
Once we validate that these changes meet current user needs and open opportunities for new user communities, we plan to implement these changes in Terra on Google.
Current costs of using Terra on Azure
Working in the cloud in Terra has infrastructure and resource costs, outlined below. Terra passes along Azure cloud resource charges with no markup.
Infrastructure cloud costs (per Terra Billing project)
For maximum control over where your data is stored and to increase flexibility (for example, to easily integrate additional analysis apps), we’ve transitioned some infrastructure from Terra-owned to user-owned.
- When you (or an IT Admin or collaborator) create a Terra Billing Project, Terra launches Azure infrastructure resources that currently cost about $5 per day and are shared across all workspaces in the billing project.
- Creating a new workspace has a fixed infrastructure cost (currently about $5 per day) for resources that power data tables. Additional charges will apply based on storage and compute usage within the workspace.
These infrastructure cloud costs accrue as long as you have a Terra Billing project/workspaceWe have not yet released support for deleting a Terra Billing project once created. If you want to pause resources on your billing project to reduce costs after you start working in Terra on Azure Preview, please reach out to support@terra.bio for assistance.
It is our top priority to reduce this cost while balancing performance needs in the near future.
Note that this cost model differs from that of Terra on Google, where there is currently no cost associated with creating a billing project (see Overview: Terra costs and Billing - GCP). As we develop multi-cloud Terra, we expect these cost models to align.
Variable cloud costs (per Terra workspace)
Adding data to storage and running analyses will incur additional fees to cover the cloud resources used in the workspace. These costs are calculated following Azure’s pricing (see pricing in Overview: Costs and billing in Terra on Azure). Terra passes these costs along to users without any markup.
JupyterLab
Terra on Azure includes access to JupyterLab supported by Azure Data Science Virtual Machines (DSVM). This offering includes flexible VM and disk size configuration options and a convenient file syncing service that automatically saves your notebook files to and from your workspace blob storage.
Select from four pre-configured cloud compute profiles
and specify the VM disk size in the Azure Cloud
Environment setup pane. Cost estimates for the
configuration will display in the blue bar at the top.
New functionality coming soon!
We will release additional JupyterLab functionality soon, including persistent disk storage and optional Graphics Processing Units (GPUs).
Workspace data tables
Data tables help store and organize data in an integrated, spreadsheet-like format. Primary data - including clinical data, demographics, or phenotypic data - can all be stored in data tables. Data tables can also keep links to genomic data files in cloud storage (workspace or external).
Data tables are hosted in a private relational database set up by the workspace owner. This makes data tables more scalable and gives you complete control over where (what geographic location) your data lives in Azure.
Who can see data tables?
In this early release, data tables will only be visible to the workspace creator. Subsequent feature releases will focus on a collaborative data table experience and database cost optimization.
WDL workflows with Cromwell
Terra on Azure includes three COVID-19-related workflows in every workspace created after February 22, 2023. The workflows are automatically included when the workspace is created.
To learn more, see How to set up and run a workflow.
For a tutorial and step-by-step instructions, see the COVID-19-Surveillance tutorial workspace and the accompanying step-by-step guide.
We hope to provide the ability to bring your own workflow very soon.
Workspace collaboration & sharing
Terra on Azure workspaces supports multiple users with owner, reader, and writer permissions.
What is shared by all collaborators?
In the first release, dashboard content, workspace blob storage files, and notebooks are visible to all collaborators.
Data tables and workflows are a single-user-per-workspace experience for now. Only the workspace creator can see and use both data tables and Cromwell for launching workflows. Our next release focuses on providing a collaborative analysis experience.
What is copied (when cloning a workspace)?
When you clone a workspace, the dashboard and notebook files are copied into the new workspace, along with three pre-configured workflows. Each workspace has its own blob storage, but files stored there are not copied to the cloned workspace.
Next steps
Ready to get started using Terra on Azure? Follow the three steps below.
1. Set up an account on Terra
- Request to be added to the public preview allow list (see request form here).
- Register for a Terra account (see How to set up an account in Terra on Azure).
2. Set up billing
Finance admins/users with access to an existing Azure subscription must set up cloud billing and link it to a Terra Billing Project following step-by-step instructions.
3. Explore a tutorial workspace
Featured workspaces let you try out the platform with pre-configured sample data, analysis tools, and documentation to guide you.
See the Bulk and single cell RNA Seq Analysis with Bioconductor workspace.
Note that you will need to make your own clone of the Featured Workspace!