How to set up Terra on Azure (billing and admins)

Updated November 27, 2023 21:30

Step-by-step instructions for team finance admins (with access to the Azure subscription that will cover cloud costs) to set up cloud billing for Terra on Azure. Once you go through these steps, you will be able to set up workspaces to store and analyze data in Terra.

Steps to set up billing (admins)

Terra can only be deployed to US regionsTerra is available in the global Azure marketplace. However, at this time, the Terra AMA (Azure Marketplace Application) can only be deployed to US regions. The default region is South Central US. If you need to deploy Terra on Azure to a different region, please contact support (support@terra.bio).

Step 1: Set Azure subscription prerequisites

Before you can create the Terra Azure Managed Application, you must be a member of at least one organization’s subscription in the Microsoft Azure Portal.

If your organization already uses Azure

You will need to request access to an Azure Subscription you can use for working in Terra from your IT admin.

If your organization is new to Azure

To create an account and get started using Azure, see Azure documentation here.

Subscription prerequisites

Your Terra environment relies on several Azure services, listed below, for tasks like authentication, setting up storage and compute, and accessing cost information. The Terra managed application is empty when you deploy it in the Azure portal.

Terra and your Azure resource group

For Terra to work, your Azure subscription must have a number of resource providers enabled. Resources get deployed in the Managed Resource Group by the Terra Control Plane (which runs externally and is operated by the Broad Institute) when you perform certain actions in Terra UI. Thus you can think of Terra as a SaaS offering that happens to run some infrastructure in users’ Azure subscriptions.

Compute resources

AKS for running core analysis services and user-defined applications
Compute Data Science VMs for running tools like JupyterLab
Batch for running batch jobs (e.g. via Cromwell or Nextflow)
Azure Relay for proxying inbound http traffic

Storage resources

Storage Account + Blob storage
Postgres Flexible

Additional infrastructure

Log Analytics
App Insight
VNet and dedicated subnets for each compute resource

Resource providers that must be enabled

Some of these may already be enabled. Others may be disabled by your organization by default, and you will have to get admin approval to override. For additional guidance, see Additional Resources below.

Microsoft.Storage
Terra leverages Storage Accounts and Blob Containers for holding unstructured data in Terra workspaces.
Microsoft.Compute
Terra allocates Linux Virtual Machines to run JupyterLab and other analysis applications in Terra workspaces.
Microsoft.Authorization
Registered by default. Terra authenticates through Azure Resource Manager (ARM) for managing cloud resources in your subscription.
Microsoft.Batch
Terra leverages Azure Batch to run batch workflows using Cromwell in Terra workspaces.
Microsoft.OperationalInsights
Terra provisions a Log Analytics Workspace to collect system logs, audit logs, and user application logs.
Microsoft.OperationsManagement
Terra deploys a Data Collection Rule to connect Azure Kubernetes Service to the Log Analytics Workspace.
Microsoft.Insights
Terra leverages Application Insights to monitor system health of applications in Terra workspaces.
Microsoft.Network
Terra allocates a Virtual Network (VNet) and subnets to host applications running on compute resources.
Microsoft.DBforPostgreSQL
Terra allocates a PostgreSQL Flexible Server for hosting SQL Databases for applications in Terra workspaces.
Microsoft.ContainerService
Terra leverages Azure Kubernetes Service (AKS) to run applications in Terra workspaces, such as Cromwell and Workspace Data Service.
Microsoft.Relay
Terra leverages Azure Relay to allow end users to securely access Terra workspace applications via https.
Microsoft.ManagedIdentity
Terra creates User-Assigned Managed Identities and assigns them to compute resources, to allow for authentication and data access.

Additional resources

For more information about what each resource does, see Microsoft’s documentation on Matching resource providers to service.
For more information on how Terra uses your data, see Terra’s Terms of Service.
For more information on data privacy and security, see Terra's Security Posture.

Step 2: Create the Azure Managed Application

2.1. Go to the Terra application in the Managed Application Marketplace in the Azure portal under your subscription.

Can't find it?
The application can also be found under the Analytics or Compute categories or by searching for The Broad Institute publisher name in the marketplace.

ToA-Billing-1.2_Terra-in-Azure_Marketplace_Screenshot.png

Troubleshooting: If you cannot find the Terra ApplicationYou may need to explicitly add the Terra B2C tenant to your Azure tenant as an Enterprise Application.

What to do
Go to Home > Enterprise Applications > Consent and Permissions and select the Allow user consent for apps radio button.
ToA-EnterpriseApplications_Consent-and-permissions_Screenshot.png

2.2. Click on the Terra application and then click Create to surface the Terra application in Azure portal (screenshot below).

ToA-Billing-1.3_Creating-the-Terra-Managed-app_Screenshot.png

2.3. Enter the required information in the creation wizard.

Screenshot of the creation wizard

Required information explanation

Subscription: The Azure subscription to bill to.
Resource Group: The parent collection of resources that the Terra application will belong to. This can be a new group or an existing one within the subscription. If you are creating a new one, it is useful to give a name that helps you easily identify the resource group. For example, if you have different resource groups in different locations, include the location in the name.
Location: The region where the Terra application and its resources will be located.

Location caveatsIf you have data residency requirements Note that this is the region where all data - including data in tables and in workspace blob storage - will be stored.

The default region is South Central US. If you need to deploy Terra on Azure to a different region, please contact support (support@terra.bio).
Authorized Terra User: The email of the Terra user who will link a Terra billing project to this managed application. This can be a comma-separated list of multiple email addresses.
NOTE: This could be the IT/Finance admin setting up billing, or you may designate a different person to set up on the Terra side (team lead or PI).
Microsoft and Google Authorized Terra UsersAn Authorized Terra User can be any active Terra account user, regardless of whether they use Microsoft or Google to sign into Terra.
Application Name and Managed Resource Group: These are user-configurable values that refer back to the Terra application within the Azure portal.
Names are limited to 40 charactersNote that this cannot be validated on the marketplace side. In order to avoid problems, you will need to keep track.

2.4. Click Review and Create, accept the license terms, and then click Create.

ToA-Billing-1.5_Create-Managed-app-final-step_Screenshot.PNG

It can take an hour or two to deploy the application We recommend waiting at least an hour before proceeding to the next step. If you run into problems following the next steps, please wait another hour or two before contacting support.

Step 3: Connect Terra to the Azure Managed Application

Note that if you haven’t yet registered for a Terra account, you will need to register first.

3.1. Wait until the managed app has been deployed in the Azure portal.

ToA-Billing-1.6_Deployment-complete_Screenshot.PNG

3.2. Go to the Terra Billing page (https://app.terra.bio/#billing) and sign in with the credentials for the Authorized Terra User (from 2.3 above).

3.3. Click the Create button, and (3) select Azure Billing Project.

ToA-Billing-2.2_Create-Terra-on-Azure-Billing-Project_Screenshot.png

3.4. Follow the steps to link your Azure Subscription to Terra. Enter an Azure subscription ID, add additional users (optional), and enter a Billing Project name in the form to create a new Terra Billing Project.

ToA_Screenshot-of-UI-to-link-an-Azure-subscription-to-Terra.png

Finding the subscription ID

You'll find the subscription on the Azure Managed Application homepage shown in step 3.1.

ToA-Billing-2.3_Subscription-ID-inManaged-Application_-homepage_Screenshot.png

If you don’t see the Azure Managed ApplicationMake sure your email address matches the “Authorized Terra User” entered in Step 1.4.

Colleagues don’t need to be on the Billing Project to collaborate It’s always best to start by giving colleagues minimal access and grant additional permissions as needed. The admin or PI who sets up billing can create workspaces and add collaborators as co-owners or writers. How you share funding, data, and analysis tools ultimately depends on your group's needs.

3.5. Click the Create button. You should now see the Azure Terra Billing Project in the list to the left.

ToA-Billing-2.4_Terra-on-Azure-Billing-Project-on-billing-page_Screenshot.png

This operation takes approximately 15 minutes to completeYou’ll see a loading spinner next to the project to indicate progress.

This is when you start to accrue the fixed workbench infrastructure cost ($10/day).

Troubleshooting

If you get an error message that includes the phrase “Missing required providers,” you will need to go back and make sure the indicated providers are enabled. These may be disabled by default for your organization. If so, you will need to ask to have them enabled.

What to do

See Step 1: Set up Azure subscription prerequisites for additional guidance.

Next steps: Set up workspaces/team access

You can now use the Terra Billing project to create workspaces where your team can collaborate in Terra.

To try out working in Terra

Check out Terra on Azure Featured Workspaces.
See How to create or clone a workspace for step-by-step instructions.
Once you create a workspace, you can add colleagues who can work collaboratively.

Shared workspace concepts to consider

Sharing workspaces is a way to share a funding source that gives less control to individual team members.
All workspace costs are paid by the linked Azure Marketplace subscription via the associated Terra Billing project.
The workspace owner (creator) controls exactly what each collaborator can do in the workspace (i.e., reader, writer, owner roles).
For a detailed description of shared workspace roles, see Sharing data and tools (workspace permissions.
Collaborators can only accrue costs (run an analysis or store or egress data) if the workspace owner gives sufficient workspace permission (writer or owner).
Colleagues cannot create workspaces of their own unless they are on a Terra Billing project.

Controlling cost and access

Owners have fine-grained control of what collaborators in a shared workspace are able to do. When you share the workspace, you give each person or managed group reader, writer, or owner access.