Terra expenses in the Azure Portal

Allie Cliffe
  • Updated

If you're interested in using Terra on Azure, please email terra-enterprise@broadinstitute.org.

Find out how to optimize your spend by aligning cloud costs in Azure Portal with actions performed in Terra. This document also includes information about actions you can perform in Terra, which ones cost money, and which are free of charge. All Google Cloud costs in Terra are calculated per workspace and paid via the Terra Billing project assignen you create it. 

Please reference Overview: Controlling Cloud costs on Terra as a companion to this article.

Cost Transparency with Azure Portal

Terra and your Azure subscription

Organizations working with Terra on Azure deploy the Terra Managed Application using a subscription in their Azure tenant. Once deployed, an IT admin in your organization creates a Terra Billing Project, which spins up resources that power your own Terra Environment and its workspaces. In this article, we describe how a member with permission to view an org’s Azure subscription can view and monitor costs of actions their group takes in Terra.

Who can view costs?

Users with permission to view the Azure Subscription linked to the Terra Billing Project can view costs with the Azure Cost Management tool.

Connecting Terra terms/functionality to Azure Services

Azure Portal’s Cost Analysis exposes two resource groups created when you spin up your Terra Environment. Read on for a summary of what Azure services are deployed under these two groups. This will help you navigate the cloud costs of resources in your Terra Environment.

Managed Resource Group (MRG)

  • Naming: Often starts with the mrg- prefix (however, this depends on what the person that created the resource group named it).
  • Function: Contains all Terra Environment costs except for Workspace Services
    • Databases for managing data and logging
    • Cloud environment VM 
    • Cloud storage 
Azure Service

Terra Environment
(Terra Managed Application)

Cost Considerations
Postgres Server A shared server that enables a database per workspace. Workspace data tables and workflow services both use the database to provide users with data management features

Because this server is used by all workspaces, the server is always on.

The server size is a fixed cost. When you add data to your database, it does not increase cost.

Log Analyics A tool in the Azure portal to edit and run log queries from data collected by Azure Monitor logs. This service provides the ability to retrieve records that match particular criteria, identify trends, analyze patterns, and provide various insights into your data. This can include data access logs that are required when working with protected data.

As users perform more actions across the Terra Instance, log file sizes will grow.

  • $2.50/GB for standard Terra Environments

  • $4.50/GB for Terra Environments that use Protected Data
Service Bus Networking  
Virtual Network Networking  
Virtual Machine When a notebook is created or a workflow is submitted, this launches a short-term Virtual Machine that is used to complete the requested analysis Pay as you use
Load Balancer Workflow execution Pay as you use
Storage Account For workspace storage containers Cost increases when you store data in your workspaces cloud storage or persistent disk

Workspace services

  • Automatically created by Azure
  • Naming: Most often the name of your MRG and an ‘-aks’ suffix
    - [Managed_Resource_Group]_aks (Terra Environments created after Nov 2023)
    - MC_[Manage Resource Group_] (Terra Environments created before Nov 2023)
  • Function: Manages all of the resources for your workspace services
Azure Service

Terra Environment
(Terra Managed Application)

Cost Considerations
Virtual Machine Azure automatically creates a resource group specific to the Azure Kubernetes Service (AKS). Workspace services (data tables, workflow managed, workflow engines) share AKS resources. AKS automatically scales up resources when a user creates a new workspace or starts a new service. The more workspaces a Billing Project has, and the more services running within those workspaces, the more resources AKS needs.
Storage
Load Balancer
Virtual Network

1. Spend reporting in Terra

Permission requirements to view spend in Terra

  • Terra Billing Project Owner
  • Azure Subscription Administrator

Instructions to find spend reporting in Terra

1.1. Go to your billing page (https://app.terra.bio/#billing) from the main menu at the top left. 

1.2. Select the Terra Billing Project corresponding to your Terra Environment of interest.

1.3. Click on Spend report to see the total spend over the last 7, 30, or 90 days. 

ToA-Expense-in-Azure-Portal_1.3-spend-report-in-Terra_Screenshot.png

2. More granular cost reporting (Azure Portal)

To drill down to specific costs that are merged in the spend reporting in Terra, you will go to the associated Terra Managed Application in the Microsoft Azure portal.

Step-by-step instructions

2.1. From the Terra Billing page, click the View project resources in Azure Portal link under the Billing Project name.

Terra-on-Azure-Expenses-in-Azure-Portal_2.1-click-link_Screenshot.png

2.2. This takes you to the Azure portal where you'll see a list of your Terra Environment resources.

Terra-on-Azure-Expenses-in-Azure-Portal_2.2_Resources-in-portal_Screenshot.png

There are two resource groups that make up your Terra Environment“mrg-[your custom name]”
This includes all resources except for the resources that power Workspace Services.

"mrg-[your custom name]_aks”
All resources to power Workspace Services (data tables, workflow service).

Note: If your Terra billing project was created before Nov 2023, the workspace services resource group was called MC_[your custom name]”.

2.3. Use the Subscription and two Resource Group names to navigate to different views to narrow down on specific costs or specific timeframes. Scroll down for a few examples to help you get started.

Granular cost analysis examples

Below are step-by-step instructions for identifying Cloud costs associated with your Terra environment and workspaces.

What is the total spend of my Terra Environment?
This is similar to the Spend Report in Terra Billing Project, but you can customize timeframes and understand where costs are derived.

How much am I currently paying in storage?
This is similar to the Spend Report in Terra Billing Project, but with more customizable timeframes.

What is the compute cost of workflows run in my Terra Environment?

3. Total Terra Environment spend (example)

In the Azure portal, your Terra Environment is called the Terra Managed Application. The cost of the entire Terra Managed Application includes two resources groups: mrg-[your custom name] + mrg-[your custom name]_aks.

Step-by-step instructions

3.1. In the view in the prior section, the name of the managed application is bethcosttest1.

3.2. From the Azure portal URL provided in your Terra Billing Project, select the Subscription these resources are linked to.

3.3. This takes you to a view of the entire subscription.

ToA-Expenses-in-th-e-Azure-Portal_3.3-subscription-overview_Screenshot.png

Remember, that a subscription could pay for one or more Terra Environments (Terra Managed Applications). If you have more than one Terra Environment, you need to narrow your search criteria, which we will demonstrate in this tutorial.

3.4. To look at cost, click Cost Analysis on the left-hand side.

oA-Expenses-in-th-e-Azure-Portal_3.4-Cost-Analysis_Screenshot.png

3.5. In this tutorial, we'll look at the cost per day. On the cost analysis screen, set View to DailyCosts.

oA-Expenses-in-th-e-Azure-Portal_3.5-daily-costs_Screenshot.png

3.6. You can select the calendar icon, and then custom date range to select a timeframe of interest.

ToA-Expenses-in-th-e-Azure-Portal_3.6-Custom-dates_Screenshot.png

3.7. Apply a filter for the Terra Environment of interest by clicking “Add Filter”.

oA-Expenses-in-th-e-Azure-Portal_3.7-add-filter_Screenshot.png

3.8. Filter on Resource group name and type in the custom name of your Resource Groups.

oA-Expenses-in-th-e-Azure-Portal_3.8-terra-resource-groups-filter_Screenshot.png

3.9. You'll see two resource groups: mrg-[your custom name] and mrg-[your custom name]_aks. Select both.

3.10. Now, you can see the entire cost of this Terra Environment per day, or over any time period of interest.

Example: Find the starting cost of a Terra Environment

To isolate the starting costs of a Terra Environment, follow the instructions above, setting the custom time as the first one or two days (before creating a workspace). This example shows the current starting cost of a Terra Environment deployed in east-us.

For 1 day on Dec 1

  • mrg-[your custom name] = $5.09
  • mrg-[your custom name]_aks = $5.49

Total daily cost = $10.58

  • For 2 days (Dec 1 - 2)
  • The total cost is $21.68

4. Monitor workspace costs (example)

Workspace services (data tables, workflows) are powered by a shared Azure Kubernetes Service (aks), and all of the resources in this cluster are under the resource group with the `_aks` suffix in your Terra Environment.

When you first create your Terra Environment, there is a starting size for Azure Kubernetes Service (aks) that can fit a few workspaces. Once you create several workspaces, resources automatically scale, and your workspace infrastructure cost will increase. You can monitor the cost of workspaces by looking at the daily cost of the resource group with the `_aks` suffix.

Step-by-step instructions

4.1. From the Cost Analysis screen, filter only on the Resource Group ending with -aks in the Terra Environment of interest.

ToA-Expenses-in-th-e-Azure-Portal_4.1-aks-filter_Screenshot.png

4.2. Using the same DailyCosts view for the time period Oct 17 - 21, you can see a scale event occurred on mid-day on December 8 when an additional workspace was created.

December 9 shows the cost of the new resource for a 24-hour period.

ToA-Expenses-in-th-e-Azure-Portal_4.2-Workspace-costs-by-day_Screenshot.png

4.3. Below is a starting example of the resources within the resource group with the `_aks` suffix: Virtual Machine, Storage, Load Balancer, Virtual Network.ToA-Expenses-in-th-e-Azure-Portal_4.3-resources-in-the-aks-resource-group_Screenshot.png

You can change the filter name to Resource and see that these resources are for AKS (Azure’s Managed Kubernetes service).

4.4. You can see how this total cost changes over time as you add more workspaces. Currently, an AKS scale event adds about $5/day.

5. How to isolate storage costs

This view shows the total cost of all files stored in all workspaces.

5.1. From the cost analysis page, select the Scope as your mrg-[resource_group_name]. This resource group includes storage associated with your workspace blobs.

Step-by-step instructions

5.2. Filter by resource type and type in storage account.

ToA-Expenses-in-th-e-Azure-Portal_5.2-filter-by-resource-type-storage-account_Screenshot.png

5.3. Narrow to the date range for the period of interest.

6. Find the cost of a workflow run (Batch VM+Disk)

When a workflow is launched by the Workflow Service in a workspace, Azure Virtual Machines and attached Disks are created while the workflow runs. These costs are associated with the Azure Batch resource group in your Terra Managed Application.

Step-by-step instructions

6.1. From the cost analysis page, select the Scope as your mrg-[resource_group_name]. This resource group will include any Azure-batch resources requested by Terra.

6.2. Select a time period when you believe the workflow ran (for some workflows, this can be greater than 24 hours). In the example below, we selected December 11-13.

ToA-Expenses-in-the-Azure-Portal_6.2-Select-time-period_Screenshotpng.png

6.3. Use the add filter option to narrow to Resource type = microsoft.batch/batchaccounts

ToA-Expenses-in-Azure-Portal_6.3-Filter-by-resource-type-batchaccounts_Screenshot.png

6.4. The time period selected used batch resources totaling $2.25.

ToA-Expenses-in-Azure-Portal_6.4-Isolate-a-single-workflow_Screenshot.png

To benchmark costs for a single runIf you run multiple workflows during the same period, they will be combined in this view. If you wish to identify the cost of a single run, we recommend launching a single workflow and waiting for a 24 hour period to see the cost reported in the portal.

Other notable costs: Logs

The examples above cover direct workflows costs onlyWhen you run workflows, there are notable indirect costs to consider as well. If you’re working in a Terra Environment, your organization is paying for the cost of Log Analytics, an Azure service that provides the ability to query logs for troubleshooting and compliance.

Examples of when log files increase

Troubleshooting logs

  • When you create a Terra Environment
  • When you create a workspace
  • When you turn on a workspace service (workflows)

Compliance logs

  • When you upload files to your data table or workspace cloud storage
  • When you import data from a repository
  • When you access a file (or files) when running a notebook or workflow
  • Note that each individual read operation creates a distinct log event
  • When you download files from your workspace storage
  • When you add data to a data table

To monitor the costs of logs (standard Terra/Billing Project)

1. Go to the Cost Analysis page in your subscription.

2. Filter on View: Daily Costs, a date range of interest, Resource group name: mrg-<your name here>, and Service Name: Log Analytics.

Working in FedRAMP-moderate Terra Environments (additional Security Monitoring enabled)If you are working with controlled-access data, your organization is required to use a Terra Environment with additional security monitoring enabled (FedRAMP-moderate Terra). This requirement increases the costs of logs, which will be reflected on the Cost Analysis page for the Sentinel service.

To monitor the costs of logs (FedRAMP-moderate Terra/Billing Project)

If you are working with FISM controlled data and have additional security logging enabled, follow the steps below to isolate the costs of the additional logging. 

1. Go to the Cost Analysis page in your subscription.

2. Filter on View: Daily Costs, a date range of interest, Resource group name: mrg-<your name here>, and Service Name: Sentinel.

Other notable costs:  Network Usage

When you spin up resources in your Terra Environment, they communicate with one another via network usage. In some scenarios, your network usage could spike. We explain below how to monitor network usage.

Examples of when networking increases

General Terra usage

  • When you create and use workspaces 

When moving data

  • When workspace data is downloaded to your computer
  • When workspace data is accessed from another region in Azure
  • When workspace data is accessed from another cloud provider

To monitor networking costs

1. Go to the Cost Analysis page in your subscription.

To monitor general Terra usage networking

2. Filter on View: Daily Costs, a date range of interest (for example: the date you ran a workflow), Resource group name: mrg-<your name here>, and Service Name: Load Balancer, Service Bus.

To monitor data movement networking

2. Filter on View: Daily Costs, a date range of interest (for example: the date you ran a workflow), Resource group name: mrg-<your name here>, and Service Name: Bandwidth, Service Bus.

 

Was this article helpful?

0 out of 0 found this helpful

Comments

0 comments

Please sign in to leave a comment.