How to create a TDR Billing Profile (Azure)

Learn how to create and list Azure-backed TDR Billing Profiles, and share them with others. Creating a Billing Profile is the first step for data custodians/stewards in the Terra Data Repo. To learn how to set up a Google-backed TDR billing profile, see How to create a TDR Billing Profile (GCP).

Overview

Before you can begin doing anything billable in the Terra Data Repository - adding datasets, ingesting data, or creating snapshots through Swagger - you will need to create a Billing Profile. The Billing profile UUID links actions with a cost in TDR to an Azure billing account.

A single Billing Profile can be reused across datasets, and a single dataset can use multiple profiles to bill for different snapshots of data.

Before you start: Prerequisites

Make sure to have these in place before trying to follow the instructions below.

• You are a member of at least one organization’s subscriptions in the Microsoft Azure Portal.
• The Azure subscription has the following resource providers enabled:
  • "Microsoft.Storage"
  • "Microsoft.Compute"
  • "Microsoft.Authorization"
  • "Microsoft.Batch"
  • "Microsoft.OperationalInsights"
  • "Microsoft.OperationsManagement"
  • "Microsoft.Insights"
  • "Microsoft.Network"
  • "Microsoft.DBforPostgreSQL"
  • "Microsoft.ContainerRegistry"
  • "Microsoft.KeyVault"
  • "Microsoft.ContainerService"
  • "Microsoft.Relay"
  • "Microsoft.SecurityInsights"

Terra requires a US-based subscriptionCurrently, a US-based subscription is needed to deploy the Terra AMA (Azure Marketplace Application.)

Step 1: Create an Azure Managed Application

To view the TDR offering in the Marketplace, you will need to be actively invitedCurrently, you will need to ask to the TDR offering by emailing support@terra.bio.

1.1. Go to the Managed Application Marketplace in the Azure portal under your subscription.

1.2. Search for the tdr-prod offering and click on it.

If you don’t see the tdr-prod offeringTry changing directories by clicking on your avatar in the top right and the Switch directories link.

1.3. In the tdr-prod (preview) offering, click on the blue Create button.

Screenshot of TDR application in Azure portal

1.4. In the Basics tab, enter the required information in the creation wizard. You can specify the two fields highlighted in orange below (you’ll select the other two - preconfigured - from a drop-down).

Screenshot of the creation wizard (Basics tab)

Required information details

• Subscription: The Azure subscription to bill to. Select from dropdown.
• Resource Group: The parent collection of resources the Terra application will belong to. This can be a new group or an existing one within the subscription. If your organization has one, it will show up in a dropdown. If you create a new group, only lowercase letters are allowed.
• Region: The region where the Terra application and its resources will be located. You should leave the default (East US).
• Application Name: This is a user-specified reference to your TDR Managed App. You can name it anything.
• Managed Resource Group: This is a pre-configured value that will store the data ingested into TDR within the Azure portal.

Region caveats: If you have data residency requirementsNote that this is the region where all data - including data in tables and in workspace blob storage - will be stored. Note that Terra on Azure currently only supports the East US region.

1.5. In the Terra Data Repository Settings tab, enter the required information in the creation wizard.

Screenshot of creation wizard (Terra Data Repository Settings tab)

Required information details

• Storage account name prefix: User-defined nUser-defined name to help identify your storage within the deployment (since the storage account names created by TDR will be random strings).
• Email: The user ID of the person who will create the TDR Billing Profile in Terra. Note that this has to be an individual user (no groups are allowed).

1.6. Click Review and Create (bottom), accept the license terms, and then click Create.

It can take an hour or two to deploy the applicationWe recommend waiting at least an hour before proceeding to the next step. If you run into problems following the next steps, wait another hour or two before contacting support.

Once it’s done, you’ll get a nice notification, and you can proceed to the next step.

Step 2: Gather information to link the account

You will need some parameters to be able to link the Managed Account to TDR. Once your managed app has been deployed, go back to the portal to get the following parameters. Make sure to write these values down somewhere easy to access.

Step-by-step instructions

2.1. Click on Home (top left) to get the variables you’ll need for the next step.

2.2. Under Resources (recent), click on the managed app you just created.

2.3. From the next screen, gather the following variables (used in step 3).

Values to write down

• PublisherTenant ID: Listed on the managed app Properties page. You can navigate to the properties page using the panel on the left. The tenant id is under the “properties” section of this page.
• Subscription ID: Listed on the managed app overview page
• Resource group: Can be an existing resource group or a newly created one you selected in the setup process, listed as the “Resource group” on the managed app overview page
• Application DeploymentName = Managed Application Name: The name at the top left of the page

Step 3: Create Terra Billing profile (Swagger APIs)

Once you have created the Managed Application and collected the values for the variables above, you’ll be able to use the createProfile API endpoint to create a new Billing profile.

3.1. Click on the createProfile API endpoint link or go directly to the endpoint at https://data.terra.bio/swagger-ui.html#/profiles/createProfile.

3.2. Authenticate Swagger.

3.3. Click Try it out to unlock the API.

What to expect

If you're properly authenticated, you'll be able to edit any input fields, such as the main request body, and you'll see an execute button somewhere below that field.

3.3. Modify the request body with the appropriate code, as shown below.

Do not use the default response body in SwaggerTo execute successfully, make sure to copy/paste the response body below!

createProfile request body (copy/paste below)

  {
  "profileName": "string",
  "description": "string",
  "biller": "direct",
  "cloudPlatform": "azure",
  "tenantId": "UUID",
  "subscriptionId": "UUID",
  "resourceGroupName": "string",
  "applicationDeploymentName": "string"
  }

Request body parameters

• profileName: This string is user-defined and will be how you find and refer to the Billing Profile in TDR. Wherever the above example code says "string", you can put whatever arbitrary set of characters you like to help you keep track of things. The ProfileName can be useful if you use multiple billing accounts. As you'll see below, when you're looking up Billing Profile information, having a clearly named Billing Profile saves you a lot of trouble if you're trying to remember which Billing Profile ID goes with which Billing Account ID.
• description: You can use this for a more detailed description.
• biller: Should generally be set to "direct."
• tenant ID: See section 2: Gathering information above.
• subscriptionID: See section 2: Gathering information above.
• resourceGroupName: See section 2: Gathering information above.
• applicationDeploymentName = Managed Application Name: See section 2: Gathering information above.

What to expect

The createProfile API endpoint can take a while to execute. Once it has executed, you will see a series of responses from the server, including error or success codes. It can be challenging to decipher these responses to know whether the job was successful, or if it is still running. 

You can check whether you successfully created a billing profile in two ways:

Check on the billing profile with Swagger

1. Wait until you see messages in the Response section of the Swagger UI (below the Execute button). 

2. Go to the enumerateProfiles API endpoint, and click Execute. Note that you may need to re-authenticate Swagger first, if your authentication has timed out.

3. If the profile you just requested is listed in the response body, it was successfully created.

Check on the billing profile through the TDR website

1. Log into https://data.terra.bio/.

2. Click on the Activity tab.

3. Locate the job you just ran (the class should be "ProfileCreateFlight")

4. If the job has succeeded, the status will be completed. 

Troubleshooting

The createProfile API endpoint can take a while to execute, and it can be challenging to know if something is wrong or if the job has just not finished yet. To monitor how your job is progressing, you can use the jobs endpoints. 

For more details and step-by-step instructions, see the troubleshooting section in How to authenticate/troubleshoot in Swagger. 

How to look up billing profile information

If you lose track of your Billing profile ID, you can always find it using the enumerateProfiles API endpoint to list out all of the profiles to which you have access.

If you've properly authenticated using the Authorize button near the top of the page, clicking Execute on this endpoint will list all Billing Profiles associated with the credentials you used to log in. Info for each Billing Profile will appear in the response body that appears beneath the request body after you've executed. The relevant information in most cases is the number corresponding to the "id" field:

Next Steps

Once you've set up your TDR billing profile, you can add collaborators to your profile or start adding data to TDR by creating a TDR dataset through the TDR website or Swagger APIs.