How to create a TDR Billing Profile (Azure)

Learn how to create and list TDR (Azure) Billing Profiles, and share them with others. Creating a Billing Profile is the first step for data custodians/stewards in the Terra Data Repo.

Overview

Before you can begin doing anything billable in the Terra Data Repository - adding datasets, ingesting data, or creating snapshots through Swagger - you will need to create a Billing Profile. The Billing profile UUID links actions with a cost in TDR to a Google Cloud billing account.

A single Billing Profile can be reused across datasets, and a single dataset can use multiple profiles to bill for different snapshots of data.

Before you start: Prerequisites

Make sure to have these in place before trying to follow the instructions below.

• You are a member of at least one organization’s subscriptions in the Microsoft Azure Portal.
• The Azure subscription has the following resource providers enabled:
  • "Microsoft.Storage"
  • "Microsoft.Compute"
  • "Microsoft.Authorization"
  • "Microsoft.Batch"
  • "Microsoft.OperationalInsights"
  • "Microsoft.OperationsManagement"
  • "Microsoft.Insights"
  • "Microsoft.Network"
  • "Microsoft.DBforPostgreSQL"
  • "Microsoft.ContainerRegistry"
  • "Microsoft.KeyVault"
  • "Microsoft.ContainerService"
  • "Microsoft.Relay"

Terra requires a US-based subscriptionCurrently, a US-based subscription is needed to deploy the Terra AMA (Azure Marketplace Application.)

Step 1: Create an Azure Managed Application

To view the TDR offering in the Marketplace, you will need to be actively invitedCurrently, you will need to ask to the TDR offering by emailing support@terra.bio.

1.1. Go to the Managed Application Marketplace in the Azure portal under your subscription.

1.2. Search for the tdr-prod offering and click on it.

If you don’t see the tdr-prod offeringTry changing directories by clicking on your avatar in the top right and the Switch directories link.

1.3. In the tdr-prod (preview) offering, click on the blue Create button.

Screenshot of TDR application in Azure portal

1.4. In the Basics tab, enter the required information in the creation wizard. You can specify the two fields highlighted in orange below (you’ll select the other two - preconfigured - from a drop-down).

Screenshot of the creation wizard (Basics tab)

Required information details

• Subscription: The Azure subscription to bill to. Select from dropdown.
• Resource Group: The parent collection of resources the Terra application will belong to. This can be a new group or an existing one within the subscription. If your organization has one, it will show up in a dropdown. If you create a new group, only lowercase letters are allowed.
• Region: The region where the Terra application and its resources will be located. You should leave the default (East US).
• Application Name: This is a user-specified reference to your TDR Managed App. You can name it anything.
• Managed Resource Group: This is a pre-configured value that will store the data ingested into TDR within the Azure portal.

Region caveats: If you have data residency requirementsNote that this is the region where all data - including data in tables and in workspace blob storage - will be stored. Note that Terra on Azure currently only supports the East US region.

1.5. In the Terra Data Repository Settings tab, enter the required information in the creation wizard.

Screenshot of creation wizard (Terra Data Repository Settings tab)

Required information details

• Storage account name prefix: User-defined nUser-defined name to help identify your storage within the deployment (since the storage account names created by TDR will be random strings).
• Email: The user ID of the person who will create the TDR Billing Profile in Terra. Note that this has to be an individual user (no groups are allowed).

1.6. Click Review and Create (bottom), accept the license terms, and then click Create.

It can take an hour or two to deploy the applicationWe recommend waiting at least an hour before proceeding to the next step. If you run into problems following the next steps, wait another hour or two before contacting support.

Once it’s done, you’ll get a nice notification, and you can proceed to the next step.

Step 2: Gather information to link the account

You will need some parameters to be able to link the Managed Account to TDR. Once your managed app has been deployed, go back to the portal to get the following parameters. Make sure to write these values down somewhere easy to access.

Step-by-step instructions

2.1. Click on Home (top left) to get the variables you’ll need for the next step.

2.2. Under Resources (recent), click on the managed app you just created.

2.3. From the next screen, gather the following variables (used in step 3).

Values to write down

• PublisherTenant ID: Listed on the managed app Properties page. You can navigate to the properties page using the panel on the left. The tenant id is under the “properties” section of this page.
• Subscription ID: Listed on the managed app overview page
• Resource group: Can be an existing resource group or a newly created one you selected in the setup process, listed as the “Resource group” on the managed app overview page
• Application DeploymentName = Managed Application Name: The name at the top left of the page

Step 3: Create Terra Billing profile (Swagger APIs)

Once you have created the Managed Application and collected the values for the variables above, you’ll be able to use the createProfile API endpoint to create a new Billing profile.

3.1. Click on the createProfile API endpoint link or go directly to the endpoint at https://data.terra.bio/swagger-ui.html#/profiles/createProfile.

3.2. Authenticate Swagger (check off three default scopes).

Remember to authorize Swagger every time you use itClick “Authorize” near the top of the page, check all four boxes (including the last one about billing, which may not be checked by default) in the pop-up, and hit “Authorize” again. Then input the appropriate credentials to authenticate. Make sure you close the subsequent pop-up without clicking the “Sign Out” button.

You should now be able to execute the commands below by clicking the “Try it out” button next to the command of your choice. For a more detailed description of this authentication step, see How to authenticate/troubleshoot Swagger.

3.2. Click Try it out to unlock the API.

What to expect

If you're properly authenticated, you'll be able to edit any input fields, such as the main request body, and you'll see an execute button somewhere below that field.

3.3. Modify the request body with the appropriate code, as shown below.

Do not use the default response body in SwaggerTo execute successfully, make sure to copy/paste the response body below!

createProfile request body (copy/paste below)

  {
  "id": "UUID",
  "profileName": "string",
  "description": "string",
  "biller": "direct",
  "cloudPlatform": "azure",
  "tenantId": "UUID",
  "subscriptionId": "UUID",
  "resourceGroupName": "string",
  "applicationDeploymentName": "string"
  }

Request body parameters

• id field (optional):  If you don’t assign one, Swagger will generate a UUID for you. This ID will be required for most of the downstream steps that involve creating resources -- such as creating datasets, ingesting files, creating snapshots, and more. (don't worry, the section below explains how to look it up if you lose track of it). If you won’t assign a profileID, make sure to delete this line in the request body.
  Requirements: the id must be a regex string, which is a string that follows this pattern:[a-z0-9]{8}-[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{12}. You can generate a regex string using a website like https://www.uuidgenerator.net/ or a library like uuidgen or uuid and ensuring it is lower case.
• profileName: This string is user-defined and will be how you find and refer to the Billing Profile in TDR. Wherever the above example code says "string", you can put whatever arbitrary set of characters you like to help you keep track of things. The ProfileName can be useful if you use multiple billing accounts. As you'll see below, when you're looking up Billing Profile information, having a clearly named Billing Profile saves you a lot of trouble if you're trying to remember which Billing Profile ID goes with which Billing Account ID.
• description: You can use this for a more detailed description.
• biller: Should generally be set to "direct."
• tenant ID: See section 2: Gathering information above.
• subscriptionID: See section 2: Gathering information above.
• resourceGroupName: See section 2: Gathering information above.
• applicationDeploymentName = Managed Application Name: See section 2: Gathering information above.

Troubleshooting

The createProfile API endpoint can take a while to execute, and it can be challenging to know if something is wrong or if the job has just not finished yet. To monitor how your job is progressing, you can use the jobs endpoints. 

For more details and step-by-step instructions, see the troubleshooting section in How to authenticate/troubleshoot in Swagger. 

How to look up billing profile information

If you lose track of your Billing profile ID, you can always find it using the enumerateProfiles API endpoint to list out all of the profiles to which you have access.

What to expect

If you've properly authenticated using the Authorize button near the top of the page, clicking Execute on this endpoint will list all Billing Profiles associated with the credentials you used to log in. Info for each Billing Profile will appear in the response body that appears beneath the request body after you've executed. The relevant information in most cases is the number corresponding to the "id" field:

Sharing your Billing profile with someone else

Another way to gain access to a Billing profile (or to give someone else access) is to use the addProfilePolicyMember API endpoint to add a person via their email to a Billing profile to which you have access.

You'll need the Billing profile ID (above) and the target user's email address.