How to create a TDR Billing Profile (GCP)

Anton Kovalsky
  • Updated

Learn how to create and list Google-backed TDR Billing Profiles, and share them with others. Creating a Billing Profile is the first step for data custodians/stewards in the Terra Data Repo. To learn how to set up an Azure-backed TDR billing profile, see How to create a TDR Billing Profile (Azure).

Overview

Before you can begin doing anything billable in the Terra Data Repository - adding datasets, ingesting data, or creating snapshots through Swagger - you will need to create a Billing Profile. The Billing profile UUID links actions with a cost in TDR to a Google Cloud billing account.

A single Billing Profile can be reused across datasets, and a single dataset can use multiple profiles to bill for different snapshots of data.

Step-by-step instructions to create a billing profile

1.1. Set up a Google Billing Account

Before creating a billing profile, you will need access to a Google Cloud Billing account. For details, see Instructions to create a Google Cloud Billing Account.

Prerequisite

Note that you need to have the Billing Account User role on that Cloud Billing Account.

1.2. Add the Data Repo service account as a user on your account

To give the Data Repo access to your billing, go to your GCP Billing page to grant the Data Repository Service account (datarepo-jade-api@terra-datarepo-production.iam.gserviceaccount.com) the Billing Account User role on the account as shown below.

Screenshot of billing management page in GCP console

Screen_Shot_2021-09-21_at_3.55.23_AM.png

Screenshot of Add Principal popup in GCP console

Screen_Shot_2021-09-21_at_3.55.34_AM.png

If you are working in the alpha environment, you’ll need to grant the following Data Repository Service account access as well: datarepo-jade-api@terra-datarepo-alpha.iam.gserviceaccount.com.

1.3. Create a Billing Profile via Swagger API

Once the service account has access to a billing source, you’ll be able to use the createProfile API endpoint to create a new Billing profile.

Click on the title or go directly to the endpoint at https://data.terra.bio/swagger-ui.html#/profiles/createProfile) and click “Try it out” to unlock it:

Remember to authorize Swagger every time you use itSee How to authenticate/troubleshoot Swagger for TDR for step-by-step instructions.

To create a GCP billing profile, you must check all four scopesThe last scope, https://www.googleapis.com/auth/cloud-billing.readonlycloud billing authorization is not checked off by default. You will have to manually check it.

If you get the error
"{
"message": "Could not check permissions on billing account '006351-1F5067-BF7B50'",
"errorDetail": []
}

it is because you are missing the last scope. Go back and re-authenticate, checking all four boxes.

What to expect

If you're properly authenticated, you'll be able to edit any input fields, such as the main request body, and you'll see an execute button somewhere below that field.

Modify the request body with the appropriate code, as shown below, and then click Execute.

Do not use the default response body in SwaggerTo execute successfully, make sure to copy/paste the response body below!

createProfile request body (copy/paste below)

{
"biller": "direct",
  "billingAccountId": "/*The Google Billing Account ID you can find on your GCP Billing page*/",
"cloudPlatform": "gcp",
"description": "string",
"profileName": "string"
}

Request body parameters

  • The "biller" field should generally be set to "direct"
  • The "billingAccountId" field should be set to the Google Billing Account ID that you can find on your GCP Billing page.
  • Wherever the above example code says "string", you can put whatever arbitrary set of characters you like to help you keep track of things. The ProfileName can be useful if you use multiple billing accounts. As you'll see below, when you're looking up Billing Profile information, having a clearly named Billing Profile saves you a lot of trouble if you're trying to remember which Billing Profile ID goes with which Billing Account ID.

Screen capture for Try it out

Sep-21-2021_04-07-55.gif

What to expect

The createProfile API endpoint can take a while to execute. Once it has executed, you will see a series of responses from the server, including error or success codes. It can be challenging to decipher these responses to know whether the job was successful, or if it is still running. 

You can check whether you successfully created a billing profile in two ways:

Check on the billing profile with Swagger

1. Wait until you see messages in the Response section of the Swagger UI (below the Execute button). 

2. Go to the enumerateProfiles API endpoint, and click Execute. Note that you may need to re-authenticate Swagger first, if your authentication has timed out.

3. If the profile you just requested is listed in the response body, it was successfully created.

Check on the billing profile through the TDR website

1. Log into https://data.terra.bio/.

2. Click on the Activity tab.

3. Locate the job you just ran (the class should be "ProfileCreateFlight")

4. If the job has succeeded, the status will be completed. 

Troubleshooting

If your job is taking awhile to run, you can monitor your job's progress with the jobs endpoints:

1. Copy your job's id from the "id" field in the createProfile job's response body:

Screenshot of the response body for an example createProfile request job. The 'id' field corresponds to the job's UUID, which can be used to monitor the job's progress.

2. Go to the retrieveJob API endpoint, enter the job id in the "id" field, and click Execute

3. If  the job_status is "succeeded," your billing profile was successfully created.

For more details and step-by-step instructions, see the troubleshooting section in How to authenticate/troubleshoot in Swagger

How to look up billing profile information

If you lose track of your billing profile ID, you can always find it using the enumerateProfiles API endpoint to list out all of the profiles to which you have access.

What to expect

If you've properly authenticated using the Authorize button near the top of the page, clicking Execute on this endpoint will list all Billing Profiles associated with the credentials you used to log in. Info for each Billing Profile will appear in the response body that appears beneath the request body after you've executed. The relevant information in most cases is the number corresponding to the "id" field:

Sep-21-2021_05-41-38.gif

Next Steps

Once you've set up your TDR billing profile, you can add collaborators to your profile or start adding data to TDR by creating a TDR dataset through the TDR website or Swagger APIs. See How to use the TDR support articles to locate instructions for these and other actions in TDR. 

Was this article helpful?

Comments

0 comments

Please sign in to leave a comment.