Learn how to create and list TDR Billing Profiles, and share them with others. Creating a Billing Profile is the first step for data custodians/stewards in the Terra Data Repo.
Before you can begin doing anything billable in the Terra Data Repository - adding datasets, ingesting data, or creating snapshots through Swagger - you will need to create a Billing Profile. The Billing profile UUID links actions with a cost in TDR to a Google Cloud billing account.
A single Billing Profile can be reused across datasets, and a single dataset can use multiple profiles to bill for different snapshots of data.
Step-by-step instructions to create a billing profile
1.1. Set up a Google Billing Account
Before creating a billing profile, you will need access to a Google Cloud Billing account. For details, see Instructions to create a Google Cloud Billing Account.
Note that you need to have the Billing Account User role on that Cloud Billing Account.
1.2. Add the Data Repo service account as a user on your account
To give the Data Repo access to your billing, go to your GCP Billing page to grant the Data Repository Service account (
firstname.lastname@example.org) the Billing Account User role on the account as shown below.
Screenshot of billing management page in GCP console
Screenshot of Add Principal popup in GCP console
If you are working in the alpha environment, you’ll need to grant the following Data Repository Service account access as well:
1.3. Create a Billing Profile via Swagger API
Once the service account has access to a billing source, you’ll be able to use the createProfile API endpoint to create a new Billing profile.
Click on the title or go directly to the endpoint at https://data.terra.bio/swagger-ui.html#/profiles/createProfile) and click “Try it out” to unlock it:
Remember to authorize Swagger every time you use it- Click Authorize near the top of the page.
- Choose an authentication method: googleoauth or oidc.
- If you're running the createProfile endpoint to create a google-backed billing profile, use the googleoauth authentication and check all four boxes (including the last one about billing, which may not be checked by default).
- Otherwise, choose either authentication method, but not both.
- click Authorize again.
- Input the appropriate credentials.
- When you close the pop-up window, do not click Sign Out.
To create a GCP billing profile, you must check all four scopesThe last scope,
https://www.googleapis.com/auth/cloud-billing.readonlycloud billing authorization is not checked off by default. You will have to manually check it.
If you get the error
"message": "Could not check permissions on billing account '006351-1F5067-BF7B50'",
it is because you are missing the last scope. Go back and re-authenticate, checking all four boxes.
What to expect
If you're properly authenticated, you'll be able to edit any input fields, such as the main request body, and you'll see an execute button somewhere below that field.
Modify the request body with the appropriate code, as shown below, and then click Execute.
Do not use the default response body in SwaggerTo execute successfully, make sure to copy/paste the response body below!
createProfile request body (copy/paste below)
"billingAccountId": "/*The Google Billing Account ID you can find on your GCP Billing page*/",
Request body parameters
- The "biller" field should generally be set to "direct"
- The "billingAccountId" field should be set to the Google Billing Account ID that you can find on your GCP Billing page.
- Wherever the above example code says "string", you can put whatever arbitrary set of characters you like to help you keep track of things. The ProfileName can be useful if you use multiple billing accounts. As you'll see below, when you're looking up Billing Profile information, having a clearly named Billing Profile saves you a lot of trouble if you're trying to remember which Billing Profile ID goes with which Billing Account ID.
Screen capture for Try it out
What to expect
The createProfile API endpoint can take a while to execute. Once it has executed, you will see a series of responses from the server, including error or success codes. It can be challenging to decipher these responses to know whether the job was successful, or if it is still running.
You can check whether you successfully created a billing profile in two ways:
Check on the billing profile with Swagger
1. Wait until you see messages in the Response section of the Swagger UI (below the Execute button).
2. Go to the enumerateProfiles API endpoint, and click Execute. Note that you may need to re-authenticate Swagger first, if your authentication has timed out.
3. If the profile you just requested is listed in the response body, it was successfully created.
Check on the billing profile through the TDR website
1. Log into https://data.terra.bio/.
2. Click on the Activity tab.
3. Locate the job you just ran (the class should be "ProfileCreateFlight")
4. If the job has succeeded, the status will be completed.
If your job is taking awhile to run, you can monitor your job's progress with the jobs endpoints:
1. Copy your job's id from the "id" field in the createProfile job's response body:
2. Go to the retrieveJob API endpoint, enter the job id in the "id" field, and click Execute.
3. If the job_status is "succeeded," your billing profile was successfully created.
For more details and step-by-step instructions, see the troubleshooting section in How to authenticate/troubleshoot in Swagger.
How to look up billing profile information
If you lose track of your billing profile ID, you can always find it using the enumerateProfiles API endpoint to list out all of the profiles to which you have access.
What to expect
If you've properly authenticated using the Authorize button near the top of the page, clicking Execute on this endpoint will list all Billing Profiles associated with the credentials you used to log in. Info for each Billing Profile will appear in the response body that appears beneath the request body after you've executed. The relevant information in most cases is the number corresponding to the "id" field: