Learn how to create and list TDR Billing Profiles, and share them with others. Creating a Billing Profile is the first step for data custodians/stewards in the Terra Data Repo.
Overview
Before you can begin doing anything billable in the Terra Data Repository - adding datasets, ingesting data, or creating snapshots through Swagger - you will need to create a Billing Profile. The Billing profile UUID links actions with a cost in TDR to a Google Cloud billing account.
A single Billing Profile can be reused across datasets, and a single dataset can use multiple profiles to bill for different snapshots of data.
Step-by-step instructions to create a billing profile
1.1. Set up a Google Billing Account
Before creating a billing profile, you will need access to a Google Cloud Billing account. For details, see Instructions to create a Google Cloud Billing Account.
Prerequisite
Note that you need to have the Billing Account User role on that Cloud Billing Account.
1.2. Add the Data Repo service account as a user on your account
To give the Data Repo access to your billing, go to your GCP Billing page to grant the Data Repository Service account (datarepo-jade-api@terra-datarepo-production.iam.gserviceaccount.com) the Billing Account User role on the account as shown below.
Screenshot of billing management page in GCP console
Screenshot of Add Principal popup in GCP console
If you are working in the alpha environment, you’ll need to grant the following Data Repository Service account access as well: datarepo-jade-api@terra-datarepo-alpha.iam.gserviceaccount.com.
1.3. Create a Billing Profile via Swagger API
Once the service account has access to a billing source, you’ll be able to use the createProfile API endpoint to create a new Billing profile.
Click on the title or go directly to the endpoint at https://data.terra.bio/swagger-ui.html#/profiles/createProfile) and click “Try it out” to unlock it:
Remember to authorize Swagger every time you use itClick “Authorize” near the top of the page, check all four boxes (including the last one about billing, which may not be checked by default) in the pop-up, and hit “Authorize” again. Then input the appropriate credentials to authenticate. Make sure you close the subsequent pop-up without clicking the “Sign Out” button.
You should now be able to execute the commands below by clicking the “Try it out” button next to the command of your choice. For a more detailed description of this authentication step, see How to authenticate/troubleshoot Swagger.
To create a GCP billing profile, you must check all four scopesThe last scope, https://www.googleapis.com/auth/cloud-billing.readonlycloud billing authorization is not checked off by default. You will have to manually check it.
If you get the error
"{
"message": "Could not check permissions on billing account '006351-1F5067-BF7B50'",
"errorDetail": []
}
it is because you are missing the last scope. Go back and re-authenticate, checking all four boxes.
What to expect
If you're properly authenticated, you'll be able to edit any input fields, such as the main request body, and you'll see an execute button somewhere below that field.
Modify the request body with the appropriate code, as shown below, and then click Execute.
Do not use the default response body in SwaggerTo execute successfully, make sure to copy/paste the response body below!
createProfile request body (copy/paste below)
{
"biller": "direct",
"billingAccountId": "/*The Google Billing Account ID you can find on your GCP Billing page*/",
"cloudPlatform": "gcp",
"description": "string",
"id": "/*a string object with a regex pattern*/",
"profileName": "string"
}
Request body parameters
- The "biller" field should generally be set to "direct"
- The "billingAccountId" field should be set to the Google Billing Account ID that you can find on your GCP Billing page.
- Wherever the above example code says "string", you can put whatever arbitrary set of characters you like to help you keep track of things. The ProfileName can be useful if you use multiple billing accounts. As you'll see below, when you're looking up Billing Profile information, having a clearly named Billing Profile saves you a lot of trouble if you're trying to remember which Billing Profile ID goes with which Billing Account ID.
- The "id" field must be a regex string, which is a string that follows this pattern:[a-z0-9]{8}-[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{12}.
How to generate a regex string
This might be auto-generated by the Swagger UI, but if, for some reason, the UI isn't generating a new unique regex string here, you may have to do it yourself. You can do this using a library like uuidgen or uuid and ensuring it is lower case. This ID will be required for most of the downstream steps that involve creating resources -- such as creating datasets, ingesting files, creating snapshots, and more. (don't worry, the section below explains how to look it up if you lose track of it).
Command to generate a regex string in a terminal
uuidgen | tr 'A-Z' 'a-z'
Screen capture for Try it out
Troubleshooting
The createProfile API endpoint can take a while to execute, and it can be challenging to know if something is wrong or if the job has just not finished yet. To monitor how your job is progressing, you can use the jobs endpoints.
For more details and step-by-step instructions, see the troubleshooting section in How to authenticate/troubleshoot in Swagger.
How to look up billing profile information
If you lose track of your Billing profile ID, you can always find it using the enumerateProfiles API endpoint to list out all of the profiles to which you have access.
What to expect
If you've properly authenticated using the Authorize button near the top of the page, clicking Execute on this endpoint will list all Billing Profiles associated with the credentials you used to log in. Info for each Billing Profile will appear in the response body that appears beneath the request body after you've executed. The relevant information in most cases is the number corresponding to the "id" field:
Sharing your Billing profile with someone else
Another way to gain access to a Billing profile (or to give someone else access) is to use the addProfilePolicyMember API endpoint to add a person via their email to a Billing profile to which you have access.
You'll need is the Billing profile ID (above) and the target user's email address.