Step-by-step instructions for setting up Google spend reporting in Terra. For most people, there are three steps (including verifying that you have the right permission) to setting up cost reporting. Once you complete these, the spend reporting will appear automatically.
For more background, see Displaying Google cloud costs in Terra.
Not able to follow these directions?You must be an "Owner" of the Terra Billing Project and “Owner” or "Admin" on the Google Cloud Billing account. If at any point you are not able to follow the directions below, or do not see the options in the screenshots (or they are greyed out), it is likely because you do not have sufficient permissions on the Google Cloud Billing account. You will need to ask the owner for admin privileges on the Cloud Billing account or to set up a BigQuery dataset for billing export (follow directions below).
This may be the case, for example, if you are using a third-party resellers such as Onix.
Step 1. Before you start: Verify your permissions
Follow the steps below (three screens) by clicking the Next button before going on to Step 1.
-
1.1: Verify you are a Terra Billing project "Owner"
Go to your billing page (app.terra.bio/#billing) and find the Terra Billing project you wish to enable spend reporting on.
If the billing project is listed in the Owned by you section, you're all set (go to 2: Verify you are the Cloud Billing account Owner or Admin).
Not the Billing project owner?
If the billing project is listed in the Shared with you section, you can click on the Members tab to see who you would need to ask to grant you permission.
-
1.2: Verify you are a Cloud Billing account owner or admin
Go to billing in Google Cloud console (https://console.cloud.google.com/billing) and click on the name of the Cloud Billing account linked to the Terra Billing project.
If you see the current month's summary for this Cloud Billing account, you are all set! You can move on to Step 1: Set up billing export (once per Cloud Billing account).
If you see an error message "You don’t have permission to view costs for this billing account" you will need to ask the owner to grant you admin permission on the Cloud Billing account.
If you use a third-party reseller, contact them for help.
-
1.3: What to do if you don't have Cloud billing account permission
Follow the steps below to determine who to ask for Admin permission.
1.3.1. Click on the Billing account Manage link to the right (above the Cloud billing account name - where the arrow is pointing)
1.3.2. Expand the Billing Account Administrator section to see who to ask.
Step 2. Set up billing export (once per Cloud Billing account)
Follow the steps below by clicking the Next button before going on to Step 3.
-
Terra needs to be able to access Google's cost reports (a spend reporting billing dataset on BigQuery) to enable the workflows spend reporting. Google Cloud account owners will set this up in Google Cloud.
You will only need to do this once per Google Cloud Billing account.
Who can skip this step Some users already have billing export set up for their Cloud Billing account. Broad Google Cloud Billing account users and SADA users can skip this first step and go directly to Step 2: Configure the workflow spend report.
-
2.1. Navigate to the billing account management page in Google Cloud Console.
2.2. Click on the name of the billing account you would like to enable billing exports for.
If you have more than one Cloud Billing accountYou will need to repeat for each one associated with a Terra Billing Project.
-
2.3. Click on Billing export in the left sidebar.
-
2.4. Under Standard usage cost, click the Edit Settings button.
-
2.5. Next you will create/configure a project for the billing export BigQuery dataset.
Why do you need a Google project for the export dataset? Storing the export data and querying the dataset will cost a (small amount) of money. The dataset project tracks how much and where to bill the charges.
-
2.6. Select the Google project to host the billing export BigQuery dataset from the dropdown.
The Billing project cannot be a Terra-generated workspace projectTo check, make sure the ID in the dropdown does not start with
terra-
. If the project has that formatting, it was created by Terra and will not work for this step.
Example: projects created by Terra
In that case, you will need to create a Google-native project first.If you don't have a project in the Cloud Billing account
Google will prompt you to create one.
-
If you only have Terra-generated projects
You will need to create a project to host your dataset.
1. Go to IAM & Admin > Manage resources from the main menu (three horizontal lines at top left of the page).
2. Click Create project.
3. In the New Project form, fill in the name of your export billing project and select the Cloud Billing account and organization (if applicable).
Don't start your project name with
terra-
! That will make it hard to separate what is Terra-generated and what you created. -
Unable to create a project in Google?If you are unable to follow the instructions (especially if what's on your screen looks different than what's here), you most likely don't have the right permission.
What to do
If you are part of an organization, reach out to your Google Cloud Billing account admin or central organization. See instructions above for how to find the right person to ask. If you use a third-party reseller, reach out to the reseller to ask for help. -
2.8. From the dropdown menu, select (or create) the BigQuery dataset to store the billing export data.
If you don't already have a dataset in this Google project, Google will prompt you to create oneFrom the menu, click on Create new dataset, fill in the form (screenshot below), and select the Create dataset button at the bottom.
-
2.9. Click Save.
-
2.10. In the BigQuery export tab, click on the dataset name.
-
2.11. From the dataset tab, copy the Google project ID (to the left of the period) and BigQuery dataset name (to the right of the period) to a safe place. You will need it to configure spend reporting in Terra (2.4 below).
Example: Google Project ID and Dataset ID
Google Project ID
billing-export-project-367217
and Dataset IDspend-reporting-in-terra
The format is
Google project ID
.BigQuery dataset name
(separated by a period) -
2.12. Go to the sharing dropdown above and to the right of the Dataset info and click Permissions.
If this option is greyed out, it is most likely because your role doesn't allow you to share the dataset. You will need to ask the owner of the Google Cloud Billing account to grant you permission to share a BigQuery dataset.
-
2.13. Select the Add principals button in the Dataset Permissions section.
-
2.14. In the Add principals popup, type
spend-reporting@terra.bio
into the New Principals field (at arrow in screenshot below) and select BigQuery Data Viewer from the Role dropdown (circled). This grants Terra permission to access the dataset you've just set up. -
2.15. Click the Save button (at the bottom of the form).
Google BigQuery billing exports are now configured for this Cloud Billing account. You can confirm by expanding the menu under BigQuery Data Viewer (below).
-
The next step will be Step 3: Configure workflow spend report in Terra.
Before you move on to step 2If you just completed Step 1 we recommend you wait several hours to complete Step 2 in order for billable activity to be recorded in BigQuery or you may receive an error that the dataset cannot be found.
Step 3. Configure workflow spend reporting in Terra
Follow the steps below by clicking the Next button.
-
Once billing data export is set up in Google Cloud, you'll set up (configure) spend reporting in Terra. You will only need to do this once per Terra Billing project.
Do you have multiple Terra billing projects?If you have workspaces in multiple billing projects that you want to set up with spend reporting, be sure to repeat the following steps for each billing project.
-
3.1. Go to the Billing page by first clicking your name and selecting Billing from the main navigation menu (top left of any page in Terra).
-
3.2. Select the Terra Billing project associated with the workspace where you're running your workflow analysis.
Note that you need to be an owner to follow these steps. You'll know you're the owner if you see the Terra billing project listed under Owned by You in the top left column.
-
3.3. Click the pencil icon beside Workflow Spend Report Configuration to edit.
-
3.4. Fill in the Dataset Project ID and Dataset Name from Google Cloud console (step 1.11 above) in the Configure Spend Reporting form.
Blank form
Filled-in form (example 1.11 above)
Example Google Project ID.BigQuery dataset name (separated by a period)
careful-griffin-360513.daily_cost_detail from Google Cloud console
-
3.5. Click the OK button to save.
What to expect
You will not get a confirmation message, but as long as you don't get an error message, your configuration should be saved.
If you get an error that looks like this
Error updating workflow spend report configuration The billing export table <name here> in dataset <dataset name> could not be found
This error occurs when there is no data in the dataset.
To remedy this, try the following 1. Run a small workflow in the workspace.
2. Wait 2-3 hours and follow steps 3.1 - 3.5 again.