Displaying Google Cloud costs in Terra

Allie Cliffe
  • Updated

Executing jobs on the cloud can be a little scary if you don't know how much you're spending. For peace of mind that you're not going over a project budget, read on for an overview of how to see the actual Google cloud costs of running workflows and the storage and compute costs of your top ten workspaces. 

For step-by-step instructions, see How to set up spend reporting

Overview: Cloud cost reporting in Terra

To help make cloud computing costs more transparent, Billing project owners can surface storage and analysis costs in two ways:

  1. View the costs for the top ten workspaces (in the Billing page)
  2. View the cost of individual workflow submissions (in the Job History page)

Workspace compute and storage costs (top 10)

Billing project owners can see the total spend, total compute, and total storage costs of the top ten workspaces in the Billing page
Screenshot of spend report in billing page for example AnVIL billing project showing total, compute, and storage costs for top ten workspaces. Total spend is $17.20; total compute is $13.64; total storage is $2.16

Workflow submission costs

1. To view the cost of an individual workflow submission, navigate to the Job History page of the associated workspace.
Screenshot of the tabs available for an individual workspace. The Job History tab is highlighted with an orange rectangle.
This page includes all workflow submissions for the workspace.

2. Click the submission of interest in the far left column.
Screenshot showing the contents of an example workspace's Job History tab. Each row of the Job History table refers to an individual workflow. An orange rectangle and an orange arrow highlight the left-hand column of the first row of this table. Clicking on this cell will provide more information about the associated submission, including its cost.

3. If your spend reporting has been correctly set up, you will find the Total Run Cost at the top right corner.  Note that the spend report can take up to 24 hours to appear in Terra. 
Screenshot showing detailed information about an individual workflow submission. An orange rectangle highlights the section that displays the cost to run this submission.
 If your submission included more than one execution, each will be listed separately under Run Cost.

Cost reporting caveats

Because Terra derives cost information from Google's billing export services, there are a few things to keep in mind.

Cost reports take a few hours to show up

Cost information from Google isn't available until some hours after the analysis has completed. It's best to allow a day to pass before checking in on the cost.

Costs are only reported for completed workflows

The cost is a true cost, not an estimate. It is only available after the workflow is complete and is not available for failed workflows. 

Reports cover the total workflow cost

It is not possible to access task-level or instance-level costs (i.e. the cost of running a particular task within a workflow) with built-in cost reporting. 

How accurate is the cost report? Terra's built-in cost reports come directly from Google. This is the actual cost for the given workflow (i.e. not an estimate). They are generated by accessing Google's billing repository, so they can take several hours to log. 

For more granular costs or cost for workflows in progress, you could try running a cost-estimate notebook. See Cost-estimate notebook (deprecated). Note that this archived notebook is broken and no longer supported. However it could provide guidance for an alternate approach. 

How to set up cost reporting in Terra

For most people, there are two steps to setting up cost reporting. Once you complete these, the spend reporting will appear automatically. 

Step 1: Set up billing export on Google

Most users will need to set up Google Cloud so that Terra can access Google's cost reports. This must be done once per Cloud Billing account. Users with a Broad Google sub-account can skip this step. 

Step 2: Configure workflow spend reporting in Terra

Once you have set up billing export, you will need to specify the Project ID and BigQuery dataset name in Terra. You will need to do this once for each Terra Billing project. 

For step-by-step instructions, see How to set up spend reporting

Was this article helpful?

0 out of 0 found this helpful

Comments

0 comments

Please sign in to leave a comment.