Executing jobs on the cloud can be a little scary if you don't know how much you're spending. For peace of mind that you're not going over a project budget, read on for an over view of how to see the actual Google cloud costs of running workflows and storage and compute costs of your top ten workspaces.
For step-by-step instructions, see How to set up spend reporting.
Overview: Cloud cost reporting in Terra
To help make cloud computing costs more transparent, Terra has two ways Billing project owners can surface storage and analysis costs in Terra:
- Cost of the top ten workspaces (in the Billing page)
- Cost of individual workflow submissions (in the Job History page)
Workspace compute and storage costs (top 10)
Billing project owners can see the total spend, total compute, and total storage costs of the top ten workspaces in the Billing page.
Workflow submission costs
1. Navigate to the Job History page of the workspace.
This page includes all workflow submissions for the workspace.
2. Click the submission of interest in the far left column.
3. If your spend reporting has been correctly set up, you will find the Total Run Cost at the top right corner. Note that the spend report can take up to 24 hours to appear in Terra, as Google Cloud costs reports have some delay.
If your submission included more than one execution, each will be listed separately under Run Cost.
Cost reporting caveats
Because Terra derives cost information from Google's billing export services, there are a few things to keep in mind.
Cost reports take a few hours to show up
Cost information from Google isn't available until some hours after the analysis has completed. It's best to allow a day to pass before checking in on the cost.
Costs are only reported for completed workflows
The cost is a true cost, not an estimate. It is only available after the workflow is complete and is not available for failed workflows.
Reports cover the total workflow cost
It is not possible to access task-level or instance-level costs (i.e. the cost of running a particular task within a workflow) with built-in cost reporting.
How accurate is the cost report? Terra's built-in cost reports come directly from Google. This is the actual cost for the given workflow (i.e. not an estimate). They are generated by accessing Google's billing repository, so they can take several hours to log.
For more granular costs or cost for workflows in progress, you could try running a cost-estimate notebook. See Cost-estimate notebook (deprecated). Note that this archived notebook is broken and no longer supported. However it could provide guidance for an alternate approach.
How to set up cost reporting in Terra
For most people, there are two steps to setting up cost reporting. Once you complete these, the spend reporting will appear automatically.
Step 1: Set up billing export on Google (once per Cloud billing account)
Most users will need to do a one-time (per Cloud Billing account) setup on Google Cloud, to allow Terra access to Google's cost reports. Users with a Broad Google sub-account can skip this step.
Step 2: Configure workflow spend reporting in Terra
Once you have set up billing export, you will need to specify the Project ID and BigQuery dataset name in Terra. You will need to do this once for each Terra Billing project.
For step-by-step instructions, see How to set up spend reporting.