Executing jobs on the cloud can be a little scary if you don't know how much you're spending. For peace of mind that you're not going over a project budget, read on for an overview of how to see the actual Google cloud costs of running workflows and the storage and compute costs of your top ten workspaces.
For step-by-step instructions, see How to set up spend reporting.
Overview: Cloud cost reporting in Terra
To help make cloud computing costs more transparent, Billing project owners can surface storage and analysis costs in two ways:
- View the costs for the top ten workspaces (in the Billing page)
- View the cost of individual workflow submissions (in the Job History page)
Workspace compute and storage costs (top 10)
Billing project owners can see the total spend, total compute, and total storage costs of the top ten workspaces in the Billing page.
Workflow submission costs
1. To view the cost of an individual workflow submission, navigate to the Job History page of the associated workspace.
This page includes all workflow submissions for the workspace.
2. Click the submission of interest in the far left column.
3. If your spend reporting has been correctly set up, you will find the Total Run Cost at the top right corner. Note that the spend report can take up to 24 hours to appear in Terra.
If your submission included more than one execution, each will be listed separately under Run Cost.
Cost reporting caveats
Because Terra derives cost information from Google's billing export services, there are a few things to keep in mind.
Cost reports take a few hours to show up
Cost information from Google isn't available until some hours after the analysis has completed. It's best to allow a day to pass before checking in on the cost.
Costs are only reported for completed workflows
The cost is a true cost, not an estimate. It is only available after the workflow is complete and is not available for failed workflows.
Reports cover the total workflow cost
It is not possible to access task-level or instance-level costs (i.e. the cost of running a particular task within a workflow) with built-in cost reporting.
How accurate is the cost report? Terra's built-in cost reports come directly from Google. This is the actual cost for the given workflow (i.e. not an estimate). They are generated by accessing Google's billing repository, so they can take several hours to log.
For more granular costs or cost for workflows in progress, you could try running a cost-estimate notebook. See Cost-estimate notebook (deprecated). Note that this archived notebook is broken and no longer supported. However it could provide guidance for an alternate approach.
How to set up cost reporting in Terra
For most people, there are two steps to setting up cost reporting. Once you complete these, the spend reporting will appear automatically.
Step 1: Set up billing export on Google
Most users will need to set up Google Cloud so that Terra can access Google's cost reports. This must be done once per Cloud Billing account. Users with a Broad Google sub-account can skip this step.
Step 2: Configure workflow spend reporting in Terra
Once you have set up billing export, you will need to specify the Project ID and BigQuery dataset name in Terra. You will need to do this once for each Terra Billing project.
For step-by-step instructions, see How to set up spend reporting.