How much did my workflow cost?

Allie Cliffe

Executing workflows on the cloud can be a little scary if you don't know how much you're spending. For peace of mind that you're not going over a project budget, read on for instructions to set up cost reporting for workflows.  

Overview: Workflow cost reporting

To make cloud computing costs more transparent, you can have Terra display the cost for each executed workflow (including failed and aborted workflows) in the Job History page.

To access spend reporting, you'll need to configure spend reporting for your Terra Billing Project. Most users will also need to do a one-time setup on Google Cloud, to allow Terra access to Google's cost reports. A few things to keep in mind are below.

Built-in cost reports take a few hours to show up

Results from Terra's built-in cost report aren't available until some hours after the workflow has completed (this is a lag in reporting on the Google side). It's best to allow a day to pass before checking in on the cost.

Costs are only reported for completed workflows

The cost is a true cost, not an estimate. It is only available after the workflow is complete.

Reports cover the total job cost

It is not possible with built-in cost reporting to access task-level or instance-level costs (i.e. the cost of running a particular task within a workflow). 

How accurate is the cost report? Terra's built-in cost reports come directly from Google. This is the actual cost for the given workflow (i.e. not an estimate). They are generated by accessing Google's billing repository, so they can take several hours to log. 

For more granular costs or cost for workflows in progress, you could try running a cost-estimate notebook. See Cost-estimate notebook (deprecated). Note that this archived notebook is broken and no longer supported, however it could provide guidance for an alternate approach. 

Before you start: Verify your permissions

Not able to follow these directions?You must have "Owner" permission of the Terra Billing Project and “Owner” or "Admin" permission on the Google Cloud Billing account. If you are not able to follow the directions below, or do not see the options in the screenshots (or they are greyed out), it is likely because you do not have sufficient permissions on the GCP Billing account. You will need to ask the owner for admin privileges on the Cloud Billing account or to set up a BigQuery dataset for billing export (follow directions below).

This may be the case, for example, if you are using a third-party resellers such as Onix.

1. Verify you are a Billing project Owner 

  • Go to your billing page (
  • If the billing project is listed in the Owned by you section, you're all set.
    To find the Google Billing account associated with this Billing project: 1. click on the billing project link  2. Look next to Billing Account.

  • If the Billing Project is under Shared with You, you can click on the Members tab to see who you would need to ask to get that permission. 

2. Verify you are a Google Cloud billing account Owner or Admin

Go to billing in Google Cloud console ( and click on the Billing account link. 

If get an error message "You don’t have permission to view costs for this billing account", you will need to ask an owner to expand your role to be able to follow the directions below. 

What to do if you don't have Cloud billing account permission Click on the Billing account Manage link to the right and expand the Billing Account Administrator section to see who to ask.

If you use third-party resellers, contact your reseller directly for help.

Step 1. Set up Cloud account billing export (once)

If you are not using a Broad Institute Google Cloud Billing account, you will need to grant Terra access to cost reports on GCP console to enable the workflows spend reporting following the steps below. 

You will only need to do this once per Google Cloud Billing account. 

  • 1.1. Navigate to the billing account management page in Google Cloud Console.

    1.2. Click on the name of the billing account you would like to enable billing exports for. If you have more than one, you will need to repeat for each one associated with a Terra Billing Project.

    1.3. Click on Billing export in the left sidebar. 

    1.4. Under Standard usage cost, click the Edit Settings button.

    1.5. Next you will configure/create the BigQuery dataset to hold the exported billing data. 

    Requirements to set up the BigQuery billing export dataset The Billing project you select cannot be a Terra-generated workspace project. To check, make sure the ID in the dropdown is not of the format <Terra-Billing-project-name>--<workspace-name>. If the project name has that formatting, it was created by Terra and will not work for this step.

    In that case, you will need to create a GCP-native project first. You can find step-by-step instructions here.

    If you don't have a project, Google will prompt you to create one. 
    Screen shot of Google prompt to create a project for cost reporting

    What to do if you are unable to create a project in Google consoleIf you are unable to follow the instructions (especially if what's on your screen looks different than what's in these instructions), you most likely don't have the right permission.

    What to do
    If you are part of an organization, reach out to your Google Cloud Billing account admin or central organization. See instructions above for how to find the right person to ask. If you use a third-party reseller, reach out to the reseller to ask for help.

    1.6. Create the project (if you already have an existing non-Terra Google project, you may skip this step). We suggest using daily_cost_detail_bq, but you can use any name that makes sense to you. Note that appending _bq to your project name will help you identify it later!  

    1.7. Select the Google project to host the billing export BigQuery dataset (cannot be a workspace project created by Terra - see 1.5 above)

    1.8. From the dropdown menu, select (or create) the BigQuery dataset to store the billing export data.

    If you don't already have a dataset in this Google project, Google will prompt you to create oneFrom the menu above (1.8), click on Create new dataset, fill in the form (screenshot below), and select the Create dataset button at the bottom.
    Screenshot of Create dataset screen on Google Cloud console

    1.9. Click Save.

    1.10. In the BigQuery export tab, click on the dataset name.

    1.11. From the dataset tab, copy the Google project ID (to the left of the period) and BigQuery dataset name (to the right of the period) to a safe place. You will need it to configure spend reporting in Terra (2.4 below).

    Example Dataset ID (careful-griffin-360513.daily_cost_detail)


    The format is Google project ID . BigQuery dataset name (separated by a period)

    1.12. Go to the sharing dropdown above and to the right of the Dataset info and click Permissions

    If this option is greyed out, it is most likely because your role doesn't allow you to share the dataset. You will need to ask the owner of the Google Cloud Billing account to grant you permission to share a BigQuery dataset. 

    1.13. Select the Add principals button in the Dataset Permissions section.

    1.14. Type into the New Principals field (circled in screenshot below) and select BigQuery Data Viewer from the Role dropdown (arrow). This grants Terra permission to access the dataset you've just set up.

    1.15. Click the Save button (at the bottom of the form).

    Google BigQuery billing exports are now configured for this Cloud Billing account. You can confirm by expanding the menu under BigQuery Data Viewer (below).

    The next step will be to configure workflow spend report in Terra.

  • If you use STRIDES

    1.1. Navigate to and select your project ID.

    1.2. Select Menu > BigQuery.

    1.3. Follow the instructions to create a dataset daily_cost_reporting.

    1.4. Share the daily_cost_reporting dataset with as BigQuery Dataset Viewer.

    If you use a third-party reseller

    You may need to ask them to set up the spend reporting (in GCP console) for you.

    1.1. Create a BigQuery dataset daily_cost_reporting for exporting Google billing data.

    1.2. Share the daily_cost_reporting dataset with as BigQuery Dataset Viewer.

    1.3. Give you the Google project ID of the dataset (to use in Step 2 below).

  • Broad GCP Billing account users can skip this first step and go directly to Step 2: Configure the workflow spend report.

Before you move on to step 2If you just completed Step 1 we recommend you wait several hours to complete Step 2 in order for billable activity to be recorded in BigQuery or you may receive an error that the dataset cannot be found.

Step 2. Configure workflow spend reporting in Terra

Now that you have set up billing data export in Google Cloud, you will set up (configure) spend reporting in Terra. You will only need to do this once per Terra Billing project. 

Do you have multiple billing projects?If you have workspaces in multiple billing projects that you want to set up with spend reporting, be sure to repeat the following steps for each billing project.

2.1. Go to the Billing page by first clicking your name and selecting Billing from the main navigation menu (top left of any page in Terra).

2.2. Select the Terra Billing project associated with the workspace where you're running your workflow analysis.

Note that you need to be an owner to follow these steps. You'll know you're the owner if you see the Terra billing project listed under Owned by You in the top left column. 

2.3. Click the pencil icon beside Workflow Spend Report Configuration to edit. 

2.4. Fill in the Dataset Project ID and Dataset Name from GCP console (step 1.11 for general users or step 1.9 for STRIDES - above) in the Configure Spend Reporting form.

Example Dataset ID (careful-griffin-360513.daily_cost_detail) from Google Cloud console

The format is Google project ID . BigQuery dataset name (separated by a period)

2.5. Click the OK button to save.

What to expect

You will not get a confirmation message, but as long as you don't get an error message, your configuration should be saved. 

If you get an error that looks like this


Example error text

Error updating workflow spend report configuration
The billing export table <name here> in dataset <dataset name> could not be found

Full error:

 "causes": [],    "message": "The billing export table gcp_villing_export_v1_012061_174696_8A47BC in dataset terra_daily_cost_details could not be found.",    "source": "rawls",    "stackTrace": [],
 "statusCode": 400,    "timestamp": 1631824489662 }

This error occurs when there is currently no data in the dataset. 

To remedy this, try the following

  1. Run a small workflow in the workspace.

  2. Wait 2-3 hours and follow steps 2.1 - 2.5 again. 

How to find built-in workflow cost reporting

1. Navigate to the Job History page of the workspace.
This page includes all workflow submissions for the workspace

2. Click the submission of interest in the far left column.

3. If your spend reporting has been correctly set up, you will find the Total Run Cost at the top right corner.  Note that the spend report can take up to 24 hours to appear in Terra, as GCP costs reports have some delay. 
 If your submission included more than one execution, each will be listed separately under Run Cost.

For more granular cost details

Step-by-step instructions for querying the billing export dataset directly here

Was this article helpful?

1 out of 3 found this helpful



  • Comment author
    Kamil Slowikowski

    Could I please ask if you might consider sharing a specific example of how to retrieve the data for a specific Terra workspace?

    For example, you included a screenshot of a CSV file. Could you please take screenshots of all of the steps that led to that point?

  • Comment author
    Andrew Davidson

    I happened to stumble across this document today by accident. Any idea why stuff like this is not sent out in the newsletter or 

    Given terra is still in beta knowing a new vignette has been posted or that some part of the documentation has changed is really important.

    The purpose of writing documentation is to reduce support costs. If we do not know about changes we all just loose time and money

  • Comment author
    Andrew Davidson
    • Edited

    How is this related to the 'Workflow Spend Report Configuration:' on

  • Comment author
    Geraldine Van der Auwera

    Hi Andrew Davidson, that's absolutely a fair point. We've had this particular item in our blog/newsletter backlog for a while and finally got to it this week; I'm putting the finishing touches to the blog post right now and it should come out tomorrow if all goes well. Going forward we're going to try to report on these kinds of things more consistently as they come out.

    Note that you can get notified when new documentation articles are posted by clicking the blue "Follow" button near the top of each category page (such as this one). Annoyingly you have to do this for each category of the knowledge base, as opposed to just being able to subscribe to the whole thing in one click, which I believe is a limitation of the ZenDesk software, but at least this way you can get notifications about categories of documentation that you care about. 

    Regarding your second question, the 'Workflow Spend Report Configuration' item on gives you a way to specify which BigQuery dataset to associate with a given billing project from within Terra rather than by going through the Google Cloud console (which is described in the "Grant Terra Billing project access" folding section). This approach is described under the folding section titled "Configure workflow spend report in UX" (which should probably be UI rather than UX, sorry about that typo). 

    I hope this helps, let us know if you have any thoughts on how we could make the docs clearer and more helpful. 

  • Comment author
    Shyamsundar Ravishankar

    Thanks for this article it was very easy to follow and both methods worked great! 

    I wanted to confirm for Option 2 cost estimation, is the currency USD or does it automatically change to the local currency the workspace / billing account is in? Our Workspaces and compute instances are in the Australian region. 

    I would think Option 1's built-in reporting would be AUD since it is getting it directly from the billing account. 





  • Comment author
    Andrew Davidson
    • Edited

    I had trouble following the 1-time general instructions. I do not know anything about gcp administration. 

    I had trouble creating a new project. There are instructions on 

    I also had trouble with step 1.12. The shares icon was not present. When I tried again the next day it magically appeared. 

    Once I got past the one-time set the setting up the report was easy




  • Comment author
    Brendan Reardon

    We've literally been asking for this for years and I am so happy to see it finally in production. I enabled this last night and it is working smoothly. Thank you! 

  • Comment author
    Ash O'Farrell
    • Edited

    Did permissions change relatively recently? Previously, I was able to view costs after workflows run on billing projects on which I'm not the owner. Now I see N/A for everything (after waiting for the usual few-hour delay for costs to show up).

  • Comment author
    Mark Godek

    Step 2.4 was a little confusing.

    "The format is Google project ID . BigQuery dataset name"

    On the Google page, it says the entire thing is the ID but the Terra configuration prompt asks for them to be separate.

    I was putting:

    Dataset Project ID: GoogleProjectID.BigQueryDatasetName

    Dataset name: BigQueryDatasetName

    As Google labels the whole string as the dataset ID:

    Dataset Project ID: GoogleProjectID

    Dataset name: BigQueryDatasetName

    It would be useful if the article could be updated to exactly reflect the terminology of the Google page.

    Google project ID  instead of "Dataset Project ID". BigQuery dataset name instead of "Dataset Name"


Please sign in to leave a comment.