Executing jobs on the cloud can be a little scary if you don't know how much you're spending. For peace of mind that you're not going over a project budget, read on for instructions to set up cost reporting for workflows.
Overview: Workflow cost reporting
To help make cloud computing costs more transparent, you can have Terra display the cost for each executed workflow (including failed and aborted workflows) in the Job History page. In order to access spend reporting, you will configure spend reporting for your Terra Billing Project. Most users will also need to do a one-time setup on GCP, to allow Terra access to GCP's cost reports.
Built-in cost reports take a few hours to compile
Results from Terra's built-in cost report aren't available until some hours after the workflow has completed. It's best to allow a day to pass before checking in on the cost.
Costs are only reported for completed workflows
The cost is a true cost, not an estimate. It is only available after the workflow is complete.
Reports cover the total job cost
It is not possible with built-in cost reporting to access task-level or instance-level costs (i.e. the cost of running a particular task within a workflow).
How accurate is the cost report? Terra's built-in cost reports come directly from Google. This is the actual cost for the given workflow (i.e. not an estimate). They are generated by accessing Google's Billing Repository, and can take several hours to log.
For more granular costs or cost for workflows in progress, you could try running a cost-estimate notebook. See Cost-estimate notebook (deprecated). Note that this archived notebook is broken and no longer supported, however it could provide guidance for an alternate approach.
Step 1. Set up access (once per Cloud Billing account)
If you are not using a Broad Institute Google Cloud Billing account, you will need to grant Terra access to cost reports on GCP console to enable the workflows spend reporting.
Not able to follow these directions? You must have "Owner" permission of the Terra Billing Project and “Owner” or "Admin" permission on the Google Cloud Billing account. If you are not able to follow the directions below, or do not see the options in the screenshots (or they are greyed out), it is likely because you do not have sufficient permissions on the GCP Billing account. You will need to ask the owner for admin privileges on the Cloud Billing account or to set up a BigQuery dataset for billing export (follow directions below).
This may be the case, for example, if you are using a third-party resellers such as Onix.
-
1.1. Navigate to the billing account management page in Google Cloud Console.
1.2. Click on the name of the billing account you would like to enable billing exports for. If you have more than one, you will need to repeat for each one associated with a Terra Billing Project.
1.3. Click on Billing export in the left sidebar.
1.4. Under Standard usage cost, click the Edit Settings button.
1.5. Next you will configure/create the BigQuery dataset to export the billing data to.
Note that you must have an active Google project - not a Terra Billing project - tied to this billing account for this step.
The Billing project you select cannot be a Terra-generated workspace project To check, make sure the ID in the dropdown is not of the format <Terra-Billing-project-name>--<workspace-name>. If the project name has that formatting, it was created by Terra and will not work for this step.
In that case, you will need to create a GCP-native project first. You can find step-by-step instructions here.If you don't have a project, Google will prompt you to create one.
1.6. Create the project.
(if you already have an existing Google project, you may skip this step).
1.7. Select the Google project. This will be used to host the billing export BigQuery dataset (cannot be a workspace project created by Terra - see 1.5 above).
1.8. From the dropdown menu, select (or create) the BigQuery dataset to store the billing export data.
If you don't already have a dataset in this Google project, Google will prompt you to create one: From this menu, click on Create new dataset as shown above (1.8), fill in this form and select the Create dataset button at the bottom.
1.9. Click Save.
1.10. To view the BigQuery dataset, click on the link (name of your BigQuery dataset) in the BigQuery export tab - to the right of Dataset name).
1.11. From the dataset tab, copy the Google project ID (to the left of the period) and BigQuery dataset name (to the right of the period) to a safe place. You will need it in the next step (2.4) in Terra.
The format is Google project ID : BigQuery dataset name (separated by a colon)
1.12. Hover over the person icon to the right of the project and dataset name and click Share dataset.
If this option is greyed out, it is most likely because you don't have the right permission. You will need to ask the owner of the Google Cloud Billing account to grant you permission to share a BigQuery dataset.1.13. Type
spend-reporting@terra.biointo the Add principals field and select BigQuery Data Viewer from the Select a role dropdown. This grants Terra permission to access the dataset you've just set up.1.14. Click Add (to the right of the dropdown), then Done (at the bottom of the form).
Google BigQuery billing exports are now configured for this Cloud Billing account. You can confirm by expanding the menu under BigQuery Data Viewer (below).
The next step will be to configure workflow spend report.
-
1.1. Navigate to the billing account management page in Google Cloud Console
1.2. Click on the name of the STRIDES billing account you would like to enable billing exports for. It will have the form
NIH.NHLBI.BDC.Cohort#.Fellow.00#.1.3. Click on "Billing export" in the left sidebar.
1.4. Under Standard usage cost, click the "Edit Settings" button.
1.6. From the dropdown menu under Dataset ID, select the Billing dataset. This is the pre-configured BigQuery dataset to store the billing export data.1.7. Click Save.
1.8. To view the BigQuery dataset, click on the Billing link in the BigQuery export tab - to the right of Dataset name under Standard usage cost.
1.9. Copy the Google project ID to a safe place. You will need it in the next step (2.4) in Terra.
1.10. Hover over the person icon to the right of the project and dataset name and click Share dataset.
1.11. Type
spend-reporting@terra.biointo the Add principals field and select BigQuery Data Viewer from the Select a role dropdown. This grants Terra permission to access the dataset you've just set up.1.12. Click Add (to the right of the dropdown), then Done (at the bottom of the form).
Google BigQuery billing exports are now configured for the STRIDES Cloud billing account. You can confirm by expanding the menu under BigQuery Data Viewer (below).
-
If you use a third-party reseller, you will need to ask them to set up the spend reporting (in GCP console) for you following the steps below.
1. Create a BigQuery dataset for exporting billing data on GCP.
2. Share the dataset with
spend-reporting@terra.bioas BigQuery Dataset Viewer.3. Tell you the name and project of the dataset (to use in Step 2 below).
-
Broad GCP Billing account users can skip this first step and go directly to Step 2: Configure the workflow spend report.
Before you move on to step 2If you just completed Part 1 we recommend you wait several hours to complete Part 2 in order for billable activity to be recorded in BigQuery or you may receive an error that the dataset cannot be found.
Step 2. Set up workflow spend reporting in Terra
Now that you have set up the GCP billing data export, you will set up spend reporting in Terra. You will only need to do this once per Terra Billing project.
Do you have multiple billing projects?If you have workspaces in multiple billing projects that you want to set up with spend reporting, be sure to repeat the following steps for each billing project.
2.1. Go to the Billing page by first clicking your name and selecting Billing from the main navigation menu (top left of any page in Terra).
2.2. Select the Terra Billing project associated with the workspace where you're running your workflow analysis.
Note that you need to be an owner to follow these steps. You'll know you're the owner if you see the Terra billing project listed under Owned by You in the top left column. 
2.3. Click the pencil icon beside Workflow Spend Report Configuration to edit. 
2.4. Fill in the Dataset Project Name (Project ID) and Dataset Name from GCP console (step 1.11 for general users or step 1.9 for STRIDES - above).
2.5. Click the OK button to save.
You will not get a confirmation message,but as long as you don't get an error message, your configuration should be saved.
If you get an error that looks like this
It is because there is currently no data in the dataset.
To remedy this, try the following
1. Run a small workflow in the workspace.
2. Wait 2-3 hours and follow steps 2.1 - 2.5 again.
How to find built-in workflow cost reporting
1. Navigate to the Job History page of the workspace.
This page includes all workflow submissions for the workspace
2. Click the submission of interest in the far left column.
3. If your spend reporting has been correctly set up, you will find the Total Run Cost at the top right corner. Note that the spend report can take up to 24 hours to appear in Terra, as GCP costs reports have some delay. 
If your submission included more than one execution, each will be listed separately under "Run Cost"
