(09-01-2021) This article has been edited with changes for the upcoming Project per Workspace functionality. You can update with additional changes, but please do not publish! Once the PPW feature is out, all the articles will be released at one time!
Slack Allie Cliffe (@acliffe) with any questions.
Understanding how to manage and control costs is an important consideration for a team with a shared funding source. Read on for two ways to share funding resources and which one to use - depending on your collaboration needs.
For information on best practices for sharing data resources, see this article.
To learn about sharing GCP Billing accounts (if colleagues need to be able to create their own Terra Billing projects, or look at detailed cost breakdown and reports), see this article.
Overview: How costs are calculated in Terra
Working in Terra incurs GCP fees that are passed along to users without any markup via a Terra billing project. GCP costs are based on how much storage, egress and compute is done in the project workspace (bottom level). These workspace costs are covered via a Terra billing project (gray box, middle level), funded by a GCP Billing Account (top level). Terra billing projects can include many different workspaces, such as one for each member of a team with shared funding.
Workspaces created prior to September 24, 2021
The Terra billing project maps 1:1 to a Google project. Since all GCP costs are calculated per Google project, costs are summed across all the workspaces in the Terra Billing project.
Workspaces created after September 24, 2021
Terra creates a GCP project for each workspace, so GCP costs are calculated per workspace, and summed across all workspaces associated with the Terra billing. The workspace Google project is mostly invisible to the Terra user, unless you are interfacing directly with GCP console (when setting up workflow spend reporting or Google budget alerts, for example).
Note: All Terra components in the diagram are gray and all GCP components are blue.
Consider the case of several collaborators with the same funding source. The funding will be dispersed through a GCP Billing account (top level above). Collaborators can access the shared funds at any level of the billing hierarchy, depending on whether they will be using their own or a shared workspace. For a detailed description of the structure of costs and permissions in Terra, see this article.
Below are two options for managing shared team costs.
Working in shared team workspaces (PIs have most control)
Workspace owners (i.e. lab PIs) can share a workspace with an individual or a managed group. The workspace owner controls what each collaborator can do in the workspace. Collaborators can only accrue costs if they have the right workspace permission (to add data, for example, or compute an analysis).
TIP: Note that workspace owners can change collaborator roles at any time.
How to set up a shared team workspace
1. Create the workspace.
2. Make a managed group for the team by going to "Profile" -> "Groups" from the main navigation.
3. Return to the team workspace and click the three vertical dots.
4. Open the "Share workspace" form by selecting "Share" in the dropdown.
5. Add the group (start typing in the group name in the "User email" field).
6. Select "enter" or "return" after entering the group name.
7. Select the group's permissions.
Don't forget to save!
|Note that everyone in the group shares the same workspace permissions.
To assign more granular permissions, add individual collaborators one at a time.
To learn more about managing shared resources with groups, click here.
Controlling spending in a shared workspace
The workspace owner controls what collaborators can do by assigning workspace roles/permissions when sharing a workspace. Workspace permissions give the most control over spend, as owners can change or remove an individual or group's permission at any time, and it takes effect immediately.
Operations that have a cost
- Move, copy and store data
- Run an interactive analysis (Jupyter notebook)
- Run a workflow (note that this will generate data, including intermediate files, that will be stored by default in the workspace bucket, which also has a cost)
|Note that it is not possible to get cost breakdowns by user for different users working in the same workspace|
How to add/remove individuals from a shared workspace
2. If adding someone, you need to hit "enter" (or "return") after entering their user email.
3. Choose their role, or delete the person from the collaborator list.
4. Don't forget to save after changing!
How to add/remove a user from a workspace shared with a Group
1. Go to "Your Name" > "Groups" from the main navigation menu
2. Edit the person's role in the group (you must have the right permissions, of course!)
Working in separate workspaces with team billing
Owners and administrators of the GCP Billing account can create one or more Terra billing projects for each GCP billing account - one for the entire team or separate one for different collaborators or different work in Terra. One advantage of this is it is easy to cut off spend by Billing project.
A warning about controlling costs
A Terra billing project user can create their own workspaces. The workspace creator is the "owner" by default, and will be able to store and analyze data in any workspaces they create. The GCP costs will be billed to the shared Terra Billing project.
TIP: Being on a team Billing project does not allow the collaborator to work in any workspaces created by others in the team - even under the shared billing project! That requires the workspace owner to grant the appropriate workspace permission to the user (option 1, above).
|If your Terra Billing project will be used by a group (such as members in a research lab, or scientists with a common funding source), make sure to have more than one individual with "owner" permission. That way you will not be locked out of the Billing project if the sole owner leaves the group.
Add a second owner in the UI
Learn more about creating and using a managed group here.
Step-by-step instructions to create a Terra Billing project
2. Click on "Create a New Project."
3. If prompted, click to enable billing permissions, select the Google ID of the Google Billing account owner, and click "Allow".
4. Enter a unique name for your Terra Billing Project. Names must follow these rules:
- Must be between 6 and 30 characters in length
- May only contain alphanumeric characters, underscores, and dashes
4. Select a GCP Billing account to associate with the Terra billing project.
Note: You may see multiple GCP Billing accounts. If you need to locate a GCP Billing account ID, navigate to the Google Developers Console and click on Billing. Look for the number below "Billing account ID".
Add a second billing project owner (click for step-by-step instructions)
You'll find it under the main navigation (top left of any page) in the drop down under your name:
2. Click on the billing project.
3. Under the "Users" tab, select the blue "Add Users" button.
4. Update their role to owner by selecting the "Edit role" icon
How to add/remove collaborators on a Terra Billing project
1. Navigate to "Billing" from the main navigation menu (Main menu > user name > "Biling").
2. Select the Billing Project from the list at the left.
3. To add a collaborator, click on the "+" icon on the "Add a User" card.
4. To remove or edit the role of a collaborator, click the link at the right of their user ID.
|Removing a colleague from billing
Remember that removing someone from a billing project does not keep them from spending money in an existing workspace that has been shared with them! It removes their ability to create new workspaces.
We repeat - Users can still run workflows and notebooks in a workspace even after you remove them from the billing project!!
After removing billing project permissions, to avoid accruing additional charges, you must eliminate all possible sources of continuing costs (at the workspace and billing project level), following the checklist below.
Disabling billing (at the project level)
If you are no longer using a shared Terra Billing project, you can disable billing for any projects (and all the associated workspaces) using the Swagger API. This will remove the GCP Billing account immediately from a Terra Billing project (i.e. you will not be able to clone a workspace). This will also disable billing on all workspaces created prior to September 24, 2021 immediately.
Workspaces created after September 24, 2021: Terra will then asynchronously remove the billing account from all the Google projects of all workspaces created under the Terra billing project.
Scope: This will prevent additional costs accruing in all workspaces under this Terra billing project. Once you disable billing, no one will be able to start a workflow or notebook in any workspace associated with this project (if you disable the Terra billing project) - effective immediately.
See step-by-step directions for disabling billing on a Terra Billing project using Swagger here.
In order to disable billing, you need to have owner or admin permission on the GCP Billing account. If you are using STRIDES or third party resellers such as Onix, you may not be able to follow the steps unless you request admin privileges from the third-party reseller (who is the Owner).
What should I do next?
GCP costs are billed by billing project (i.e. by Terra workspace), not by user.
This means, if the workspace still has a valid billing project, any user with "can-compute" permissions will still be able to run workflows or notebooks in workspaces.
What can someone do without billing permissions?