Often it's useful to connect data or Cloud resources based in Terra with data and resources outside of Terra. Learn how to connect these services using service accounts, and manage your service accounts using Terra Groups.
Overview: When should you use a service account?
Terra uses service accounts to access resources on your behalf, in a secure way. This is useful in three main situations:
1. You're running a Terra-supported application (such as a workflow or the Terra Data Repository) that needs to access data stored in a Terra workspace.
2. You're running a Terra-supported application that needs to access data stored outside of Terra.
3. You're running a third-party application (such as GitHub actions) that needs to access data stored in a Terra workspace.
Learn more about these situations by reading Best practices for using service accounts in Terra.
In all of these cases, Terra cannot use your user ID to access your data, for security reasons. Instead, Terra creates additional Google accounts - called "pet service accounts" - to use as a proxy for your Terra account. Your pet service account has the format:
To make it easier to manage these proxies, Terra stores them in a "Proxy Group," which is a Google group that only stores your service accounts. In many cases, Terra creates and manages these service accounts for you automatically; however, in some cases you need to create and manage your own service account. Follow the steps below to do so.
Step 1. Create a service account
1.1. Create or be ask to be added to a Google Cloud project.
1.2. Follow the step-by-step instructions in the Google Cloud support doc (external link) to set up your service account.
1.3. Click manage roles to select which roles to assign to this service account.
Choosing the service account roleHopefully you already know which role you need for the task you want the service account to do (often in the instructions for the action). There are hundreds of roles you can grant to a service account so take some time to see which scopes and permissions you may want to assign to the service account (note that roles can be updated later).
Most common services and permissions on Terra
- Google Cloud Storage (viewer, reader, writer, owner)
- Google BigQuery (viewer, reader, writer, owner)
- Google Compute Engine (can-compute)
These are the most common roles and permissions. There is no one size fits all solution. If you think you may require a less-used role, see this doc (Google reference) which outlines all the different types of roles you can assign a service account.
1.4. After you've finished the initial account set up, save the email associated with the service account:
1.5. Before moving to step 2 (registration), go to the "keys" tab to create and download a key for your service account onto your local machine.
Note: Use the recommended JSON format to download your key
Step 2. Register your service account
You will need the key file and service account email from part 1 (above) to register the service account. Registration allows the service account to call Terra APIs shared with the service account.
2.1. Run the script to register a service account (you can find the script on the Terra FireCloud tools page).
To run the script you will need
- -j <file path on a local machine to the JSON file containing the key created for service account>
- -e <email address for the service account>
2.2. In your local terminal, run "git clone https://github.com/broadinstitute/firecloud-tools" with the following code. Cloning the FireCloud repository will allow you to run FireCloud tools used by Terra.
"git clone https://github.com/broadinstitute/firecloud-tools"
To learn how to install and use a terminal on your local machine, see How to move data between local and workspace storage (Step 1: Install gsutil locally).
2.3. Change the directory to "firecloud-tools" using the below code.
2.4. Run the following script in the firecloud-tools directory to register your service account.
./run.sh scripts/register_service_account/register_service_account.py -j <path to your service account credentials json file> -e <email address for owner of this service account, it's where notifications will go>
Replace the bolded sections with the file path to the key you downloaded in step 1 and the email associated with the service account.
Note: You will need to run this code in the Firecloud-tools directory on your local machine.
Example registration code
./run.sh scripts/register_service_account/register_service_account.py -j /Users/yduggal/downloads/setting-up-service-account-890a643c6881.json -e firstname.lastname@example.org
The service account is now registered with FireCloud! You are now able to share workspaces with this address or use it to call APIs in Terra.
Step 3. Manage your service account with a Terra group
Terra will add your new service account to a Proxy Group. But, the pre-defined Proxy Group identifier is not very human-friendly. If you're try to manage the users whose service accounts have access to a resource, seeing a Proxy Group named something like
PROXY_11564882405514439@firecloud.org is not very helpful.
Therefore, we recommend creating a Terra Group (with a sensible name) as a proxy for your Proxy Group. Terra Groups include the proxy service account by default and are much easier to manage!
Terra groups are not necessary in all casesIf you're using a service account to give a third-party application access to data stored on Terra, you generally do not need to add the service account to a Terra group.
1. Log into Terra, and go to your Groups page ("Main menu" --> "Groups" from the top left of any page in Terra).
2. In the Create a New Group card, click on the blue + icon.
3. Enter your human-friendly user-ID (can be the same as your Terra login) and click the Create Group button.
Naming your Terra group
Your Terra group should have a name that is intuitive and understandable. A good practice is to base it off of your Terra account name. For example, if your registered Terra account is
email@example.com, create a Terra Group named
Terra creates a mirrored Google group that includes your Proxy (which already includes your user ID). When you (or the owner/admin) grant access on a Google Cloud resource to the <terra-group>@firecloud.org group, both your end-user credentials and your pet service accounts have access to the resource.
To allow access to external resources, owners can now grant permission to your Terra group (i.e. firstname.lastname@example.org) in GCP console.
You can share resources with groups of Terra users by first setting up a group for each person's Terra proxy (following the steps above).
Then create a managed group (i.e.
your-lab-group) that includes all these personal groups.
To give permission to everyone in the group, you would grant access to
email@example.com. Since you can add or remove group members right in Terra, it's easy to adjust who has access.
Follow these instructions in case you ever need to access your non-intuitively-named proxy group on Terra:
1. On the upper left-hand side, click on the main menu (three lines at the top of any page on Terra).
2. Next to your name, click the drop-down arrow.
3. Click to expand the profile section under your name.
4. You'll see your proxy group listed near the bottom left.
Want to learn how to access advanced Google Cloud features not (yet) available in Terra?
- WRITE to BigQuery
- Interact with Cloud Storage buckets other than the workspace bucket
- Run dsub jobs
- Run Cloud Dataflow jobs
- Run Cloud ML engine jobs