Learn how to set up a service account to allow Terra to access resources on your behalf such as ingesting data into the Terra Data Repository, using API calls, or automating some aspects of bulk (workflow) analysis.
How Terra uses service accounts to interface with Google Cloud
To maintain the high security required for working with controlled-access data in the cloud, Terra cannot use your user ID when interacting with cloud resources external to Terra. Instead, Terra creates and uses additional Google accounts (specific to you) called pet service accounts. Terra also puts all of these pet service accounts into a convenient Google Group (specific to you) called a Proxy Group.
Pet service account details
A service account is a special type of Google account that lets Terra interface directly with Google Cloud on your behalf. For example, Terra uses a service account - rather than your user ID - to let you run workflows, create workspaces, make Data Repo datasets and snapshots; anything that uses Application Programming Interface calls.
Your pet service account has the format: pet-7293562825402802834639@.iam.gserviceaccount.com
How service accounts differ from your user account (Google ID)
- Service accounts do not have passwords and cannot be logged in via browsers or cookies.
Instead, they are associated with public and private key pairs used for authentication to Google. Service accounts are not associated with your Google account. Assets created by a service account are not created in your Google account's domain (Google Workplace Domain). - A service account is tied to a Google project.
You will need to have separate service accounts for different actions in Terra such as calling APIs or running workflows. After you create a service account, you cannot move it to a different project. By default, each project can have up to 100 service accounts that control access to resources.
When to use a service account in Terra
When building any automation in Terra, it is best practice to use service accounts rather than private credentials as an individual. Read on for step-by-step instructions to register a service account to use in Terra.
If you are looking to access your own private external-to-Terra GCP resources from within Terra (i.e., get data from an external Google bucket, run ML VMs in a notebook, etc.), use a Terra-managed group instead of a service account. To learn more, see Best practices for accessing external GCP resources.
Step 1: Create a service account
1.1. Create or be added to a Google Cloud project.
1.2. Follow the step-by-step instructions in the Google Cloud support doc (external link) to set up your service account.
1.3. Click manage roles to select which roles to assign to this service account.
Choosing the service account roleHopefully you already know which role you need for the task you want the service account to do (often in the instructions for the action). There are hundreds of roles you can grant to a service account so take some time to see which scopes and permissions you may want to assign to the service account (note that roles can be updated later).
Most common services and permissions on Terra
- Google Cloud Storage (viewer, reader, writer, owner)
- Google BigQuery (viewer, reader, writer, owner)
- Google Compute Engine (can-compute)
These are the most common roles and permissions. There is no one size fits all solution. If you think you may require a less-used role, see this doc (Google reference) which outlines all the different types of roles you can assign a service account.
1.4. After you've finished the initial account set up save the email associated with the service account:
service-account-name@PROJECT_ID.iam.gserviceaccount.com
1.5. Before moving to step 2 (registration), go to the "keys" tab to create and download a key for your service account onto your local machine.
Note: Use the recommended .JSON format to download your key
Step 2: Register a service account
You will need the key file and service account email from part 1 (above) to register the service account. Registration allows the service account to call Terra APIs shared with the service account.
2.1. Run the script to register a service account (you can find the script on the Terra FireCloud tools page).
To run the script you will need
- -j <file path on a local machine to the JSON file containing the key created for service account>
- -e <email address for the service account>
2.2. In your local terminal, run "git clone https://github.com/broadinstitute/firecloud-tools" with the following code. Cloning the FireCloud repository will allow you to run FireCloud tools used by Terra.
"git clone https://github.com/broadinstitute/firecloud-tools"
To learn how to install and use a terminal on your local machine, see How to move data between local and workspace storage (Step 1: Install gsutil locally).
2.3. Change the directory to "firecloud-tools" using the below code.
cd firecloud-tools
2.4. Run the following script in the firecloud-tools directory to register your service account.
./run.sh scripts/register_service_account/register_service_account.py -j <path to your service account credentials json file> -e <email address for owner of this service account, it's where notifications will go>
Replace the bolded sections with the file path to the key you downloaded in step 1 and the email associated with the service account.
Note: You will need to run this code in the Firecloud-tools directory on your local machine.
Example registration code
./run.sh scripts/register_service_account/register_service_account.py -j /Users/yduggal/downloads/setting-up-service-account-890a643c6881.json -e creating-a-service-account-903@setting-up-service-account.iam.gserviceaccount.com
The service account is now registered with FireCloud! You are now able to share workspaces with this address or use it to call APIs in Terra.