How to use a service account in Terra

Often it's useful to connect data or Cloud resources based in Terra with data and resources outside of Terra.  Learn how to connect these services using service accounts, and manage your service accounts using Terra Groups.

Overview: When should you use a service account?

Terra uses service accounts to access resources on your behalf, in a secure way. This is useful in three main situations:

1. You're running a Terra-supported application (such as a workflow or the Terra Data Repository) that needs to access data stored in a Terra workspace.

2. You're running a Terra-supported application that needs to access data stored outside of Terra.

3. You're running a third-party application (such as GitHub actions) that needs to access data stored in a Terra workspace.

Learn more about these situations by reading Best practices for using service accounts in Terra.

In all of these cases, Terra cannot use your user ID to access your data, for security reasons. Instead, Terra creates additional Google accounts - called "pet service accounts" - to use as a proxy for your Terra account. Your pet service account has the format:    
pet-7293562825402802834639@.iam.gserviceaccount.com.

To make it easier to manage these proxies, Terra stores them in a "Proxy Group," which is a Google group that only stores your service accounts. In many cases, Terra creates and manages these service accounts for you automatically; however, in some cases you need to create and manage your own service account. Follow the steps below to do so.

Step 1. Create a service account

1.1.  Create or ask to be added to a Google Cloud project.

1.2. Follow the step-by-step instructions in the Google Cloud support doc (external link) to set up your service account. 

1.3. Click manage roles to select which roles to assign to this service account.

Choosing the service account roleHopefully you already know which role you need for the task you want the service account to do (often in the instructions for the action). There are hundreds of roles you can grant to a service account so take some time to see which scopes and permissions you may want to assign to the service account (note that roles can be updated later). 

Most common services and permissions on Terra
 - Google Cloud Storage (viewer, reader, writer, owner)
 - Google BigQuery (viewer, reader, writer, owner)
 - Google Compute Engine (can-compute)

These are the most common roles and permissions. There is no one size fits all solution. If you think you may require a less-used role, see this doc (Google reference) which outlines all the different types of roles you can assign a service account.

1.4. After you've finished the initial account set up, save the email associated with the service account:

service-account-name@PROJECT_ID.iam.gserviceaccount.com

1.5. Before moving to step 2 (registration), go to the "keys" tab to create and download a key for your service account onto your local machine. 

Note: Use the recommended JSON format to download your key

Step 2. Register your service account

You will need the key file from part 1 (above) to register the service account. Registration allows the service account to call Terra APIs shared with the service account.

2.1. Set up your computer:

• Install a terminal program. To learn how to install and use a terminal on your local machine, see How to move data between local and workspace storage (Step 1: Install gsutil locally).
• Install git. You can check whether git is already installed by opening your local terminal and typing git version. 
• Download Python. To check whether Python is already installed, open your local terminal and type python -V or python3 -V.

2.2. Gather some information to register your service account: 

• The path on your computer to the JSON file that you downloaded in step 1.5. For example, if you downloaded the file to your 'downloads' folder on a Windows computer, your path might be C:\Users\yourname\Downloads\jsonfilename.json 
• The email address for the owner of the service account.

2.3. In your local terminal, run the following command to clone the Terra tools repository.

git clone https://github.com/broadinstitute/terra-tools.git

2.4.  Change the directory to "terra-tools" using the code below.

cd terra-tools

2.5. Run the following script in the terra-tools directory to register your service account.  

python scripts/register_service_account/register_service_account.py -j PATH TO YOUR SERVICE ACCOUNT CREDENTIALS JSON FILE -e EMAIL ADDRESS FOR THE SERVICE ACCOUNT OWNER

Replace the bolded and capitalized sections with the file path to the key you downloaded in step 1 and the email associated with the service account. 

Note: You will need to run this code in the terra-tools directory on your local machine. 

Example registration code

python scripts/register_service_account/register_service_account.py -j C:\Users\yourname\serviceaccountcredentials.json -e accountowner@email.com

Once the service account is successfully registered, you should see a message in the terminal saying The service account foo-example-sa@terra-foo-foo-dev.iam.gserviceaccount.com is now registered with FireCloud. You can share workspaces with this address, or use it to call APIs.

Troubleshooting service account registration

Expand the sections below to learn how to troubleshoot common service account registration errors.

• If running the code in step 2.5 produces an error like "Python was not found," it's possible that Python is not installed on your computer or that it is installed under a different name.
  1. If you have not installed Python, download the latest version of Python and follow the installation instructions.
  2. If you have installed Python, try the command from step 2.5 again -- this time replace python with python3 or py, which are a common alternatives.
• If running the code in step 2.5 produces an error that a specific module or library does not exist, that indicates that you need to separately install that Python package. You can do this using pip. For example, to install the oauth2client module, run the command pip install oauth2client in your terminal.

Step 3. Manage your service account with a Terra group

Since Google service accounts can have long, complex names that are difficult to remember, consider using a Terra group to manage the service account's permissions.

Terra groups are not necessary in all casesIf you're using a service account to give a third-party application access to data stored on Terra, you generally do not need to add the service account to a Terra group.

1. Log into Terra, and go to your Groups page ("Main menu" --> "Groups" from the top left of any page in Terra).

2. In the Create a New Group card, click on the blue + icon.

3. Enter your human-friendly user-ID (can be the same as your Terra login) and click the Create Group button.

Naming your Terra group

Your Terra group should have a name that is intuitive and understandable. A good practice is to base it off of your Terra account name. For example, if your registered Terra account is j_doe@someplace.org, create a Terra Group named j_doe_at_someplace_org. 

What happens

Terra creates a mirrored Google group that includes your Proxy (which already includes your user ID). When you (or the owner/admin) grant access on a Google Cloud resource to the <terra-group>@firecloud.org group, both your end-user credentials and your pet service accounts have access to the resource.  

Enabling access

To allow access to external resources, owners can now grant permission to your Terra group (i.e.  j_doe_at_someplace_org@firecloud.org) in GCP console.  

Additional resources

• You can share resources with groups of Terra users by first setting up a group for each person's Terra proxy (following the steps above).
  
  Then create a managed group (i.e.your-lab-group) that includes all these personal groups.
  
  To give permission to everyone in the group, you would grant access to your-lab-group@firecloud.org. Since you can add or remove group members right in Terra, it's easy to adjust who has access.

• Follow these instructions in case you ever need to access your non-intuitively-named proxy group on Terra:

  1. On the upper left-hand side, click on the main menu (three lines at the top of any page on Terra).

  2. Next to your name, click the drop-down arrow. 

  3. Click to expand the profile section under your name. 

  

  4. You'll see your proxy group listed near the bottom left.

  

• Want to learn how to access advanced Google Cloud features not (yet) available in Terra?

  • WRITE to BigQuery
  • Interact with Cloud Storage buckets other than the workspace bucket
  • Run dsub jobs
  • Run Cloud Dataflow jobs
  • Run Cloud ML engine jobs

  See Accessing Google Cloud features that are not in the Terra UI.

• See Best practices for accessing external resources (Google buckets, Google Cloud VMs, etc.).