How to use a service account in Terra

Yashasvika Duggal
  • Updated

Often it's useful to connect data or Cloud resources based in Terra with data and resources outside of Terra.  Learn how to connect these services using service accounts, and manage your service accounts using Terra Groups.

Overview: When should you use a service account?

Terra uses service accounts to access resources on your behalf, in a secure way. This is useful in three main situations:

1. You're running a Terra-supported application (such as a workflow or the Terra Data Repository) that needs to access data stored in a Terra workspace.

2. You're running a Terra-supported application that needs to access data stored outside of Terra.

3. You're running a third-party application (such as GitHub actions) that needs to access data stored in a Terra workspace.

Learn more about these situations by reading Best practices for using service accounts in Terra.

In all of these cases, Terra cannot use your user ID to access your data, for security reasons. Instead, Terra creates additional Google accounts - called "pet service accounts" - to use as a proxy for your Terra account. Your pet service account has the format:    
pet-7293562825402802834639@.iam.gserviceaccount.com.

To make it easier to manage these proxies, Terra stores them in a "Proxy Group," which is a Google group that only stores your service accounts. In many cases, Terra creates and manages these service accounts for you automatically; however, in some cases you need to create and manage your own service account. Follow the steps below to do so.

Step 1. Create a service account

1.1.  Create or be ask to be added to a Google Cloud project.

1.2. Follow the step-by-step instructions in the Google Cloud support doc (external link) to set up your service account. 

1.3. Click manage roles to select which roles to assign to this service account.

Screenshot of Create a service account page on GCP console with a popup for selecting a role in the grant this service account access to the project step and the manage button circled.

Choosing the service account roleHopefully you already know which role you need for the task you want the service account to do (often in the instructions for the action). There are hundreds of roles you can grant to a service account so take some time to see which scopes and permissions you may want to assign to the service account (note that roles can be updated later). 

Most common services and permissions on Terra
 - Google Cloud Storage (viewer, reader, writer, owner)
 - Google BigQuery (viewer, reader, writer, owner)
 - Google Compute Engine (can-compute)

These are the most common roles and permissions. There is no one size fits all solution. If you think you may require a less-used role, see this doc (Google reference) which outlines all the different types of roles you can assign a service account.

1.4. After you've finished the initial account set up, save the email associated with the service account:

service-account-name@PROJECT_ID.iam.gserviceaccount.com

1.5. Before moving to step 2 (registration), go to the "keys" tab to create and download a key for your service account onto your local machine. 

Screenshot of the keys tab in the create a service account step on GCP console with Create new key option circled under the add key tab.

Note: Use the recommended JSON format to download your key

Step 2. Register your service account

You will need the key file and service account email from part 1 (above) to register the service account. Registration allows the service account to call Terra APIs shared with the service account.

2.1. Run the script to register a service account (you can find the script on the Terra FireCloud tools page).

To run the script you will need

  • -j <file path on a local machine to the JSON file containing the key created for service account>
  • -e <email address for the service account>

2.2. In your local terminal, run "git clone https://github.com/broadinstitute/firecloud-tools" with the following code. Cloning the FireCloud repository will allow you to run FireCloud tools used by Terra.

"git clone https://github.com/broadinstitute/firecloud-tools"

To learn how to install and use a terminal on your local machine, see How to move data between local and workspace storage (Step 1: Install gsutil locally).

2.3.  Change the directory to "firecloud-tools" using the below code.

cd firecloud-tools

2.4. Run the following script in the firecloud-tools directory to register your service account.  

./run.sh scripts/register_service_account/register_service_account.py 
-j <path to your service account credentials json file> 
-e <email address for owner of this service account, it's where notifications will go>

Replace the bolded sections with the file path to the key you downloaded in step 1 and the email associated with the service account. 

Note: You will need to run this code in the Firecloud-tools directory on your local machine. 

Example registration code

./run.sh scripts/register_service_account/register_service_account.py 
-j /Users/yduggal/downloads/setting-up-service-account-890a643c6881.json
-e creating-a-service-account-903@setting-up-service-account.iam.gserviceaccount.com

The service account is now registered with FireCloud! You are now able to share workspaces with this address or use it to call APIs in Terra.

Step 3. Manage your service account with a Terra group

Terra will add your new service account to a Proxy Group. But, the pre-defined Proxy Group identifier is not very human-friendly. If you're try to manage the users whose service accounts have access to a resource, seeing a Proxy Group named something like PROXY_11564882405514439@firecloud.org is not very helpful.

Therefore, we recommend creating a Terra Group (with a sensible name) as a proxy for your Proxy Group. Terra Groups include the proxy service account by default and are much easier to manage!

Terra groups are not necessary in all casesIf you're using a service account to give a third-party application access to data stored on Terra, you generally do not need to add the service account to a Terra group.

1. Log into Terra, and go to your Groups page ("Main menu" --> "Groups" from the top left of any page in Terra).
screenshot of Groups page on Terra highlighting vertical 3 lines on top of groups page with arrow and numeral 1, also highlighting the word Groups with numeral 2

2. In the Create a New Group card, click on the blue + icon.
screenshot of Groups page on Terra highlighting the words Create a New Group with an arrow pointing to a plus sign

3. Enter your human-friendly user-ID (can be the same as your Terra login) and click the Create Group button.
screenshot of Groups page on Terra with  Enter a unique name box overlaid on page

Naming your Terra group

Your Terra group should have a name that is intuitive and understandable. A good practice is to base it off of your Terra account name. For example, if your registered Terra account is j_doe@someplace.org, create a Terra Group named j_doe_at_someplace_org

What happens

Terra creates a mirrored Google group that includes your Proxy (which already includes your user ID). When you (or the owner/admin) grant access on a Google Cloud resource to the <terra-group>@firecloud.org group, both your end-user credentials and your pet service accounts have access to the resource.  

Enabling access

To allow access to external resources, owners can now grant permission to your Terra group (i.e.  j_doe_at_someplace_org@firecloud.org) in GCP console.  
screenshot of Groups page on Terra with Group Management text showing the name of a fictional Terra group

Additional resources

  • You can share resources with groups of Terra users by first setting up a group for each person's Terra proxy (following the steps above).

    Then create a managed group (i.e.your-lab-group) that includes all these personal groups.

    To give permission to everyone in the group, you would grant access to your-lab-group@firecloud.org. Since you can add or remove group members right in Terra, it's easy to adjust who has access.

  • Follow these instructions in case you ever need to access your non-intuitively-named proxy group on Terra:

    1. On the upper left-hand side, click on the main menu (three lines at the top of any page on Terra).

    2. Next to your name, click the drop-down arrow

    3. Click to expand the profile section under your name. 

    screenshot of highlighted word Profile on upper lefthand corner of profile page in Terra

    4. You'll see your proxy group listed near the bottom left.

    screenshot of highlighted words Proxy Group near bottom lefthand corner of Terra page

  • Want to learn how to access advanced Google Cloud features not (yet) available in Terra?

    • WRITE to BigQuery
    • Interact with Cloud Storage buckets other than the workspace bucket
    • Run dsub jobs
    • Run Cloud Dataflow jobs
    • Run Cloud ML engine jobs

    See Accessing Google Cloud features that are not in the Terra UI.

  • See Best practices for accessing external resources (Google buckets, Google Cloud VMs, etc.).

Was this article helpful?

Comments

0 comments

Please sign in to leave a comment.