Best practices for accessing external buckets, GCP VMs, and machine-learning tools

Learn how to access to resources outside the Terra platform - using data stored in external buckets, running Google Cloud Platform (GCP) VMs or machine-learning tools. This article outlines how to harness Terra's back-end infrastructure while keeping it easy to manage (i.e. using human-friendly IDs versus strings of random variables to represent users).   

Overview - Accessing external resources

The Terra platform is designed to remove some of the barriers of moving to the cloud, like interfacing directly with Google so you don't have to. Behind-the-scenes, Terra uses a special kind of Google account - called a service account - to access data, including external Google buckets, as well as other Google Cloud Platform resources (VMs that power Cloud Environments and workflows) directly from the Terra UI.

Every Terra user has one or more of these "pet" service accounts (one for each  Billing Project), which are used when interfacing with the cloud outside of Terra. 

       - Accessing a non-Terra GCS bucket, BQ dataset, GCR docker image, etc.     
       - Running workflows or notebooks (interactive analyses) on virtual machines (VMs

In all of these instances, Terra assumes the identity of the service account - rather than your user ID credentials - to call Google APIs. Using an anonymous service account is required for data and workspace security, but means that the back-end interfacing includes a lot of non-human friendly details. 

When and why to use Terra managed groups

What is a managed group? A managed group is a set of individuals defined in the Terra UI to streamline resource management. A managed group could include everyone in a research team, for example, who might need access to the same workspace or billing project. Once created, owners can assign permissions (roles) to a managed group as well as to an individual. This is especially useful since teams often change, since updating the group membership updates all permissions on all resources shared with the group. Terra Groups can be used within Terra for    - Adding to billing projects    - Sharing workspaces    - Restricting access to workspaces (Authorization Domains)    - Granting access to a non-Terra GCS bucket, BQ dataset, GCR docker image, etc. Using a personalized Terra group (for just one person) for easy sharing Always use Terra groups for accessing external resources, even for one user! With a Terra group, you can manage your Terra group within the Terra UI and Terra handles all the non-human-friendly back-end. To learn more about Terra groups, see this article.

Best practices for individuals accessing external resources

Use human-friendly groups 
The service accounts that Terra uses behind the scenes to interface with GCP have the format PROXY_<long-number>@firecloud.org. Although any user can use these pre-defined groups, the long string of numbers makes them not recommended (imagine you're a resource owner trying to identify who has access to the data in your external bucket. It's tough when the list is a bunch of PROXY_<long-number>@firecloud.org) 

Best practices is to create a Terra user group with a human-friendly name that includes you and any other users who need access to the external resource, and use it for interfacing with GCP (i.e. granting access to external buckets).

	Example: Terra group for single user (User ID:  j_doe@someplace.org)
	- Create a Terra Group: j_doe_at_someplace_org - Don't add anyone else to this group - Make grants to j_doe_at_someplace_org@firecloud.org

1. Set up a human-friendly custom group to make it easier to track shared resources

Always use Terra groups for accessing external resources, even for one user! With a Terra group, you can manage your Terra group within the Terra UI and Terra handles all the non-human-friendly back-end.

Create your personal Terra group in four steps

1.1. Go to your Groups page ("Main menu" --> "Groups" from the top left of any page in Terra)

1.2. In the "Create a New Group" card, click on the blue "+" icon 

1.3. Enter a name for your personalized group of one and click the "Create Group" button

1.4. You can now use your personal Terra group (in this example:  j_doe_at_someplace_org@firecloud.org) for accessing external resources

If multiple users need access to the same external resources, simply add them as members of the group.

2. Grant permissions to the Terra Group

	Before you start: Only resource owners or admins can grant access
	If what you see on the console does not look like the screenshots, it is most likely because you do not have the right permissions for the Google bucket or other resources. You will need to ask the resource owner or admin to grant permission to your Terra group, following the steps below.

Step-by-step instructions

2.1. From the GCP console select the resource to be shared (i.e. a particular bucket in https://console.cloud.google.com/storage/browser)

2.2. Go to Permissions

2.3. View by "Members" and select the "Add" icon

2.4. Add the full name of your Terra group (i.e. j_doe_at_someplace_org@firecloud.org) as a New Member and select 1. the resource type (left column - i.e. "Cloud Storage") and 2. the appropriate roles (right column).

"Storage Object Viewer" allows you to read from the bucket "Storage Object Creator" allows you to write to the bucket

You will see your Terra group and role in the Members Permissions

Best practices for groups to access external resources 

Managed groups are the best way to share resources (workspaces and billing as well as external resources) amongst a group of individuals, such as everyone in a lab. Sharing with a managed group instead of a long list of individuals saves time and avoids errors. The groups can be updated in the Terra UI when people are added to - or leave - the lab or project.

Best practices is to create a Terra user group with a human-friendly name that includes all users who need access to the external resource, and use it for interfacing with GCP (i.e. granting access to external buckets).

	Example: Terra group for lab (User ID: my_lab@someplace_org)
	- Create a Terra Group: my_lab_at_someplace_org - Include the Terra user ID of everyone in the lab - Make grants to my_lab_at_someplace_org@firecloud.org

1. Set up a Terra managed group for all collaborators

Manage your group members within the Terra UI and Terra handles all the non-human-friendly back-end.

2. Grant permissions to the Terra Group

	Before you start: Only resource owners or admins can grant access
	If what you see on the console does not look like the screenshots, it is most likely because you do not have the right permissions for the Google bucket or other resources. You will need to ask the resource owner or admin to grant permission to your Terra group, following the steps below.

Step-by-step instructions

2.1. From the GCP console select the resource to be shared (i.e. a particular bucket in https://console.cloud.google.com/storage/browser)

2.2. Go to Permissions

2.3. View by "Members" and select the "Add" icon

2.4. Add the full name of your Terra group as a New Member and select the resource type (left column - i.e. "Cloud Storage") and the appropriate roles right column). 

"Storage Object Viewer" allows everyone in the group to read from the bucket "Storage Object Creator" allows everyone in the group to write to the bucket

 You will see your Terra group and role in the Members Permissions.