Accessing Google Cloud features that are not (yet!) available in Terra

Allie Cliffe

Learn to perform Google Cloud operations like writing to a BigQuery dataset, running dsub jobs, and more. Did you know you can do many of these things in Terra already? This article explains how to leverage Terra notebooks and workflows to access additional Google Cloud features in Terra.

Getting Started with advanced Google Cloud features

The Terra platform is designed to remove some of the barriers of moving to the cloud: Terra interfaces directly with Google. However, there are many other Google Cloud features not yet on the Terra platform.  Some are on the horizon, others are niche capabilities that may never be integrated with Terra. 

Examples of what you can do 

  • WRITE to BigQuery
  • Interact with Cloud Storage buckets other than the workspace bucket
  • Run dsub jobs
  • Run Cloud Dataflow jobs
  • Run Cloud Machine Learning (ML) engine jobs

How to do it

Just because they aren't in Terra doesn't mean you cannot use them. You can access these advanced features through a Google project that you will set up on the Google Cloud console. You can connect to Terra with a human-friendly personal Terra group following the steps below. 

Diagram of three steps showing how to access advanced Google Cloud-features: 1. Set up an external cloud-native project on Google console 2. Create your human-friendly Terra managed group and 3. Add your group as editor on the Google project.

Once you follow these three setup steps, you'll be able to use the Google Cloud project to leverage advanced Google Cloud features by running notebooks and workflows on Terra.

Before you start! Your Terra user ID must have a Google Cloud Billing accountTo set up a Google Cloud-native project on Google Cloud console, you need to be an owner or a user on a Google Cloud Billing account linked to Terra. If what you see on the console does not look like the screenshots, it is because you do not have the right permissions on a Google Cloud Billing account.

To learn how to set up Google Cloud billing, and access $300 in free credits from Google, see How to set up billing in Terra

How to set up your Google Project in three steps

1. Set up a Google Cloud-native project (on Google Cloud console)

1.1. From the main menu (three horizontal lines at the top left of the Google Cloud console pagego to the IAM & Admin > Manage resources page.
screenshot of Google Cloud features page with Manage Resources highlighted

1.2. On the "Select organization" drop-down at the top of the page, select the organization in which you want to create a project. Free trial users can skip this step, as this list does not appear.

1.3. Select Create project.
Screenshot of Google Cloud features page with 'Create Project' at the top of the page highlighted

1.4. In the New Project window, enter a project name and select a Billing account. This is the Cloud Billing account that will cover all Google Cloud costs incurred in your Google project.

Screenshot of Google Cloud new project pane for entering new project details
  • If you don't see a Billing account in the drop-down, you can set one up following these instructions.

  • Note: A project name can contain only letters, numbers, single quotes, hyphens, spaces,
    or exclamation points, and must be between 4 and 30 characters.

1.5. Enter the parent organization or folder in the Location box. That resource will be the hierarchical parent of the new project.

1.6. When you're finished entering new project details, click Create.

2. Create a human-friendly personal Terra group in three steps

Why use a Terra group for external access? Each Terra user has a prebuilt "Proxy" Group for accessing resources outside of Terra.

However, your proxy group is not very human-friendly. If you're looking at a list of users with access to an external Google Cloud bucket, seeing that there's a grant to PROXY_11564882405514439@firecloud.org is not helpful unless you happen to have a way to figure out what user is associated with that Proxy Group.

Instead, you can create a Terra group (with a sensible name) as an alias for your proxy. if your registered Terra account is j_doe@someplace.org, create a Terra Group named j_doe_at_someplace_org. Don't add anyone else to this group. You can make grants to j_doe_at_someplace_org@firecloud.org.

This group contains one member, namely the proxy group for j_doe@someplace.org. This is much easier for a human to recognize and remember.

2.1. Go to your Groups page (Your name > "Groups" from the main navigation menu at top left of any page in Terra).
Screenshot of Groups page in Terra highlighting main navigation menu - three vertical lines - and Groups section

2.2. Click on the blue Create a new group button.
Screenshot of Groups page in Terra pointing out the 'Create a new group' button at the top

2.3. Enter your human-friendly user-ID (can be your Terra login - see screenshot below) and click the Create Group button.
screenshot-of-Groups-page-in-Terra-with-overlay-of-Create-New-Group-box
Terra creates a mirrored Google group (your Terra ID plus your built-in proxy) for interfacing directly with Google Cloud that you can use as well.

You'll see the full name in your list of Groups (below). In the next step, you'll grant permission for this group to access the Google Cloud-native project you created in step 1:
Screenshot of Groups page in Terra with new group name highlighted

3. Add your Terra group on the Google project

This step allows you to work in Terra (i.e., a Terra notebook), while Terra acts on your behalf (as your "proxy") behind the scenes in the project you just set up in Google Cloud. 

You will give your personal Terra group "Editor" permission (for more information about Google Cloud permissions, see IAM basic and predefined roles reference).

Consider group membership before giving editor permission Note: If your Terra group includes additional people, be careful of what permissions you grant to the group. Be aware editors can turn on a large number of services, including ones that can be expensive!

3.1. Go to IAM >Manage Resources in your new Google Cloud project and select Add Member.
Screenshot of Google Cloud IAM Manage Resources page with 'Add Member' button highlighted

3.2. Add your human-friendly personal Terra group as a member in your project permissions.
Screenshot of Google Cloud resource page with new members box highlighted

3.3. Give the group Editor permission.
Screenshot  of Google Cloud page with project editor role selected.

Once these three steps are complete, you can do many advanced Google Cloud tasks. In many cases, Terra will interface with Google Cloud on your behalf! Read on for details of how to do specific tasks. We will continue to add to this list. 

Step-by-step instructions and template notebooks

Below are a series of requested features that are not (yet!) available in Terra. Expand each section for step-by-step instructions - or a link to a notebook in the public workspace.

Create an external Google Cloud bucket accessible by your Terra workspaces

Why use an external bucket?To learn more about the benefits of using external buckets for storing shared data resources, see IAM basic and predefined role reference.

1. Go to Google Cloud Storage Console.

2. Select your Google Cloud-native project from the dropdown and click Create bucket.
screenshot of Google Cloud page gighlighting the project name in the top menu and pointing to the 'Create bucket' button below the project name

External Google Cloud bucket configuration tips In general, you can use the default values when setting up your external bucket.

For customization details, see the Google documentation.

When you are done, you will see your external bucket in the console!
Screenshot of newly created external bucket in Google cloud console

Set an external Google Cloud bucket to autodelete

Why set bucket to autodelete?When you're testing code, you may generate a lot of data that you don't want to keep (or pay for). To avoid cleaning up at the end of the day, set your storage bucket to delete the contents every day with the following steps.

1. Go to Google Cloud Storage console.

2. Select the bucket you want to set to automatically delete data by clicking the bucket name.
Screenshot of Google Cloud storage page with bucket name highlighted

3. Select the Lifecycle tab.
Screenshot of Google Cloud page with lifecycle tab highlighted

4. Choose Add a Rule.
Screenshot of Google Cloud storage page with 'Add a rule' tab highlighted

5. Follow the instructions to set up a custom rule.

If you set up a rule to delete contents after 1 day, for example, you will see this:

Screenshot of custom rule to autodelete

Interact with a cloud storage bucket other than a Workspace bucket (template notebook)

There are times when you may not want to keep shared data in a Workspace bucket (particularly if you're sharing large numbers of large data files with a large group - see IAM basic and predefined roles reference for good reasons).

Why use external buckets? To learn more about sharing large numbers of large data files with large groups, see Best practices for sharing and protecting data resources.
Example notebook
For an end-to-end example of interacting with an external bucket, see this template notebook.

Create a BigQuery dataset (in Google Cloud console)

1. Go to BigQuery in the Google Cloud console and select the Google Cloud-native project you created above.

2. Select Create Dataset to the right of the project name.
Screenshot-of-Google-Cloud-page-with-create-dataset-tab-highlighted

3. In the dataset creation form, choose a unique dataset name and select the default table expiration.
In general, you would choose "Never". But if you are testing queries and saving those results as tables, you may generate a lot of tables that you don't want to keep (or pay for). To avoid having to clean up those tables at the end of the day, you can create a BigQuery dataset for test results that auto deletes its tables after your selected time period.
screenshot of Google Cloud dataset creation form with 'default table expiration' option highlighted

4. You will see your new Big Query dataset in the Resources section on the far left.
Screenshot of Google Cloud page with BigQuery dataset autodelete in far left section highlighted

How to load data to BigQuery (template notebook)

Note: Before you can load data to BigQuery, you must have (at least) WRITE access permission to an existing BQ dataset. If you set up your own BigQuery dataset (above), you automatically have those permissions.

Example notebook

See an example notebook in a public Terra workspace.

Additional resources

dsub

See this Google Cloud tutorial on running dsub jobs in Python.

Cloud Dataflow

See this Google Cloud Quickstart on running Dataflow in Python.

Cloud ML

See Google Cloud documentation on ML or tensorflow.

Was this article helpful?

0 out of 0 found this helpful

Have more questions? Submit a request

Comments

3 comments

  • Comment author
    Laura Egolf
    • Edited

    Can the GCP project created using these instructions be linked to a Terra billing project or workspace? My organization wants to create a GCP project for me so that they can have better control of billing on their end, but I don't see a way to link this GCP project to a Terra project/workspace.

    0
  • Comment author
    Allie Hajian

    Laura Egolf Unfortunately, you cannot create a Google project on GCP and add/link it to Terra. Billing projects See this article (the section on the relationship between GCP projects and Terra Billing projects).

    I completely understand your organization wanting to have control over billing and it's possible! The key is Ownership of the GCP Billing account linked to your Terra account. here are the steps:

    1. Your organization can create a GCP Billing account 
    2. They then link it to Terra 
    3. Then within Terra, you would create a Terra Billing project funded by that GCP account that you and they would be able to see on GCP console.

    As the owners of the GCP billing, they would have access to all billing and spend information about that Terra Billing project (on GCP and in the Terra UI, as available). For more detail, see this article. Hope this helps!

    1
  • Comment author
    William Grisaitis
    • Edited

    Allie Hajian what permissions do i need so that the drop-down for selecting a billing account shows up in the "create project" dialog on GCP? i am able to create [terra] workspaces in [terra] billing projects, but i see that i can't enable many GCP APIs... so it sounds like i need to create a new GCP project. but, when i follow the directions above and pursue "create priject" (under IAM & Permissions > mMagage Resources), i don't see that drop-down for selecting my lab's billing project. Someone else on my team (who is an owner on the billing project) does see the dropdown. so, do I need to be an owner? is there any other way?

    my goal - all i want to do is have a normal GCP project where i can enable GCP APIs and have billing linked to my lab's terra billing project. what's the simplest way to do that? can i just create a GCP project on my own (not via terra) and then link it to a terra billing project later? would there be any downside to doing this? (would data in my buckets not be accessible for running terra pipelines?)

    EDIT - i asked this question here, too: https://support.terra.bio/hc/en-us/community/posts/4409556648475-What-permissions-do-I-need-to-create-GCP-projects-on-GCP-not-on-terra-linked-to-a-Terra-billing-project-

    0

Please sign in to leave a comment.