Do you want to perform Google Cloud Platform (GCP) operations not currently available in the Terra UI? You can do many of these things in Terra already! This article explains how to leverage Terra notebooks and workflows to access additional Google Cloud Platform (GCP) features in Terra.
- WRITE to BigQuery
- Interact with Cloud Storage buckets other than the workspace bucket
- Run dsub jobs
- Run Cloud Dataflow jobs
- Run Cloud ML engine jobs
Getting Started with advanced GCP features
The Terra platform is designed to remove some of the barriers of moving to the cloud: Terra interfaces directly with Google so you don't have to. However, there are many GCP features that are not included in the platform. Some are on the horizon, others are niche capabilities that may never be integrated with the Terra UI.
Just because they aren't in the UI doesn't mean you cannot use them, however. You can access these advanced features through a GCP project, which you will set up on the GCP console and connect to Terra with a human-friendly personal Terra group following the steps below. Once you follow these three setup steps, you'll be able to use the GCP project to leverage advanced GCP features by running notebooks and workflows on Terra.
|In order to set up a GCP-native project on GCP console, you need to be an owner or a user on a GCP Billing account linked to Terra. If what you see on the console does not look like the screenshots, it is most likely because you do not have the right permissions on a GCP Billing account.
To learn how to set up GCP billing, and access $300 in free credits from Google, see this article.
1. Set up a GCP-native project (on GCP console)
1.2. On the "Select organization" drop-down at the top of the page, select the organization in which you want to create a project. Free trial users can skip this step, as this list does not appear.
1.3. Select Create project.
1.4. In the New Project window, enter a project name and select a Billing account. This is the Cloud Billing account that will cover all GCP costs incurred in your Google project.
1.5. Enter the parent organization or folder in the Location box. That resource will be the hierarchical parent of the new project.
1.6. When you're finished entering new project details, click Create.
2. Create a human-friendly personal Terra group in three steps
Each Terra user has a pre-built "Proxy" Group for accessing resources that exist outside of Terra.
However, your proxy group is not very human-friendly. If you're
is not helpful unless you happen to have a way to figure out what
Instead, you can create a Terra group (with a sensible name) as an
2.1. Go to your Groups page (Your name > "Groups" from the main menu at top left of any page in Terra).
2.2. Click on the blue Create a new group button.
2.3. Enter your human-friendly user-ID (can be your Terra login - see screenshot below) and click the Create Group button.
Terra creates a mirrored Google group (your Terra ID plus your built-in proxy) for interfacing directly with GCP that you can use as well.
You'll see the full name in your list of Groups (below). In the next step, you'll grant permission for this group to access the cloud-native GCP you created in step 1:
3. Add your Terra group on the Google project
You will give your personal Terra group "Editor" permission (for more information about GCP permissions, see this article).
Note that if your Terra group includes additional people, you will want to be careful what permissions you grant the group. This is because editors can turn on a large number of services, including ones that can be expensive!
3.1. Go to IAM >Manage Resources in your new GCP project and select Add Member.
3.2. Add your human-friendly personal Terra group as a member in your project permissions.
3.3. Give the group Editor permission.
Once these three steps are complete, you'll be able to do many advanced GCP tasks. In many cases, Terra will interface with GCP on your behalf! Read on for details of how to do specific tasks. We will continue to add to this list.
Step-by-step instructions and template notebooks
Below are a series of features users have asked about that are not (yet!) available in Terra. Expand each section for step-by-step instructions - or a link to a notebook in the public workspace.
Create an external GCP bucket accessible by your Terra workspaces
1. Go to GCP Storage Console.
2. Select your GCP-native project from the dropdown and click Create bucket.
In general, you can use the default values when setting up your external bucket.
When you are done, you will see your external bucket in the console!
Set an external GCP bucket to auto-delete
1. Go to GCP Storage console.
2. Select the bucket you want to set to automatically delete data by clicking the bucket name.
3. Select the Lifecycle tab.
4. Choose Add a Rule.
5. Follow the instructions to set up a custom rule.
If you set up a rule to delete contents after 1 day, for example, you will see this:
Interact with Cloud storage buckets other than Workspace bucket (template notebook)
There are times when you may not want to keep shared data in a Workspace bucket (particularly if you're sharing large numbers of large data files with a large group - see this article for why).
To learn more about sharing large numbers of large data files with large groups, see this article.
For an end-to-end example of interacting with an external bucket, see this template notebook.
Create a BigQuery dataset (in GCP console)
1. Go to BigQuery in the GCP console and select the native GCP project you created above.
2. Select Create Dataset to the right of the project name.
3. In the dataset creation form, choose a unique dataset name and select the default table expiration.
In general, you would choose "Never". But if you are testing queries and saving those results as tables, you may generate a lot of tables that you don't want to keep (or pay for). To avoid having to clean up those tables at the end of the day, you can create a BigQuery dataset for test results that auto deletes its tables after a period of time has elapsed.
4. You will see your new Big Query dataset in the Resources section on the far left.
How to load data to BigQuery (template notebook)
Note that before you can load data to BigQuery, you must have (at least) WRITE access permission to an existing BQ dataset. If you have set up your own BigQuery dataset (above), you will automatically have those permissions.
|See an example notebook in a public Terra workspace|
Other things you can do in a GCP Project