Google Cloud quotas: What are they and how do you request more?

Is your analysis running slow? Have you gotten an error message that includes the word "quota" or when trying to run a large analysis (workflow or interactive)? Or a "precondition failed" error when creating a new workspace? It could be because you've exceeded the default quota for a particular kind of resource. Read on to understand GCP quotas that can affect your work, and how to ask for more when you need them. 

Overview

In the Google Cloud Platform (GCP), quotas limit the GCP resources any one user can consume. Quotas prevent unforeseen spikes in usage, making sure resources are available to the community at all times.

There are two categories of quotas that may affect your work on Terra. 

Resource quotas (VM compute and storage limits)
These limit how many resources like central processing units (CPUs and GPUs) and persistent disks (PDs), can be used by a single Google project at any given time. Resource quotas affects your ability to spin up a large VM to run a workflow or interactive analysis. They can also impact the speed of your analysis, since tasks will pause or slow as you run up against a compute or disk quota.

In Terra these limits will apply per workspace (for those created after September 27, 2021) or per Terra billing project (for workspace created before September 27th).

Project quotas
These limit the number of Google projects that can be created by a single user (default is 25) or under a single GCP Billing account (default is five). This impacts the number of workspaces you can create in Terra.

Why do quotas matter in Terra?

Quotas will limit how much of a particular cloud resource you can use. Resource quotas limit the CPUs, GPUs, preemptible CPUs, persistent disk, and IP addresses you can use. Exceeding these quotas can make your analysis fail or run slowly. Project quotas limit the number of Google projects a single user or Billing account can have. Exceeding these quotas affect your ability to create workspaces, since each Terra workspace has an associated Google project. 

Resource quotas

All workflows (or "methods") and Cloud Environments that run in Terra are affected by GCP compute and disk quotas. Google enforces default resource quotas for Terra Billing projects (before September 27, 2021) and workspaces (after September 27, 2021) based on a user's GCP billing reputation.

Examples of resource quotas include the following.

• CPUs: how many CPUs you can use at once across all tasks
• GPUs: how many GPUs you can use at once across all tasks
• Preemptible CPUs: the pool of CPUs that would only be used by preemptible instances. You can learn more about this quota here and about preemptible instances here.
• Persistent disk standard(GB): how much total disk (non-SSD) you can have attached at once to your task VMs
• Persistent disk SSD(GB): how much total SSD disk you can have attached at once to your task VMs
• Local SSD(GB): how much SSD is attached directly to the server running the task VMs. You can learn more here. This quota only applies if you are using local SSD in your task.

If you bump up against your resource quota, Terra will not be able to secure the CPUs, GPUs or PD requested, and your workflow or Cloud Environment analysis will run very slowly or not at all. 

Google project quotas

Google enforces two project quotas that typically impact Terra users: a limit on the number of projects per GCP Billing account and a limit on the number of projects a single user can create (technically there's a third, which is projects per organization, but not many Terra users run up against this one). 

Note that since Terra creates a Google project for each workspace, this is effectively a limit on the number of workspaces per Terra user or per Google Cloud Billing account.

Google projects you create on GCP and projects that Terra creates for you when you create a workspace all count against this limit. 

To learn more about Google project quotas, see the Google documentation.  

When would I need to request additional quota?

Quota limits are not always easy to diagnose! Below are some behaviors and error messages that indicate you may need a quota increase. 

Resource quota symptoms (makes your analysis stall or run slow)

You may experience one of the following after launching a workflow analysis if there is not enough resource (i.e. VM compute or disk capacity) in your quota:

1. Tasks within your workflow will run slow while they wait on quota availability. For example, if you requested 1,000 tasks with eight CPUs each, and your quotas allow 24 CPUs at once, you can only run three tasks at a time. Each subsequent task is queued.
   
   
2. A task within your workflow may fail because it requested more resources than allowed by your  quotas. For example, if you requested 60 CPUs in your task and your quota is capped at 24 CPUs at once, your workflow may fail to launch.

When to request more 
Please note that unless you are seeing errors, you do not need to update quotas - your analysis will simply run more slowly. If your analysis runs more slowly than you expect, or if you see errors/messages related to quota in your logs, you may want to request an increase. 

Project quota symptoms (affect workspace creation)

If you have exceeded your Google project quota and try to create a workspace in Terra, your workspace will fail to create and you will get an error message in the create workspace modal.

You will not get an error when you create a Terra Billing project, as Terra billing projects are on the Terra back-end only and no longer map to Google projects (after September 27, 2021). 

How do I check my quotas?

Checking resource quotas 

If your analysis is running slow or failing (see the list of symptoms here), you can check the resources quota of that workspace on the GCP console.

You need to have Owner permission for the Terra billing project in order to view the resource quotas on the GCP console.

1. Go to  https://console.cloud.google.com/iam-admin/quotas?project=project_id where project_id is the workspace Google project ID (find in the Dashboard page on the right under "Workspace Information.")

In the GCP console you'll see a long list of quotas for the workspace.

2. Identify quotas that are close to the limit. In the example below, the CPU quota in region "us-central1" is maxed out (the orange bar near 100%) and you'll need to request more (see steps below).

Note that quotas are defined per region. To run your analysis across multiple regions (e.g. us-east1 and us-central1), you need to request a larger quota in both.

Scroll down to see how to request more resource quota. 

Terra workspace (i.e Google project) quotas

Per GCP Billing account 
Since there is a 1:1 relationship between the number of Terra workspaces and Google projects, the number of workspaces you can create is constrained by your Google project-per-billing-account quota. Because the default quota is small (default is five), you are most likely to bump up against this when creating new workspaces. Unfortunately, there is no way to check your present per-billing-account project quota limit.  

You can check how many projects are attached to a billing account (you must be the GCP Billing account owner or admin) by going into the billing account in GCP console. This page will list all projects (even disabled ones) created under the selected billing account.  

Per individual user
To see how much of your per-user quota you have left, go to the "Create project" page on GCP console here. The number of projects you have in your quota will be at the top of the page. 

How much quota will I need?

GCP Resource quotas

The right amount of quota is a function of the number of workflows being launched, the number of concurrent tasks running within each workflow, and the resources being requested by those tasks.

To calculate the quota needed for the workflows, you need to do a bit of diving into your WDL to examine what it is doing.

Example resource quota estimation (three task WDL)
Consider a three-task WDL that will run on one to many samples. You would need to look across all three tasks to determine what the maximum amount of CPU and PD you expect to need at any given time.

• Task 1: uses 10 CPUs and 10GB of PD
• Task 2: uses 1 CPU, 1GB of PD and scatters 10-ways wide
• Task 3: uses 10 CPU, 10GB of PD and scatters 10-ways wide

In this example, tasks 1 and 2 are using the same amount of resources because task 2 scatters. Task 3, however, uses more resources than Task 1 or 2.

When running task 3 on ten samples at once, the task requests a total of 100 CPUs and 100GB of PD (due to scattering 10-ways wide) for one sample. Because it is running on ten samples, it will need ten times those resources at once -- 1000 CPUs and 1TB of persistent disk. If the current resources quota is set at 24 CPUs and 100GB of PD and you want this workflow to run as quickly as possible, you will need to make a request for at least 1000 CPUs and 1TB of PD.

Google project quota 

By default, Google Cloud only allows the creation of 5 projects per billing account. Since there is a 1:1 relationship between Terra workspaces and Google projects, this limits the number of workspaces per billing account to five (assuming you haven't also created any on GCP console). This number is insufficient for most Terra users, let alone groups with many users on the same billing account. 

The number of projects we recommend asking for (200) is an estimate based on the highest usage levels we have seen so far for large groups using a single billing account. You may need to increase it further if you have a very large group that creates a lot of workspaces; for example, if you run workshops or classes where you provide billing account access to students. 

To request an increase in your project quota, see the step-by-step instructions below. 

How do I request a quota increase?

Request an increased resource (compute/storage) quota

You can make a resource quota increase request by sending us a message through the Contact Us module in the Terra UI, or by emailing support@terra.bio.

Please be sure to provide the following information:

• Your Google Project ID, found on the right of the Dashboard page under Workspace Information.
  
• Which quota(s) you want to increase (i.e. CPUs, PDs, etc.)
• What you want your new quota(s) to be (see this section on guidance for how to estimate)
• Which regions you want the increase applied to, if applicable (e.g. us-central-1, us-east-3, etc.)
  • Terra uses us-central-1 by default in most cases. If you don't specify your region for workflow submissions, this is likely the region you want the quota(s) increased for.
• Rationale for increase (research purpose)

We will create a request to Google on your behalf. Depending on the quota and how much of an increase you are requesting, it may take 2-3 business days for Google to process and respond to the request.

Request an increased project quota (to be able to create more workspaces)

You will need to ask Google directly for an increase in your project quota (Terra cannot make the request on your behalf). Note that you need to be the GCP Billing account owner or admin to make this request. 

1. Go to https://support.google.com/code/contact/billing_quota_increase.
2. Fill out the form details. See below for how to complete each field (in bold).

• Name: Your name
• Email addresses that will be used to create projects: terra-billing@terra.bio
• How many projects are being requested: 200
• What kind of services will these projects use? Paid services
• Billing account ( note that this field appears when you select the paid services radio button)
  You can find the 18-digit GCP billing account ID here.
• Any other things we need to be aware of to help us understand the request (please input this exact wording in the input field, as it will expedite the approval process):
  I use a platform (Terra at app.terra.bio) that creates Google projects via an api to handle data security and processing. In order to use Terra, I need sufficient quota so that projects created by the platform can be transferred over to my billing account.

You will receive a confirmation email from Google within a day or two. They may ask for additional information. before granting the request.