Is your analysis running slow? Have you gotten an error message that includes the word "quota" or when trying to run a large analysis (workflow or interactive)? Or a "precondition failed" error when creating a new workspace? It could be because you've exceeded the default quota for a particular kind of resource. Read on to understand GCP quotas that can affect your work, and how to ask for more when you need them.
Overview: Why do quotas matter in Terra?
In the Google Cloud Platform (GCP), quotas limit how much of a particular GCP cloud resource you can use. Quotas prevent unforeseen spikes in usage, making sure resources are available to the community at all times.
There are two categories of quotas that may affect your work on Terra.
Resource quotas (VM compute and storage limits) impact your analyses (including speed)
These limit how many resources like central processing units (CPUs and GPUs) and persistent disks (PDs), can be used by a single Google project at any given time. Resource quotas affects your ability to spin up a large VM to run a workflow or interactive analysis. They can also impact the speed of your analysis, since tasks will pause or slow as you run up against a compute or disk quota.
In Terra these limits will apply per workspace (for those created after September 27, 2021) or per Terra billing project (for workspace created before September 27th).
Project quotas impact your ability to create new workspaces
These limit the number of Google projects that can be created by a single user (default is 25) or under a single Google Cloud Billing account (default is five). Since Terra generates a unique Google project for each workspace, this impacts the number of workspaces you can create in Terra.
All workflows (or "methods") and Cloud Environments that run in Terra are affected by GCP compute and disk quotas. Google enforces default resource quotas for Terra Billing projects (before September 27, 2021) and workspaces (after September 27, 2021) based on a user's GCP billing reputation.
Examples of resource quotas:
- CPUs: how many CPUs you can use at once across all tasks
- GPUs: how many GPUs you can use at once across all tasks
- Preemptible CPUs: the pool of CPUs that would only be used by preemptible instances. You can learn more about this quota here and about preemptible instances here.
- Persistent disk standard(GB): how much total disk (non-SSD) you can have attached at once to your task VMs
- Persistent disk SSD(GB): how much total SSD disk you can have attached at once to your task VMs
- Local SSD(GB): how much SSD is attached directly to the server running the task VMs. You can learn more in Google's documentation. This quota only applies if you are using local SSD in your task.
What happens when you reach a resource quota?
If you bump up against your resource quota, Terra will not be able to secure the CPUs, GPUs or PD requested, and your workflow or Cloud Environment analysis will run very slowly or not at all.
Google project quotas
Google enforces two project quotas that typically impact Terra users:
- A limit on the number of projects per GCP Billing account
- A limit on the number of projects a single user can create
Note that since Terra creates a Google project for each workspace, this is effectively a limit on the number of workspaces per Terra user and per Google Cloud Billing account.
Google projects you create on GCP and projects that Terra creates for you when you create a workspace all count against this limit.
To learn more about Google project quotas, see the Google documentation.
When would I need to request additional quota?
Quota limits are not always easy to diagnose! Below are some behaviors and error messages that indicate you may need a quota increase.
|Symptom/error message||What's happening||Action to resolve|
|The server was not able to produce a timely response to your request. Error message "Please try again in a short while!"||Resource quota exceeded||Ask support to request a resource quota increase|
|Workflow tasks running very slow, especially if they ran fine in the past||Resource quota exceeded||Ask support to request a resource quota increase|
|Multiple instances of "worker assigned"/"worker released" cycles in the timing diagram.||Resource quota exceeded||Ask support to request a resource quota increase|
|Workflow fails to launch||Workflow requested more resources than allowed||Ask support to request a resource quota increase|
|Workspace fails to create. Error message will include the phrase "Precondition check failed" and maybe "QuotaFailure"||Exceeded GCP project quota (25 per user or 5 per Billing account)||User requests a project quota increase directly from Google|
Resource quota symptoms (analysis stalls or slow)
You may experience one of the following after launching a workflow analysis if there is not enough resource (i.e. VM compute or disk capacity) in your quota:
- Tasks within your workflow will run slow while they wait on quota availability.
For example, if you requested 1,000 tasks with eight CPUs each, and your quotas allow 24 CPUs at once, you can only run three tasks at a time. Each subsequent task is queued.
- A task in your workflow may fail when it requests more resources than your quota allows.
For example, if you requested 60 CPUs in your task and your quota is capped at 24 CPUs at once, your workflow may fail to launch.
When to request more resource quota Please note that unless you are seeing errors, you do not need to update quotas - your analysis will simply run more slowly. If your analysis runs more slowly than you expect, or if you see errors/messages related to quota in your logs, you may want to request an increase.
Note that when a user's GCP project reaches a quota limit, Cromwell continues to submit jobs and Life Sciences acknowledges them as created even if the physical VM cannot yet start. Cromwell now detects this condition in the backend and reports
AwaitingCloudQuota in the Job History Dashboard. If you suspect that you have exceeded your resource quota, you can check the Job History Dashboard to see.
Project quota symptoms (workspace fails to create)
If you have exceeded your Google project quota and try to create a workspace in Terra, your workspace will fail to create and you will get an error message in the create workspace modal.
How do I check my resource quota?
If your analysis is running slow or failing (see the list of symptoms here), you can check the resources quota of that workspace on the GCP console.
Step 1: Check Dashboard
When a user's GCP project reaches a quota limit, Cromwell continues to submit jobs and Life Sciences acknowledges them as created even if the physical VM cannot yet start. Cromwell now detects this condition in the backend and reports AwaitingCloudQuota in the Job History Dashboard.
1.1. Go to the Job History tab of your workspace.
1.2. Click on the Dashboard icon of the workflow that is running slow while it is running.
1.3. If you are running up against a resource cloud quota, you will see two notifications:
AwaitingCloudQuotamessage in the Total call Status Counts section
- An orange icon in the Status column and the message "Submitted. Awaiting Cloud Quota"
The status is informational and does not require any actionTo maximize throughput, you can use
AwaitingCloudQuota as an indication you should check quota in Cloud Console (step 2, below) and request a quota increase from GCP.
Step 2: Check GCP console
Once you know you're running up against a resource quota limit, you can check GCP console for the details and submit a quota increase request.
You need to have Owner permission for the Terra billing project in order to view the resource quotas on the GCP console.
2.1. Go to
https://console.cloud.google.com/iam-admin/quotas?project=project_id where project_id is the workspace Google project ID.
The workspace Google project in the workspace Dashboard page in the Cloud Information section:
2. Identify quotas that are close to the limit. In the example below, the CPU quota in region "us-central1" is maxed out (the orange bar near 100%) and you'll need to request more (see steps below).
A long list of quotas for the workspace in GCP console:
Note that quotas are defined per regionTo run your analysis across multiple regions (e.g. us-east1 and us-central1), you need to request a larger quota in both.
Scroll down to see how to request more resource quota.
How do I check my Terra workspace (i.e Google Project) quota?
There are two distinct Google Project quotas that limit how many projects can be under a single Google Cloud Billing account and how many projects a single individual can create. Terra users are most likely to be impacted by the first.
Since there is a 1:1 relationship between the number of Terra workspaces and Google projects, the number of workspaces you can create is constrained by your Google project-per-billing-account quota. Because the default quota is small (five Google Projects per Cloud Billing account), you are most likely to bump up against this when creating new workspaces. Unfortunately, there is no way to check your current per-billing-account project quota limit.
You can check how many projects are attached to a Google Cloud Billing account (you must be the Billing account owner or admin) by going into the billing account in GCP console. This page will list all projects (even disabled ones) created under the selected billing account.
To see how much of your per-user quota you have left, go to the "Create project" page on GCP console here. The number of projects you have in your quota will be at the top of the page.
Note that a user can have plenty of Google projects left but still bump up against the per-Google-Billing account quota limit. If the Google Billing account has an established history (i.e. has accrued and paid GCP charges), you should be able to request a quota increase.
How much quota will I need?
GCP Resource quotas
The right amount of quota is a function of the number of workflows being launched, the number of concurrent tasks running within each workflow, and the resources being requested by those tasks.
To calculate the quota needed for the workflows, you need to do a bit of diving into your WDL to examine what it is doing.
Example resource quota estimation (three task WDL)
Consider a three-task WDL that will run on one to many samples. You would need to look across all three tasks to determine what the maximum amount of CPU and PD you expect to need at any given time.
- Task 1: uses 10 CPUs and 10GB of PD
- Task 2: uses 1 CPU, 1GB of PD and scatters 10-ways wide
- Task 3: uses 10 CPU, 10GB of PD and scatters 10-ways wide
In this example, tasks 1 and 2 are using the same amount of resources because task 2 scatters. Task 3, however, uses more resources than Task 1 or 2.
When running task 3 on ten samples at once, the task requests a total of 100 CPUs and 100GB of PD (due to scattering 10-ways wide) for one sample. Because it is running on ten samples, it will need ten times those resources at once -- 1000 CPUs and 1TB of persistent disk. If the current resources quota is set at 24 CPUs and 100GB of PD and you want this workflow to run as quickly as possible, you will need to make a request for at least 1000 CPUs and 1TB of PD.
Google project quota
By default, Google Cloud only allows the creation of five projects per billing account. Since there is a 1:1 relationship between Terra workspaces and Google projects, this limits the number of workspaces per billing account to five (assuming you haven't also created any on GCP console). This number is insufficient for most Terra users, let alone groups with many users on the same billing account.
We recommend asking for a small (five) quota to avoid triggering an abuse investigation. When you have established a reputation with Google based on work in GCP (in Terra), you can ask for more quota later, as your project needs grow.
To request an increase in your project quota, see the step-by-step instructions below.
How do I request a quota increase?
Request an increased resource (compute/storage) quota
You can make a resource quota increase request by sending us a message through the Contact Us module in the Terra UI, or by emailing firstname.lastname@example.org.
Please be sure to provide the following information:
- Your Google Project ID, found on the right of the Dashboard page under Workspace Information.
- Which quota(s) you want to increase (i.e. CPUs, PDs, etc.)
- What you want your new quota(s) to be (see this section above for guidance on how to estimate)
- Which regions you want the increase applied to, if applicable (e.g. us-central-1, us-east-3, etc.)
- Terra uses us-central-1 by default in most cases. If you don't specify your region for workflow submissions, this is likely the region you want the quota(s) increased for.
- Rationale for increase (research purpose)
We will create a request to Google on your behalf. Depending on the quota and how much of an increase you are requesting, it may take 2-3 business days for Google to process and respond to the request.
Request an increased project quota (to be able to create more workspaces)
You will need to ask Google directly for an increase in your project quota (Terra cannot make the request on your behalf). Note that you need to be the GCP Billing account owner or admin to make this request.
1. Go to https://support.google.com/code/contact/billing_quota_increase.
2. Fill out the form details. See below for how to complete each field (in bold).
- Name: Your name
- Email addresses that will be used to create projects: use the Google Cloud Billing account owner's address
How many projects are being requested: 5
Why five projects?As a new billing account with little to no reputation, asking for an extremely high number of additional projects is a trigger for an abuse investigation. We recommend starting with the minimum quota that can meet your needs, and then asking for more quota later, as your project needs grow.
- What kind of services will these projects use? Paid services
- Billing account ( note that this field appears when you select the paid services radio button)
You can find the 18-digit GCP billing account ID here.
Any other things we need to be aware of to help us understand the request? Please input your own version of the text below in the input field
I use a platform (Terra at app.terra.bio) that creates Google projects via an api to handle data security and processing. In order to use Terra, I need sufficient quota so that projects created by the platform can be transferred over to my billing account.
Why it's important to use your own wordsAn exact copy and paste can flag an abuse investigation. Note that we are working with GCP on a way to expedite this process for genuine Terra users:
You will receive a confirmation email from Google within a day or two. They may ask for additional information. before granting the request.