Common issue/error "cheat sheet"

Lin Betancourt
  • Updated

Learn how to identify and solve some common errors you may experience in Terra.

Web Portal: Internal Server Error or Gateway Timeout

What it means

Terra may be temporarily down. Please wait five minutes and refresh your browser page. If you still see this message, please let us know through the Terra Community Forum

Error message: PAPI error code 9

The Error

Task <task_id> failed. The job was stopped before the command finished. 
PAPI error code 9. 
Please check the log file for more details: <link_to_gs_log_file.log>.

What it means 

PAPI error code 9 is an indicator that a task has failed in the VM. The error code can manifest in numerous ways. 

What to do

There are a number of possible scenarios for this error message. See Error message: PAPI code 9 for detailed troubleshooting tips. 

Error message: PAPI error code 10

The error text

PAPI error code 10. 
The assigned worker has failed to complete the operation.

What it means 

In general, if your workflow fails and you see PAPI error code 10, the assigned worker has failed to complete the operation. One of two things has happened:

  1. Your machine actually failed.
  2. The machine was preempted in such a way that it looks like machine failure.

This error code 10 is sort of a catch-all error message. This error doesn't get retried.

Workarounds

  1. You can choose to turn preemptibles off, as this will take away the possibility of silent preemption failures.
  2. Use the attribute maxRetries to retry tasks that failed after your command fails with a non-zero return code, so that transient type failures get more attempts without manual intervention.
  3. PAPI error code 10 can also occur for reasons related to insufficient memory or disk space. If the problem does not appear to be related to preemptibles or retries, we recommend increasing the allocated memory to see if you get farther in your task or if it resolves the issue. If increasing the memory does not resolve the issue, you can try increasing the disk size.
    • Insufficient disk space is a common cause of the error when not allocating enough for the localization of the bams you are trying to process in the workflow.

What to look for

Sometimes your machine has truly crashed and has nothing to do with preemption. In such cases, it's worth looking into your logs to see if the job always crashes a certain number of minutes into the process, or always when copying inputs/outputs or when pulling docker images. Those patterns could help indicate why your tasks are failing with error code 10.

This error applies to Terra Billing projects created before 09/27/2021 at 4:00 pm EDT. 

You will see this error when you try to create a new Terra Billing project but have reached your Google quota.

You may need to request additional projects. See see Google quotas: What are they and how do I request more for step-by-step instructions.

Google Cloud Billing account does not appear in the Billing Management screen

What to do 

  1. Verify that terra-billing@firecloud.org is a billing account user on your Google Cloud Billing account. Refer to instructions to add Terra as a user on your Google Cloud Billing account in this article. After you add Terra as a billing account user, you should be able to create a Terra Billing project.
  2. Another reason you may not see your Google Cloud Billing account?  You may have enabled Terra billing permissions for a Google ID (e.g., Gmail address) that is different from your Terra user ID. This could happen if you have both a professional and a personal Google account, and are signed into your personal account (not the account used for your Terra user ID) in the browser where you're working.

How to switch Google accounts in Chrome

Please enable Terra billing permissions for the Google ID under which you created your Cloud Billing account. You may need to close your Terra session and/or log out of the incorrect Google account before re-enabling Terra billing permissions.

  1. In your Chrome browser, sign in to Google.
  2. On the top right (to the left of the three vertical dots), select your profile image or initial.

On the menu, select or add the account corresponding to your Terra user ID.

Expired Google Billing account trial

Google Billing accounts set up using a free trial will automatically expire after 60 days unless you upgrade to a paid account. If your Google Billing account expired, you will be unable to create a new Terra Billing Project.

To verify that your Google Billing account is active, log in to the Google Developers Console Billing page. You can select Show all accounts to display an expired Google Billing account. Google will prompt you to upgrade your free account to a full account backed by a credit card or bank account. 

If your Google Billing account is inactive, we suggest that you create a new Google Billing account.

“Resource has been exhausted” error

This error applies to Terra Billing projects created before 09/27/2021 at 4:00 pm EDT. 

You will see this error when you try to create a new Terra Billing project but have reached your Google quota. To learn more, see Google quotas: What are they and how do I request more

You may need to request additional projects from the Google Console as described in this Google Help article.

Billing 

Error creating Billing project- “The caller does not have permission”

This error message applies only to Terra Billing projects created before September 29, 2021 4:00 EDT.

The Error (screenshot)

When creating a Billing project, you may see a red exclamation point icon pop up with an error similar to the following:

9a33117d-ebbf-415f-bb6f-ad3db557f420.png

Error text

project <project_name> creation finished 
with errors: 
{"ResourceType":"cloudresourcemanage
r.v1.project","ResourceErrorCode":"403","
ResourceErrorMessage":
{"code":403,"message":"The caller does 
not have 
permission","status":"PERMISSION_DENI
ED","statusMessage":"Forbidden","reques
tPath":"https://cloudresourcemanager.g
oogleapis.com/v1/projects/my-
project","httpMethod":"GET"}} - code 
RESOURCE_ERROR

The Cause

At the time of Billing project creation, Terra also creates a Google project of the same name. Google requires that all Google projects have unique names. If you create a Billing project with a name matching a current Google project, that Google project already exists, but Terra does not have permission for it. This is why you see messages like The caller does not have permission and PERMISSION_DENIED.

The Solution

Create a Billing project with a name that is totally unique across Google. You can consider implementing one or more of the following suggestions:

  • Include the name of your lab, team, department, and/or institution
  • Prepend terra- to the beginning or append -terra to the end of the name

Didn't find your problem/solution? Try these resources

  • Bugs & Feature Requests: List of known bugs, limitations, and requested enhancements.
  • Forum: Ask the support team a question directly or see if other users have asked your question. The support team tries to respond to every question within one business day.

 

Was this article helpful?

0 out of 0 found this helpful

Have more questions? Submit a request

Comments

0 comments

Please sign in to leave a comment.