Common issue/error "cheat sheet"

Learn how to identify and solve some common errors you may experience in Terra.

Web Portal: Internal Server Error or Gateway Timeout

What it means

Terra may be temporarily down.

What to do

Please wait five minutes and refresh your browser page. If you still see this message, please let us know through the Terra Community Forum. 

Error message: PAPI error code 9

The Error text

Task <task_id> failed. The job was stopped before the command finished. 
PAPI error code 9. 
Please check the log file for more details: <link_to_gs_log_file.log>.

What it means 

PAPI error code 9 is an indicator that a workflow task has failed in the VM. The error can result from several possible problems, which can make it difficult to diagnose the root cause. 

What to do

There are a number of possible scenarios for this error message. See Error message: PAPI code 9 for detailed troubleshooting tips. 

Error message: PAPI error code 10

The error text

PAPI error code 10. 
The assigned worker has failed to complete the operation.

What it means 

In general, if your workflow fails and you see PAPI error code 10, the assigned worker has failed to complete the operation. One of two things has happened:

1. Your virtual machine actually failed.
2. The virtual machine was preempted in such a way that it looks like machine failure.

This error code 10 is sort of a catch-all error message. This error doesn't get retried.

Workarounds

1. You can choose to turn preemptible VMs off, as this will take away the possibility of silent preemption failures.
2. Use the attribute maxRetries to retry tasks that failed after your command fails with a non-zero return code, so that transient type failures get more attempts without manual intervention.
3. PAPI error code 10 can also occur for reasons related to insufficient memory or disk space. If the problem does not appear to be related to preemptibles or retries, we recommend increasing the allocated memory to see if you get farther in your task or if it resolves the issue. If increasing the memory does not resolve the issue, you can try increasing the disk size.
   • Insufficient disk space is a common cause of the error when not allocating enough for the localization of the bams you are trying to process in the workflow.

What to look for

Sometimes your machine has truly crashed and has nothing to do with preemption. In such cases, it's worth looking into your logs to see if the job always crashes a certain number of minutes into the process, or always when copying inputs/outputs or when pulling docker images. Those patterns could help indicate why your tasks are failing with error code 10.

Error message: AccessDeniedException

The error text

The error might look something like this:

AccessDeniedException: 403 serviceaccountname@serviceaccountdomain.iam.gserviceaccount.com does not have storage.objects.list access to the Google Cloud Storage bucket. Permission ‘storage.objects.list’ denied on resource (or it may not exist).

What's happening

You might encounter this error when running a workflow that uses a pet service account to access data stored outside of your Terra workspace.

This means that an account that needs to access your data for an analysis doesn’t have the necessary access permissions. This could happen for several reasons:

1. You never gave the account permissions to access the Google Cloud bucket where the data is stored.
2. The account isn’t registered for Terra.
3. You added the account to the Google Cloud bucket, but the underlying Google id has since changed.

What to do

1. Check whether the account has the necessary permissions for the Google Cloud bucket--and adjust the permissions if necessary--by following the instructions in How to access external Google Cloud resources. 

• • Note: you must have an owner or admin role on the Google Cloud bucket in order to edit the bucket's permissions.

2. Register the account in Terra. If the error persists after verifying that the account has permission to access the Google bucket, it's possible that the account has not been registered in Terra. 

• • For service accounts, follow the instructions to register a service account for use in Terra. 
  • For individual users who need to access a bucket, follow the instructions in How to register on Terra.

3. Start over with a new service account. If the error persists after you've completed steps 1 and 2, the account may be attached to an orphaned Terra account. This can happen if a service account was deleted and then re-created (for example, when someone left an institution). In this case, the best solution is to create a new service account and add it to the Google bucket instead. For step-by-step instructions, see How to use a service account in Terra and Best practices for using service accounts in Terra.

Google Cloud Billing account does not appear in the Billing menu

What to do 

1. Verify that terra-billing@firecloud.org is a billing account user on your Google Cloud Billing account. Refer to the instructions in How to set up billing in Terra to add Terra as a user on your Google Cloud Billing account. After doing this, you should be able to create a Terra Billing project and link it to your Google Cloud Billing account.
2. Another reason you may not see your Google Cloud Billing account? You may have enabled Terra billing permissions for a Google ID (e.g., Gmail address) that is different from your Terra user ID. This could happen if you have both a professional and a personal Google account, and are signed into your personal account (not the account used for your Terra user ID) in the browser where you're working.

How to switch Google accounts in Chrome

Please enable Terra billing permissions for the Google ID under which you created your Cloud Billing account. You may need to close your Terra session and/or log out of the incorrect Google account before re-enabling Terra billing permissions.

1. In your Chrome browser, sign in to Google.

2. On the top right (to the left of the three vertical dots), select your profile image or initial.

3. On the menu, select or add the account corresponding to your Terra user ID.

Expired Google Billing account trial

What's happening

Google Billing accounts set up using a free trial will automatically expire after 60 days unless you upgrade to a paid account. If your Google Billing account expired, you will be unable to create a new Terra Billing Project.

What to do

To verify that your Google Billing account is active, log in to the Google Developers Console Billing page. You can select Show all accounts to display an expired Google Billing account. Google will prompt you to upgrade your free account to a full account backed by a credit card or bank account. 

If your Google Billing account is inactive, we suggest that you create a new Google Billing account.

Didn't find your problem/solution? Try these resources

• Bugs & Feature Requests: List of known bugs, limitations, and requested enhancements.
• Forum: Ask the support team a question directly or see if other users have asked your question. The support team tries to respond to every question within one business day.