Error message: PAPI error code 9

Jason Cerrato
  • Updated

The Error

Task <task_id> failed. The job was stopped before the command finished. 
PAPI error code 9. 
Please check the log file for more details: <link_to_gs_log_file.log>.

What it means 

PAPI error code 9 is an indicator that a task has failed in the VM. The error code can manifest in numerous ways. 

How to troubleshoot the exact problem

In all scenarios, it is important to check the relevant .log file to understand the reason for the error.

How to find the log files

You can pull up a pop-up view of the log file from the Job Manager by clicking the “backend log” button (file with a cloud icon) on the right-hand side of the Error tab information.

screenshot_1.png

Screenshot of log file

screenshot_2.png

How to find the log files (option 2)

Alternatively, you can select the “execution directory” icon (green folder icon) to navigate to the job IDs associated Google Bucket location, which includes the relevant .log file.

screenshot_3.png

How to interpret the log file

Often, the last lines of the log will report that the task attempted to delocalize a particular file, then report that the file does not exist (this is not always the case—see Scenario 4 below). This is a file that should have been generated, but failed for one of the reasons detailed below. 

2020/01/01 00:12:30 Starting delocalization.
2020/01/01 00:12:31 Delocalizing output <path_to_output_file> -> <workspace_bucket>
Required file output '<path/to/file>' does not exist.

Frequently, looking at the information a few lines up from the end of the file provides the relevant details about what went wrong. Scroll down for the most common failure scenarios. 

Scenario 1 - “Killed” or “Out of memory”

How to know if this is the problem

If your task runs out of memory - or the task utilizes the bulk of the VM’s memory leaving little-to-none for delocalization - the .log file may abruptly end mid-process or show the message “Killed.”

What to do

If this scenario fits your situation, we recommend trying another run with twice the amount of memory, or set up an out-of-memory retry (see instructions here). 

Scenario 2 - Invalid commands

How to know if this is the problem

Depending on the software you're using, there may be helpful information in the log when a command is incorrect (see highlighted error message "'BWAMINdexImageCreator' is not a valid command" below).

screenshot_4.png

If a command was not entered correctly, the expected output file will not be created, and a PAPI error code 9 will appear.

Please note that an incorrect command is still possible even if there is no explicit feedback about this being the reason for the error—the error reporting will vary application to application.

Scenario 3 - Invalid arguments

Similar to Scenario 2, providing invalid arguments to commands can result in PAPI error code 9.

What to do

Check the .log file to see if there are any error messages related to a provided argument not being valid, found, etc.

Scenario 4 - AccessDeniedException

Clues that this is the problem

You may see a message in your .log that looks something like this:

  AccessDeniedException: 403 pet-012345678901234567890@broad-firecloud-dsde.iam.gserviceaccount.com
  does not have storage.objects.list access to <gs_bucket_location>.

What to do

This scenario is fairly straightforward—you will need access to the specified resource in order to localize it. See Best practices for accessing external resources for help with accessing data from external Google buckets and Error message: AccessDeniedException for help troubleshooting persistent errors of this category.

Scenario 5 - Unknown reason or no stderr

Clues that this is the problem

If neither your .log nor stderr provide information about why you received the error, you can take a look at the “compute details” button (the ‘i’ icon) to see if there is error information there. If you don't see it at the task level, you may need to click down to the shard level, or the shard attempt level.

screenshot_5.png

What to do

If this error message is not helpful, it may be worth taking a look at your script to ensure that all outputs are configured properly. For example, if you write your script in such a way that it creates a file with one name, and then later outputs a file using a different name, PAPI will not be able to find it and you will receive a PAPI error code 9.

Was this article helpful?

1 out of 2 found this helpful

Comments

0 comments

Please sign in to leave a comment.