The Error
Task <task_id> failed. The job was stopped before the command finished. PAPI error code 9. Please check the log file for more details: <link_to_gs_log_file.log>.
What it means
PAPI error code 9 is an indicator that a task has failed in the VM. The error code can manifest in numerous ways.
How to troubleshoot the exact problem
In all scenarios, it is important to check the relevant .log file to understand the reason for the error.
How to find the log files
You can pull up a pop-up view of the log file from the Job Manager by clicking the “backend log” button (file with a cloud icon) on the right-hand side of the Error tab information.
Screenshot of log file
How to find the log files (option 2)
Alternatively, you can select the “execution directory” icon (green folder icon) to navigate to the job IDs associated Google Bucket location, which includes the relevant .log file.
How to interpret the log file
Often, the last lines of the log will report that the task attempted to delocalize a particular file, then report that the file does not exist (this is not always the case—see Scenario 4 below). This is a file that should have been generated, but failed for one of the reasons detailed below.
2020/01/01 00:12:30 Starting delocalization.
2020/01/01 00:12:31 Delocalizing output <path_to_output_file> -> <workspace_bucket>
Required file output '<path/to/file>' does not exist.
Frequently, looking at the information a few lines up from the end of the file provides the relevant details about what went wrong. Scroll down for the most common failure scenarios.
Scenario 1 - “Killed” or “Out of memory”
How to know if this is the problem
If your task runs out of memory - or the task utilizes the bulk of the VM’s memory leaving little-to-none for delocalization - the .log file may abruptly end mid-process or show the message “Killed.”
What to do
If this scenario fits your situation, we recommend trying another run with twice the amount of memory, or set up an out-of-memory retry (see instructions here).
Scenario 2 - Invalid commands
How to know if this is the problem
Depending on the software you're using, there may be helpful information in the log when a command is incorrect (see highlighted error message "'BWAMINdexImageCreator'
is not a valid command
" below).
If a command was not entered correctly, the expected output file will not be created, and a PAPI error code 9 will appear.
Please note that an incorrect command is still possible even if there is no explicit feedback about this being the reason for the error—the error reporting will vary application to application.
Scenario 3 - Invalid arguments
Similar to Scenario 2, providing invalid arguments to commands can result in PAPI error code 9.
What to do
Check the .log file to see if there are any error messages related to a provided argument not being valid, found, etc.
Scenario 4 - AccessDeniedException
Clues that this is the problem
You may see a message in your .log that looks something like this:
AccessDeniedException: 403 pet-012345678901234567890@broad-firecloud-dsde.iam.gserviceaccount.com does not have storage.objects.list access to <gs_bucket_location>.
What to do
This scenario is fairly straightforward—you will need access to the specified resource in order to localize it. See Best practices for accessing external resources for help with accessing data from external Google buckets and Error message: AccessDeniedException for help troubleshooting persistent errors of this category.
Scenario 5 - Unknown reason or no stderr
Clues that this is the problem
If neither your .log nor stderr provide information about why you received the error, you can take a look at the “compute details” button (the ‘i’ icon) to see if there is error information there. If you don't see it at the task level, you may need to click down to the shard level, or the shard attempt level.
What to do
If this error message is not helpful, it may be worth taking a look at your script to ensure that all outputs are configured properly. For example, if you write your script in such a way that it creates a file with one name, and then later outputs a file using a different name, PAPI will not be able to find it and you will receive a PAPI error code 9.