Task <task_id> failed. The job was stopped before the command finished. PAPI error code 9. Please check the log file for more details: <link_to_gs_log_file.log>.
What it means
PAPI error code 9 is an indicator that a task has failed in the VM. The error code can manifest in numerous ways. This article describes the most common reasons for the error, and outlines steps you can take to remedy the issue.
In all scenarios, it is important to check the relevant .log file to understand the reason for the error. You can pull up a pop-up view of the log file from the Job Manager by clicking the “backend log” button (file with a cloud icon) on the right-hand side of the Error tab information.
You can alternatively select the “execution directory” icon (green folder icon) to navigate to the job id’s associated Google Bucket location, which includes the relevant .log file.
The last lines of the log will often report that the task attempted to delocalize a particular file, then report that the file does not exist (this is not always the case—see Scenario 4 below). This is a file that should have been generated, but failed to for one of the reasons detailed below.
2020/01/01 00:12:30 Starting delocalization.
2020/01/01 00:12:31 Delocalizing output <path_to_output_file> -> <workspace_bucket>
Required file output '<path/to/file>' does not exist.
Looking at the information a few lines up from the end of the file typically provides the relevant details about what went wrong.
Scenario 1 - “Killed” or “Out of memory”
If your task runs out of memory, or the task utilizes the bulk of the VM’s memory leaving little-to-none for delocalization, the .log file may abruptly end mid-process or show the message “Killed.”
If this scenario fits your situation, we recommend trying another run with twice the amount of memory.
Scenario 2 - Invalid commands
Depending on the software you are using, you may receive helpful information in the log when a command is incorrect.
If a command was not entered correctly, the expected output file will not be created, and a PAPI error code 9 will be thrown. Please note that an incorrect command is still possible even if there is no explicit feedback about this being the reason for the error—the error reporting will vary application-to-application.
Scenario 3 - Invalid arguments
Similar to Scenario 2, providing invalid arguments to commands can result in PAPI error code 9. Please check the .log file to see if there are any error messages related to a provided argument not being valid, found, etc.
Scenario 4 - AccessDeniedException
You may see a message in your .log that looks something like this:
AccessDeniedException: 403 firstname.lastname@example.org
does not have storage.objects.list access to <gs_bucket_location>.
This scenario is fairly straightforward—you will need access to the specified resource in order to localize it. See this article for help with accessing data from external Google buckets.
Scenario 5 - Unknown reason or no stderr
If neither your .log nor stderr appear to provide any information about why you received the error, you can also take a look at the “compute details” button (the ‘i’ icon) to see if there is any error information there. If you do not see it at the task level, you may need to click down to the shard level, or the shard attempt level.
If this error message is not helpful, it may be worth taking a look at your script to ensure that all outputs are configured properly. For example, if you write your script in such a way that it creates a file with one name, and then later outputs a file using a different name, PAPI will not be able to find it and you will receive a PAPI error code 9.