Workflows can fail for a lot of different reasons. If you're lucky, you just have the wrong inputs plugged into the configuration, which is typically fast to fail and fast to fix (to learn more about how to avoid these sorts of errors, see this article). More complex errors come from the workflow code: bugs, or limitations in the analysis software package you're calling on. And of course you can fall victim to transient errors when something goes wrong on the Google cloud itself (even giants sometimes stumble).

In this document we'll go over some basic strategies to investigate failed workflows on Terra. This isn’t a guide for solving all errors but a doc to help diagnose failed submissions. The information below will help you drill down to find the root cause of these more complex errors, so you can be up and running quickly. Descriptions of more complicated errors are always welcomed on the Terra Forum where our team is happy to help.
 

Contents

1. High-level status on submissions
2. Workflow-level errors 
3. More detail, and metadata, on each task
4. sterr and stout logs
5. Summary 

At this point we will assume that you have a workspace setup with a data table and workflow configuration loaded. You’ve launched your workflow, but your submission has failed. Don’t despair! There is some information you can gather that will be helpful in getting your workflow up and running.

1. High-level status on submissions

You can find high-level status on the success or failure of your submissions, along with a few columns of metadata, in the Job History tab. 

When you click on Job History, you’ll see a list of all the workflow or workflows within the submission. For details about a particular submission, click on the link in the "Submission" column. 

2. Workflow-level status

That will take you to this screen, which contains further details about workflows within the submission and their status (failed, queued, etc.). To display any error messages, click “View” located to the the left of Failures.
The message listed under Failures isn't always short and sweet, and, if interpreted incorrectly, will lead you down the wrong debugging path. Instead, use the message to identify and investigate which task failed.

3. Job Manager: checking log files for individual tasks

If it's not immediately obvious what failed, the best sources of information are log files, which you can access directly in the Job Manager interface.  Click View on the task that failed to gain access to three very useful files for debugging: stderr, stdout, JES log. These files are generated by Cromwell when executing any task and are placed in the task's folder along with its output. In Terra we add quick links to these files to make troubleshooting easier.

Here you can preview the tail end of your logs by clicking on the icons (shown here in the card view):

Shown here in the list view:

Click on the icons to preview the log, with an option to expand to the full log.

4. Log functions

In addition to the error message (in the red triangle), the log functions, going from left to right icons, are defined below:

• Backend (Cromwell) log - A step-by-step report of actions during the execution of the task. These details include information about Docker setup, localization (the step of copying files from your google bucket into the Docker container), stdout from tools run within the command block of the task, and finally, the delocalization and Docker shutdown steps.
  
  
• Execution directory - Clicking on this icon will redirect you to the exact folder/directory where you can find your stderr, stdout, and backend logs in the Google cloud storage bucket. From there, you can open those files to view their contents or you can download them. If your task generates outputs, this directory is where you can find them as well. 
  
  
  • Standard Out (stdout): A file containing log outputs generated by commands in the task. Not all commands generate log outputs and so this file may be empty.
  • Standard Error (stderr): A file containing error messages produced by the commands executed in the task. A good place to start for a failed task, as many common task level errors are indicated in the stderr file.
  • JES log: A log file tracking the events that occurred in performing the task such as downloading docker, localizing files, etc.. Occasionally a workflow will fail without a stderr and stdout files, leaving you with only a JES log. More on this on the next section.
    
    
• Compute details - A report of the actions taken on the Google side by the Pipelines API (PAPI) to execute the task, including things like the request we send to Google, the exact events as tracked by Google, timestamps of what happened when, and if there were errors. This is where you would find information about errors that are unrelated to the WDL code or configuration. This information is great for debugging when failures happen before a task starts or after a task completes. 
  
  
• TaskName.log (formerly JESLog) - Often this log is difficult to decipher, so its better to proceed to the other log files. However, in some cases your submitted job will fail with no stderr or stdout files. In these cases you’ll have to suck it up and unravel the meaning behind the TaskName.log messages.

  Below we’ve provided some common errors and their possible meaning as aid. There isn’t a solution for all of them, so feel free to post your error on the Terra forum so the team could help you through the message.

  • PAPI (JES) Error 10 Message 15

    • The provided memory or disk space isn’t sufficient to load your input files and Docker container, thus the task instance exited abruptly. Increasing the disk space should avoid the error message.

  • PAPI (JES) Error 10 Message 14

    • This error is associated with preemptible VMs, but we've observed that it occasionally applies when non-preemptible machines fail. Retry the workflow. If you see this particular error on the same task repeatedly, we recommend using the runtime attribute maxRetries to retry transient failures.

  • PAPI Error 10 Message 13

    • The submission has been aborted with many possible reasons, sometimes it's due to preemption.

  • PAPI Error 5: Message 10

    • This message does not mean that the files failed to delocalize, but rather that something went wrong upstream and the files it is trying to delocalize were never generated. Check the stderr or stdout for possible reasons why the files were not generated.

  • PAPI Error 2

    • The submission could not start possibly due to preemption. There may be a description indicating the reason in the error message.

  • Cannot find credentials for RawlsUser(RawlsUserSubjectId(******),RawlsUserEmail(******))
    
    Refresh your browser window, and you will see a yellow banner at the top of the page that says "Your offline credentials are missing or out-of-date. Your workflows may not run correctly until they have been refreshed. Refresh now...", click the link to "Refresh now..." and follow the prompts. This will update your credentials in the system and should make this error message go away. This prompt is needed in order to maintain compliance with certain security standards required for hosting sensitive data.

Summary

• The error examples discussed above are pretty simple and very common. Be sure to check your inputs before launching them to avoid failed workflows.
  
  
• Remember that when troubleshooting, you should automatically head towards to the Monitor tab and check stdout,stderr, and JES log for your failed task.
  
  
• In cases where there isn’t a stdout or stderr file, use the common JES log and the message explanations in this document to help you solve the problem.
  
  
• Of course, if you are having any trouble with Terra troubleshooting, you can ask your question on the Terra forum.