Learn some basic resources to help investigate failed workflows on Terra. This isn’t a guide for solving all errors, but it can help diagnose failed submissions.
Overview
Workflows fail for many different reasons. If you're lucky, you have the wrong inputs plugged into the configuration, which is typically fast to fail and fast to fix (to learn how to avoid these sorts of errors, see Workflow setup: Configuring inputs). More complex errors come from the workflow code: bugs, or limitations in your analysis software package. And, of course, you can fall victim to transient errors when something goes wrong on the Azure Cloud itself (even giants sometimes stumble). The information below will help you as you drill down to find the root cause of more complex errors.
At this point, we assume you have a workspace set up with a data table and workflow configuration loaded. You’ve launched your workflow, but your submission has failed. Don’t despair! Information in the Submissions History can help get your workflow up and running - as long as you know how to access and use the information.
If the error messages and log files seem like they're written in a foreign language, don't despair! Our team at frontline support is here to help. The messages and log files are full of useful information that they can use to tease out the root of your problems. Contact them directly at support@terra.bio.
Accessing troubleshooting tools (logs)
Each submission you make to Terra consists of one or more workflows, each of which contains one or more tasks. You can access increasingly more granular information - starting with submission-level status and digging into workflow-level information (within a submission) and tasks-level (within each workflow).
interest.
1. Click the Submission History tab at the left side of the Workflows page to find your submission.
2. Click on the submission name to go to the submission detail page.
3. From the submission detail page, you can view the inputs, outputs, and workflow execution log for each individual workflow in your submission. Errors in the execution log can indicate problems parsing your WDL or Terra systems issues.
4. Click on the name in the Sample ID column (at left in screenshot above) for more granular information about the workflow.
5. In the Workflows Details page, you'll find helpful troubleshooting information at the top right (circled below). The execution log is the same as the one linked from the Submission Details page, and may indicate WDL parsing or Terra Systems issues.
The table at the bottom of the Workflow Details page contains one row per task in the workflow. Each task has its own inputs, outputs, and logs which you can access via the indicated links. These task inputs, outputs, and logs are not the same as the workflow ones (from step 3).
Clicking the Logs link under Task Data (with the arrow pointing to it in the bottom right of the screenshot above) surfaces the logs generated for that task. There are multiple types of logs generated for each task at different stages of its execution, and certain logs may not exist if the task did not complete successfully.
What is in the different logs?
Roughly speaking, the Task and Exec logs show the output of the code executed by your task, the Download logs contain information about downloading the task input and docker image, and the Upload logs contain information about uploading the results of your task.
Troubleshooting with the call caching debug wizard
If you have a workflow you believe should have benefited from call caching, you can run the Call Caching Debug Wizard found in the Workflow Dashboard for the workflow of interest.
1. Click the Submission History tab at the left side of the Workflows page to find your submission.
2. Click on the submission name to go to the submission detail page.
3. In the Sample ID column of the Submission Details page, click on the input for the run you think should have benefited from call caching.
4. In the Workflows Details page, click on the magnifying glass icon under Call Caching Result (right-hand side).
How to abort a workflow
There are many reasons you may want to abort a running workflow. Perhaps you included a wrong input or have a failed task that you debugged and now want to abort, make the edits, and re-submit.
In the Submission History page, click on the three-dot action icon to see the abort workflow option.