PAPI error code 2 failures

Post author
David Heiman

I'm getting errors with very little detail for a task I've recently run on much larger files:

Task Funcotator.Funcotate:NA:1 failed. The job was stopped before the command finished. PAPI error code 2. Execution failed: generic::unknown: action 3: waiting for container: starting container: running ["docker" "start" "-ai" "3319a676186064b1463bf519fa5282d0fad536127763fcae14382f9c4bcb8f24"]: exit status 125

The data is protected, so I can't give workspace access. Several failed workflow IDs are:

Logfiles simply stop in the middle of tool execution with no errors.

Comments

7 comments

  • Comment author
    Jason Cerrato
    • Edited
    • Official comment

    Hi David et. al.,

    Google has confirmed this to be a problem on their end and they are looking to release a fix within the week. I will post all updates to the situation in our Known Issues thread here: https://support.terra.bio/hc/en-us/community/posts/360062450471-PAPI-Error-Code-2

    Kind regards,

    Jason

  • Comment author
    Jason Cerrato

    Hi David,

    Thanks for writing in. I'll take a look at these workflows as soon as I am able and get back to you!

    Kind regards,

    Jason

    0
  • Comment author
    Lucas Taniguti

    Hello!

    I'm having exactly the same failure:

    "Task UnmappedBamToAlignedBam.SortSampleBam:NA:1 failed. The job was stopped before the command finished. PAPI error code 2. Execution failed: generic::unknown: action 1: waiting for container: starting container: running ["docker" "start" "-ai" "5cf4a943e0048f5cd7089551759e246a8c0413a565eb97dce1c847188b6eeb3f"]: exit status 125"

    I'm using my own cromwell-server, so now I think it is related to PAPIv2, as once described here.

    0
  • Comment author
    Liudmila Elagina

    I am having the same issue with PAPI error code 2

    0
  • Comment author
    Jason Cerrato

    Hi all,

    Thank you for the additional reports. We have reached out to Google to investigate as the number of reports for PAPI error code 2 have greatly spiked over the course of the past month, and especially within this past week. I will keep you updated as we hear more.

    Kind regards,

    Jason

    0
  • Comment author
    David Heiman

    Thanks Jason,

    This really smells like PAPI misreporting preemption, especially as there are zero errors in the log, rather they seem to simple be cut off mid-execution.

    Best,

    David

    0
  • Comment author
    Jason Cerrato

    Hi all,

    Google has reported that the fix for this PAPI Error Code 2 has finished rolling out. It should no longer occur for new pipelines going forward but those in flight might still encounter it. If you submit a brand new workflow after receipt of this message and still run into the issue, please let us know.

    Kind regards,

    Jason

    0

Please sign in to leave a comment.