Sporadic problems pulling Docker images from quay.io causing issues with workflows

Post author
Samantha (she/her)

We've received several reports of users receiving the following error when running workflows that pull Docker images from quay.io.

The job was stopped before the command finished. PAPI error code 9. Execution failed: generic::failed_precondition: pulling image: docker pull: running ["docker" "pull" "quay.io/staphb/snp-dists@sha256:a8ee1104259dc8b6707c20bbb0ad7128b280ba58b2a5f89b0090f6f9187f6c96"]: exit status 1 (standard error: "Error response from daemon: manifest for quay.io/staphb/snp-dists@sha256:a8ee1104259dc8b6707c20bbb0ad7128b280ba58b2a5f89b0090f6f9187f6c96 not found: manifest unknown: manifest unknown\n")

 

Our engineers are currently investigating this issue and working on a fix. Please contact Terra Support if you run into the error.

Comments

4 comments

  • Comment author
    Kyler Anderson

    Hello,

    I have two plink2 workflows, single tasks, that use a quay.io container (quay.io/biocontainers/plink2). Since plink2 is in alpha, I specified the tag for alpha 5. I've run these workflows multiple times and had no issue with the docker.

    When I added these tasks to a longer workflow, Terra reports the aforementioned error. The longer workflow has two other preceding tasks, one using quay.io/biocontainers/bcftools, and another using a dockerhub container mixing plink and plink2. The error occurs whether or not a cached result is used for the preceding tasks.

    Is there maybe some issue with using multiple docker sources in a single workflow?

    0
  • Comment author
    Samantha (she/her)

    Hi Kyler Anderson,

    I don't believe the use of multiple docker sources in a single workflow would be the cause for that error. If you are able to share your workspace with us, we'll be happy to take a closer look. 

    The Share option is in the three-dots menu at the top-right.

    1. Toggle the "Share with support" button to "Yes"
    2. Click Save

    Please provide us with

    1. A link to your workspace
    2. The relevant submission ID
    3. The relevant workflow ID

    Best,

    Samantha

    Upcoming Terra Support out-of-office: December 22-January 1

    0
  • Comment author
    Kyler Anderson

    Hello,

    Sorry to message during out-of-office. I've shared the workspace to support. The workspace is https://app.terra.bio/#workspaces/DFCI-aldubayan-lab/KAnderson_Multi-Ancestry-PRS_WES_1KG/, submission id c594d7df-0e1a-40db-83cf-08b6d6e5e74c and workflow PCA_PGS_OLS.

    I've since changed the dockers for the tasks to all be from docker hub and the workflow runs fine, though it would still be nice to know what else may have been wrong here. The workflow version in question is version 11. The first optional task does not run as configured, so that does not make a quay.io request.

    Thank you,

    Kyler

    0
  • Comment author
    Curtis Kapsak

    Kyler Anderson I don't think you've done anything wrong, I think it's just that quay as a container repository is unreliable. The Broad team has said that quay has the highest error/failure rate of any container repo. I reported the error shared via the original post above.

    My team ended up switching to using google container registry for almost all of our docker image hosting and generally try to avoid quay. We've had much more success using GCR. Dockerhub has less errors than quay, but some containers have rate-limiting on the number of pulls you can do per minute.

    0

Please sign in to leave a comment.