Need Help?

Search our documentation and community forum

Terra is a cloud-native platform for biomedical researchers to access data, run analysis tools, and collaborate.
Terra powers important scientific projects like FireCloud, AnVIL, and BioData Catalyst. Learn more.

Ongoing workflow failures with error “key not found”

Featured

Comments

8 comments

  • Avatar
    Sushma Chaluvadi

    Hello All,

     

    The 10/15 Docker Hub outage had an impact on Cromwell that extended into 10/16 and 10/17. This impact has now been resolved.

     

    Please confirm if you are able to successfully run Workflows.

    1
    Comment actions Permalink
  • Avatar
    Adam Nichols

    Hi James - thank you for the feedback, and apologies for the inconvenience.

    1
    Comment actions Permalink
  • Avatar
    James Gatter

    Working fine for me! Thanks!

    0
    Comment actions Permalink
  • Avatar
    Adam Haber

    Hi, still getting errors (see below)
    done
     true

     

    error : code 2

     

    details Array[0] [] message "Execution failed: pulling image: docker pull: running ["docker" "pull" "cellranger:3.0.2"]: exit status 1 (standard error: "Error response from daemon: pull access denied for cellranger, repository does not exist or may require 'docker login': denied: requested access to the resource is denied\n")"

     

    metadata : @type "type.googleapis.com/google.genomics.v2alpha1.Metadata" createTime "2019-10-18T14:17:15.849863Z" endTime "2019-10-18T14:19:04.808435277Z"

     

    events Array[6] [{"description":"Worker released","details":{"@type":"type.googleapis.com/google.genomics.v2alpha1.WorkerReleasedEvent","instance":"google-pipelines-worker-239222145a5c2a5edd4aa7e6d1097164","zone":"us-west1-b"},"timestamp":"2019-10-18T14:19:04.808435277Z"},{"description":"Execution failed: pulling image: docker pull: running [\"docker\" \"pull\" \"cellranger:3.0.2\"]: exit status 1 (standard error: \"Error response from daemon: pull access denied for cellranger, repository does not exist or may require 'docker login': denied: requested access to the resource is denied\\n\")","details":{"@type":"type.googleapis.com/google.genomics.v2alpha1.FailedEvent","cause":"Execution failed: pulling image: docker pull: running [\"docker\" \"pull\" \"cellranger:3.0.2\"]: exit status 1 (standard error: \"Error response from daemon: pull access denied for cellranger, repository does not exist or may require 'docker login': denied: requested access to the resource is denied\\n\")","code":"UNKNOWN"},"timestamp":"2019-10-18T14:19:03.532399522Z"},{"description":"Started pulling \"cellranger:3.0.2\"","details":{"@type":"type.googleapis.com/google.genomics.v2alpha1.PullStartedEvent","imageUri":"cellranger:3.0.2"},"timestamp":"2019-10-18T14:18:25.017346022Z"},{"description":"Stopped pulling \"gcr.io/google.com/cloudsdktool/cloud-sdk:264.0.0-slim\"","details":{"@type":"type.googleapis.com/google.genomics.v2alpha1.PullStoppedEvent","imageUri":"gcr.io/google.com/cloudsdktool/cloud-sdk:264.0.0-slim"},"timestamp":"2019-10-18T14:18:23.226581956Z"},{"description":"Started pulling \"gcr.io/google.com/cloudsdktool/cloud-sdk:264.0.0-slim\"","details":{"@type":"type.googleapis.com/google.genomics.v2alpha1.PullStartedEvent","imageUri":"gcr.io/google.com/cloudsdktool/cloud-sdk:264.0.0-slim"},"timestamp":"2019-10-18T14:18:02.126281492Z"},{"description":"Worker \"google-pipelines-worker-239222145a5c2a5edd4aa7e6d1097164\" assigned in \"us-west1-b\"","details":{"@type":"type.googleapis.com/google.genomics.v2alpha1.WorkerAssignedEvent","instance":"google-pipelines-worker-239222145a5c2a5edd4aa7e6d1097164","machineType":"custom-1-2048","zone":"us-west1-b"},"timestamp":"2019-10-18T14:17:16.761218289Z"}]

     

    labels Object {"cromwell-workflow-id":"cromwell-0f658af3-51d4-4ada-a5ee-b0a51bd8c02a","terra-submission-id":"terra-16957d9a-9f58-4c2d-aed1-175cd42bc496","wdl-task-name":"generate-bcl-csv"}

    0
    Comment actions Permalink
  • Avatar
    James Gatter

    Try fixing your docker parameter in the WDL:

    docker: "cumulusprod/cellranger:3.0.2"

    0
    Comment actions Permalink
  • Avatar
    Adam Nichols

    Hi Adam - thanks for your post.

    While it definitely looks like something is going wrong with your workflow, the "key not found" error is not present so I would suggest that you make a new thread.

    0
    Comment actions Permalink
  • Avatar
    Charlie Hatton

    I think I may be having a problem related to this issue. On 10/16 and 10/17, I started three jobs (each running a workflow on each of 5 inputs) and got a "key not found" error and Done/Failed status for each. I only realized later that I've been incurring high computing costs since then (with no other analyses running), and tracked it to 15 Compute Engine VMs that appear to have been running since I started those jobs. I can see these in Google's console, but evidently I don't have permission to delete them (though I own the workspace that created them and the job history shows each process as "Failed").

    Viewing the Job Manager for each job, I see that some steps in each job do appear to still be running -- but I see no way to kill those from the interface. What would be the best way to abort these jobs/VMs? Thanks in advance for any advice.

    0
    Comment actions Permalink
  • Avatar
    Adam Nichols

    Hi Charlie,

    If you can believe this, I received the Zendesk email notification for your message at 12:35 PM on January 3, 2020.

    I'm guessing you figured out the issue by now, but I wanted to apologize for the lack of response.

    Adam

    0
    Comment actions Permalink

Please sign in to leave a comment.

Powered by Zendesk