Issue when docker pulls an image: "no space left on device"

Post author
Jean Monlong

Hi,

I'm experiencing a new error on a task of one of our workflows. The error message says:

Task GiraffeDeepVariantFromGAM.runDeepVariantCallVariants:23:6 failed. The job was stopped before the command finished. PAPI error code 9. Execution failed: generic::failed_precondition: pulling image: docker pull: running ["docker" "pull" "google/deepvariant:1.3.0-gpu"]: exit status 1 (standard error: "write /var/lib/docker/tmp/GetImageBlob1540024992: no space left on device\n")

The docker container exists and, from the message, it's not about a network issue, permissions, or dockerhub quotas. Also, this same workflow, using this exact same image, used to work before the holidays.

I tried twice and got the same error. I also get the error consistently for all tasks that use this image.

The second time I allocated more disk space (+50Gb) and got the same, so I don't think it's about the instance disk space but something around the docker image management system.

I read that the default image size limit is 10 Gb and that larger images could cause a similar-looking issue. This image is 7.4 Gb though. Is there any chance the image size limit got decreased recently?

The most promising solutions that I found online suggest running 'docker system prune'.

Anyone else got this issue?

Jean

 

Comments

4 comments

  • Comment author
    Samantha (she/her)

    Hi Jean Monlong,

    Thank you for writing in about this issue. Can you share the workspace where you are seeing this issue with Terra Support by clicking the Share button in your workspace? The Share option is in the three-dots menu at the top-right.

    1. Toggle the "Share with support" button to "Yes"
    2. Click Save

    Please provide us with

    1. A link to your workspace
    2. The relevant submission ID
    3. The relevant workflow ID

    We’ll be happy to take a closer look as soon as we can!

    Kind regards,

    Samantha

    0
  • Comment author
    Jean Monlong

    Thanks for the help!

    Here is the info:

    - workspace: https://app.terra.bio/#workspaces/firecloud-cgl/hprc-evaluation
    - submission ID: 63f1e57a-5e9f-4413-b2be-53822fc1b8e5
    - workflow ID: 46a167c5-d263-4779-a277-d1cb5999356d

    Jean

    0
  • Comment author
    Samantha (she/her)

    Hi Jean Monlong,

    I think you may have been impacted by a service incident discovered yesterday, Service Incident - January 9, 2023 (Workflows), which has since been resolved. Can you try resubmitting your workflow and see if you still encounter any issues/errors?

    Best,

    Samantha

    0
  • Comment author
    Jean Monlong

    Hi Samantha,

    That's good news. I've resubmitted my workflow, and it looks like it's working now. Some of the problematic tasks have finished. The others are running. No sign of the issue anymore.

    Best,

    Jean

    0

Please sign in to leave a comment.