Docker Hub login error on scattered task
I have a workflow with a scattered task that scatters to approximately 50 instances. It appears that in three of those instances, the images failed to pull from Docker Hub. They are all using the same image identified by a specific sha256 instead of a tag that may change, so in theory, they shouldn't need to be pulled 50 times, but it seems something might be getting rate limited somewhere.
The tasks that error out say:
Task assoc_agg.assoc_aggregate:28:1 failed. The job was stopped before the command finished. PAPI error code 14. Execution failed: generic::unavailable: pulling image: docker login: retry budget exhausted (10 attempts): running ["docker" "login" "-u" "firecloud" "--password-stdin"]: exec: already started (standard error: "Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\n")
The other tasks are returning 0 as expected.
Added the firecloud support group. It's submission ID 5fb05fb8-6b69-43fe-8ad9-9542694715ae and in a workspace called "assoc PUBLIC". https://app.terra.bio/#workspaces/anvil-stage-demo/assoc%20PUBLIC/job_history/5fb05fb8-6b69-43fe-8ad9-9542694715ae
Comments
4 comments
Hi Aisling,
Thanks for writing in. We are going to take a look at your inquiry and get back to you as soon as we can.
Best,
Emil
Hi Aisling,
Could you please try running your job again with call caching enabled and let us know if you still receive the same errors?
Best,
Emil
I haven't been able to replicate the error, no. Perhaps it was a temporary limit?
Hi Aisling,
I'm glad to hear it seems to be working. Yes, it sounds like the limit you reached was only temporary.
If we can help with anything else, please let us know.
Kind regards,
Emil
Please sign in to leave a comment.