Docker authentication issues
Hi Terra team,
I've been running workflows today with no issues until ~30 minutes ago, when all tasks started failing with PAPI error code 7 and error messages like this:
The job was stopped before the command finished. PAPI error code 7. Execution failed: generic::permission_denied: pulling image: docker login: running ["docker" "login" "-u" "firecloud" "--password-stdin"]: exit status 1 (standard error: "Error response from daemon: Get \"https://registry-1.docker.io/v2/\": unauthorized: incorrect username or password\n")
I have successfully used the same docker image many times earlier today with no issues, so I don't think it's anything wrong with the docker image itself.
Any idea what could be going on? Is this due to maintenance and, if so, do you happen to know when we might expect this issue to clear up?
Thanks!
Ryan
Comments
4 comments
Hi Team,
I think I am encountering a related or at least similar issues. Some of my jobs randomly failed and it says
Hi Ryan and Ruolin,
Thank you both for writing in! As it turns out, there was a Service Incident last night involving Docker containers used with workflows. The issue started yesterday around 4PM and ended around 8:20PM. If either of you were seeing issues during that window, then I would suggest trying to run the workflows again and see if the issue has been resolved.
Please let me know if you have any questions.
Best,
Josh
Hi Josh,
Thank you for the update! This is useful to know.
For what it's worth, I can confirm that my resubmitted workflows have not experienced any of these errors so far this morning, so the issue seems to have stabilized for now at least.
Thanks,
Ryan
Hi Ryan,
That's great news! I'm glad the issue has been resolved for you for now.
Please let me know if you have any questions.
Best,
Josh
Please sign in to leave a comment.