Description of problem: docker-compose pull worked with private repos in F25 but is broken in F26. Version-Release number of selected component (if applicable): 1.14.0, build c7bdf9e How reproducible: always Steps to Reproduce: 1. Create test.yml file with private images 2. docker-compose -f test.yml pull 3. Observed failed download Actual results: Download fails Expected results: Download works as it did with F25 Additional info: docker pull works just fine.
I added an upstream bug that I found related to this issue.
Unfortunately I wasn't able to reproduce, therefore closing. (more detailed info in upstream issue: https://github.com/docker/compose/issues/4807#issuecomment-321770563)
(In reply to Tomas Tomecek from comment #2) > Unfortunately I wasn't able to reproduce, therefore closing. > > (more detailed info in upstream issue: > https://github.com/docker/compose/issues/4807#issuecomment-321770563) Are you using the packages that come with Fedora? Or the ones from Docker? The output in yours doesn't show trying to pull from the Fedora repos like in my output, so it appears that there's some difference between the config your using and the config that I'm seeing after installing the packages from Fedora.
I am using packages that come from Fedora. It's true that I edited /etc/sysconfig/docker, this is how mine looks: OPTIONS='-l debug --log-driver=journald' if [ -z "${DOCKER_CERT_PATH}" ]; then DOCKER_CERT_PATH=/etc/docker fi INSECURE_REGISTRY='--insecure-registry 172.30.0.0/16' I have commented out the ADD_REGISTRY value: # ADD_REGISTRY='--add-registry registry.access.redhat.com'
Even with vanilla /etc/sysconfig/docker I am getting the same output: $ docker-compose pull Pulling x (tomastomecek/secret-repo:latest)... Trying to pull repository docker.io/tomastomecek/secret-repo ... sha256:8573b4a813d7b90ef3876c6bec33db1272c02f0f90c406b25a5f9729169548ac: Pulling from docker.io/tomastomecek/secret-repo Digest: sha256:8573b4a813d7b90ef3876c6bec33db1272c02f0f90c406b25a5f9729169548ac Status: Downloaded newer image for docker.io/tomastomecek/secret-repo:latest
(In reply to Tomas Tomecek from comment #5) > Even with vanilla /etc/sysconfig/docker I am getting the same output: > > $ docker-compose pull > Pulling x (tomastomecek/secret-repo:latest)... > Trying to pull repository docker.io/tomastomecek/secret-repo ... > sha256:8573b4a813d7b90ef3876c6bec33db1272c02f0f90c406b25a5f9729169548ac: > Pulling from docker.io/tomastomecek/secret-repo > Digest: > sha256:8573b4a813d7b90ef3876c6bec33db1272c02f0f90c406b25a5f9729169548ac > Status: Downloaded newer image for docker.io/tomastomecek/secret-repo:latest That output still doesn't show it trying to pull from the Fedora repos, so it appears that there's still something different in your config from a vanilla install.
I have vanilla config right now and can easily prove it: $ rpm -q -Vv docker-common ......... c /etc/sysconfig/docker Care to share your config then? $ cat /etc/sysconfig/docker $ cat /etc/containers/registries.conf And care to share if it's vanilla? $ rpm -q -Vv docker docker-common
(In reply to Tomas Tomecek from comment #7) > Care to share your config then? > > $ cat /etc/sysconfig/docker > $ cat /etc/containers/registries.conf $ grep -v "^#" /etc/sysconfig/docker OPTIONS='--selinux-enabled --log-driver=journald' if [ -z "${DOCKER_CERT_PATH}" ]; then DOCKER_CERT_PATH=/etc/docker fi $ grep -v "^#" /etc/containers/registries.conf registries: - registry.fedoraproject.org - registry.access.redhat.com > And care to share if it's vanilla? > > $ rpm -q -Vv docker docker-common ......... /etc/docker ......... /etc/docker/certs.d ......... /etc/docker/certs.d/redhat.com ......... /etc/docker/certs.d/redhat.com/redhat-ca.crt ......... /etc/docker/certs.d/redhat.io ......... /etc/docker/certs.d/redhat.io/redhat-ca.crt ......... /etc/docker/docker-lvm-plugin ......... /etc/docker/seccomp.json ......... c /etc/sysconfig/docker-network S.5....T. c /etc/sysconfig/docker-storage ......... /usr/bin/docker-current ......... /usr/bin/docker-storage-setup ......... /usr/bin/dockerd-current ......... /usr/lib/systemd/system/docker-containerd.service ......... /usr/lib/systemd/system/docker-storage-setup.service ......... /usr/lib/systemd/system/docker.service ......... /usr/lib/udev/rules.d/80-docker.rules ......... /usr/libexec/docker/docker-containerd-current ......... /usr/libexec/docker/docker-containerd-shim-current ......... /usr/libexec/docker/docker-ctr-current ......... /usr/libexec/docker/docker-init-current ......... /usr/libexec/docker/docker-proxy-current ......... /usr/libexec/docker/docker-runc-current ......... /usr/share/bash-completion/completions/docker ......... /usr/share/doc/docker ......... d /usr/share/doc/docker/AUTHORS ......... d /usr/share/doc/docker/CHANGELOG.md ......... d /usr/share/doc/docker/CONTRIBUTING.md ......... d /usr/share/doc/docker/MAINTAINERS ......... d /usr/share/doc/docker/NOTICE ......... d /usr/share/doc/docker/README-novolume-plugin.md ......... d /usr/share/doc/docker/README-vim-syntax.md ......... d /usr/share/doc/docker/README.md ......... c /usr/share/docker-storage-setup/docker-storage-setup-atomichost ......... c /usr/share/docker-storage-setup/docker-storage-setup-cloud ......... c /usr/share/docker-storage-setup/docker-storage-setup-default ......... c /usr/share/docker-storage-setup/docker-storage-setup-server ......... c /usr/share/docker-storage-setup/docker-storage-setup-workstation ......... /usr/share/licenses/docker ......... l /usr/share/licenses/docker/LICENSE ......... l /usr/share/licenses/docker/LICENSE-novolume-plugin ......... l /usr/share/licenses/docker/LICENSE-vim-syntax ......... d /usr/share/man/man1/docker-attach.1.gz ......... d /usr/share/man/man1/docker-build.1.gz ......... d /usr/share/man/man1/docker-commit.1.gz ......... d /usr/share/man/man1/docker-cp.1.gz ......... d /usr/share/man/man1/docker-create.1.gz ......... d /usr/share/man/man1/docker-diff.1.gz ......... d /usr/share/man/man1/docker-events.1.gz ......... d /usr/share/man/man1/docker-exec.1.gz ......... d /usr/share/man/man1/docker-export.1.gz ......... d /usr/share/man/man1/docker-history.1.gz ......... d /usr/share/man/man1/docker-images.1.gz ......... d /usr/share/man/man1/docker-import.1.gz ......... d /usr/share/man/man1/docker-info.1.gz ......... d /usr/share/man/man1/docker-inspect.1.gz ......... d /usr/share/man/man1/docker-kill.1.gz ......... d /usr/share/man/man1/docker-load.1.gz ......... d /usr/share/man/man1/docker-login.1.gz ......... d /usr/share/man/man1/docker-logout.1.gz ......... d /usr/share/man/man1/docker-logs.1.gz ......... d /usr/share/man/man1/docker-network-connect.1.gz ......... d /usr/share/man/man1/docker-network-create.1.gz ......... d /usr/share/man/man1/docker-network-disconnect.1.gz ......... d /usr/share/man/man1/docker-network-inspect.1.gz ......... d /usr/share/man/man1/docker-network-ls.1.gz ......... d /usr/share/man/man1/docker-network-rm.1.gz ......... d /usr/share/man/man1/docker-pause.1.gz ......... d /usr/share/man/man1/docker-port.1.gz ......... d /usr/share/man/man1/docker-ps.1.gz ......... d /usr/share/man/man1/docker-pull.1.gz ......... d /usr/share/man/man1/docker-push.1.gz ......... d /usr/share/man/man1/docker-rename.1.gz ......... d /usr/share/man/man1/docker-restart.1.gz ......... d /usr/share/man/man1/docker-rm.1.gz ......... d /usr/share/man/man1/docker-rmi.1.gz ......... d /usr/share/man/man1/docker-run.1.gz ......... d /usr/share/man/man1/docker-save.1.gz ......... d /usr/share/man/man1/docker-search.1.gz ......... d /usr/share/man/man1/docker-start.1.gz ......... d /usr/share/man/man1/docker-stats.1.gz ......... d /usr/share/man/man1/docker-stop.1.gz ......... d /usr/share/man/man1/docker-storage-setup.1.gz ......... d /usr/share/man/man1/docker-tag.1.gz ......... d /usr/share/man/man1/docker-top.1.gz ......... d /usr/share/man/man1/docker-unpause.1.gz ......... d /usr/share/man/man1/docker-update.1.gz ......... d /usr/share/man/man1/docker-version.1.gz ......... d /usr/share/man/man1/docker-wait.1.gz ......... d /usr/share/man/man1/docker.1.gz ......... d /usr/share/man/man5/Dockerfile.5.gz ......... d /usr/share/man/man5/docker-config-json.5.gz ......... d /usr/share/man/man8/docker-lvm-plugin.8.gz ......... d /usr/share/man/man8/docker-novolume-plugin.8.gz ......... d /usr/share/man/man8/dockerd.8.gz ......... d /usr/share/man/man8/rhel-push-plugin.8.gz ......... /usr/share/rhel/secrets .M....... /var/lib/docker ......... c /etc/sysconfig/docker ......... /usr/bin/docker ......... /usr/share/doc/docker-common ......... d /usr/share/doc/docker-common/README-docker-common I tried removing docker through dnf, removing the .rpmsave files and then re-installing and it still has the same problem.
I think it's time to move this to docker component, since I'm getting clueless.
Dave, could you also please post logs from docker when the faulty behavior occurs (journalctl -u docker) and which versions of docker, docker-compose and python*-docker you are using? Thank you.
(In reply to Tomas Tomecek from comment #10) > Dave, could you also please post logs from docker when the faulty behavior > occurs (journalctl -u docker) and which versions of docker, docker-compose > and python*-docker you are using? Thank you. NOTE: I replaced the repo and container names in these logs to protect any potential sensitivities. journalctl output when running docker-compose pull (failed): Aug 18 07:50:16 JohansenDev dockerd-current[27659]: time="2017-08-18T07:50:16.951538320-06:00" level=info msg="{Action=create, Username=dlj, LoginUID=1000, PID=9681}" Aug 18 07:50:18 JohansenDev dockerd-current[27659]: time="2017-08-18T07:50:18.558016708-06:00" level=error msg="Error trying v2 registry: manifest unknown: manifest unknown" Aug 18 07:50:18 JohansenDev dockerd-current[27659]: time="2017-08-18T07:50:18.558116856-06:00" level=error msg="Attempting next endpoint for pull after error: manifest unknown : manifest unknown" Aug 18 07:50:18 JohansenDev dockerd-current[27659]: time="2017-08-18T07:50:18.558235910-06:00" level=info msg="Translating \"manifest unknown: manifest unknown\" to \"manifest for registry.fedoraproject.org/my_repo/my_container:dev not found\"" Aug 18 07:50:19 JohansenDev dockerd-current[27659]: time="2017-08-18T07:50:19.838385871-06:00" level=error msg="Not continuing with pull after error: unknown: Not Found" Aug 18 07:50:21 JohansenDev dockerd-current[27659]: time="2017-08-18T07:50:21.399352042-06:00" level=error msg="Not continuing with pull after error: errors:\ndenied: requeste d access to the resource is denied\nunauthorized: authentication required\n" Aug 18 07:50:21 JohansenDev dockerd-current[27659]: time="2017-08-18T07:50:21.399389950-06:00" level=info msg="Ignoring extra error returned from registry: unauthorized: authe ntication required" Aug 18 07:50:21 JohansenDev dockerd-current[27659]: time="2017-08-18T07:50:21.399435955-06:00" level=info msg="Translating \"denied: requested access to the resource is denied \" to \"repository docker.io/my_repo/my_container not found: does not exist or no pull access\"" Aug 18 07:50:36 JohansenDev dockerd-current[27659]: time="2017-08-18T07:50:36.786006933-06:00" level=info msg="{Action=_ping, Username=dlj, LoginUID=1000, PID=9780}" journalctl output when running docker pull (worked): Aug 18 07:50:16 JohansenDev dockerd-current[27659]: time="2017-08-18T07:50:16.951538320-06:00" level=info msg="{Action=create, Username=dlj, LoginUID=1000, PID=9681}" Aug 18 07:50:18 JohansenDev dockerd-current[27659]: time="2017-08-18T07:50:18.558016708-06:00" level=error msg="Error trying v2 registry: manifest unknown: manifest unknown" Aug 18 07:50:18 JohansenDev dockerd-current[27659]: time="2017-08-18T07:50:18.558116856-06:00" level=error msg="Attempting next endpoint for pull after error: manifest unknown : manifest unknown" Aug 18 07:50:18 JohansenDev dockerd-current[27659]: time="2017-08-18T07:50:18.558235910-06:00" level=info msg="Translating \"manifest unknown: manifest unknown\" to \"manifest for registry.fedoraproject.org/my_repo/my_container:dev not found\"" Aug 18 07:50:19 JohansenDev dockerd-current[27659]: time="2017-08-18T07:50:19.838385871-06:00" level=error msg="Not continuing with pull after error: unknown: Not Found" Aug 18 07:50:21 JohansenDev dockerd-current[27659]: time="2017-08-18T07:50:21.399352042-06:00" level=error msg="Not continuing with pull after error: errors:\ndenied: requeste d access to the resource is denied\nunauthorized: authentication required\n" Aug 18 07:50:21 JohansenDev dockerd-current[27659]: time="2017-08-18T07:50:21.399389950-06:00" level=info msg="Ignoring extra error returned from registry: unauthorized: authe ntication required" Aug 18 07:50:21 JohansenDev dockerd-current[27659]: time="2017-08-18T07:50:21.399435955-06:00" level=info msg="Translating \"denied: requested access to the resource is denied \" to \"repository docker.io/my_repo/my_container not found: does not exist or no pull access\"" Aug 18 07:50:36 JohansenDev dockerd-current[27659]: time="2017-08-18T07:50:36.786006933-06:00" level=info msg="{Action=_ping, Username=dlj, LoginUID=1000, PID=9780}" $ dnf list installed | grep docker docker.x86_64 2:1.13.1-21.git27e468e.fc26 @updates docker-common.x86_64 2:1.13.1-21.git27e468e.fc26 @updates docker-compose.noarch 1.15.0-1.fc26 @updates docker-rhel-push-plugin.x86_64 2:1.13.1-21.git27e468e.fc26 @updates python2-dockerfile-parse.noarch 0.0.5-9.fc26 @fedora python3-docker.noarch 2.4.2-1.fc26 @updates python3-docker-pycreds.noarch 0.2.1-4.fc26 @fedora python3-dockerpty.noarch 0.4.1-6.fc26 @fedora
I finally managed to reproduce! (realized the difference in configuration was caused by my drop-in for docker systemd unit file) So I can reliably reproduce: $ docker-compose pull Pulling x (tomastomecek/secret-repo:latest)... Trying to pull repository registry.fedoraproject.org/tomastomecek/secret-repo ... Trying to pull repository registry.access.redhat.com/tomastomecek/secret-repo ... Trying to pull repository docker.io/tomastomecek/secret-repo ... ERROR: repository docker.io/tomastomecek/secret-repo not found: does not exist or no pull access Luckily, this has a very simple work-around, just change the image reference to 'docker.io/*': $ docker-compose pull Pulling x (docker.io/tomastomecek/secret-repo:latest)... Trying to pull repository docker.io/tomastomecek/secret-repo ... sha256:8573b4a813d7b90ef3876c6bec33db1272c02f0f90c406b25a5f9729169548ac: Pulling from docker.io/tomastomecek/secret-repo Digest: sha256:8573b4a813d7b90ef3876c6bec33db1272c02f0f90c406b25a5f9729169548ac Status: Downloaded newer image for docker.io/tomastomecek/secret-repo:latest I suspect the issue is when docker-compose calls pull method of docker-py and this is then follow by some authconfig misconfiguration. Not sure if on side of docker-compose or docker. Dave, is the workaround sufficient for you? I don't think that docker-compose upstream would accept a patch which is suppose to fix an issue in our downstream docker patch.
(In reply to Tomas Tomecek from comment #12) > I finally managed to reproduce! (realized the difference in configuration > was caused by my drop-in for docker systemd unit file) > > So I can reliably reproduce: > > $ docker-compose pull > Pulling x (tomastomecek/secret-repo:latest)... > Trying to pull repository > registry.fedoraproject.org/tomastomecek/secret-repo ... > Trying to pull repository > registry.access.redhat.com/tomastomecek/secret-repo ... > Trying to pull repository docker.io/tomastomecek/secret-repo ... > ERROR: repository docker.io/tomastomecek/secret-repo not found: does not > exist or no pull access > > > Luckily, this has a very simple work-around, just change the image reference > to 'docker.io/*': > > $ docker-compose pull > Pulling x (docker.io/tomastomecek/secret-repo:latest)... > Trying to pull repository docker.io/tomastomecek/secret-repo ... > sha256:8573b4a813d7b90ef3876c6bec33db1272c02f0f90c406b25a5f9729169548ac: > Pulling from docker.io/tomastomecek/secret-repo > Digest: > sha256:8573b4a813d7b90ef3876c6bec33db1272c02f0f90c406b25a5f9729169548ac > Status: Downloaded newer image for docker.io/tomastomecek/secret-repo:latest This didn't work for me and I'm now getting 401 errors even with docker pull. I did docker logout/login and it said it was successful, so I'm not sure what's going on there. > I suspect the issue is when docker-compose calls pull method of docker-py > and this is then follow by some authconfig misconfiguration. Not sure if on > side of docker-compose or docker. > > > Dave, is the workaround sufficient for you? I don't think that > docker-compose upstream would accept a patch which is suppose to fix an > issue in our downstream docker patch. These .yml files are used on multiple platforms and work on all the rest (with some of those being Fedora before 26), so this should be fixed instead of requiring everyone to put in a workaround for a regression in Fedora 26.
Ok, I figured out why I'm getting the 401s. When I run `docker login` and use my Docker Hub credentials, it says it was successful but saves them as fedoraproject ones. When I try running `docker login docker.io`, it errors out. It appears that the config for the fedora specific repos that were add Fedora 26 are messing things up. If this isn't going to be fixed really soon, then can these Fedora specific configs be removed/disabled until they are fixed to not break the non-Fedora repos?
This message is a reminder that Fedora 26 is nearing its end of life. Approximately 4 (four) weeks from now Fedora will stop maintaining and issuing updates for Fedora 26. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '26'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 26 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
Fedora 26 changed to end-of-life (EOL) status on 2018-05-29. Fedora 26 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.