This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 1471319 - docker-compose pull does not work with private repos
docker-compose pull does not work with private repos
Status: NEW
Product: Fedora
Classification: Fedora
Component: docker (Show other bugs)
26
Unspecified Unspecified
urgent Severity urgent
: ---
: ---
Assigned To: Antonio Murdaca
Fedora Extras Quality Assurance
: Reopened
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-07-14 21:47 EDT by Dave Johansen
Modified: 2017-09-10 11:19 EDT (History)
17 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-08-11 05:43:47 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Github docker/compose/issues/4807 None None None 2017-07-25 11:16 EDT

  None (edit)
Description Dave Johansen 2017-07-14 21:47:37 EDT
Description of problem:
docker-compose pull worked with private repos in F25 but is broken in F26.

Version-Release number of selected component (if applicable):
1.14.0, build c7bdf9e

How reproducible:
always

Steps to Reproduce:
1. Create test.yml file with private images
2. docker-compose -f test.yml pull
3. Observed failed download

Actual results:
Download fails

Expected results:
Download works as it did with F25

Additional info:
docker pull works just fine.
Comment 1 Dave Johansen 2017-07-25 11:16:21 EDT
I added an upstream bug that I found related to this issue.
Comment 2 Tomas Tomecek 2017-08-11 05:43:47 EDT
Unfortunately I wasn't able to reproduce, therefore closing.

(more detailed info in upstream issue: https://github.com/docker/compose/issues/4807#issuecomment-321770563)
Comment 3 Dave Johansen 2017-08-12 10:36:11 EDT
(In reply to Tomas Tomecek from comment #2)
> Unfortunately I wasn't able to reproduce, therefore closing.
> 
> (more detailed info in upstream issue:
> https://github.com/docker/compose/issues/4807#issuecomment-321770563)

Are you using the packages that come with Fedora? Or the ones from Docker? The output in yours doesn't show trying to pull from the Fedora repos like in my output, so it appears that there's some difference between the config your using and the config that I'm seeing after installing the packages from Fedora.
Comment 4 Tomas Tomecek 2017-08-14 02:31:40 EDT
I am using packages that come from Fedora.

It's true that I edited /etc/sysconfig/docker, this is how mine looks:
OPTIONS='-l debug --log-driver=journald'
if [ -z "${DOCKER_CERT_PATH}" ]; then
    DOCKER_CERT_PATH=/etc/docker
fi
INSECURE_REGISTRY='--insecure-registry 172.30.0.0/16'

I have commented out the ADD_REGISTRY value:
# ADD_REGISTRY='--add-registry registry.access.redhat.com'
Comment 5 Tomas Tomecek 2017-08-14 07:27:27 EDT
Even with vanilla /etc/sysconfig/docker I am getting the same output:

$ docker-compose pull
Pulling x (tomastomecek/secret-repo:latest)...
Trying to pull repository docker.io/tomastomecek/secret-repo ...
sha256:8573b4a813d7b90ef3876c6bec33db1272c02f0f90c406b25a5f9729169548ac: Pulling from docker.io/tomastomecek/secret-repo
Digest: sha256:8573b4a813d7b90ef3876c6bec33db1272c02f0f90c406b25a5f9729169548ac
Status: Downloaded newer image for docker.io/tomastomecek/secret-repo:latest
Comment 6 Dave Johansen 2017-08-15 22:17:11 EDT
(In reply to Tomas Tomecek from comment #5)
> Even with vanilla /etc/sysconfig/docker I am getting the same output:
> 
> $ docker-compose pull
> Pulling x (tomastomecek/secret-repo:latest)...
> Trying to pull repository docker.io/tomastomecek/secret-repo ...
> sha256:8573b4a813d7b90ef3876c6bec33db1272c02f0f90c406b25a5f9729169548ac:
> Pulling from docker.io/tomastomecek/secret-repo
> Digest:
> sha256:8573b4a813d7b90ef3876c6bec33db1272c02f0f90c406b25a5f9729169548ac
> Status: Downloaded newer image for docker.io/tomastomecek/secret-repo:latest

That output still doesn't show it trying to pull from the Fedora repos, so it appears that there's still something different in your config from a vanilla install.
Comment 7 Tomas Tomecek 2017-08-16 01:59:02 EDT
I have vanilla config right now and can easily prove it:

$ rpm -q -Vv docker-common
.........  c /etc/sysconfig/docker


Care to share your config then?

$ cat /etc/sysconfig/docker
$ cat /etc/containers/registries.conf

And care to share if it's vanilla?

$ rpm -q -Vv docker docker-common
Comment 8 Dave Johansen 2017-08-16 23:19:17 EDT
(In reply to Tomas Tomecek from comment #7)
> Care to share your config then?
> 
> $ cat /etc/sysconfig/docker
> $ cat /etc/containers/registries.conf

$ grep -v "^#" /etc/sysconfig/docker

OPTIONS='--selinux-enabled --log-driver=journald'
if [ -z "${DOCKER_CERT_PATH}" ]; then
    DOCKER_CERT_PATH=/etc/docker
fi


$ grep -v "^#" /etc/containers/registries.conf 




registries:
  - registry.fedoraproject.org
  - registry.access.redhat.com


> And care to share if it's vanilla?
> 
> $ rpm -q -Vv docker docker-common

.........    /etc/docker
.........    /etc/docker/certs.d
.........    /etc/docker/certs.d/redhat.com
.........    /etc/docker/certs.d/redhat.com/redhat-ca.crt
.........    /etc/docker/certs.d/redhat.io
.........    /etc/docker/certs.d/redhat.io/redhat-ca.crt
.........    /etc/docker/docker-lvm-plugin
.........    /etc/docker/seccomp.json
.........  c /etc/sysconfig/docker-network
S.5....T.  c /etc/sysconfig/docker-storage
.........    /usr/bin/docker-current
.........    /usr/bin/docker-storage-setup
.........    /usr/bin/dockerd-current
.........    /usr/lib/systemd/system/docker-containerd.service
.........    /usr/lib/systemd/system/docker-storage-setup.service
.........    /usr/lib/systemd/system/docker.service
.........    /usr/lib/udev/rules.d/80-docker.rules
.........    /usr/libexec/docker/docker-containerd-current
.........    /usr/libexec/docker/docker-containerd-shim-current
.........    /usr/libexec/docker/docker-ctr-current
.........    /usr/libexec/docker/docker-init-current
.........    /usr/libexec/docker/docker-proxy-current
.........    /usr/libexec/docker/docker-runc-current
.........    /usr/share/bash-completion/completions/docker
.........    /usr/share/doc/docker
.........  d /usr/share/doc/docker/AUTHORS
.........  d /usr/share/doc/docker/CHANGELOG.md
.........  d /usr/share/doc/docker/CONTRIBUTING.md
.........  d /usr/share/doc/docker/MAINTAINERS
.........  d /usr/share/doc/docker/NOTICE
.........  d /usr/share/doc/docker/README-novolume-plugin.md
.........  d /usr/share/doc/docker/README-vim-syntax.md
.........  d /usr/share/doc/docker/README.md
.........  c /usr/share/docker-storage-setup/docker-storage-setup-atomichost
.........  c /usr/share/docker-storage-setup/docker-storage-setup-cloud
.........  c /usr/share/docker-storage-setup/docker-storage-setup-default
.........  c /usr/share/docker-storage-setup/docker-storage-setup-server
.........  c /usr/share/docker-storage-setup/docker-storage-setup-workstation
.........    /usr/share/licenses/docker
.........  l /usr/share/licenses/docker/LICENSE
.........  l /usr/share/licenses/docker/LICENSE-novolume-plugin
.........  l /usr/share/licenses/docker/LICENSE-vim-syntax
.........  d /usr/share/man/man1/docker-attach.1.gz
.........  d /usr/share/man/man1/docker-build.1.gz
.........  d /usr/share/man/man1/docker-commit.1.gz
.........  d /usr/share/man/man1/docker-cp.1.gz
.........  d /usr/share/man/man1/docker-create.1.gz
.........  d /usr/share/man/man1/docker-diff.1.gz
.........  d /usr/share/man/man1/docker-events.1.gz
.........  d /usr/share/man/man1/docker-exec.1.gz
.........  d /usr/share/man/man1/docker-export.1.gz
.........  d /usr/share/man/man1/docker-history.1.gz
.........  d /usr/share/man/man1/docker-images.1.gz
.........  d /usr/share/man/man1/docker-import.1.gz
.........  d /usr/share/man/man1/docker-info.1.gz
.........  d /usr/share/man/man1/docker-inspect.1.gz
.........  d /usr/share/man/man1/docker-kill.1.gz
.........  d /usr/share/man/man1/docker-load.1.gz
.........  d /usr/share/man/man1/docker-login.1.gz
.........  d /usr/share/man/man1/docker-logout.1.gz
.........  d /usr/share/man/man1/docker-logs.1.gz
.........  d /usr/share/man/man1/docker-network-connect.1.gz
.........  d /usr/share/man/man1/docker-network-create.1.gz
.........  d /usr/share/man/man1/docker-network-disconnect.1.gz
.........  d /usr/share/man/man1/docker-network-inspect.1.gz
.........  d /usr/share/man/man1/docker-network-ls.1.gz
.........  d /usr/share/man/man1/docker-network-rm.1.gz
.........  d /usr/share/man/man1/docker-pause.1.gz
.........  d /usr/share/man/man1/docker-port.1.gz
.........  d /usr/share/man/man1/docker-ps.1.gz
.........  d /usr/share/man/man1/docker-pull.1.gz
.........  d /usr/share/man/man1/docker-push.1.gz
.........  d /usr/share/man/man1/docker-rename.1.gz
.........  d /usr/share/man/man1/docker-restart.1.gz
.........  d /usr/share/man/man1/docker-rm.1.gz
.........  d /usr/share/man/man1/docker-rmi.1.gz
.........  d /usr/share/man/man1/docker-run.1.gz
.........  d /usr/share/man/man1/docker-save.1.gz
.........  d /usr/share/man/man1/docker-search.1.gz
.........  d /usr/share/man/man1/docker-start.1.gz
.........  d /usr/share/man/man1/docker-stats.1.gz
.........  d /usr/share/man/man1/docker-stop.1.gz
.........  d /usr/share/man/man1/docker-storage-setup.1.gz
.........  d /usr/share/man/man1/docker-tag.1.gz
.........  d /usr/share/man/man1/docker-top.1.gz
.........  d /usr/share/man/man1/docker-unpause.1.gz
.........  d /usr/share/man/man1/docker-update.1.gz
.........  d /usr/share/man/man1/docker-version.1.gz
.........  d /usr/share/man/man1/docker-wait.1.gz
.........  d /usr/share/man/man1/docker.1.gz
.........  d /usr/share/man/man5/Dockerfile.5.gz
.........  d /usr/share/man/man5/docker-config-json.5.gz
.........  d /usr/share/man/man8/docker-lvm-plugin.8.gz
.........  d /usr/share/man/man8/docker-novolume-plugin.8.gz
.........  d /usr/share/man/man8/dockerd.8.gz
.........  d /usr/share/man/man8/rhel-push-plugin.8.gz
.........    /usr/share/rhel/secrets
.M.......    /var/lib/docker
.........  c /etc/sysconfig/docker
.........    /usr/bin/docker
.........    /usr/share/doc/docker-common
.........  d /usr/share/doc/docker-common/README-docker-common

I tried removing docker through dnf, removing the .rpmsave files and then re-installing and it still has the same problem.
Comment 9 Tomas Tomecek 2017-08-17 03:41:50 EDT
I think it's time to move this to docker component, since I'm getting clueless.
Comment 10 Tomas Tomecek 2017-08-17 03:44:03 EDT
Dave, could you also please post logs from docker when the faulty behavior occurs (journalctl -u docker) and which versions of docker, docker-compose and python*-docker you are using? Thank you.
Comment 11 Dave Johansen 2017-08-18 09:58:21 EDT
(In reply to Tomas Tomecek from comment #10)
> Dave, could you also please post logs from docker when the faulty behavior
> occurs (journalctl -u docker) and which versions of docker, docker-compose
> and python*-docker you are using? Thank you.

NOTE: I replaced the repo and container names in these logs to protect any potential sensitivities.

journalctl output when running docker-compose pull (failed):
Aug 18 07:50:16 JohansenDev dockerd-current[27659]: time="2017-08-18T07:50:16.951538320-06:00" level=info msg="{Action=create, Username=dlj, LoginUID=1000, PID=9681}"
Aug 18 07:50:18 JohansenDev dockerd-current[27659]: time="2017-08-18T07:50:18.558016708-06:00" level=error msg="Error trying v2 registry: manifest unknown: manifest unknown"
Aug 18 07:50:18 JohansenDev dockerd-current[27659]: time="2017-08-18T07:50:18.558116856-06:00" level=error msg="Attempting next endpoint for pull after error: manifest unknown : manifest unknown"
Aug 18 07:50:18 JohansenDev dockerd-current[27659]: time="2017-08-18T07:50:18.558235910-06:00" level=info msg="Translating \"manifest unknown: manifest unknown\" to \"manifest  for registry.fedoraproject.org/my_repo/my_container:dev not found\""
Aug 18 07:50:19 JohansenDev dockerd-current[27659]: time="2017-08-18T07:50:19.838385871-06:00" level=error msg="Not continuing with pull after error: unknown: Not Found"
Aug 18 07:50:21 JohansenDev dockerd-current[27659]: time="2017-08-18T07:50:21.399352042-06:00" level=error msg="Not continuing with pull after error: errors:\ndenied: requeste d access to the resource is denied\nunauthorized: authentication required\n"
Aug 18 07:50:21 JohansenDev dockerd-current[27659]: time="2017-08-18T07:50:21.399389950-06:00" level=info msg="Ignoring extra error returned from registry: unauthorized: authe ntication required"
Aug 18 07:50:21 JohansenDev dockerd-current[27659]: time="2017-08-18T07:50:21.399435955-06:00" level=info msg="Translating \"denied: requested access to the resource is denied \" to \"repository docker.io/my_repo/my_container not found: does not exist or no pull access\""
Aug 18 07:50:36 JohansenDev dockerd-current[27659]: time="2017-08-18T07:50:36.786006933-06:00" level=info msg="{Action=_ping, Username=dlj, LoginUID=1000, PID=9780}"

journalctl output when running docker pull (worked):
Aug 18 07:50:16 JohansenDev dockerd-current[27659]: time="2017-08-18T07:50:16.951538320-06:00" level=info msg="{Action=create, Username=dlj, LoginUID=1000, PID=9681}"
Aug 18 07:50:18 JohansenDev dockerd-current[27659]: time="2017-08-18T07:50:18.558016708-06:00" level=error msg="Error trying v2 registry: manifest unknown: manifest unknown"
Aug 18 07:50:18 JohansenDev dockerd-current[27659]: time="2017-08-18T07:50:18.558116856-06:00" level=error msg="Attempting next endpoint for pull after error: manifest unknown : manifest unknown"
Aug 18 07:50:18 JohansenDev dockerd-current[27659]: time="2017-08-18T07:50:18.558235910-06:00" level=info msg="Translating \"manifest unknown: manifest unknown\" to \"manifest  for registry.fedoraproject.org/my_repo/my_container:dev not found\""
Aug 18 07:50:19 JohansenDev dockerd-current[27659]: time="2017-08-18T07:50:19.838385871-06:00" level=error msg="Not continuing with pull after error: unknown: Not Found"
Aug 18 07:50:21 JohansenDev dockerd-current[27659]: time="2017-08-18T07:50:21.399352042-06:00" level=error msg="Not continuing with pull after error: errors:\ndenied: requeste d access to the resource is denied\nunauthorized: authentication required\n"
Aug 18 07:50:21 JohansenDev dockerd-current[27659]: time="2017-08-18T07:50:21.399389950-06:00" level=info msg="Ignoring extra error returned from registry: unauthorized: authe ntication required"
Aug 18 07:50:21 JohansenDev dockerd-current[27659]: time="2017-08-18T07:50:21.399435955-06:00" level=info msg="Translating \"denied: requested access to the resource is denied \" to \"repository docker.io/my_repo/my_container not found: does not exist or no pull access\""
Aug 18 07:50:36 JohansenDev dockerd-current[27659]: time="2017-08-18T07:50:36.786006933-06:00" level=info msg="{Action=_ping, Username=dlj, LoginUID=1000, PID=9780}"


$ dnf list installed | grep docker
docker.x86_64                              2:1.13.1-21.git27e468e.fc26  @updates
docker-common.x86_64                       2:1.13.1-21.git27e468e.fc26  @updates
docker-compose.noarch                      1.15.0-1.fc26                @updates
docker-rhel-push-plugin.x86_64             2:1.13.1-21.git27e468e.fc26  @updates
python2-dockerfile-parse.noarch            0.0.5-9.fc26                 @fedora 
python3-docker.noarch                      2.4.2-1.fc26                 @updates
python3-docker-pycreds.noarch              0.2.1-4.fc26                 @fedora 
python3-dockerpty.noarch                   0.4.1-6.fc26                 @fedora
Comment 12 Tomas Tomecek 2017-08-24 05:39:30 EDT
I finally managed to reproduce! (realized the difference in configuration was caused by my drop-in for docker systemd unit file)

So I can reliably reproduce:

$ docker-compose pull
Pulling x (tomastomecek/secret-repo:latest)...
Trying to pull repository registry.fedoraproject.org/tomastomecek/secret-repo ...
Trying to pull repository registry.access.redhat.com/tomastomecek/secret-repo ...
Trying to pull repository docker.io/tomastomecek/secret-repo ...
ERROR: repository docker.io/tomastomecek/secret-repo not found: does not exist or no pull access


Luckily, this has a very simple work-around, just change the image reference to 'docker.io/*':

$ docker-compose pull
Pulling x (docker.io/tomastomecek/secret-repo:latest)...
Trying to pull repository docker.io/tomastomecek/secret-repo ...
sha256:8573b4a813d7b90ef3876c6bec33db1272c02f0f90c406b25a5f9729169548ac: Pulling from docker.io/tomastomecek/secret-repo
Digest: sha256:8573b4a813d7b90ef3876c6bec33db1272c02f0f90c406b25a5f9729169548ac
Status: Downloaded newer image for docker.io/tomastomecek/secret-repo:latest


I suspect the issue is when docker-compose calls pull method of docker-py and this is then follow by some authconfig misconfiguration. Not sure if on side of docker-compose or docker.


Dave, is the workaround sufficient for you? I don't think that docker-compose upstream would accept a patch which is suppose to fix an issue in our downstream docker patch.
Comment 13 Dave Johansen 2017-08-26 11:43:41 EDT
(In reply to Tomas Tomecek from comment #12)
> I finally managed to reproduce! (realized the difference in configuration
> was caused by my drop-in for docker systemd unit file)
> 
> So I can reliably reproduce:
> 
> $ docker-compose pull
> Pulling x (tomastomecek/secret-repo:latest)...
> Trying to pull repository
> registry.fedoraproject.org/tomastomecek/secret-repo ...
> Trying to pull repository
> registry.access.redhat.com/tomastomecek/secret-repo ...
> Trying to pull repository docker.io/tomastomecek/secret-repo ...
> ERROR: repository docker.io/tomastomecek/secret-repo not found: does not
> exist or no pull access
> 
> 
> Luckily, this has a very simple work-around, just change the image reference
> to 'docker.io/*':
> 
> $ docker-compose pull
> Pulling x (docker.io/tomastomecek/secret-repo:latest)...
> Trying to pull repository docker.io/tomastomecek/secret-repo ...
> sha256:8573b4a813d7b90ef3876c6bec33db1272c02f0f90c406b25a5f9729169548ac:
> Pulling from docker.io/tomastomecek/secret-repo
> Digest:
> sha256:8573b4a813d7b90ef3876c6bec33db1272c02f0f90c406b25a5f9729169548ac
> Status: Downloaded newer image for docker.io/tomastomecek/secret-repo:latest

This didn't work for me and I'm now getting 401 errors even with docker pull. I did docker logout/login and it said it was successful, so I'm not sure what's going on there.

> I suspect the issue is when docker-compose calls pull method of docker-py
> and this is then follow by some authconfig misconfiguration. Not sure if on
> side of docker-compose or docker.
> 
> 
> Dave, is the workaround sufficient for you? I don't think that
> docker-compose upstream would accept a patch which is suppose to fix an
> issue in our downstream docker patch.

These .yml files are used on multiple platforms and work on all the rest (with some of those being Fedora before 26), so this should be fixed instead of requiring everyone to put in a workaround for a regression in Fedora 26.
Comment 14 Dave Johansen 2017-08-26 15:45:45 EDT
Ok, I figured out why I'm getting the 401s. When I run `docker login` and use my Docker Hub credentials, it says it was successful but saves them as fedoraproject ones. When I try running `docker login docker.io`, it errors out.

It appears that the config for the fedora specific repos that were add Fedora 26 are messing things up. If this isn't going to be fixed really soon, then can these Fedora specific configs be removed/disabled until they are fixed to not break the non-Fedora repos?

Note You need to log in before you can comment on or make changes to this bug.