Bug 1506291

Summary: Without docker.io listed in insecure+add registires, pushes to docker.io fail with "auth required"
Product: [Fedora] Fedora Reporter: Erik M Jacobs <ejacobs>
Component: dockerAssignee: Antonio Murdaca <amurdaca>
Status: CLOSED EOL QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 25CC: adimania, admiller, amurdaca, dwalsh, fkluknav, ichavero, jcajka, lsm5, marianne, nalin, santiago, tsweeney, vbatts
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-12-12 10:05:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Erik M Jacobs 2017-10-25 15:41:38 UTC
Description of problem:
Even after removing ~/.docker/config.json and re-authenticating only to docker.io, pushes result in an "auth required" error message. Once adding docker.io to both insecure+add registries, the push succeeds.

Version-Release number of selected component (if applicable):
Client:
 Version:         1.12.6
 API version:     1.24
 Package version: docker-common-1.12.6-7.gitae7d637.fc25.x86_64
 Go version:      go1.7.6
 Git commit:      ae7d637/1.12.6
 Built:           Tue Jul 18 16:18:12 2017
 OS/Arch:         linux/amd64

How reproducible:
100%

Steps to Reproduce:
Do not have docker.io listed in /etc/sysconfig/docker, then:
root@t460 ~]# docker images | grep samba
docker.io/thoraxe/openshift-centos-samba                                              0.1                 510fda34fceb        3 days ago          242 MB

[root@t460 ~]# rm -rf ~/.docker/
[root@t460 ~]# docker login docker.io
Username: thoraxe
Password: 
Login Succeeded
[root@t460 ~]# cat ~/.docker/config.json 
{
        "auths": {
                "https://index.docker.io/v1/": {
                        "auth": "REDACTED"
                }
        }
}[root@t460 ~]# docker push docker.io/thoraxe/openshift-centos-samba:0.1
The push refers to a repository [docker.io/thoraxe/openshift-centos-samba]
9c3584bd1ab0: Preparing 
bcd7eca3bee4: Preparing 
cf516324493c: Preparing 


Actual results:
unauthorized: authentication required


Expected results:
push succeeded

Additional info:
working /etc/sysconfig/docker:
[root@t460 ~]# cat /etc/sysconfig/docker | grep -v \#

OPTIONS='--selinux-enabled --log-driver=journald --insecure-registry docker.io --insecure-registry 172.30.0.0/16 --insecure-registry brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888 --insecure-registry registry.ops.openshift.com' 
DOCKER_CERT_PATH=/etc/docker

ADD_REGISTRY='--add-registry docker.io --add-registry brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888 --add-registry registry.ops.openshift.com --add-registry registry.access.redhat.com'

Comment 1 Antonio Murdaca 2017-10-25 16:31:45 UTC
I'll look into this asap, sounds like a bug

Comment 2 Tom Sweeney 2017-10-25 16:44:07 UTC
What's weird Antonio is that I was not able to reproduce this on my Fedora machine last night using my registry (not Thoraxe) on dockerhub.  The only time I ran into that error was when I'd logged in on my secondary dockerhub account and tried to push to a registry in my primary dockerhub account.

In case it helps, here's my docker/machine info. 


docker version
Client:
 Version:         1.12.6
 API version:     1.24
 Package version: docker-common-1.12.6-7.gitae7d637.fc25.x86_64
 Go version:      go1.7.6
 Git commit:      ae7d637/1.12.6
 Built:           Tue Jul 18 16:18:12 2017
 OS/Arch:         linux/amd64

Server:
 Version:         1.12.6
 API version:     1.24
 Package version: docker-common-1.12.6-7.gitae7d637.fc25.x86_64
 Go version:      go1.7.6
 Git commit:      ae7d637/1.12.6
 Built:           Tue Jul 18 16:18:12 2017
 OS/Arch:         linux/amd64


# cat /etc/*release
Fedora release 25 (Twenty Five)
NAME=Fedora
VERSION="25 (Server Edition)"
ID=fedora
VERSION_ID=25
PRETTY_NAME="Fedora 25 (Server Edition)"
ANSI_COLOR="0;34"
CPE_NAME="cpe:/o:fedoraproject:fedora:25"
HOME_URL="https://fedoraproject.org/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=25
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=25
PRIVACY_POLICY_URL=https://fedoraproject.org/wiki/Legal:PrivacyPolicy
VARIANT="Server Edition"
VARIANT_ID=server
Fedora release 25 (Twenty Five)
Fedora release 25 (Twenty Five)

Comment 3 Tom Sweeney 2017-10-25 16:52:48 UTC
and I should have added, I'd made no changes to /etc/sysconfig/docker.

Comment 4 Antonio Murdaca 2017-10-27 10:07:03 UTC
(In reply to Tom Sweeney from comment #2)
> What's weird Antonio is that I was not able to reproduce this on my Fedora
> machine last night using my registry (not Thoraxe) on dockerhub.  The only
> time I ran into that error was when I'd logged in on my secondary dockerhub
> account and tried to push to a registry in my primary dockerhub account.
> 

Please provide the steps you used. I can't understand the steps you did Tom, "push to a registry in my primary dockerhub account" isn't helping, which registry? the dockderhub is the registry :/

FWIW I can't reproduce on a freshly provisioned Fedora 25 VM

Comment 5 Antonio Murdaca 2017-10-27 10:09:23 UTC
```
[root@fedora ~]# cat /proc/`pidof dockerd-current`/cmdline
/usr/bin/dockerd-current--add-runtimeoci=/usr/libexec/docker/docker-runc-current--default-runtime=oci--containerd/run/containerd.sock--exec-optnative.cgroupdriver=systemd--userland-proxy-path=/usr/libexec/docker/docker-proxy-current--selinux-enabled--log-driver=journald
```
I'm not running with any --add-registry or --insecure-registry as you can see above. Docker is the same version as yours + same operating system and version. I can't reproduce.

I'm logged in into my personal account, I build and tag an image and I can push it:

```
[root@fedora ~]# cat ~/.docker/config.json
{
        "auths": {
                "https://index.docker.io/v1/": {
                        "auth": "REDACTED"
                }
        }
[root@fedora ~]# docker push docker.io/runcom/testbz0:latest
The push refers to a repository [docker.io/runcom/testbz0]
0348e21b9b3c: Pushed
6a749002dd6a: Pushed
latest: digest: sha256:03143daf02975912bbb89493c2ac635042fec76f0bd69f363b76d7d0750a8ec7 size: 734
```

Comment 6 Antonio Murdaca 2017-10-27 10:12:16 UTC
no wait, I reproduced, but you need to have one other registry in add-registry :

```
[root@fedora ~]# docker push docker.io/runcom/testbz1:latest
The push refers to a repository [docker.io/runcom/testbz1]
0348e21b9b3c: Preparing
6a749002dd6a: Preparing
unauthorized: authentication required

```

```
[root@fedora ~]# cat /proc/`pidof dockerd-current`/cmdline
/usr/bin/dockerd-current--add-runtimeoci=/usr/libexec/docker/docker-runc-current--default-runtime=oci--containerd/run/containerd.sock--exec-optnative.cgroupdriver=systemd--userland-proxy-path=/usr/libexec/docker/docker-proxy-current--selinux-enabled--log-driver=journald--add-registrytest.example.com:5000
```

Comment 7 Antonio Murdaca 2017-10-27 11:13:32 UTC
The patch is already backported here https://github.com/projectatomic/docker/commit/146d0e3c335143bc156c18bea6332969f71cf6c1

The thing, we never rebuilt docker-1.12.6 in Fedora (as just F25 has it).

Lokesh could you rebuild?

Comment 8 Tom Sweeney 2017-10-28 18:44:42 UTC
@runcom, sorry just getting back to you on this.  I think you're all set.  Also I keep interchanging repository/registry.  I was using my private registry on tomsweeneyredhat/testing on dockerhub.com.  I was using the same commands that Erik was and wasn't able to recreate it.  I'd not change /etc/syconfig/docker and it sounds like changes there triggers the problem.

Comment 9 Erik M Jacobs 2017-10-30 13:48:27 UTC
If you look at my config from the original post, I had brew and registry.access listed as additional.

With only brew+access listed, and no docker.io, I had the "authentication required" problem.

With brew+access+docker.io the problem goes away.

If the problem does not exist with a "pure" config (eg: zero additional registries listed) then perhaps there is some weird thing going on with the additionals, if that makes any sense. 

It sounds like @runcom reproduced and confirms this behavior in #6

Comment 10 Fedora End Of Life 2017-11-16 19:47:28 UTC
This message is a reminder that Fedora 25 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 25. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '25'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 25 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

Comment 11 Fedora End Of Life 2017-12-12 10:05:25 UTC
Fedora 25 changed to end-of-life (EOL) status on 2017-12-12. Fedora 25 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.