Bug 1371423
Summary: | [atomic registry] fail to docker push with "authentication required" error against exposed docker-registry route. | ||
---|---|---|---|
Product: | OpenShift Container Platform | Reporter: | Johnny Liu <jialiu> |
Component: | Installer | Assignee: | Samuel Munilla <smunilla> |
Status: | CLOSED ERRATA | QA Contact: | Johnny Liu <jialiu> |
Severity: | high | Docs Contact: | |
Priority: | high | ||
Version: | 3.3.0 | CC: | abutcher, aos-bugs, aweiteka, bleanhar, jialiu, jliggitt, jokerman, miminar, mmccomas, yapei |
Target Milestone: | --- | Keywords: | TestBlocker |
Target Release: | --- | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | If docs needed, set a value | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2016-09-27 09:46:48 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Johnny Liu
2016-08-30 08:07:51 UTC
According to https://docs.openshift.org/latest/install_config/install/docker_registry.html#exposing-the-registry, seem like need configure secure docker-registry to run docker push command. Then I try to deploy a secure docker-registry, then docker push succeed. This might be the same root cause as this BZ[1]. Please confirm by adding this environment var to the registry: oc env dc/docker-registry \ REGISTRY_AUTH_OPENSHIFT_TOKENREALM="http://<REGISTRY_ROUTE>" [1] https://bugzilla.redhat.com/show_bug.cgi?id=1367610 (In reply to Aaron Weitekamp from comment #2) > This might be the same root cause as this BZ[1]. Please confirm by adding > this environment var to the registry: > > oc env dc/docker-registry \ > REGISTRY_AUTH_OPENSHIFT_TOKENREALM="http://<REGISTRY_ROUTE>" > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1367610 Just try your suggestion, but still no any change. And I go through all the comments in BZ#1367610, I do not think they are the same root cause. In BZ#1367610, it is saying user can not docker login due to docker client does not have visibility to the service IP. While in my test scenarios, docker login successfully against exposed registry route, but failed when docker pushing images. # curl -v docker-registry-default.0829-90c.qe.rhcloud.com/v2/ * About to connect() to docker-registry-default.0829-90c.qe.rhcloud.com port 80 (#0) * Trying 10.14.6.214... * Connected to docker-registry-default.0829-90c.qe.rhcloud.com (10.14.6.214) port 80 (#0) > GET /v2/ HTTP/1.1 > User-Agent: curl/7.29.0 > Host: docker-registry-default.0829-90c.qe.rhcloud.com > Accept: */* > < HTTP/1.1 401 Unauthorized < Content-Type: application/json; charset=utf-8 < Docker-Distribution-Api-Version: registry/2.0 < Www-Authenticate: Bearer realm="http://docker-registry-default.0829-90c.qe.rhcloud.com:80/openshift/token" < Date: Wed, 31 Aug 2016 02:03:27 GMT < Content-Length: 87 < Set-Cookie: 9490b2ade541f0db80f4061c983cef9c=b5d4e8b7c9e3c6ac9d99c26e07e84226; path=/; HttpOnly < {"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]} * Connection #0 to host docker-registry-default.0829-90c.qe.rhcloud.com left intact See the above output, we could see the redirect is useing the hostname as what is addressed in BZ#1367610. To be sure I understand the various combinations that work, is this right? 1. Latest docker client pushing to insecure <service-ip>:5000 works? 2. Latest docker client pushing to secure <route-host> works? 3. Docker client 1.7.1 pushing to insecure <route-host> works? And is the failing case: 1. insecure route 2. latest docker client pushing to <route-host> (or <route-host>:80?) Are you setting the <route-host> as an insecure registry in the docker client? (In reply to Jordan Liggitt from comment #4) > To be sure I understand the various combinations that work, is this right? > > 1. Latest docker client pushing to insecure <service-ip>:5000 works? > 2. Latest docker client pushing to secure <route-host> works? > 3. Docker client 1.7.1 pushing to insecure <route-host> works? yes. > > > And is the failing case: > 1. insecure route > 2. latest docker client pushing to <route-host> (or <route-host>:80?) yes. > > Are you setting the <route-host> as an insecure registry in the docker > client? yes. I'm not sure why the http exposed registry isn't able to push. By way of workaround I recommend using the secure registry endpoint with route tls termination: passthrough per documentation[1]. A typical deployment would be configured this way. This also resolves https://bugzilla.redhat.com/show_bug.cgi?id=1371031 [1] https://access.redhat.com/documentation/en/openshift-enterprise/3.2/paged/installation-and-configuration/chapter-2-installing#securing-the-registry looks like an issue with port normalization in the docker client for default ports (:80 for http, or :443 for https). issue at https://github.com/docker/docker/issues/18469 fix at https://github.com/docker/distribution/pull/1868 can you try setting --insecure-registry and omitting ":80"? (In reply to Jordan Liggitt from comment #12) > can you try setting --insecure-registry and omitting ":80"? It will fail which was tracked in BZ#1371031 https://github.com/openshift/openshift-ansible/pull/2409 secures the registry Verified this bug with openshift-ansible-playbooks-3.3.22-1.git.0.6c888c2.el7.noarch, PASS. Now when deployment_subtype=registry, a secure registry will be deployed, docker push successfully. # oc get route NAME HOST/PORT PATH SERVICES PORT TERMINATION docker-registry docker-registry-default.0905-ef2.qe.rhcloud.com docker-registry 5000-tcp passthrough registry-console registry-console-default.0905-ef2.qe.rhcloud.com registry-console registry-console passthrough # docker login -p M9nJlQFXTIhS94c80evUiLkhHEwnhYwsot1590Yto_c -e unused -u unused docker-registry-default.0905-ef2.qe.rhcloud.com WARNING: login credentials saved in /root/.docker/config.json Login Succeeded # docker tag busybox docker-registry-default.0905-ef2.qe.rhcloud.com/jialiu2/test2 # docker push docker-registry-default.0905-ef2.qe.rhcloud.com/jialiu2/test2 The push refers to a repository [docker-registry-default.0905-ef2.qe.rhcloud.com/jialiu2/test2] 8ac8bfaff55a: Pushed latest: digest: sha256:d90946bdf65877e4ea40d7901ca084281300012d5be430054c1c147223932080 size: 2089 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:1933 There's a workaround described in https://bugzilla.redhat.com/show_bug.cgi?id=1383439#c11 which allows to use exposed insecure registry without :80 port suffixes. |