Description of problem: Following the instruction on registry console web UI, try to docker push image to docker registry via exposed docker-registry route, failed, saying "unauthorized: authentication required". Version-Release number of selected component (if applicable): openshift3/registry-console:3.3 (57a566a04a79) openshift-ansible-3.3.16-1.git.0.9ed3fd2.el7.noarch docker-1.10.3-46.el7.10.x86_64 How reproducible: Always Steps to Reproduce: 1.Run ansible installer to install standalone atomic registry. deployment_type=openshift-enterprise deployment_subtype=registry openshift_cockpit_deployer_prefix=brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/ 2. After installation, check registry console is created. # oc get po NAME READY STATUS RESTARTS AGE docker-registry-2-djlbn 1/1 Running 4 3h registry-console-1-5esrx 1/1 Running 4 3h router-1-ihf7r 1/1 Running 6 3h # oc get svc NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE docker-registry 172.30.163.44 <none> 5000/TCP 23h kubernetes 172.30.0.1 <none> 443/TCP,53/UDP,53/TCP 23h registry-console 172.30.239.73 <none> 9000/TCP 23h router 172.30.157.225 <none> 80/TCP,443/TCP,1936/TCP 23h # oc get route NAME HOST/PORT PATH SERVICES PORT TERMINATION docker-registry docker-registry-default.0829-90c.qe.rhcloud.com docker-registry 5000-tcp registry-console registry-console-default.0829-90c.qe.rhcloud.com registry-console registry-console passthrough 3. Log in, create a test project (aa) on web UI. Try docker push some images to docker-registry (using :80 for workaround of 1371031). # docker login -p EIfBxE_Jn0oEoX1FWampQT73UsFizxbnQDB8H1N5fp4 -e unused -u unused docker-registry-default.0829-90c.qe.rhcloud.com:80 # docker pull busybox # docker tag busybox docker-registry-default.0829-90c.qe.rhcloud.com:80/aa/test:v1 # docker push docker-registry-default.0829-90c.qe.rhcloud.com:80/aa/test:v1 Actual results: # docker push docker-registry-default.0829-90c.qe.rhcloud.com:80/aa/test:v1 The push refers to a repository [docker-registry-default.0829-90c.qe.rhcloud.com:80/aa/test] 8ac8bfaff55a: Pushing [==================================================>] 1.125 MB unauthorized: authentication required Registry log: 10.1.0.1 - - [30/Aug/2016:00:53:11 -0400] "GET /v2/ HTTP/1.1" 401 87 "" "docker/1.10.3 go/go1.6.2 git-commit/2a93377-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64" time="2016-08-30T00:53:11.359312461-04:00" level=debug msg="authorizing request" go.version=go1.6.2 http.request.host=docker-registry-default.0829-90c.qe.rhcloud.com http.request.id=46148b40-b46e-4052-9c16-735177ec6a36 http.request.method=GET http.request.remoteaddr=10.14.6.214 http.request.uri="/v2/" http.request.useragent="docker/1.10.3 go/go1.6.2 git-commit/2a93377-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64" instance.id=212cc675-893f-4bbd-a874-8ae8475df8f4 time="2016-08-30T00:53:11.359580346-04:00" level=error msg="error authorizing context: authorization header required" go.version=go1.6.2 http.request.host=docker-registry-default.0829-90c.qe.rhcloud.com http.request.id=46148b40-b46e-4052-9c16-735177ec6a36 http.request.method=GET http.request.remoteaddr=10.14.6.214 http.request.uri="/v2/" http.request.useragent="docker/1.10.3 go/go1.6.2 git-commit/2a93377-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64" instance.id=212cc675-893f-4bbd-a874-8ae8475df8f4 time="2016-08-30T00:53:11.373768864-04:00" level=info msg="response completed" go.version=go1.6.2 http.request.host=docker-registry-default.0829-90c.qe.rhcloud.com http.request.id=1c94e2ed-6b01-43a0-93f0-efc9137a2f56 http.request.method=GET http.request.remoteaddr=10.14.6.214 http.request.uri="/openshift/token?account=unused&scope=repository%3Aaa%2Ftest%3Apush%2Cpull" http.request.useragent="docker/1.10.3 go/go1.6.2 git-commit/2a93377-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64" http.response.contenttype="application/json" http.response.duration=4.969789ms http.response.status=200 http.response.written=117 instance.id=212cc675-893f-4bbd-a874-8ae8475df8f4 10.1.0.1 - - [30/Aug/2016:00:53:11 -0400] "GET /openshift/token?account=unused&scope=repository%3Aaa%2Ftest%3Apush%2Cpull HTTP/1.1" 200 117 "" "docker/1.10.3 go/go1.6.2 git-commit/2a93377-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64" time="2016-08-30T00:53:11.382612309-04:00" level=debug msg="authorizing request" go.version=go1.6.2 http.request.host=docker-registry-default.0829-90c.qe.rhcloud.com http.request.id=806d2bb9-4b8f-4679-bfe2-d6e186924f4a http.request.method=POST http.request.remoteaddr=10.14.6.214 http.request.uri="/v2/aa/test/blobs/uploads/" http.request.useragent="docker/1.10.3 go/go1.6.2 git-commit/2a93377-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64" instance.id=212cc675-893f-4bbd-a874-8ae8475df8f4 vars.name="aa/test" time="2016-08-30T00:53:11.382873965-04:00" level=debug msg="Origin auth: checking for access to repository:aa/test:pull" go.version=go1.6.2 http.request.host=docker-registry-default.0829-90c.qe.rhcloud.com http.request.id=806d2bb9-4b8f-4679-bfe2-d6e186924f4a http.request.method=POST http.request.remoteaddr=10.14.6.214 http.request.uri="/v2/aa/test/blobs/uploads/" http.request.useragent="docker/1.10.3 go/go1.6.2 git-commit/2a93377-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64" instance.id=212cc675-893f-4bbd-a874-8ae8475df8f4 vars.name="aa/test" time="2016-08-30T00:53:11.389018237-04:00" level=debug msg="Origin auth: checking for access to repository:aa/test:push" go.version=go1.6.2 http.request.host=docker-registry-default.0829-90c.qe.rhcloud.com http.request.id=806d2bb9-4b8f-4679-bfe2-d6e186924f4a http.request.method=POST http.request.remoteaddr=10.14.6.214 http.request.uri="/v2/aa/test/blobs/uploads/" http.request.useragent="docker/1.10.3 go/go1.6.2 git-commit/2a93377-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64" instance.id=212cc675-893f-4bbd-a874-8ae8475df8f4 vars.name="aa/test" time="2016-08-30T00:53:11.393488648-04:00" level=debug msg="(*linkedBlobStore).Writer" go.version=go1.6.2 http.request.host=docker-registry-default.0829-90c.qe.rhcloud.com http.request.id=806d2bb9-4b8f-4679-bfe2-d6e186924f4a http.request.method=POST http.request.remoteaddr=10.14.6.214 http.request.uri="/v2/aa/test/blobs/uploads/" http.request.useragent="docker/1.10.3 go/go1.6.2 git-commit/2a93377-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64" instance.id=212cc675-893f-4bbd-a874-8ae8475df8f4 vars.name="aa/test" time="2016-08-30T00:53:11.405152377-04:00" level=debug msg="filesystem.PutContent(\"/docker/registry/v2/repositories/aa/test/_uploads/41a29f21-91a4-45bd-9c87-183499270818/startedat\")" go.version=go1.6.2 http.request.host=docker-registry-default.0829-90c.qe.rhcloud.com http.request.id=806d2bb9-4b8f-4679-bfe2-d6e186924f4a http.request.method=POST http.request.remoteaddr=10.14.6.214 http.request.uri="/v2/aa/test/blobs/uploads/" http.request.useragent="docker/1.10.3 go/go1.6.2 git-commit/2a93377-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64" instance.id=212cc675-893f-4bbd-a874-8ae8475df8f4 trace.duration=11.519494ms trace.file="/builddir/build/BUILD/atomic-openshift-git-0.2772547/_output/local/go/src/github.com/openshift/origin/vendor/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/openshift/origin/vendor/github.com/docker/distribution/registry/storage/driver/base.(*Base).PutContent" trace.id=9f0f3cd4-99d5-4670-8f9b-703d5999b783 trace.line=95 vars.name="aa/test" time="2016-08-30T00:53:11.408699931-04:00" level=debug msg="filesystem.Writer(\"/docker/registry/v2/repositories/aa/test/_uploads/41a29f21-91a4-45bd-9c87-183499270818/data\", false)" go.version=go1.6.2 http.request.host=docker-registry-default.0829-90c.qe.rhcloud.com http.request.id=806d2bb9-4b8f-4679-bfe2-d6e186924f4a http.request.method=POST http.request.remoteaddr=10.14.6.214 http.request.uri="/v2/aa/test/blobs/uploads/" http.request.useragent="docker/1.10.3 go/go1.6.2 git-commit/2a93377-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64" instance.id=212cc675-893f-4bbd-a874-8ae8475df8f4 trace.duration=3.390136ms trace.file="/builddir/build/BUILD/atomic-openshift-git-0.2772547/_output/local/go/src/github.com/openshift/origin/vendor/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/openshift/origin/vendor/github.com/docker/distribution/registry/storage/driver/base.(*Base).Writer" trace.id=5c1d1ce1-4c6a-46c2-925e-4c1107ad1d6e trace.line=124 vars.name="aa/test" time="2016-08-30T00:53:11.41999941-04:00" level=debug msg="filesystem.PutContent(\"/docker/registry/v2/repositories/aa/test/_uploads/41a29f21-91a4-45bd-9c87-183499270818/hashstates/sha256/0\")" go.version=go1.6.2 http.request.host=docker-registry-default.0829-90c.qe.rhcloud.com http.request.id=806d2bb9-4b8f-4679-bfe2-d6e186924f4a http.request.method=POST http.request.remoteaddr=10.14.6.214 http.request.uri="/v2/aa/test/blobs/uploads/" http.request.useragent="docker/1.10.3 go/go1.6.2 git-commit/2a93377-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64" instance.id=212cc675-893f-4bbd-a874-8ae8475df8f4 trace.duration=10.722674ms trace.file="/builddir/build/BUILD/atomic-openshift-git-0.2772547/_output/local/go/src/github.com/openshift/origin/vendor/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/openshift/origin/vendor/github.com/docker/distribution/registry/storage/driver/base.(*Base).PutContent" trace.id=2ba7e25e-d5f3-4751-8b30-6164da633004 trace.line=95 vars.name="aa/test" time="2016-08-30T00:53:11.420449703-04:00" level=info msg="response completed" go.version=go1.6.2 http.request.host=docker-registry-default.0829-90c.qe.rhcloud.com http.request.id=806d2bb9-4b8f-4679-bfe2-d6e186924f4a http.request.method=POST http.request.remoteaddr=10.14.6.214 http.request.uri="/v2/aa/test/blobs/uploads/" http.request.useragent="docker/1.10.3 go/go1.6.2 git-commit/2a93377-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64" http.response.duration=42.048728ms http.response.status=202 http.response.written=0 instance.id=212cc675-893f-4bbd-a874-8ae8475df8f4 10.1.0.1 - - [30/Aug/2016:00:53:11 -0400] "POST /v2/aa/test/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/1.10.3 go/go1.6.2 git-commit/2a93377-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64" time="2016-08-30T00:53:11.454768411-04:00" level=debug msg="authorizing request" go.version=go1.6.2 http.request.host=docker-registry-default.0829-90c.qe.rhcloud.com http.request.id=50975ed0-0de2-4cf2-bb6c-3b138850125b http.request.method=PATCH http.request.remoteaddr=10.14.6.214 http.request.uri="/v2/aa/test/blobs/uploads/41a29f21-91a4-45bd-9c87-183499270818?_state=w4HGqt0K2XVfO57Kl7QI7JvNnKmlbV5oTuoGN9oBcvt7Ik5hbWUiOiJhYS90ZXN0IiwiVVVJRCI6IjQxYTI5ZjIxLTkxYTQtNDViZC05Yzg3LTE4MzQ5OTI3MDgxOCIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAxNi0wOC0zMFQwNDo1MzoxMS4zOTM1NTM3MjdaIn0%3D" http.request.useragent="docker/1.10.3 go/go1.6.2 git-commit/2a93377-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64" instance.id=212cc675-893f-4bbd-a874-8ae8475df8f4 vars.name="aa/test" vars.uuid=41a29f21-91a4-45bd-9c87-183499270818 time="2016-08-30T00:53:11.455011307-04:00" level=error msg="error authorizing context: authorization header required" go.version=go1.6.2 http.request.host=docker-registry-default.0829-90c.qe.rhcloud.com http.request.id=50975ed0-0de2-4cf2-bb6c-3b138850125b http.request.method=PATCH http.request.remoteaddr=10.14.6.214 http.request.uri="/v2/aa/test/blobs/uploads/41a29f21-91a4-45bd-9c87-183499270818?_state=w4HGqt0K2XVfO57Kl7QI7JvNnKmlbV5oTuoGN9oBcvt7Ik5hbWUiOiJhYS90ZXN0IiwiVVVJRCI6IjQxYTI5ZjIxLTkxYTQtNDViZC05Yzg3LTE4MzQ5OTI3MDgxOCIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAxNi0wOC0zMFQwNDo1MzoxMS4zOTM1NTM3MjdaIn0%3D" http.request.useragent="docker/1.10.3 go/go1.6.2 git-commit/2a93377-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64" instance.id=212cc675-893f-4bbd-a874-8ae8475df8f4 vars.name="aa/test" vars.uuid=41a29f21-91a4-45bd-9c87-183499270818 10.1.0.1 - - [30/Aug/2016:00:53:11 -0400] "PATCH /v2/aa/test/blobs/uploads/41a29f21-91a4-45bd-9c87-183499270818?_state=w4HGqt0K2XVfO57Kl7QI7JvNnKmlbV5oTuoGN9oBcvt7Ik5hbWUiOiJhYS90ZXN0IiwiVVVJRCI6IjQxYTI5ZjIxLTkxYTQtNDViZC05Yzg3LTE4MzQ5OTI3MDgxOCIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAxNi0wOC0zMFQwNDo1MzoxMS4zOTM1NTM3MjdaIn0%3D HTTP/1.1" 401 194 "" "docker/1.10.3 go/go1.6.2 git-commit/2a93377-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64" Expected results: Docker push should succeed. Additional info: 1. If push images to docker-registry svc endpoint, it succeed. # docker tag busybox 172.30.163.44:5000/aa/test:v1 # docker push 172.30.163.44:5000/aa/test:v1 The push refers to a repository [172.30.163.44:5000/aa/test] 8ac8bfaff55a: Pushed v1: digest: sha256:62cf2abb8ebdd10e9e7f025c42a609bd9fa5677f179e20c11411ff3acdfcd990 size: 2079 2. If I downgrade client docker version to 1.7.1, it succeed.
According to https://docs.openshift.org/latest/install_config/install/docker_registry.html#exposing-the-registry, seem like need configure secure docker-registry to run docker push command. Then I try to deploy a secure docker-registry, then docker push succeed.
This might be the same root cause as this BZ[1]. Please confirm by adding this environment var to the registry: oc env dc/docker-registry \ REGISTRY_AUTH_OPENSHIFT_TOKENREALM="http://<REGISTRY_ROUTE>" [1] https://bugzilla.redhat.com/show_bug.cgi?id=1367610
(In reply to Aaron Weitekamp from comment #2) > This might be the same root cause as this BZ[1]. Please confirm by adding > this environment var to the registry: > > oc env dc/docker-registry \ > REGISTRY_AUTH_OPENSHIFT_TOKENREALM="http://<REGISTRY_ROUTE>" > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1367610 Just try your suggestion, but still no any change. And I go through all the comments in BZ#1367610, I do not think they are the same root cause. In BZ#1367610, it is saying user can not docker login due to docker client does not have visibility to the service IP. While in my test scenarios, docker login successfully against exposed registry route, but failed when docker pushing images. # curl -v docker-registry-default.0829-90c.qe.rhcloud.com/v2/ * About to connect() to docker-registry-default.0829-90c.qe.rhcloud.com port 80 (#0) * Trying 10.14.6.214... * Connected to docker-registry-default.0829-90c.qe.rhcloud.com (10.14.6.214) port 80 (#0) > GET /v2/ HTTP/1.1 > User-Agent: curl/7.29.0 > Host: docker-registry-default.0829-90c.qe.rhcloud.com > Accept: */* > < HTTP/1.1 401 Unauthorized < Content-Type: application/json; charset=utf-8 < Docker-Distribution-Api-Version: registry/2.0 < Www-Authenticate: Bearer realm="http://docker-registry-default.0829-90c.qe.rhcloud.com:80/openshift/token" < Date: Wed, 31 Aug 2016 02:03:27 GMT < Content-Length: 87 < Set-Cookie: 9490b2ade541f0db80f4061c983cef9c=b5d4e8b7c9e3c6ac9d99c26e07e84226; path=/; HttpOnly < {"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]} * Connection #0 to host docker-registry-default.0829-90c.qe.rhcloud.com left intact See the above output, we could see the redirect is useing the hostname as what is addressed in BZ#1367610.
To be sure I understand the various combinations that work, is this right? 1. Latest docker client pushing to insecure <service-ip>:5000 works? 2. Latest docker client pushing to secure <route-host> works? 3. Docker client 1.7.1 pushing to insecure <route-host> works? And is the failing case: 1. insecure route 2. latest docker client pushing to <route-host> (or <route-host>:80?) Are you setting the <route-host> as an insecure registry in the docker client?
(In reply to Jordan Liggitt from comment #4) > To be sure I understand the various combinations that work, is this right? > > 1. Latest docker client pushing to insecure <service-ip>:5000 works? > 2. Latest docker client pushing to secure <route-host> works? > 3. Docker client 1.7.1 pushing to insecure <route-host> works? yes. > > > And is the failing case: > 1. insecure route > 2. latest docker client pushing to <route-host> (or <route-host>:80?) yes. > > Are you setting the <route-host> as an insecure registry in the docker > client? yes.
I'm not sure why the http exposed registry isn't able to push. By way of workaround I recommend using the secure registry endpoint with route tls termination: passthrough per documentation[1]. A typical deployment would be configured this way. This also resolves https://bugzilla.redhat.com/show_bug.cgi?id=1371031 [1] https://access.redhat.com/documentation/en/openshift-enterprise/3.2/paged/installation-and-configuration/chapter-2-installing#securing-the-registry
looks like an issue with port normalization in the docker client for default ports (:80 for http, or :443 for https). issue at https://github.com/docker/docker/issues/18469 fix at https://github.com/docker/distribution/pull/1868
can you try setting --insecure-registry and omitting ":80"?
(In reply to Jordan Liggitt from comment #12) > can you try setting --insecure-registry and omitting ":80"? It will fail which was tracked in BZ#1371031
https://github.com/openshift/openshift-ansible/pull/2409 secures the registry
Verified this bug with openshift-ansible-playbooks-3.3.22-1.git.0.6c888c2.el7.noarch, PASS. Now when deployment_subtype=registry, a secure registry will be deployed, docker push successfully. # oc get route NAME HOST/PORT PATH SERVICES PORT TERMINATION docker-registry docker-registry-default.0905-ef2.qe.rhcloud.com docker-registry 5000-tcp passthrough registry-console registry-console-default.0905-ef2.qe.rhcloud.com registry-console registry-console passthrough # docker login -p M9nJlQFXTIhS94c80evUiLkhHEwnhYwsot1590Yto_c -e unused -u unused docker-registry-default.0905-ef2.qe.rhcloud.com WARNING: login credentials saved in /root/.docker/config.json Login Succeeded # docker tag busybox docker-registry-default.0905-ef2.qe.rhcloud.com/jialiu2/test2 # docker push docker-registry-default.0905-ef2.qe.rhcloud.com/jialiu2/test2 The push refers to a repository [docker-registry-default.0905-ef2.qe.rhcloud.com/jialiu2/test2] 8ac8bfaff55a: Pushed latest: digest: sha256:d90946bdf65877e4ea40d7901ca084281300012d5be430054c1c147223932080 size: 2089
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:1933
There's a workaround described in https://bugzilla.redhat.com/show_bug.cgi?id=1383439#c11 which allows to use exposed insecure registry without :80 port suffixes.