Created attachment 1382748 [details] log from dc/docker-registry Description of problem: Launch an openshift cluster behind proxy, cannot login openshift container registry Version-Release number of selected component (if applicable): openshift v3.9.0-0.20.0 kubernetes v1.9.1+a0ce1bc657 etcd 3.2.8 How reproducible: ALways Steps to Reproduce: 1.Launch an openshift cluster behind proxy 2.Wait registry pod is ready, login openshift container registry Actual results: cannot login openshift container registry # docker login -u dyan -pGsiRdMqRIqzRNPP1VtA_EJu8ZUusBbLtmIABUFEsQ1Y 172.30.114.12:5000 Error response from daemon: Get https://172.30.114.12:5000/v1/users/: Forbidden Expected results: could login openshift container registry Additional info: could login openshift container registry successfully without proxy
Check the proxy variables on the docker-registry. # oc env dc/docker-registry --list |grep -i proxy NO_PROXY=.cluster.local,.svc,169.254.169.254,172.16.120.100,172.16.120.162 HTTP_PROXY=http://xxx.redhat.com:3128 HTTPS_PROXY=http://xxx.redhat.com:3128 Once appending the cluster CIDR and service CIDR (`172.30.0.0/16,10.128.0.0/14`) to NO_PROXY, build can succeed. # oc env dc/docker-registry NO_PROXY=.cluster.local,.svc,169.254.169.254,172.16.120.100,172.16.120.162,172.30.0.0/16,10.128.0.0/14 Assigning to Installer component.
The docker registry service ip address is prone to change. To work around that we've configured environments to push to the registry via the service name. Can you attempt to login to 'docker-registry.default.svc' rather than the service ip address? '.svc' should already be added to docker's NO_PROXY list in /etc/sysconfig/docker which will address this issue.
Scott, `.svc` had been already added to docker's NO_PROXY. From the logs of docker-registry, it seems that docker-registry has to talk to 172.30.0.1 that lead the failure, I think this is the root cause. We just need to make sure that cluster CIDR and service CIDR could be added to NO_PROXY by installer.
During build process, failed to push image to docker-registry.default.svc:5000, throw the same error: Error response from daemon: Get /v1/users/: Forbidden. In comment3 , ghuang has provided useful info
I believe this is same root cause as https://bugzilla.redhat.com/show_bug.cgi?id=1511870 with a proposed fix in https://github.com/openshift/openshift-ansible/pull/6215
*** This bug has been marked as a duplicate of bug 1527210 ***