Description of problem: openshift-ansible-3.10.127-1.git.0.131da09.el7.noarch ansible-2.4.6.0-1.el7ae.noarch ansible 2.4.6.0 Not able to pull images using registry IP. It works with registry name. $ docker pull X.X.X.X:5000/neuvector/neuvector-enforcer:latest Trying to pull repository X.X.X.X:5000/neuvector/neuvector-enforcer ... Get https://X.X.X.X:5000/v1/_ping: x509: certificate signed by unknown authority $ docker pull docker-registry.default.svc:5000/neuvector/neuvector-enforcer:latest Trying to pull repository docker-registry.default.svc:5000/neuvector/neuvector-enforcer ... 75bdb7117b5c: Pull complete Status: Downloaded newer image for docker-registry.default.svc:5000/neuvector/neuvector-enforcer:latest Cu has upgraded their environment from 3.9 to 3.10. Earlier in 3.9 for every new node they used to get the cert /etc/pki/ca-trust/source/anchors/openshift-ca.crt but from 3.10 this cert is not available. In 3.9 it used to work with IP as well. Now by default in 3.10 its not working but if we copy the above cert in new nodes then it starts working with IP as well. I checked in 3.10 test environment and I am getting similar error when trying to login using svc IP.
You should check /etc/docker/certs.d/ on the nodes. There should be directories for both `docker-registry.default.svc:5000` and `X.X.X.X:5000` with the CA file.
So you need to copy /etc/origin/master/ca.crt to /etc/pki/ca-trust/source/anchors/ or to /etc/docker/certs.d/X.X.X.X:5000/. Why do you want to access the integrated registry by IP?
PR: https://github.com/openshift/openshift-ansible/pull/11721
Test with openshift-ansible-3.11.129-1.git.0.11838de.el7.noarch.rpm, after fresh install, /etc/pki/ca-trust/source/anchors/openshift-ca.crt was created on each node, but registry IP still not available unless restart docker service. Here's my steps: 1. Log into the 3.11 cluster as a regular user and get the access token # oc login x.x.x.x:8443 ... Login successful. # oc whoami -t abc_xxx_xxx 2. On master host, log in to the Registry using default DNS - docker-registry.default.svc:5000, could pull images from the hosted registry. # docker login -u xxx -p abc_xxx_xxx docker-registry.default.svc:5000 Login Succeeded # docker pull docker-registry.default.svc:5000/test/ruby-ex:latest Trying to pull repository docker-registry.default.svc:5000/test/ruby-ex ... latest: Pulling from docker-registry.default.svc:5000/test/ruby-ex 1af3926bc92f: Pull complete Digest: sha256:b14dfedd3d6101c247d3835e46cdedd26d005747b35a0db614cd44609fa25c28 Status: Downloaded newer image for docker-registry.default.svc:5000/test/ruby-ex:latest 3. Log in to the Registry using registry ip, failed with "unknown authority" error # oc get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE docker-registry ClusterIP 172.30.85.230 <none> 5000/TCP 12m # docker login -u xxx -p abc_xxx_xxx 172.30.85.230:5000 Error response from daemon: Get https://172.30.85.230:5000/v1/users/: x509: certificate signed by unknown authority 4 After restart docker service, it works as expected # systemctl restart docker # docker login -u xxx -p abc_xxx_xxx 172.30.85.230:5000 Login Succeeded # docker tag 68d575f0b328 172.30.85.230:5000/test/ruby-test # docker push 172.30.85.230:5000/test/ruby-test The push refers to a repository [172.30.85.230:5000/test/ruby-test] 06dcacd247e1: Pushed latest: digest: sha256:b14dfedd3d6101c247d3835e46cdedd26d005747b35a0db614cd44609fa25c28 size: 2421
Create the ca file for registry_ip under /etc/docker/certs.d/ also work in another attempt, without restarting docker service. # docker login -u xxx -p abc_xxx_xxx 172.30.85.230:5000 Error response from daemon: Get https://172.30.85.230:5000/v1/users/: x509: certificate signed by unknown authority # mkdir /etc/docker/certs.d/172.30.85.230:5000 # ls /etc/docker/certs.d/ 172.30.85.230:5000 docker-registry.default.svc:5000 redhat.com redhat.io registry.access.redhat.com # cp /etc/docker/certs.d/docker-registry.default.svc\:5000/node-client-ca.crt /etc/docker/certs.d/172.30.85.230\:5000/ # docker login -u xxx -p abc_xxx_xxx 172.30.85.230:5000 Login Succeeded
Sorry for verifying this bug with wrong ocp version, https://github.com/openshift/openshift-ansible/pull/11721 in Comment 15 is created for release-3.11 branch, so just thought it was addressed in 3.11, this bug should be a 3.10 target one. Re-test with openshift-ansible-3.10.153-1.git.0.2363fa8.el7.noarch.rpm, after fresh install, it works as expected. /etc/pki/ca-trust/source/anchors/openshift-ca.crt was created on each node, and registry IP is also available on each node. [root@ip-172-18-5-133 ~]# ls /etc/pki/ca-trust/source/anchors/openshift-ca.crt /etc/pki/ca-trust/source/anchors/openshift-ca.crt [root@ip-172-18-5-133 ~]# docker login -u xxx -p xxx 172.30.93.115:5000 Login Succeeded [root@ip-172-18-5-133 ~]# docker pull 172.30.93.115:5000/test/ruby-ex:latest Trying to pull repository 172.30.93.115:5000/test/ruby-ex ... latest: Pulling from 172.30.93.115:5000/test/ruby-ex Digest: sha256:41587ec52f6114226692a01c86be6dc51f77417425ddec7c43e542fc1a0325f3 Status: Downloaded newer image for 172.30.93.115:5000/test/ruby-ex:latest
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:1755
*** Bug 1717428 has been marked as a duplicate of this bug. ***
*** Bug 1729393 has been marked as a duplicate of this bug. ***