Bug 1713333
| Summary: | Not able to pull images using registry IP | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Mohit <mvardhan> |
| Component: | Installer | Assignee: | Joseph Callen <jcallen> |
| Installer sub component: | openshift-ansible | QA Contact: | Gaoyun Pei <gpei> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | medium | ||
| Priority: | unspecified | CC: | aos-bugs, dyocum, jcallen, obulatov, pweil, vhernand, vjaypurk |
| Version: | 3.10.0 | ||
| Target Milestone: | --- | ||
| Target Release: | 3.10.z | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: |
Cause: The openshift ca was not added to the nodes trust store
Consequence: Inability to securely access the registry's service cluster ip address
Fix: Add to the openshift-node sync pod command (script) an additional check if the file was missing or old. If so copy or replace it.
Result: The openshift ca is added to the node's trust and able to pull images from the registry's service cluster ip address
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-07-24 13:47:19 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Mohit
2019-05-23 12:26:56 UTC
You should check /etc/docker/certs.d/ on the nodes. There should be directories for both `docker-registry.default.svc:5000` and `X.X.X.X:5000` with the CA file. So you need to copy /etc/origin/master/ca.crt to /etc/pki/ca-trust/source/anchors/ or to /etc/docker/certs.d/X.X.X.X:5000/. Why do you want to access the integrated registry by IP? Test with openshift-ansible-3.11.129-1.git.0.11838de.el7.noarch.rpm, after fresh install, /etc/pki/ca-trust/source/anchors/openshift-ca.crt was created on each node, but registry IP still not available unless restart docker service. Here's my steps: 1. Log into the 3.11 cluster as a regular user and get the access token # oc login x.x.x.x:8443 ... Login successful. # oc whoami -t abc_xxx_xxx 2. On master host, log in to the Registry using default DNS - docker-registry.default.svc:5000, could pull images from the hosted registry. # docker login -u xxx -p abc_xxx_xxx docker-registry.default.svc:5000 Login Succeeded # docker pull docker-registry.default.svc:5000/test/ruby-ex:latest Trying to pull repository docker-registry.default.svc:5000/test/ruby-ex ... latest: Pulling from docker-registry.default.svc:5000/test/ruby-ex 1af3926bc92f: Pull complete Digest: sha256:b14dfedd3d6101c247d3835e46cdedd26d005747b35a0db614cd44609fa25c28 Status: Downloaded newer image for docker-registry.default.svc:5000/test/ruby-ex:latest 3. Log in to the Registry using registry ip, failed with "unknown authority" error # oc get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE docker-registry ClusterIP 172.30.85.230 <none> 5000/TCP 12m # docker login -u xxx -p abc_xxx_xxx 172.30.85.230:5000 Error response from daemon: Get https://172.30.85.230:5000/v1/users/: x509: certificate signed by unknown authority 4 After restart docker service, it works as expected # systemctl restart docker # docker login -u xxx -p abc_xxx_xxx 172.30.85.230:5000 Login Succeeded # docker tag 68d575f0b328 172.30.85.230:5000/test/ruby-test # docker push 172.30.85.230:5000/test/ruby-test The push refers to a repository [172.30.85.230:5000/test/ruby-test] 06dcacd247e1: Pushed latest: digest: sha256:b14dfedd3d6101c247d3835e46cdedd26d005747b35a0db614cd44609fa25c28 size: 2421 Create the ca file for registry_ip under /etc/docker/certs.d/ also work in another attempt, without restarting docker service. # docker login -u xxx -p abc_xxx_xxx 172.30.85.230:5000 Error response from daemon: Get https://172.30.85.230:5000/v1/users/: x509: certificate signed by unknown authority # mkdir /etc/docker/certs.d/172.30.85.230:5000 # ls /etc/docker/certs.d/ 172.30.85.230:5000 docker-registry.default.svc:5000 redhat.com redhat.io registry.access.redhat.com # cp /etc/docker/certs.d/docker-registry.default.svc\:5000/node-client-ca.crt /etc/docker/certs.d/172.30.85.230\:5000/ # docker login -u xxx -p abc_xxx_xxx 172.30.85.230:5000 Login Succeeded Sorry for verifying this bug with wrong ocp version, https://github.com/openshift/openshift-ansible/pull/11721 in Comment 15 is created for release-3.11 branch, so just thought it was addressed in 3.11, this bug should be a 3.10 target one. Re-test with openshift-ansible-3.10.153-1.git.0.2363fa8.el7.noarch.rpm, after fresh install, it works as expected. /etc/pki/ca-trust/source/anchors/openshift-ca.crt was created on each node, and registry IP is also available on each node. [root@ip-172-18-5-133 ~]# ls /etc/pki/ca-trust/source/anchors/openshift-ca.crt /etc/pki/ca-trust/source/anchors/openshift-ca.crt [root@ip-172-18-5-133 ~]# docker login -u xxx -p xxx 172.30.93.115:5000 Login Succeeded [root@ip-172-18-5-133 ~]# docker pull 172.30.93.115:5000/test/ruby-ex:latest Trying to pull repository 172.30.93.115:5000/test/ruby-ex ... latest: Pulling from 172.30.93.115:5000/test/ruby-ex Digest: sha256:41587ec52f6114226692a01c86be6dc51f77417425ddec7c43e542fc1a0325f3 Status: Downloaded newer image for 172.30.93.115:5000/test/ruby-ex:latest Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:1755 *** Bug 1717428 has been marked as a duplicate of this bug. *** *** Bug 1729393 has been marked as a duplicate of this bug. *** |