Bug 1713333 - Not able to pull images using registry IP
Summary: Not able to pull images using registry IP
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.10.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 3.10.z
Assignee: Joseph Callen
QA Contact: Gaoyun Pei
: 1729393 (view as bug list)
Depends On:
TreeView+ depends on / blocked
Reported: 2019-05-23 12:26 UTC by Mohit
Modified: 2019-08-14 20:22 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: The openshift ca was not added to the nodes trust store Consequence: Inability to securely access the registry's service cluster ip address Fix: Add to the openshift-node sync pod command (script) an additional check if the file was missing or old. If so copy or replace it. Result: The openshift ca is added to the node's trust and able to pull images from the registry's service cluster ip address
Clone Of:
Last Closed: 2019-07-24 13:47:19 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:1755 0 None None None 2019-07-24 13:47:24 UTC

Description Mohit 2019-05-23 12:26:56 UTC
Description of problem:


Not able to pull images using registry IP.
It works with registry name.

$ docker pull X.X.X.X:5000/neuvector/neuvector-enforcer:latest
Trying to pull repository X.X.X.X:5000/neuvector/neuvector-enforcer ...
Get https://X.X.X.X:5000/v1/_ping: x509: certificate signed by unknown authority

$ docker pull docker-registry.default.svc:5000/neuvector/neuvector-enforcer:latest
Trying to pull repository docker-registry.default.svc:5000/neuvector/neuvector-enforcer ...
75bdb7117b5c: Pull complete
Status: Downloaded newer image for docker-registry.default.svc:5000/neuvector/neuvector-enforcer:latest

Cu has upgraded their environment from 3.9 to 3.10. Earlier in 3.9 for every new node they used to get the cert /etc/pki/ca-trust/source/anchors/openshift-ca.crt but from 3.10 this cert is not available. In 3.9 it used to work with IP as well.

Now by default in 3.10 its not working but if we copy the above cert in new nodes then it starts working with IP as well.

I checked in 3.10 test environment and I am getting similar error when trying to login using svc IP.

Comment 1 Oleg Bulatov 2019-05-23 16:32:47 UTC
You should check /etc/docker/certs.d/ on the nodes. There should be directories for both `docker-registry.default.svc:5000` and `X.X.X.X:5000` with the CA file.

Comment 3 Oleg Bulatov 2019-05-24 11:39:23 UTC
So you need to copy /etc/origin/master/ca.crt to /etc/pki/ca-trust/source/anchors/ or to /etc/docker/certs.d/X.X.X.X:5000/.

Why do you want to access the integrated registry by IP?

Comment 15 Joseph Callen 2019-06-26 20:21:07 UTC
PR: https://github.com/openshift/openshift-ansible/pull/11721

Comment 17 Gaoyun Pei 2019-07-11 09:35:13 UTC
Test with openshift-ansible-3.11.129-1.git.0.11838de.el7.noarch.rpm, after fresh install, /etc/pki/ca-trust/source/anchors/openshift-ca.crt was
created on each node, but registry IP still not available unless restart docker service.

Here's my steps:

1. Log into the 3.11 cluster as a regular user and get the access token

# oc login x.x.x.x:8443
Login successful.

# oc whoami -t

2. On master host, log in to the Registry using default DNS - docker-registry.default.svc:5000,
could pull images from the hosted registry.

# docker login -u xxx -p abc_xxx_xxx docker-registry.default.svc:5000
Login Succeeded

# docker pull docker-registry.default.svc:5000/test/ruby-ex:latest
Trying to pull repository docker-registry.default.svc:5000/test/ruby-ex ... 
latest: Pulling from docker-registry.default.svc:5000/test/ruby-ex
1af3926bc92f: Pull complete 
Digest: sha256:b14dfedd3d6101c247d3835e46cdedd26d005747b35a0db614cd44609fa25c28
Status: Downloaded newer image for docker-registry.default.svc:5000/test/ruby-ex:latest

3. Log in to the Registry using registry ip, failed with "unknown authority" error

# oc get svc
NAME               TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                   AGE
docker-registry    ClusterIP    <none>        5000/TCP                  12m

# docker login -u xxx -p abc_xxx_xxx
Error response from daemon: Get x509: certificate signed by unknown authority

4 After restart docker service, it works as expected

# systemctl restart docker

# docker login -u xxx -p abc_xxx_xxx
Login Succeeded

# docker tag 68d575f0b328

# docker push
The push refers to a repository []
06dcacd247e1: Pushed 
latest: digest: sha256:b14dfedd3d6101c247d3835e46cdedd26d005747b35a0db614cd44609fa25c28 size: 2421

Comment 18 Gaoyun Pei 2019-07-11 09:39:56 UTC
Create the ca file for registry_ip under /etc/docker/certs.d/ also work in another attempt, without restarting docker service.

# docker login -u xxx -p abc_xxx_xxx
Error response from daemon: Get x509: certificate signed by unknown authority

# mkdir /etc/docker/certs.d/

# ls /etc/docker/certs.d/  docker-registry.default.svc:5000  redhat.com  redhat.io  registry.access.redhat.com

# cp /etc/docker/certs.d/docker-registry.default.svc\:5000/node-client-ca.crt /etc/docker/certs.d/\:5000/

# docker login -u xxx -p abc_xxx_xxx
Login Succeeded

Comment 19 Gaoyun Pei 2019-07-16 08:17:55 UTC
Sorry for verifying this bug with wrong ocp version, https://github.com/openshift/openshift-ansible/pull/11721 in Comment 15 is created for release-3.11 branch, so just thought it was addressed in 3.11,
this bug should be a 3.10 target one.

Re-test with openshift-ansible-3.10.153-1.git.0.2363fa8.el7.noarch.rpm, after fresh install, it works as expected.

/etc/pki/ca-trust/source/anchors/openshift-ca.crt was created on each node, and registry IP is also available on each node.

[root@ip-172-18-5-133 ~]# ls /etc/pki/ca-trust/source/anchors/openshift-ca.crt 
[root@ip-172-18-5-133 ~]# docker login -u xxx -p xxx
Login Succeeded
[root@ip-172-18-5-133 ~]# docker pull
Trying to pull repository ... 
latest: Pulling from
Digest: sha256:41587ec52f6114226692a01c86be6dc51f77417425ddec7c43e542fc1a0325f3
Status: Downloaded newer image for

Comment 21 errata-xmlrpc 2019-07-24 13:47:19 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Comment 24 Joseph Callen 2019-08-14 20:19:05 UTC
*** Bug 1717428 has been marked as a duplicate of this bug. ***

Comment 25 Joseph Callen 2019-08-14 20:22:01 UTC
*** Bug 1729393 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.