Bug 1667824 - Pod created from registry with self-certificate cannot be running with 509 error
Summary: Pod created from registry with self-certificate cannot be running with 509 error
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Image Registry
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.1.0
Assignee: Ben Parees
QA Contact: Wenjing Zheng
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-01-21 08:53 UTC by Wenjing Zheng
Modified: 2019-06-04 10:42 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
undefined
Clone Of:
Environment:
Last Closed: 2019-06-04 10:42:06 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Build log (18.49 KB, text/plain)
2019-01-21 08:53 UTC, Wenjing Zheng
no flags Details
oc get cm -o yaml -n openshift-image-registry (4.94 KB, text/plain)
2019-01-22 02:56 UTC, Wenjing Zheng
no flags Details
oc get ds -o node-ca -o yaml -n openshift-image-registry (4.45 KB, text/plain)
2019-01-22 02:56 UTC, Wenjing Zheng
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:0758 0 None None None 2019-06-04 10:42:13 UTC

Description Wenjing Zheng 2019-01-21 08:53:35 UTC
Created attachment 1522048 [details]
Build log

Description of problem:
Set both spec.AdditionalTrustedCA in master API and spec.caConfigName in the
image registry config to use the correct CA configmap, pod create from registry with self-certificate cannot be running with 509 error:
F0121 08:12:15.952661       1 helpers.go:119] error: build error: Error determining manifest MIME type for docker://docker-registry-default.apps.0121-0yr.qe.rhcloud.com/test/ruby-25-centos7:latest: pinging docker registry returned: Get https://docker-registry-default.apps.0121-0yr.qe.rhcloud.com/v2/: x509: certificate signed by unknown authority

Build logs with loglevel=6 will be attached to attachment.

Version-Release number of selected component (if applicable):
4.0.0-0.1

How reproducible:
Always

Steps to Reproduce:
1.Set both spec.AdditionalTrustedCA in master API and spec.caConfigName in the
image registry config to use the correct CA configmap
2.Start a build with  DockerImage strategy with the image from secure registry like below:
  strategy:
    sourceStrategy:
      env:
      - name: BUILD_LOGLEVEL
        value: "6"
      from:
        kind: DockerImage
        name: docker-registry-default.apps.0121-0yr.qe.rhcloud.com/test/ruby-25-centos7
3.Trigger build

Actual results:
Build go failed with 509 error

Expected results:
It should be running

Additional info:
A pod started directly from image which is from the registry with self-signed certificate will also be failed to be running for 509 error:
Message
  ----     ------     ----                ----                                                -------
  Normal   Scheduled  52m                 default-scheduler                                   Successfully assigned wzheng2/my-pod to ip-10-0-174-51.us-east-2.compute.internal
  Normal   Pulling    50m (x4 over 52m)   kubelet, ip-10-0-174-51.us-east-2.compute.internal  pulling image "docker-registry-default.apps.0121-0yr.qe.rhcloud.com/test/myimage"
  Warning  Failed     50m (x4 over 52m)   kubelet, ip-10-0-174-51.us-east-2.compute.internal  Failed to pull image "docker-registry-default.apps.0121-0yr.qe.rhcloud.com/test/myimage": rpc error: code = Unknown desc = pinging docker registry returned: Get https://docker-registry-default.apps.0121-0yr.qe.rhcloud.com/v2/: x509: certificate signed by unknown authority
  Warning  Failed     50m (x4 over 52m)   kubelet, ip-10-0-174-51.us-east-2.compute.internal  Error: ErrImagePull
  Normal   BackOff    50m (x6 over 52m)   kubelet, ip-10-0-174-51.us-east-2.compute.internal  Back-off pulling image "docker-registry-default.apps.0121-0yr.qe.rhcloud.com/test/myimage"
  Warning  Failed     2m (x214 over 52m)  kubelet, ip-10-0-174-51.us-east-2.compute.internal  Error: ImagePullBackOff

Comment 1 Ben Parees 2019-01-21 14:55:44 UTC
Please provide the output of:

oc get ds -o node-ca -o yaml -n openshift-image-registry
oc get cm -o yaml -n openshift-image-registry

and then on your nodes, "tree /etc/docker/certs.d"

Comment 2 Ben Parees 2019-01-21 15:13:31 UTC
fyi the build image pull won't work until https://jira.coreos.com/browse/DEVEXP-154 is complete, so that is not a bug right now.

However pods should be able to pull images from a registry if the CAs are setup properly, the information i requested in comment 1 should help determine what is happening there.

Comment 3 Wenjing Zheng 2019-01-22 02:56:05 UTC
Created attachment 1522305 [details]
oc get cm -o yaml -n openshift-image-registry

Comment 4 Wenjing Zheng 2019-01-22 02:56:28 UTC
Created attachment 1522306 [details]
oc get ds -o node-ca -o yaml -n openshift-image-registry

Comment 5 Wenjing Zheng 2019-01-22 02:57:20 UTC
Output of /etc/docker/certs.d from my node:
[root@ip-10-0-155-184 ~]# ls /etc/docker/certs.d/ca.crt/
ca.crt
[root@ip-10-0-155-184 ~]# cat /etc/docker/certs.d/ca.crt/ca.crt
-----BEGIN CERTIFICATE-----
MIIC6jCCAdKgAwIBAgIBATANBgkqhkiG9w0BAQsFADAmMSQwIgYDVQQDDBtvcGVu
c2hpZnQtc2lnbmVyQDE1NDgxMjI1NjQwHhcNMTkwMTIyMDIwMjQ0WhcNMjQwMTIx
MDIwMjQ1WjAmMSQwIgYDVQQDDBtvcGVuc2hpZnQtc2lnbmVyQDE1NDgxMjI1NjQw
ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDDaNWKPWXn/yB+4RrDb0t4
7wjbGLk4GPwEkqtQArmNN8anbn6wdXdKySfjnlYYCWtp/T+LIWm4DShwWjH55yXq
8eD4lgDJQOhftbshpEyb42kd95bZi+hTpVdVUan6pG3WLvQSB1Ho9LAyAc6ha2KE
QvMe1D4PvrF0eOsIwrWHCGuAKQxCJ1yV030GADK984nXcYxQMDG2NZpyZSvfkJnz
k06HvobC7598DVGvjmoup0lg6QcTfOWBaYdXGDg3cFzapnq7clZOLX1DyztbJz0+
ksrpDovmKSktoNQcrr/wpd+3jkp8D91gS5lZ+/O6jteZVHQ1NbY+i/RcCl1DRuJ/
AgMBAAGjIzAhMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MA0GCSqG
SIb3DQEBCwUAA4IBAQA/MRheOL1ViVZYTase4SNlWpFwO7zbzEvJjgi3TjK0s1r+
sl050bccdNelnMwJS3P96HYpHwbrn3aI5hvdgVn+QuVaFon2ETryIWmq/HHED+Iu
nssQICtpX3XhbkgLEEOpW78UiTSzDyRjLBKGucvfu+a8EbiWcT0hSNz9DGcPO36C
+mmPBYoaZl4eijYdfZ8wZ+YcabrSCWVKnP0ynvg8zhNQYgvL1IEqTf2LKjO1Q1jO
LfsJI3Et16cSvvKsaOjDbb1t7XRtPEXN9RPWUv7YTY9ZL5j/ugCc9eg6eN6b+gwg
HEss8zlkwNeThJoIByNJmGg4qpc3YZAbFJj3j6VQ
-----END CERTIFICATE-----

Comment 6 Wenjing Zheng 2019-01-22 03:02:24 UTC
This is my test file about creating a pod:
apiVersion: v1
kind: Pod
metadata:
  name: my-pod
  labels:
    app: my-pod
spec:
  containers:
  - name: busybox
    image: docker-registry-default.apps.0122-47n.qe.rhcloud.com/test/myimage
    command: ["sh", "-c", "while true; do sleep 10; done"]
    ports:
    - containerPort: 80

Comment 7 Ben Parees 2019-01-24 15:26:43 UTC
> cat /etc/docker/certs.d/ca.crt/ca.crt

this is the issue.  The cert should be in a directory named "docker-registry-default.apps.0122-47n.qe.rhcloud.com".  Since you named your key in the configmap "ca.crt" it is not.

You need to update your configmap, the key in the configmap needs to be the name of the registry host.  In addition if you need to specify a port, you need to define the key using ".." instead of ":" (because : is not a valid character for a configmap key):

for example:
host.domain.com..5000

Comment 8 Wenjing Zheng 2019-01-28 06:43:24 UTC
Thanks, Ben!

Verified with below version:
$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE     STATUS
version   4.0.0-0.nightly-2019-01-25-205123   True        False         1h        Cluster version is 4.0.0-0.nightly-2019-01-25-205123
[root@ip-10-0-135-86 ~]# ls /etc/docker/certs.d/
docker-registry-default.apps.0128-49c.qe.rhcloud.com  image-registry.openshift-image-registry.svc:5000  image-registry.openshift-image-registry.svc.cluster.local:5000
[root@ip-10-0-135-86 ~]# ls /etc/docker/certs.d/docker-registry-default.apps.0128-49c.qe.rhcloud.com
ca.crt

[wzheng@laptop test]$ oc get pods
NAME      READY     STATUS    RESTARTS   AGE
my-pod    1/1       Running   0          34s

Events:
  Type    Reason     Age   From                                                 Message
  ----    ------     ----  ----                                                 -------
  Normal  Scheduled  3m    default-scheduler                                    Successfully assigned wzheng1/my-pod to ip-10-0-154-255.us-east-2.compute.internal
  Normal  Pulling    3m    kubelet, ip-10-0-154-255.us-east-2.compute.internal  pulling image "docker-registry-default.apps.0128-49c.qe.rhcloud.com/test/myimage"
  Normal  Pulled     3m    kubelet, ip-10-0-154-255.us-east-2.compute.internal  Successfully pulled image "docker-registry-default.apps.0128-49c.qe.rhcloud.com/test/myimage"
  Normal  Created    3m    kubelet, ip-10-0-154-255.us-east-2.compute.internal  Created container
  Normal  Started    3m    kubelet, ip-10-0-154-255.us-east-2.compute.internal  Started container

[wzheng@laptop test]$ cat busybox.yml 
apiVersion: v1
kind: Pod
metadata:
  name: my-pod
  labels:
    app: my-pod
spec:
  containers:
  - name: busybox
    image: docker-registry-default.apps.0128-49c.qe.rhcloud.com/test/myimage
    command: ["sh", "-c", "while true; do sleep 10; done"]
    ports:
    - containerPort: 80

Comment 11 errata-xmlrpc 2019-06-04 10:42:06 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758


Note You need to log in before you can comment on or make changes to this bug.