Hide Forgot
Created attachment 1522048 [details] Build log Description of problem: Set both spec.AdditionalTrustedCA in master API and spec.caConfigName in the image registry config to use the correct CA configmap, pod create from registry with self-certificate cannot be running with 509 error: F0121 08:12:15.952661 1 helpers.go:119] error: build error: Error determining manifest MIME type for docker://docker-registry-default.apps.0121-0yr.qe.rhcloud.com/test/ruby-25-centos7:latest: pinging docker registry returned: Get https://docker-registry-default.apps.0121-0yr.qe.rhcloud.com/v2/: x509: certificate signed by unknown authority Build logs with loglevel=6 will be attached to attachment. Version-Release number of selected component (if applicable): 4.0.0-0.1 How reproducible: Always Steps to Reproduce: 1.Set both spec.AdditionalTrustedCA in master API and spec.caConfigName in the image registry config to use the correct CA configmap 2.Start a build with DockerImage strategy with the image from secure registry like below: strategy: sourceStrategy: env: - name: BUILD_LOGLEVEL value: "6" from: kind: DockerImage name: docker-registry-default.apps.0121-0yr.qe.rhcloud.com/test/ruby-25-centos7 3.Trigger build Actual results: Build go failed with 509 error Expected results: It should be running Additional info: A pod started directly from image which is from the registry with self-signed certificate will also be failed to be running for 509 error: Message ---- ------ ---- ---- ------- Normal Scheduled 52m default-scheduler Successfully assigned wzheng2/my-pod to ip-10-0-174-51.us-east-2.compute.internal Normal Pulling 50m (x4 over 52m) kubelet, ip-10-0-174-51.us-east-2.compute.internal pulling image "docker-registry-default.apps.0121-0yr.qe.rhcloud.com/test/myimage" Warning Failed 50m (x4 over 52m) kubelet, ip-10-0-174-51.us-east-2.compute.internal Failed to pull image "docker-registry-default.apps.0121-0yr.qe.rhcloud.com/test/myimage": rpc error: code = Unknown desc = pinging docker registry returned: Get https://docker-registry-default.apps.0121-0yr.qe.rhcloud.com/v2/: x509: certificate signed by unknown authority Warning Failed 50m (x4 over 52m) kubelet, ip-10-0-174-51.us-east-2.compute.internal Error: ErrImagePull Normal BackOff 50m (x6 over 52m) kubelet, ip-10-0-174-51.us-east-2.compute.internal Back-off pulling image "docker-registry-default.apps.0121-0yr.qe.rhcloud.com/test/myimage" Warning Failed 2m (x214 over 52m) kubelet, ip-10-0-174-51.us-east-2.compute.internal Error: ImagePullBackOff
Please provide the output of: oc get ds -o node-ca -o yaml -n openshift-image-registry oc get cm -o yaml -n openshift-image-registry and then on your nodes, "tree /etc/docker/certs.d"
fyi the build image pull won't work until https://jira.coreos.com/browse/DEVEXP-154 is complete, so that is not a bug right now. However pods should be able to pull images from a registry if the CAs are setup properly, the information i requested in comment 1 should help determine what is happening there.
Created attachment 1522305 [details] oc get cm -o yaml -n openshift-image-registry
Created attachment 1522306 [details] oc get ds -o node-ca -o yaml -n openshift-image-registry
Output of /etc/docker/certs.d from my node: [root@ip-10-0-155-184 ~]# ls /etc/docker/certs.d/ca.crt/ ca.crt [root@ip-10-0-155-184 ~]# cat /etc/docker/certs.d/ca.crt/ca.crt -----BEGIN CERTIFICATE----- MIIC6jCCAdKgAwIBAgIBATANBgkqhkiG9w0BAQsFADAmMSQwIgYDVQQDDBtvcGVu c2hpZnQtc2lnbmVyQDE1NDgxMjI1NjQwHhcNMTkwMTIyMDIwMjQ0WhcNMjQwMTIx MDIwMjQ1WjAmMSQwIgYDVQQDDBtvcGVuc2hpZnQtc2lnbmVyQDE1NDgxMjI1NjQw ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDDaNWKPWXn/yB+4RrDb0t4 7wjbGLk4GPwEkqtQArmNN8anbn6wdXdKySfjnlYYCWtp/T+LIWm4DShwWjH55yXq 8eD4lgDJQOhftbshpEyb42kd95bZi+hTpVdVUan6pG3WLvQSB1Ho9LAyAc6ha2KE QvMe1D4PvrF0eOsIwrWHCGuAKQxCJ1yV030GADK984nXcYxQMDG2NZpyZSvfkJnz k06HvobC7598DVGvjmoup0lg6QcTfOWBaYdXGDg3cFzapnq7clZOLX1DyztbJz0+ ksrpDovmKSktoNQcrr/wpd+3jkp8D91gS5lZ+/O6jteZVHQ1NbY+i/RcCl1DRuJ/ AgMBAAGjIzAhMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MA0GCSqG SIb3DQEBCwUAA4IBAQA/MRheOL1ViVZYTase4SNlWpFwO7zbzEvJjgi3TjK0s1r+ sl050bccdNelnMwJS3P96HYpHwbrn3aI5hvdgVn+QuVaFon2ETryIWmq/HHED+Iu nssQICtpX3XhbkgLEEOpW78UiTSzDyRjLBKGucvfu+a8EbiWcT0hSNz9DGcPO36C +mmPBYoaZl4eijYdfZ8wZ+YcabrSCWVKnP0ynvg8zhNQYgvL1IEqTf2LKjO1Q1jO LfsJI3Et16cSvvKsaOjDbb1t7XRtPEXN9RPWUv7YTY9ZL5j/ugCc9eg6eN6b+gwg HEss8zlkwNeThJoIByNJmGg4qpc3YZAbFJj3j6VQ -----END CERTIFICATE-----
This is my test file about creating a pod: apiVersion: v1 kind: Pod metadata: name: my-pod labels: app: my-pod spec: containers: - name: busybox image: docker-registry-default.apps.0122-47n.qe.rhcloud.com/test/myimage command: ["sh", "-c", "while true; do sleep 10; done"] ports: - containerPort: 80
> cat /etc/docker/certs.d/ca.crt/ca.crt this is the issue. The cert should be in a directory named "docker-registry-default.apps.0122-47n.qe.rhcloud.com". Since you named your key in the configmap "ca.crt" it is not. You need to update your configmap, the key in the configmap needs to be the name of the registry host. In addition if you need to specify a port, you need to define the key using ".." instead of ":" (because : is not a valid character for a configmap key): for example: host.domain.com..5000
Thanks, Ben! Verified with below version: $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-01-25-205123 True False 1h Cluster version is 4.0.0-0.nightly-2019-01-25-205123 [root@ip-10-0-135-86 ~]# ls /etc/docker/certs.d/ docker-registry-default.apps.0128-49c.qe.rhcloud.com image-registry.openshift-image-registry.svc:5000 image-registry.openshift-image-registry.svc.cluster.local:5000 [root@ip-10-0-135-86 ~]# ls /etc/docker/certs.d/docker-registry-default.apps.0128-49c.qe.rhcloud.com ca.crt [wzheng@laptop test]$ oc get pods NAME READY STATUS RESTARTS AGE my-pod 1/1 Running 0 34s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 3m default-scheduler Successfully assigned wzheng1/my-pod to ip-10-0-154-255.us-east-2.compute.internal Normal Pulling 3m kubelet, ip-10-0-154-255.us-east-2.compute.internal pulling image "docker-registry-default.apps.0128-49c.qe.rhcloud.com/test/myimage" Normal Pulled 3m kubelet, ip-10-0-154-255.us-east-2.compute.internal Successfully pulled image "docker-registry-default.apps.0128-49c.qe.rhcloud.com/test/myimage" Normal Created 3m kubelet, ip-10-0-154-255.us-east-2.compute.internal Created container Normal Started 3m kubelet, ip-10-0-154-255.us-east-2.compute.internal Started container [wzheng@laptop test]$ cat busybox.yml apiVersion: v1 kind: Pod metadata: name: my-pod labels: app: my-pod spec: containers: - name: busybox image: docker-registry-default.apps.0128-49c.qe.rhcloud.com/test/myimage command: ["sh", "-c", "while true; do sleep 10; done"] ports: - containerPort: 80
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0758