Description of problem: Met below error if try to import CA file via adding additionalTrustedCA with image.config.openshift.io cluster: I0429 07:29:46.019269 1 apiserver.go:151] reading image import ca path: /var/run/configmaps/image-import-ca/..2019_04_29_07_29_35.208155455/docker-registry-default.apps.0429-1ow.qe.rhcloud.com, incoming err: <nil> I0429 07:29:46.019407 1 apiserver.go:151] reading image import ca path: /var/run/configmaps/image-import-ca/docker-registry-default.apps.0429-1ow.qe.rhcloud.com, incoming err: <nil> I0429 07:29:46.019412 1 apiserver.go:156] skipping dir or symlink: /var/run/configmaps/image-import-ca/docker-registry-default.apps.0429-1ow.qe.rhcloud.com Version-Release number of selected component (if applicable): 4.1.0-0.nightly-2019-04-28-064010 How reproducible: Always Steps to Reproduce: 1.$ oc create configmap registry-config --from-file=docker-registry-default.apps.0429-1ow.qe.rhcloud.com=ca.crt -n openshift-config 2.$ oc edit image.config.openshift.io cluster spec: additionalTrustedCA: name: registry-config 3.Watch apiserver pods: $oc get pods -n openshift-apiserver $oc logs pods/apiserver-lpv8m -n openshift-apiserver | grep docker-registry Actual results: The CA cannot be imported with errors in api pod Expected results: Should can be imported. Additional info:
It seems it successfully imported `/var/run/configmaps/image-import-ca/..2019_04_29_07_29_35.208155455/docker-registry-default.apps.0429-1ow.qe.rhcloud.com`. Have you checked that the api server gets x509 errors when it imports from this registry?
The message is informational, it's indicating what CA was read, and that no error ("nil") was encountered in the process. I think this is working correctly unless as Oleg asked, the import is not actually succeeding. Moving to QA to verify.
OK, seems cluster has picked up the CA file, since image can be imported with the error exists in api pod. However it is not synced to image-registry pod, since I cannot see it as below: $ oc rsh image-registry-77dc78779f-kngmp sh-4.2$ ls /etc/pki/ca-trust/source/anchors service-ca.crt So build and pod cannot be running with 509 error: Warning Failed 14s kubelet, ip-172-31-146-161.eu-west-2.compute.internal Failed to pull image "image-registry.openshift-image-registry.svc:5000/openshift-image-registry/myimage@sha256:8d750876687d9fb0adf46020e38cd43165b906208d2e81088b3a13213e751df3": rpc error: code = Unknown desc = Error reading manifest sha256:8d750876687d9fb0adf46020e38cd43165b906208d2e81088b3a13213e751df3 in image-registry.openshift-image-registry.svc:5000/openshift-image-registry/myimage: unknown: unable to pull manifest from docker-registry-default.apps.0430-usw.qe.rhcloud.com/test/myimage:latest: Get https://docker-registry-default.apps.0430-usw.qe.rhcloud.com/v2/: x509: certificate signed by unknown authority Build is pending with below error: Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 65s default-scheduler Successfully assigned openshift-image-registry/ruby-hello-world-2-build to ip-172-31-146-161.eu-west-2.compute.internal Warning FailedMount 1s (x8 over 65s) kubelet, ip-172-31-146-161.eu-west-2.compute.internal MountVolume.SetUp failed for volume "build-ca-bundles" : configmap references non-existent config key: docker-registry-default.apps.0430-usw.qe.rhcloud.com
Created attachment 1560202 [details] Deployment under openshift-image-registry
Created attachment 1560203 [details] All configmaps under openshift-config
Created attachment 1560204 [details] openshift image registry operator log
Created attachment 1560205 [details] Image config
@Wenjing Please provide the following: 1. The build and build pod YAML 2. The ConfigMaps in the build pod's namespace. I'm concerned about the build pod not starting because of a non-existent ConfigMap key. @Oleg does the registry need a copy of the trusted certs to do pull-through?
PR: https://github.com/openshift/cluster-image-registry-operator/pull/272
Verified with below version: $ oc rsh image-registry-799fd5b5cf-b9x8v sh-4.2$ ls /etc/pki/ca-trust/source/anchors docker-registry-default.apps.0505-387.qe.rhcloud.com image-registry.openshift-image-registry.svc..5000 image-registry.openshift-image-registry.svc.cluster.local..5000 $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.1.0-0.nightly-2019-05-04-210601 True False 85m Cluster version is 4.1.0-0.nightly-2019-05-04-210601
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0758