https://github.com/openshift/machine-config-operator/pull/1045 testing with the above
Pending questions to how to correctly set this up - looks like steps in comment #1 aren't correct.
also, where have instructions from comment #1 been taken from? you can lay down certs already with a MachineConfig writing to /etc/pki/ca-trust/source/anchors and after reboot it's going to be in the system bundle, I'm not sure how/if comment #1 is supposed to work (asked)
From proxy owner (additionalTrustedCA): additionalTrustedCA is a field of installer config that contains the user-provided ca bundle. This will cause the installer to create a configmap containing the cert. You must set proxy.spec.trustedCA to the name of this configmap. Unless your https proxy’s identity cert is signed by a CA from the system trust bundle, then you need to use additionalTrustedCA from the installer to provide the proxy’s CA cert and set proxy.spec.trustedCA to the name of this configmap (i.e. “user-ca-bundle”). (edited) So comment #1 is essentially wrong as a day-2 operation, day-1 that should be done using http://pastebin.test.redhat.com/789782 and testing day-2 on a non-proxy install is here https://gist.github.com/danehans/4d43adf773166b41470783e4ca4f6fb6
proxy shouldn't be involved. use the image config resource documented here to provide additional CAs to trust during image pulls: https://docs.openshift.com/container-platform/4.1/openshift_images/image-configuration.html#images-configuration-parameters_image-configuration
MCO isn't managing those certs but believe the registry operator does: https://github.com/openshift/cluster-image-registry-operator/blob/master/pkg/resource/nodecadaemon.go#L19-L91. If they're not landing on nodes, there might be an issue there somehow.
The rendered-master and rendered-worker machineconfig weren't updated after I patch additionalTrustedCA to image.config.openshift.io/cluster. Is there bug to write the additionalTrustedCA from image.config.openshift.io/cluster to rendered machineconfig? NAME GENERATEDBYCONTROLLER IGNITIONVERSION CREATED 00-master 93455625b7a681965fa45f9a4a42539afb415f07 2.2.0 56d 00-worker 93455625b7a681965fa45f9a4a42539afb415f07 2.2.0 56d 01-master-container-runtime 93455625b7a681965fa45f9a4a42539afb415f07 2.2.0 56d 01-master-kubelet 93455625b7a681965fa45f9a4a42539afb415f07 2.2.0 56d 01-worker-container-runtime 93455625b7a681965fa45f9a4a42539afb415f07 2.2.0 56d 01-worker-kubelet 93455625b7a681965fa45f9a4a42539afb415f07 2.2.0 56d 99-master-083cef8c-9c29-11e9-8893-fa163e347a63-registries 93455625b7a681965fa45f9a4a42539afb415f07 2.2.0 56d 99-master-ssh 2.2.0 56d 99-worker-083e47d6-9c29-11e9-8893-fa163e347a63-registries 93455625b7a681965fa45f9a4a42539afb415f07 2.2.0 56d 99-worker-ssh 2.2.0 56d rendered-master-0a830d26f3d0941eacd65b599bf90b60 365c1cfd14de5b0e3b85e0fc815b0060f36ab955 2.2.0 40d rendered-master-1acec7e6704b36f67432a60fa7bfe346 1509d995281c39bf114bc089d4849c3f62d28e2e 2.2.0 7d13h rendered-master-64d152df0bc7a9f574cff8c29d199578 93455625b7a681965fa45f9a4a42539afb415f07 2.2.0 165m rendered-master-658bfe21341a99ab6d3ef861e54600d6 93455625b7a681965fa45f9a4a42539afb415f07 2.2.0 110m rendered-master-6e97330ec9f8cbb7061c43acdf8c0d29 83392b13a5c17e56656acf3f7b0031e3303fb5c0 2.2.0 14d rendered-master-8ecdb4afa30fdb3eaff833e03556b5b2 83392b13a5c17e56656acf3f7b0031e3303fb5c0 2.2.0 7d13h rendered-master-91385c7b43cde02fa8652b02a95032eb 02c07496ba0417b3e12b78fb32baf6293d314f79 2.2.0 56d rendered-master-e6e0a238cdd57d9742f0ece799f9157e 365c1cfd14de5b0e3b85e0fc815b0060f36ab955 2.2.0 35d rendered-worker-14558e14062a5c5af60c1ff6547b6ec0 02c07496ba0417b3e12b78fb32baf6293d314f79 2.2.0 56d rendered-worker-353ffcd6bb53f27f1643fa6b8d23231b 93455625b7a681965fa45f9a4a42539afb415f07 2.2.0 165m rendered-worker-55abce86fcc120f164d63e9a52d71286 365c1cfd14de5b0e3b85e0fc815b0060f36ab955 2.2.0 40d rendered-worker-81264cace820df36e36b61b8c6aea7a7 365c1cfd14de5b0e3b85e0fc815b0060f36ab955 2.2.0 35d rendered-worker-9047a7d63222715036919555ef635611 83392b13a5c17e56656acf3f7b0031e3303fb5c0 2.2.0 14d rendered-worker-94104491ea0c2a1262973832ff9e3100 83392b13a5c17e56656acf3f7b0031e3303fb5c0 2.2.0 7d13h rendered-worker-c9c578c7bdf48f74309d857cb2ffe93f 1509d995281c39bf114bc089d4849c3f62d28e2e 2.2.0 7d13h rendered-worker-e47a9b1646a2b9ec20c67b8886901b8b 93455625b7a681965fa45f9a4a42539afb415f07 2.2.0 110m
(In reply to Anping Li from comment #24) > The rendered-master and rendered-worker machineconfig weren't updated after > I patch additionalTrustedCA to image.config.openshift.io/cluster. Is there > bug to write the additionalTrustedCA from image.config.openshift.io/cluster > to rendered machineconfig? > The MCO doesn't watch that from image.config.openshift.io/cluster additionalTrustBundle, err := optr.getCAsFromConfigMap("openshift-config", "user-ca-bundle", "ca-bundle.crt") if err != nil && !apierrors.IsNotFound(err) { return err } if len(additionalTrustBundle) > 0 { if !certPool.AppendCertsFromPEM(additionalTrustBundle) { return fmt.Errorf("configmap %s/%s doesn't have a valid PEM bundle", "openshift-config", "user-ca-bundle") } trustBundle = append(trustBundle, additionalTrustBundle...) } We only sync the above. I'm not sure where this is going to, I cannot understand where the problem is and what are expectations, also it's not clear how to fully reproduce this based just on one-off comments showing random information. Can you please either fix that up or create a new BZ? why has this come back to MCO? The MCO is watching image.config.openshift.io/cluster but it doesn't look for additionalTrustedCA, if it needs to do that, please report it. CC'ing the Node team for the ContainerRuntimeController which is watching the image configuration.
@Antonio, Thanks, MCO for additionalTrustedCA, so that is the root cause. @image-regsitry team, Could image.config.openshift.io/cluster use 'user-ca-bundle' or 'ca-bundle.crt' ? Shall we ask MCO to support additionalTrustedCA?
The certificates from additionalTrustedCA are deployed to nodes by the node-ca deamonset, which is managed by the image registry operator. Please check /etc/docker/certs.d/ on nodes. If node-ca doesn't put certificates there, use must-gather to collect information about the cluster.
Turn to low. As I can install ca bundle during installation.