+++ This bug was initially created as a clone of Bug #2101880 +++ [1] removed the openshift.io/run-level annotation from the CVO manifest for both the openshift-controller-manager and openshift-controller-manager-operator namespaces, but did not add the empty-string marker to ask the CVO to remove the annotation (more about that in bug 2101880). This shipped in 4.5 [2] and 4.4 [3]. [4] moved openshift-controller-manager namespace management from a cluster-version manifest into the controller-manager operator, but still neglected to clear the run-level annotation. That landed in 4.6 with no backports. This leaves clusters that were born in 4.3 and earlier with a dangling run-level annotation, and the controller-manager operator should clear it, or set it to an empty string, to avoid divergence between born-in-4.4+ and born-in-4.3- clusters updating to 4.11 and 4.12. [1]: https://github.com/openshift/cluster-openshift-controller-manager-operator/pull/143 [2]: https://bugzilla.redhat.com/show_bug.cgi?id=1806913#c6 [3]: https://bugzilla.redhat.com/show_bug.cgi?id=1807490#c6 [4]: https://github.com/openshift/cluster-openshift-controller-manager-operator/pull/153
Scott failed in an update to 4.11.0-fc.3 with: $ oc get pods -A | grep CreateContainerConfigError openshift-cloud-credential-operator cloud-credential-operator-5d79d8fd6d-vv8fr 1/2 CreateContainerConfigError 0 6m54s openshift-controller-manager controller-manager-bd2bh 0/1 CreateContainerConfigError 0 5m39s openshift-controller-manager controller-manager-btqhn 0/1 CreateContainerConfigError 0 5m39s openshift-controller-manager controller-manager-fhvxp 0/1 CreateContainerConfigError 0 5m40s Warning Failed 6m36s (x10 over 8m14s) kubelet Error: container has runAsNonRoot and image will run as root (pod: "cloud-credential-operator-5d79d8fd6d-vv8fr_openshift-cloud-credential-operator(07873435-df80-477b-95ac-835ac8d41ac8)", container: cloud-credential-operator) Warning Failed 5m49s (x12 over 8m) kubelet Error: container has runAsNonRoot and image will run as root (pod: "controller-manager-bd2bh_openshift-controller-manager(32740d0f-610a-45c9-8203-a962b43ba038)", container: controller-manager) With the following cluster history: 4.3.18 4.4.32 4.5.41 4.6.56 4.7.53 4.8.46 4.9.42 4.10.22 4.11.0-fc.3
Tested upgrade one cluster 4.3.18->4.4.33->4.5.41->4.6.60->4.7.55->4.8.46->4.9.43->4.10.24->4.12.0-0.ci-2022-07-26-140708, upgrade is successful. Cluster https://mastern-jenkins-csb-openshift-qe.apps.ocp-c1.prod.psi.redhat.com/job/ocp-common/job/Flexy-install/124285/artifact/workdir/install-dir/auth/kubeconfig/*view*/ $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.12.0-0.ci-2022-07-26-140708 True False 26m Cluster version is 4.12.0-0.ci-2022-07-26-140708 $ oc edit deploy machine-api-operator securityContext: runAsNonRoot: true runAsUser: 65534 $ oc get pods -A | grep CreateContainerConfigError
verified on 4.12.0-0.ci-2022-07-26-140708
(In reply to W. Trevor King from comment #0) > [1] removed the openshift.io/run-level annotation from the CVO manifest for > both the openshift-controller-manager and > openshift-controller-manager-operator namespaces, but did not add the > empty-string marker to ask the CVO to remove the annotation (more about that > in bug 2101880). This shipped in 4.5 [2] and 4.4 [3]. This series ended up tracking only the operand namespace. The operator namespace is being tracked in bug 2111979.
*** Bug 2111979 has been marked as a duplicate of this bug. ***
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.12.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:7399