+++ This bug was initially created as a clone of Bug #2110629 +++
+++ This bug was initially created as a clone of Bug #2101880 +++
 removed the openshift.io/run-level annotation from the CVO manifest for both the openshift-controller-manager and openshift-controller-manager-operator namespaces, but did not add the empty-string marker to ask the CVO to remove the annotation (more about that in bug 2101880). This shipped in 4.5  and 4.4 .
 moved openshift-controller-manager namespace management from a cluster-version manifest into the controller-manager operator, but still neglected to clear the run-level annotation. That landed in 4.6 with no backports.
This leaves clusters that were born in 4.3 and earlier with a dangling run-level annotation, and the controller-manager operator should clear it, or set it to an empty string, to avoid divergence between born-in-4.4+ and born-in-4.3- clusters updating to 4.11 and 4.12.
--- Additional comment from W. Trevor King on 2022-07-25 18:11:10 UTC ---
Scott failed in an update to 4.11.0-fc.3 with:
$ oc get pods -A | grep CreateContainerConfigError
openshift-cloud-credential-operator cloud-credential-operator-5d79d8fd6d-vv8fr 1/2 CreateContainerConfigError 0 6m54s
openshift-controller-manager controller-manager-bd2bh 0/1 CreateContainerConfigError 0 5m39s
openshift-controller-manager controller-manager-btqhn 0/1 CreateContainerConfigError 0 5m39s
openshift-controller-manager controller-manager-fhvxp 0/1 CreateContainerConfigError 0 5m40s
Warning Failed 6m36s (x10 over 8m14s) kubelet Error: container has runAsNonRoot and image will run as root (pod: "cloud-credential-operator-5d79d8fd6d-vv8fr_openshift-cloud-credential-operator(07873435-df80-477b-95ac-835ac8d41ac8)", container: cloud-credential-operator)
Warning Failed 5m49s (x12 over 8m) kubelet Error: container has runAsNonRoot and image will run as root (pod: "controller-manager-bd2bh_openshift-controller-manager(32740d0f-610a-45c9-8203-a962b43ba038)", container: controller-manager)
With the following cluster history:
I'm creating this as a 4.11.0 blocker+, because even though the born-in-4.3-or-earlier set of clusters is unlikely to be a large fraction of clusters updating to 4.11, we should be able to get clean updates for those clusters without making invasive changes in the operator repository. But feel free to push back if folks want to fix in an early 4.11.z :)
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.