+++ This bug was initially created as a clone of Bug #2110629 +++

+++ This bug was initially created as a clone of Bug #2101880 +++

[1] removed the openshift.io/run-level annotation from the CVO manifest for both the openshift-controller-manager and openshift-controller-manager-operator namespaces, but did not add the empty-string marker to ask the CVO to remove the annotation (more about that in bug 2101880).  This shipped in 4.5 [2] and 4.4 [3].

[4] moved openshift-controller-manager namespace management from a cluster-version manifest into the controller-manager operator, but still neglected to clear the run-level annotation.  That landed in 4.6 with no backports.

This leaves clusters that were born in 4.3 and earlier with a dangling run-level annotation, and the controller-manager operator should clear it, or set it to an empty string, to avoid divergence between born-in-4.4+ and born-in-4.3- clusters updating to 4.11 and 4.12.

[1]: https://github.com/openshift/cluster-openshift-controller-manager-operator/pull/143
[2]: https://bugzilla.redhat.com/show_bug.cgi?id=1806913#c6
[3]: https://bugzilla.redhat.com/show_bug.cgi?id=1807490#c6
[4]: https://github.com/openshift/cluster-openshift-controller-manager-operator/pull/153

--- Additional comment from W. Trevor King on 2022-07-25 18:11:10 UTC ---

Scott failed in an update to 4.11.0-fc.3 with:

$ oc get pods -A | grep CreateContainerConfigError
openshift-cloud-credential-operator                cloud-credential-operator-5d79d8fd6d-vv8fr                                  1/2     CreateContainerConfigError   0             6m54s
openshift-controller-manager                       controller-manager-bd2bh                                                    0/1     CreateContainerConfigError   0             5m39s
openshift-controller-manager                       controller-manager-btqhn                                                    0/1     CreateContainerConfigError   0             5m39s
openshift-controller-manager                       controller-manager-fhvxp                                                    0/1     CreateContainerConfigError   0             5m40s

  Warning  Failed          6m36s (x10 over 8m14s)  kubelet            Error: container has runAsNonRoot and image will run as root (pod: "cloud-credential-operator-5d79d8fd6d-vv8fr_openshift-cloud-credential-operator(07873435-df80-477b-95ac-835ac8d41ac8)", container: cloud-credential-operator)

  Warning  Failed          5m49s (x12 over 8m)  kubelet            Error: container has runAsNonRoot and image will run as root (pod: "controller-manager-bd2bh_openshift-controller-manager(32740d0f-610a-45c9-8203-a962b43ba038)", container: controller-manager)

With the following cluster history:

4.3.18
4.4.32
4.5.41
4.6.56
4.7.53
4.8.46
4.9.42
4.10.22
4.11.0-fc.3

---

I'm creating this as a 4.11.0 blocker+, because even though the born-in-4.3-or-earlier set of clusters is unlikely to be a large fraction of clusters updating to 4.11, we should be able to get clean updates for those clusters without making invasive changes in the operator repository.  But feel free to push back if folks want to fix in an early 4.11.z :)