Bug 2110629

Summary: openshift-controller-manager(-operator) namespace should clear run-level annotations
Product: OpenShift Container Platform Reporter: W. Trevor King <wking>
Component: openshift-controller-managerAssignee: Lalatendu Mohanty <lmohanty>
openshift-controller-manager sub component: controller-manager QA Contact: Jitendar Singh <jitsingh>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: high CC: bleanhar, bparees, hongkliu, jitsingh, jshu, lamarach, lmohanty, mifiedle, sdodson, slaznick, wking, zhsun
Version: 4.5Keywords: FastFix, Upgrades
Target Milestone: ---   
Target Release: 4.12.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 2101880
: 2110715 (view as bug list) Environment:
Last Closed: 2023-01-17 19:53:27 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 2110715    

Description W. Trevor King 2022-07-25 17:50:58 UTC 
+++ This bug was initially created as a clone of Bug #2101880 +++

[1] removed the openshift.io/run-level annotation from the CVO manifest for both the openshift-controller-manager and openshift-controller-manager-operator namespaces, but did not add the empty-string marker to ask the CVO to remove the annotation (more about that in bug 2101880).  This shipped in 4.5 [2] and 4.4 [3].

[4] moved openshift-controller-manager namespace management from a cluster-version manifest into the controller-manager operator, but still neglected to clear the run-level annotation.  That landed in 4.6 with no backports.

This leaves clusters that were born in 4.3 and earlier with a dangling run-level annotation, and the controller-manager operator should clear it, or set it to an empty string, to avoid divergence between born-in-4.4+ and born-in-4.3- clusters updating to 4.11 and 4.12.

[1]: https://github.com/openshift/cluster-openshift-controller-manager-operator/pull/143
[2]: https://bugzilla.redhat.com/show_bug.cgi?id=1806913#c6
[3]: https://bugzilla.redhat.com/show_bug.cgi?id=1807490#c6
[4]: https://github.com/openshift/cluster-openshift-controller-manager-operator/pull/153
Comment 1 W. Trevor King 2022-07-25 18:11:10 UTC 
Scott failed in an update to 4.11.0-fc.3 with:

$ oc get pods -A | grep CreateContainerConfigError
openshift-cloud-credential-operator                cloud-credential-operator-5d79d8fd6d-vv8fr                                  1/2     CreateContainerConfigError   0             6m54s
openshift-controller-manager                       controller-manager-bd2bh                                                    0/1     CreateContainerConfigError   0             5m39s
openshift-controller-manager                       controller-manager-btqhn                                                    0/1     CreateContainerConfigError   0             5m39s
openshift-controller-manager                       controller-manager-fhvxp                                                    0/1     CreateContainerConfigError   0             5m40s

  Warning  Failed          6m36s (x10 over 8m14s)  kubelet            Error: container has runAsNonRoot and image will run as root (pod: "cloud-credential-operator-5d79d8fd6d-vv8fr_openshift-cloud-credential-operator(07873435-df80-477b-95ac-835ac8d41ac8)", container: cloud-credential-operator)

  Warning  Failed          5m49s (x12 over 8m)  kubelet            Error: container has runAsNonRoot and image will run as root (pod: "controller-manager-bd2bh_openshift-controller-manager(32740d0f-610a-45c9-8203-a962b43ba038)", container: controller-manager)

With the following cluster history:

4.3.18
4.4.32
4.5.41
4.6.56
4.7.53
4.8.46
4.9.42
4.10.22
4.11.0-fc.3
Comment 3 sunzhaohua 2022-07-26 16:28:03 UTC 
Tested upgrade one cluster 4.3.18->4.4.33->4.5.41->4.6.60->4.7.55->4.8.46->4.9.43->4.10.24->4.12.0-0.ci-2022-07-26-140708, upgrade is successful. Cluster https://mastern-jenkins-csb-openshift-qe.apps.ocp-c1.prod.psi.redhat.com/job/ocp-common/job/Flexy-install/124285/artifact/workdir/install-dir/auth/kubeconfig/*view*/

$ oc get clusterversion                                                                                           
NAME      VERSION                         AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.12.0-0.ci-2022-07-26-140708   True        False         26m     Cluster version is 4.12.0-0.ci-2022-07-26-140708

$ oc edit deploy machine-api-operator
      securityContext:
        runAsNonRoot: true
        runAsUser: 65534

$ oc get pods -A | grep CreateContainerConfigError
Comment 4 Mike Fiedler 2022-07-26 16:47:47 UTC 
verified on  4.12.0-0.ci-2022-07-26-140708
Comment 5 W. Trevor King 2022-07-28 15:05:05 UTC 
(In reply to W. Trevor King from comment #0)
> [1] removed the openshift.io/run-level annotation from the CVO manifest for
> both the openshift-controller-manager and
> openshift-controller-manager-operator namespaces, but did not add the
> empty-string marker to ask the CVO to remove the annotation (more about that
> in bug 2101880).  This shipped in 4.5 [2] and 4.4 [3].

This series ended up tracking only the operand namespace.  The operator namespace is being tracked in bug 2111979.
Comment 7 jawed 2022-11-04 07:02:06 UTC 
*** Bug 2111979 has been marked as a duplicate of this bug. ***
Comment 10 errata-xmlrpc 2023-01-17 19:53:27 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.12.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:7399