Bug 1806918

Summary: openshift-apiserver: Some core components are in openshift.io/run-level 1 and are bypassing SCC, but should not be
Product: OpenShift Container Platform Reporter: Stefan Schimanski <sttts>
Component: openshift-apiserverAssignee: Stefan Schimanski <sttts>
Status: CLOSED NOTABUG QA Contact: Xingxing Xia <xxia>
Severity: low Docs Contact:
Priority: low    
Version: 4.4CC: aos-bugs, bleanhar, ccoleman, eparis, jialiu, lmohanty, mfojtik, nhale, nstielau, sfowler, wking, wlewis, wsun, xiyuan, xtian, xxia, ykashtan
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1805488 Environment:
Last Closed: 2022-02-25 18:00:18 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1805488, 1966621    

Comment 1 Stefan Schimanski 2020-05-19 12:29:31 UTC
No progress in 4.5 for this.

Comment 12 Xingxing Xia 2020-12-10 08:54:59 UTC
Launched 4.7 fresh env with latest 4.7 nightly payload, then check openshift-apiserver openshift-apiserver-operator namespace yaml, no label "openshift.io/run-level".

Further looking at the PR cluster-openshift-apiserver-operator/pull/323/files, the fix code says:
+    openshift.io/run-level-: "" # remove the label if previously set

This seems to need test upgrade: if before upgrade they have the label, then after upgrade they should remove the label if previously set.

So upgraded from 4.6.8 to latest 4.7.0-0.nightly-2020-12-10-034711, ns/openshift-apiserver removed, but ns/openshift-apiserver-operator still has:
$ oc get ns/openshift-apiserver-operator -o yaml 
    openshift.io/run-level: "0"

Is this expected?

Comment 13 Standa Laznicka 2020-12-10 09:51:30 UTC
Thanks, your verification proves that the fix we added to our namespace-apply logic in library-go works (that's what we're doing with the openshift-apiserver NS) and that the label does not get added to the openshift-apiserver-operator NS in new cluster deployments.

Unfortunately, IIRC, CVO uses their custom logic for these operations which apparently does not work during upgrades, or we need a different fix to our manifest. Changing component to CVO for clarification.

Comment 14 W. Trevor King 2021-01-29 23:38:00 UTC
The CVO ensures that manifest-specified annotations and labels exist on the in-cluster resource, but it does not remove in-cluster labels or annotations that do not occur in the manifest [1,2].  So the set on the in-cluster resource may be a superset of the set in the manifest, which is why [3] is not clearing the label on update.  Talking about this internally, Clayton suggested that all label/annotation keys contain explicit default values, so manifest authors can set 'openshift.io/run-level: default' (or whatever the key maintainers decide to use for "I am explicitly fine with the default value, and want the CVO to stomp any divergence").  Moving back to API-server so they can make that change (I'm not actually sure who manages the key, we may need to track those folks down so they can define a default value before the API-server folks can explicitly set it in their manifest).

[1]: https://github.com/openshift/cluster-version-operator/blob/1d30d9c07c4220d0271a37f5e8999316f67c3ce6/lib/resourcemerge/meta.go#L13-L14
[2]: https://github.com/openshift/cluster-version-operator/blob/1d30d9c07c4220d0271a37f5e8999316f67c3ce6/lib/resourcemerge/meta.go#L28-L41
[3]: https://github.com/openshift/cluster-openshift-apiserver-operator/pull/330/files#diff-8d719c9f5255dbb98d5e444d361a84a4ed918b238a766fa8149f926fd2fe6962L7