No progress in 4.5 for this.
Launched 4.7 fresh env with latest 4.7 nightly payload, then check openshift-apiserver openshift-apiserver-operator namespace yaml, no label "openshift.io/run-level". Further looking at the PR cluster-openshift-apiserver-operator/pull/323/files, the fix code says: + openshift.io/run-level-: "" # remove the label if previously set This seems to need test upgrade: if before upgrade they have the label, then after upgrade they should remove the label if previously set. So upgraded from 4.6.8 to latest 4.7.0-0.nightly-2020-12-10-034711, ns/openshift-apiserver removed, but ns/openshift-apiserver-operator still has: $ oc get ns/openshift-apiserver-operator -o yaml ... labels: ... openshift.io/run-level: "0" ... Is this expected?
Thanks, your verification proves that the fix we added to our namespace-apply logic in library-go works (that's what we're doing with the openshift-apiserver NS) and that the label does not get added to the openshift-apiserver-operator NS in new cluster deployments. Unfortunately, IIRC, CVO uses their custom logic for these operations which apparently does not work during upgrades, or we need a different fix to our manifest. Changing component to CVO for clarification.
The CVO ensures that manifest-specified annotations and labels exist on the in-cluster resource, but it does not remove in-cluster labels or annotations that do not occur in the manifest [1,2]. So the set on the in-cluster resource may be a superset of the set in the manifest, which is why [3] is not clearing the label on update. Talking about this internally, Clayton suggested that all label/annotation keys contain explicit default values, so manifest authors can set 'openshift.io/run-level: default' (or whatever the key maintainers decide to use for "I am explicitly fine with the default value, and want the CVO to stomp any divergence"). Moving back to API-server so they can make that change (I'm not actually sure who manages the key, we may need to track those folks down so they can define a default value before the API-server folks can explicitly set it in their manifest). [1]: https://github.com/openshift/cluster-version-operator/blob/1d30d9c07c4220d0271a37f5e8999316f67c3ce6/lib/resourcemerge/meta.go#L13-L14 [2]: https://github.com/openshift/cluster-version-operator/blob/1d30d9c07c4220d0271a37f5e8999316f67c3ce6/lib/resourcemerge/meta.go#L28-L41 [3]: https://github.com/openshift/cluster-openshift-apiserver-operator/pull/330/files#diff-8d719c9f5255dbb98d5e444d361a84a4ed918b238a766fa8149f926fd2fe6962L7