beta promotion in https://github.com/kubernetes/kubernetes/pull/96527 switched storage version to v1beta1, making downgrades impossible. when you start with 1.20, it will write v1beta1 to etcd. Self-upgrade downgrades to 1.19 and that cannot read the objects from etcd (it does not know v1beta1 flowschemas). > E1213 01:15:55.872313 1 task.go:81] error running apply for flowschema "openshift-etcd-operator" (72 of 670): no kind "FlowSchema" is registered for version "flowcontrol.apiserver.k8s.io/v1beta1" in scheme "k8s.io/kubernetes/pkg/api/legacyscheme/scheme.go:30" Mitigation: - we have to migrate p&f v1alpha1 objects in storage to beta in 4.8 - and make sure upstream keeps the alpha version in 1.21.
Verification steps: $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.7.0-0.nightly-2020-12-14-165231 True False 7h48m Cluster version is 4.7.0-0.nightly-2020-12-14-165231 $ oc debug node/<master node> sh-4.4# chroot /host sh-4.4# grep -nR 'flowcontrol.apiserver.k8s.io\/v1alpha1' openshift-kube-apiserver_kube-apiserver-ip-10-0-168-141.us-east-2.compute.internal_e4ce3b08-146d-43e0-934e-879692e7db5b openshift-kube-apiserver_kube-apiserver-ip-10-0-168-141.us-east-2.compute.internal_e4ce3b08-146d-43e0-934e-879692e7db5b/kube-apiserver/0.log:184:2020-12-16T06:28:59.034652522+00:00 stderr F I1216 06:28:59.034634 17 flags.go:59] FLAG: --runtime-config="flowcontrol.apiserver.k8s.io/v1alpha1=true" sh-4.4# grep -nR 'flowcontrol.apiserver.k8s.io\/v1alpha1' /var/log/pods/openshift-* | grep -v 'debug' | wc -l 3131 sh-4.4# grep -nR 'flowcontrol.apiserver.k8s.io\/v1alpha1' /var/log/pods/openshift-* | grep -v 'debug' | head -3 /var/log/pods/openshift-cluster-version_cluster-version-operator-569bb44c8d-7j8hw_50cd9a9c-7691-4c40-9f7b-aad94598061b/cluster-version-operator/0.log.20201216-061917:628:2020-12-16T00:39:58.365312441+00:00 stderr F I1216 00:39:58.365259 1 request.go:591] Throttling request took 95.17121ms, request: GET:https://api-int.kewang1671.qe.devcluster.openshift.com:6443/apis/flowcontrol.apiserver.k8s.io/v1alpha1/prioritylevelconfigurations/openshift-control-plane-operators /var/log/pods/openshift-cluster-version_cluster-version-operator-569bb44c8d-7j8hw_50cd9a9c-7691-4c40-9f7b-aad94598061b/cluster-version-operator/0.log.20201216-061917:630:2020-12-16T00:39:58.465299974+00:00 stderr F I1216 00:39:58.465250 1 request.go:591] Throttling request took 95.602294ms, request: PUT:https://api-int.kewang1671.qe.devcluster.openshift.com:6443/apis/flowcontrol.apiserver.k8s.io/v1alpha1/prioritylevelconfigurations/openshift-control-plane-operators /var/log/pods/openshift-cluster-version_cluster-version-operator-569bb44c8d-7j8hw_50cd9a9c-7691-4c40-9f7b-aad94598061b/cluster-version-operator/0.log.20201216-061917:636:2020-12-16T00:39:58.565330349+00:00 stderr F I1216 00:39:58.565283 1 request.go:591] Throttling request took 94.361185ms, request: GET:https://api-int.kewang1671.qe.devcluster.openshift.com:6443/apis/flowcontrol.apiserver.k8s.io/v1alpha1/flowschemas/openshift-monitoring-metrics sh-4.4# grep -nr 'flowcontrol.apiserver.k8s.io\/v1beta1' /var/log/pods/openshift-* | grep -v debug From above results, the flowcontrol.apiserver.k8s.io uses v1alpha1, instead of v1beta1, so move the bug VERIFIED.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:5633