Bug 1907211

Summary:	beta promotion of p&f switched storage version to v1beta1, making downgrades impossible.
Product:	OpenShift Container Platform	Reporter:	Abu Kashem <akashem>
Component:	kube-apiserver	Assignee:	Abu Kashem <akashem>
Status:	CLOSED ERRATA	QA Contact:	Ke Wang <kewang>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	4.7	CC:	aos-bugs, mfojtik, sttts, wking, xxia
Target Milestone:	---
Target Release:	4.7.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1907309 (view as bug list)		Environment:
Last Closed:	2021-02-24 15:43:00 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1907309

Description Abu Kashem 2020-12-13 16:44:36 UTC

beta promotion in https://github.com/kubernetes/kubernetes/pull/96527 switched storage version to v1beta1, making downgrades impossible.
when you start with 1.20, it will write v1beta1 to etcd. Self-upgrade downgrades to 1.19 and that cannot read the objects from etcd (it does not know v1beta1 flowschemas).

> E1213 01:15:55.872313       1 task.go:81] error running apply for flowschema "openshift-etcd-operator" (72 of 670): no kind "FlowSchema" is registered for version "flowcontrol.apiserver.k8s.io/v1beta1" in scheme "k8s.io/kubernetes/pkg/api/legacyscheme/scheme.go:30"

Mitigation:
- we have to migrate p&f v1alpha1 objects in storage to beta in 4.8
- and make sure upstream keeps the alpha version in 1.21.

Comment 2 Ke Wang 2020-12-16 08:30:36 UTC

Verification steps:

$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.0-0.nightly-2020-12-14-165231   True        False         7h48m   Cluster version is 4.7.0-0.nightly-2020-12-14-165231

$ oc debug node/<master node>

sh-4.4# chroot /host
sh-4.4# grep -nR 'flowcontrol.apiserver.k8s.io\/v1alpha1' openshift-kube-apiserver_kube-apiserver-ip-10-0-168-141.us-east-2.compute.internal_e4ce3b08-146d-43e0-934e-879692e7db5b
openshift-kube-apiserver_kube-apiserver-ip-10-0-168-141.us-east-2.compute.internal_e4ce3b08-146d-43e0-934e-879692e7db5b/kube-apiserver/0.log:184:2020-12-16T06:28:59.034652522+00:00 stderr F I1216 06:28:59.034634      17 flags.go:59] FLAG: --runtime-config="flowcontrol.apiserver.k8s.io/v1alpha1=true"

sh-4.4# grep -nR 'flowcontrol.apiserver.k8s.io\/v1alpha1' /var/log/pods/openshift-* | grep -v 'debug' | wc -l
3131

sh-4.4# grep -nR 'flowcontrol.apiserver.k8s.io\/v1alpha1' /var/log/pods/openshift-* | grep -v 'debug' | head -3
/var/log/pods/openshift-cluster-version_cluster-version-operator-569bb44c8d-7j8hw_50cd9a9c-7691-4c40-9f7b-aad94598061b/cluster-version-operator/0.log.20201216-061917:628:2020-12-16T00:39:58.365312441+00:00 stderr F I1216 00:39:58.365259       1 request.go:591] Throttling request took 95.17121ms, request: GET:https://api-int.kewang1671.qe.devcluster.openshift.com:6443/apis/flowcontrol.apiserver.k8s.io/v1alpha1/prioritylevelconfigurations/openshift-control-plane-operators
/var/log/pods/openshift-cluster-version_cluster-version-operator-569bb44c8d-7j8hw_50cd9a9c-7691-4c40-9f7b-aad94598061b/cluster-version-operator/0.log.20201216-061917:630:2020-12-16T00:39:58.465299974+00:00 stderr F I1216 00:39:58.465250       1 request.go:591] Throttling request took 95.602294ms, request: PUT:https://api-int.kewang1671.qe.devcluster.openshift.com:6443/apis/flowcontrol.apiserver.k8s.io/v1alpha1/prioritylevelconfigurations/openshift-control-plane-operators
/var/log/pods/openshift-cluster-version_cluster-version-operator-569bb44c8d-7j8hw_50cd9a9c-7691-4c40-9f7b-aad94598061b/cluster-version-operator/0.log.20201216-061917:636:2020-12-16T00:39:58.565330349+00:00 stderr F I1216 00:39:58.565283       1 request.go:591] Throttling request took 94.361185ms, request: GET:https://api-int.kewang1671.qe.devcluster.openshift.com:6443/apis/flowcontrol.apiserver.k8s.io/v1alpha1/flowschemas/openshift-monitoring-metrics

sh-4.4# grep -nr 'flowcontrol.apiserver.k8s.io\/v1beta1' /var/log/pods/openshift-* | grep -v debug

From above results, the flowcontrol.apiserver.k8s.io uses v1alpha1, instead of v1beta1, so move the bug VERIFIED.

Comment 5 errata-xmlrpc 2021-02-24 15:43:00 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633