Bug 1907309
Summary: | Migrate controlflow v1alpha1 to v1beta1 in storage | ||
---|---|---|---|
Product: | OpenShift Container Platform | Reporter: | Stefan Schimanski <sttts> |
Component: | kube-apiserver | Assignee: | Stefan Schimanski <sttts> |
Status: | CLOSED ERRATA | QA Contact: | Ke Wang <kewang> |
Severity: | high | Docs Contact: | |
Priority: | medium | ||
Version: | 4.8 | CC: | akashem, aos-bugs, jdee, kewang, mfojtik, sanchezl, sttts, wking, xxia |
Target Milestone: | --- | Flags: | mfojtik:
needinfo?
|
Target Release: | 4.8.0 | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | LifecycleReset | ||
Fixed In Version: | Doc Type: | If docs needed, set a value | |
Doc Text: | Story Points: | --- | |
Clone Of: | 1907211 | Environment: | |
Last Closed: | 2021-07-27 22:35:01 UTC | Type: | --- |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | 1907211 | ||
Bug Blocks: |
Comment 1
Michal Fojtik
2021-01-13 08:39:05 UTC
The LifecycleStale keyword was removed because the bug moved to QE. The bug assignee was notified. @kewang for verification purposes, the goal is to make sure that the kube-apiserver operator will auto-migrate the storage version of APF resources from v1alpha1 to v1beta1. See https://github.com/openshift/library-go/pull/1091 for the tests I did. It caused upgrade fail in https://issues.redhat.com/browse/OCPQE-4308 (see the log pasted in the issue: https://mastern-jenkins-csb-openshift-qe.apps.ocp4.prod.psi.redhat.com/job/upgrade_CI/14962/console), where network degraded in the upgarde with message about can't find v1alpha1 flow related resoures Refer to the https://bugzilla.redhat.com/show_bug.cgi?id=1907211#c2, the flowcontrol.apiserver.k8s.io uses v1alpha1 in OCP 4.7, After upgraded 4.7 to 4.8, confirm if the flowcontrol.apiserver.k8s.io uses v1beta1 in 4.8, $ oc get clusterversion -o json|jq ".items[0].status.history" [ { "completionTime": "2021-06-28T14:36:22Z", "image": "registry.ci.openshift.org/ocp/release:4.8.0-0.nightly-2021-06-25-182927", "startedTime": "2021-06-28T12:24:06Z", "state": "Completed", "verified": false, "version": "4.8.0-0.nightly-2021-06-25-182927" }, { "completionTime": "2021-06-28T12:16:18Z", "image": "registry.ci.openshift.org/ocp/release@sha256:33d1e084c2068dd9ff940c95f1d1b9b454debeed009c9b5846b8dfa3280b3034", "startedTime": "2021-06-28T11:43:04Z", "state": "Completed", "verified": false, "version": "4.7.0-0.nightly-2021-06-25-194529" } ] $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.8.0-0.nightly-2021-06-25-182927 True False 129m Cluster version is 4.8.0-0.nightly-2021-06-25-182927 $ oc debug node/<master node> sh-4.4# chroot /host sh-4.4# cd /var/log/pods sh-4.4# grep -nR 'flowcontrol.apiserver.k8s.io\/v1alpha1' /var/log/pods/openshift-* | grep -v 'debug' /var/log/pods/openshift-kube-apiserver_kube-apiserver-ip-10-0-55-130.us-east-2.compute.internal_6efa4434-a83c-4627-bb0d-e44eee1eae8d/kube-apiserver/0.log:613:2021-06-28T13:13:41.977032694+00:00 stderr F W0628 13:13:41.976988 19 genericapiserver.go:461] Skipping API flowcontrol.apiserver.k8s.io/v1alpha1 because it has no resources. /var/log/pods/openshift-kube-apiserver_kube-apiserver-ip-10-0-55-130.us-east-2.compute.internal_6efa4434-a83c-4627-bb0d-e44eee1eae8d/kube-apiserver/1.log:819:2021-06-28T14:28:28.705021313+00:00 stderr F W0628 14:28:28.704965 21 genericapiserver.go:461] Skipping API flowcontrol.apiserver.k8s.io/v1alpha1 because it has no resources. sh-4.4# grep -nR 'flowcontrol.apiserver.k8s.io\/v1beta1' /var/log/pods/openshift-* | grep -v 'debug' | wc -l 693 sh-4.4# grep -nR 'flowcontrol.apiserver.k8s.io\/v1beta1' /var/log/pods/openshift-* | grep -v 'debug' | head -1 openshift-kube-apiserver_kube-apiserver-ip-10-0-55-130.us-east-2.compute.internal_6efa4434-a83c-4627-bb0d-e44eee1eae8d/kube-apiserver/1.log:15535:2021-06-28T16:37:06.713311074+00:00 stderr F I0628 16:37:06.713224 21 apiaccess_count_controller.go:130] updating top flowcontrol.apiserver.k8s.io/v1beta1, Resource=prioritylevelconfigurations APIRequest counts From above, we can see the flowcontrol.apiserver.k8s.io uses v1beta1 in 4.8 after upgrade, Logged into etcd container and checked if the objects are now in the beta version, $ oc rsh -n openshift-etcd etcd-ip-10-0-55-130.us-east-2.compute.internal sh-4.4# etcdctl get /kubernetes.io/flowschemas/catch-all --prefix --print-value-only k8s 2 $flowcontrol.apiserver.k8s.io/v1beta1 FlowSchema� � catch-all�"*$837fbd70-265f-4573-9766-1288a64485472���b) !apf.kubernetes.io/autoupdate-spectruez�� ,api-priority-and-fairness-config-consumer-v1Update�$flowcontrol.apiserver.k8s.io/v1beta���FieldsV1:� �{"f:status":{"f:conditions":{".":{},"k:{\"type\":\"Dangling\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}�� ,api-priority-and-fairness-config-producer-v1Update�$flowcontrol.apiserver.k8s.io/v1betaу�FieldsV1:� �{"f:metadata":{"f:annotations":{".":{},"f:apf.kubernetes.io/autoupdate-spec":{}}},"f:spec":{"f:distinguisherMethod":{".":{},"f:type":{}},"f:matchingPrecedence":{},"f:priorityLevelConfiguration":{"f:name":{}},"f:rules":{}}}x catch-all�N� ByUser"\ ! Group� system:unauthenticated Group� system:authenticated **�* **� *2*�� � DanglingFals���"Found*`This FlowSchema references the PriorityLevelConfiguration object named "catch-all" and it exists�" Since values are stored in protobuf encoding, not JSON, but we still can see the object saved with flowcontrol.apiserver.k8s.io/v1beta version, refer to the PR https://github.com/openshift/library-go/pull/1091, it is as expected, so move the bug VERIFIED. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2438 |