Description of problem: Upgrading descheduler operator from 4.8 to 4.9 fails with error “CRD removes version v1beta1 that is listed as a stored version on the existing CRD” Version-Release number of selected component (if applicable): clusterkubedescheduleroperator.4.9.0-202108050954 How reproducible: Hit once Steps to Reproduce: 1. Install 4.6 descheduler operator 2. Upgrade from 4.6 -> 4.7 -> 4.8 -> 4.9 3. Actual results: Upgrading from 4.8 to 4.9 fails [knarra@knarra ~]$ oc get csv -n openshift-kube-descheduler-operator NAME DISPLAY VERSION REPLACES PHASE clusterkubedescheduleroperator.4.8.0-202107291502 Kube Descheduler Operator 4.8.0-202107291502 clusterkubedescheduleroperator.4.7.0-202107292319 Replacing clusterkubedescheduleroperator.4.9.0-202108050954 Kube Descheduler Operator 4.9.0-202108050954 clusterkubedescheduleroperator.4.8.0-202107291502 Pending elasticsearch-operator.5.2.0-28 OpenShift Elasticsearch Operator 5.2.0-28 elasticsearch-operator.5.1.1-27 Succeeded status: bundleLookups: - catalogSourceRef: name: qe-app-registry namespace: openshift-marketplace identifier: clusterkubedescheduleroperator.4.9.0-202108050954 path: registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-cluster-kube-descheduler-operator-bundle@sha256:f2be2b8652b6e98693cafc4ed311c95d2ce6af9c458bedad41341703dae3e1d1 properties: '{"properties":[{"type":"olm.gvk","value":{"group":"operator.openshift.io","kind":"KubeDescheduler","version":"v1"}},{"type":"olm.package","value":{"packageName":"cluster-kube-descheduler-operator","version":"4.9.0-202108050954"}}]}' replaces: clusterkubedescheduleroperator.4.8.0-202107291502 catalogSources: [] conditions: - lastTransitionTime: "2021-08-10T10:03:01Z" lastUpdateTime: "2021-08-10T10:03:01Z" message: 'risk of data loss updating "kubedeschedulers.operator.openshift.io": new CRD removes version v1beta1 that is listed as a stored version on the existing CRD' reason: InstallComponentFailed status: "False" type: Installed message: 'risk of data loss updating "kubedeschedulers.operator.openshift.io": new CRD removes version v1beta1 that is listed as a stored version on the existing CRD' phase: Failed plan: - resolving: clusterkubedescheduleroperator.4.9.0-202108050954 resource: group: operators.coreos.com kind: ClusterServiceVersion Expected results: Upgrade should be successful Additional info:
*** Bug 1992478 has been marked as a duplicate of this bug. ***
Validation is required to test adding DevPreviewLongLifecycle to the kubedescheduler cluster object as currently it fails with the below error. Once this bug moves to ON_QA we will need to validate below as well. # kubedeschedulers.operator.openshift.io "cluster" was not valid: # * spec.profiles: Unsupported value: "DevPreviewLongLifecycle": supported values: "AffinityAndTaints", "TopologyAndDuplicates", "LifecycleAndUtilization
Tried verifying the bug here but i still see the error below, @jan should i wait for next respin of descheduler operator ? Version i tried verifying is below. Could you please help confirm ? [knarra@knarra openshift-client-linux-4.6.0-0.nightly-2021-09-13-181051]$ oc get csv -n openshift-kube-descheduler-operator NAME DISPLAY VERSION REPLACES PHASE clusterkubedescheduleroperator.4.8.0-202109100832 Kube Descheduler Operator 4.8.0-202109100832 clusterkubedescheduleroperator.4.7.0-202108311835 Succeeded E0916 15:40:44.574486 1 target_config_reconciler.go:425] key failed with : storageversionmigrations.migration.k8s.io "operator-kubedescheduler-storage-version-migration" is forbidden: User "system:serviceaccount:openshift-kube-descheduler-operator:openshift-descheduler" cannot get resource "storageversionmigrations" in API group "migration.k8s.io" at the cluster scope
Moving the bug ack as the PR below [1] still has not merged yet, clearing needinfo on jan as he has confirmed the same on slack. [1] https://github.com/openshift/cluster-kube-descheduler-operator/pull/221
setting blocker '?' as any customer who tries to install or upgrade descheduler on 4.8 would hit this bug.
Tried to upgrade from 4.6 to 4.9, but after upgrading to 4.8 i could see below error in the descheduler logs. message E1004 11:14:41.092535 1 target_config_reconciler.go:425] key failed with : object *v1alpha1.StorageVersionMigration does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message E1004 11:24:41.093216 1 target_config_reconciler.go:425] key failed with : object *v1alpha1.StorageVersionMigration does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message E1004 11:25:07.156259 1 target_config_reconciler.go:425] key failed with : object *v1alpha1.StorageVersionMigration does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message E1004 11:34:41.101799 1 target_config_reconciler.go:425] key failed with : object *v1alpha1.StorageVersionMigration does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message E1004 11:41:47.164784 1 target_config_reconciler.go:425] key failed with : object *v1alpha1.StorageVersionMigration does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message E1004 11:44:41.094502 1 target_config_reconciler.go:425] key failed with : object *v1alpha1.StorageVersionMigration does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message E1004 11:54:41.095124 1 target_config_reconciler.go:425] key failed with : object *v1alpha1.StorageVersionMigration does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message E1004 11:58:27.174260 1 target_config_reconciler.go:425] key failed with : object *v1alpha1.StorageVersionMigration does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message E1004 12:04:41.094648 1 target_config_reconciler.go:425] key failed with : object *v1alpha1.StorageVersionMigration does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message E1004 12:14:41.095152 1 target_config_reconciler.go:425] key failed with : object *v1alpha1.StorageVersionMigration does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message E1004 12:15:07.182630 1 target_config_reconciler.go:425] key failed with : object *v1alpha1.StorageVersionMigration does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message E1004 12:24:41.096773 1 target_config_reconciler.go:425] key failed with : object *v1alpha1.StorageVersionMigration does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message E1004 12:31:47.192133 1 target_config_reconciler.go:425] key failed with : object *v1alpha1.StorageVersionMigration does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message E1004 12:34:41.102838 1 target_config_reconciler.go:425] key failed with : object *v1alpha1.StorageVersionMigration does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message E1004 12:44:41.104141 1 target_config_reconciler.go:425] key failed with : object *v1alpha1.StorageVersionMigration does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message E1004 12:48:27.202362 1 target_config_reconciler.go:425] key failed with : object *v1alpha1.StorageVersionMigration does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message E1004 12:54:41.106192 1 target_config_reconciler.go:425] key failed with : object *v1alpha1.StorageVersionMigration does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message
Here is the 4.8.0 descheduler version, clusterkubedescheduleroperator.4.8.0-202109281622
Both of the linked bugs have merged. Bad bot. Moving to ON_QA
Moving the bug back to assigned because i see that with the 4.8 fix in the descheduler operator now it starts to fail upgrading to 4.9 and below is the error i see. 1) upgrade from 4.6 -> 4.7 -> 4.8 -> 4.9 2) 4.8 version from where upgrade was done [knarra@knarra openshift-client-linux-4.9.0-0.nightly-2021-10-28-150616]$ oc get csv -n openshift-kube-descheduler-operator NAME DISPLAY VERSION REPLACES PHASE clusterkubedescheduleroperator.4.8.0-202110131431 Kube Descheduler Operator 4.8.0-202110131431 clusterkubedescheduleroperator.4.7.0-202110121415 Replacing clusterkubedescheduleroperator.4.9.0-202110182323 Kube Descheduler Operator 4.9.0-202110182323 clusterkubedescheduleroperator.4.8.0-202110131431 Pending Install Modes: Supported: true Type: OwnNamespace Supported: true Type: SingleNamespace Supported: false Type: MultiNamespace Supported: false Type: AllNamespaces Keywords: scheduling descheduler workload Labels: Olm - Owner - Enterprise - App: cluster-kube-descheduler-operator Olm - Status - Descriptors: cluster-kube-descheduler-operator.v4.9.0 Links: Name: Source Code URL: https://github.com/openshift/cluster-kube-descheduler-operator Maintainers: Email: support Name: Red Hat Maturity: beta Min Kube Version: 1.22.0 Provider: Name: Red Hat, Inc. Related Images: Image: registry.redhat.io/openshift4/ose-cluster-kube-descheduler-operator@sha256:0d2ff85df5662d937b4aa2d96ebcd3b0ecc8807d7170e0008965aa28f2aba1d1 Name: ose-cluster-kube-descheduler-operator Image: registry.redhat.io/openshift4/ose-descheduler@sha256:05c2392601b177dd27ac0af9e40754f793b8befcf42afed393640b67f9c8cd11 Name: ose-descheduler Replaces: clusterkubedescheduleroperator.4.8.0-202110131431 Version: 4.9.0-202110182323 Status: Cleanup: Conditions: Last Transition Time: 2021-10-29T15:50:30Z Last Update Time: 2021-10-29T15:50:30Z Message: requirements not yet checked Phase: Pending Reason: RequirementsUnknown Last Transition Time: 2021-10-29T15:50:30Z Last Update Time: 2021-10-29T15:50:30Z Message: one or more requirements couldn't be found Phase: Pending Reason: RequirementsNotMet Last Transition Time: 2021-10-29T15:50:30Z Last Update Time: 2021-10-29T15:50:30Z Message: one or more requirements couldn't be found Phase: Pending Reason: RequirementsNotMet Requirement Status: Group: operators.coreos.com Kind: ClusterServiceVersion Message: CSV minKubeVersion (1.22.0) less than server version (v1.22.0-rc.0+a44d0f0) Name: clusterkubedescheduleroperator.4.9.0-202110182323 Status: Present Version: v1alpha1 Group: apiextensions.k8s.io Kind: CustomResourceDefinition Message: CRD installed alongside other CSV(s): clusterkubedescheduleroperator.4.8.0-202110131431 Name: kubedeschedulers.operator.openshift.io Status: PresentNotSatisfied Version: v1 Group: Kind: ServiceAccount Message: Service account is owned by another ClusterServiceVersion Name: openshift-descheduler Status: PresentNotSatisfied Version: v1 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal RequirementsUnknown 23m operator-lifecycle-manager requirements not yet checked Normal RequirementsNotMet 23m (x2 over 23m) operator-lifecycle-manager one or more requirements couldn't be found must-gather could be found in the link below: ============================================== http://virt-openshift-05.lab.eng.nay.redhat.com/knarra/1991938/ Setting blocker '?' as if any customer/user tries to upgrade descheduler from currently released version of 4.8 i.e 4.8.0-202110131431 to 4.9 the upgrade is going to fail.
Hey Rama, I tried to reproduce this bug on 4.9.0-0.ci-2021-11-01-163801 build and didn't encounter this issue. Admittedly, I installed only 4.8 and upgraded to 4.9 version of descheduler. It worked just fine. Are you about to reproduce this issue consistently? Vu
Hello Vu Dinh, So i tried to reproduce again and i see that when we try to upgrade descheduler operator in a fresh installed 4.8 cluster i.e 4.8.0-202110131431 to 4.9 it does not fail, but when trying to upgrade descheduler operator from 4.6 to 4.9 it fails. when upgrading from 4.6 -> 4.7 -> 4.8 kubedescheduler CRD does not get updated to operator.openshift.io/v1 & remains at operator.openshift.io/v1beta1 in 4.8 and that might be the reason why there is a failure when upgrading from 4.8 -> 4.9 Operator logs at 4.8 contains below error logs. E1102 12:07:33.814476 1 target_config_reconciler.go:425] key failed with : customresourcedefinitions.apiextensions.k8s.io "kubedeschedulers.operator.openshift.io" is forbidden: User "system:serviceaccount:openshift-kube-descheduler-operator:openshift-descheduler" cannot get resource "customresourcedefinitions" in API group "apiextensions.k8s.io" at the cluster scope Hope this helps !! Please do let me know if you need more clarification, thanks !! Thanks kasturi
Hey Rama, So the issue is here the CRD versions. In 4.7, CRD has v1beta1 version which is both served and storage. In 4.8, CRD has v1 and v1beta1 versions and v1 is now served and storage while v1beta1 is not served nor storage which is considered compatible by OLM in term of CRD upgrade. However, in 4.9, the CRD only has v1 as v1beta1 is removed. Because v1beta1 was previously storage version (if you install 4.7 version of the operator), this removal of storage version is considered incompatible due to potential data loss as OLM doesn't do storage version/CR migration. As a result, OLM is unable to upgrade the underlying CRD which eventually leads to the failed upgrade with descheduler 4.9. If you inspect the InstallPlan that has 4.9 version, you will notice the CRD step is listed as FAILED with the message "data loss risk". In the past, OLM would handle the storage version removal on its own in certain situation. However, this practice is considered too risky without CR migration and/or user's approval. Therefore, in the case of incompatible CRD upgrade (such as CRD storage version), OLM will simply fail to upgrade the operator and user will need to manually intervene. This is a known behavior of OLM. It is up to the team on how to handle this CRD upgrade process. I can see one of merged PRs which has something to do with storageversionmigrations.migration.k8s.io so I wonder if the team will handle this migration programmatically or manually. Thanks, Vu
Hi Vu Dinh, Thanks for the detailed explanation & with respect to storageversionmigrations a fix has already been merged and the latest issue was something to do with permissions as i see below errors in the operator logs. Also i see that mike has already raised a PR [1] to fix the issue. Hopefully that will fix the issue. E1102 12:07:33.814476 1 target_config_reconciler.go:425] key failed with : customresourcedefinitions.apiextensions.k8s.io "kubedeschedulers.operator.openshift.io" is forbidden: User "system:serviceaccount:openshift-kube-descheduler-operator:openshift-descheduler" cannot get resource "customresourcedefinitions" in API group "apiextensions.k8s.io" at the cluster scope [1] https://github.com/openshift/cluster-kube-descheduler-operator/pull/228 Thanks kasturi
I tried to upgrade descheduler again from 4.6 -> 4.7 -> 4.8 -> 4.9 but i still see an issue, i do not see upgrade happening. Below is what i see from the installplan. [knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-01-19-104449]$ oc get ip -n openshift-kube-descheduler-operator NAME CSV APPROVAL APPROVED install-hqfnt clusterkubedescheduleroperator.4.9.0-202112142229 Automatic true install-jpz9q clusterkubedescheduleroperator.4.7.0-202111041636 Automatic true install-kbpng clusterkubedescheduleroperator.4.8.0-202112141153 Automatic true install-prn7s clusterkubedescheduleroperator.4.6.0-202111301700 Automatic true [knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-01-19-104449]$ oc describe ip install-hqfnt -n openshift-kube-descheduler-operator Name: install-hqfnt Namespace: openshift-kube-descheduler-operator Labels: operators.coreos.com/cluster-kube-descheduler-operator.openshift-kube-descheduler-op= Annotations: <none> API Version: operators.coreos.com/v1alpha1 Kind: InstallPlan Metadata: Creation Timestamp: 2022-01-19T17:54:12Z Generate Name: install- Generation: 1 Managed Fields: API Version: operators.coreos.com/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:generateName: f:ownerReferences: .: k:{"uid":"3dc9b9a3-fedd-428c-b245-752705c2d13e"}: f:spec: .: f:approval: f:approved: f:clusterServiceVersionNames: f:generation: Manager: catalog Operation: Update Time: 2022-01-19T17:54:12Z API Version: operators.coreos.com/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:labels: .: f:operators.coreos.com/cluster-kube-descheduler-operator.openshift-kube-descheduler-op: Manager: olm Operation: Update Time: 2022-01-19T17:54:12Z API Version: operators.coreos.com/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:status: .: f:bundleLookups: f:catalogSources: f:conditions: f:message: f:phase: f:plan: f:startTime: Manager: catalog Operation: Update Subresource: status Time: 2022-01-19T17:55:28Z Owner References: API Version: operators.coreos.com/v1alpha1 Block Owner Deletion: false Controller: false Kind: Subscription Name: cluster-kube-descheduler-operator UID: 3dc9b9a3-fedd-428c-b245-752705c2d13e Resource Version: 212102 UID: ecfdcf87-8a55-430b-b801-d72225f30959 Spec: Approval: Automatic Approved: true Cluster Service Version Names: clusterkubedescheduleroperator.4.9.0-202112142229 Generation: 4 Status: Bundle Lookups: Catalog Source Ref: Name: qe-app-registry Namespace: openshift-marketplace Identifier: clusterkubedescheduleroperator.4.9.0-202112142229 Path: registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-cluster-kube-descheduler-operator-bundle@sha256:cb84fbf8495438a637d95b21d8de981fb3ad7e37a553e24c18aab3f1b88765e2 Properties: {"properties":[{"type":"olm.gvk","value":{"group":"operator.openshift.io","kind":"KubeDescheduler","version":"v1"}},{"type":"olm.package","value":{"packageName":"cluster-kube-descheduler-operator","version":"4.9.0-202112142229"}}]} Replaces: clusterkubedescheduleroperator.4.8.0-202112141153 Catalog Sources: Conditions: Last Transition Time: 2022-01-19T17:55:28Z Last Update Time: 2022-01-19T17:55:28Z Message: risk of data loss updating "kubedeschedulers.operator.openshift.io": new CRD removes version v1beta1 that is listed as a stored version on the existing CRD Reason: InstallComponentFailed Status: False Type: Installed Message: risk of data loss updating "kubedeschedulers.operator.openshift.io": new CRD removes version v1beta1 that is listed as a stored version on the existing CRD Phase: Failed Plan: Resolving: clusterkubedescheduleroperator.4.9.0-202112142229 Resource: Group: operators.coreos.com Kind: ClusterServiceVersion Manifest: {"kind":"ConfigMap","name":"c712c8a7a2f1000d041329479e8b118fd7b01ca1370e77cbf381c87a53c94b6","namespace":"openshift-marketplace","catalogSourceName":"qe-app-registry","catalogSourceNamespace":"openshift-marketplace","replaces":"clusterkubedescheduleroperator.4.8.0-202112141153","properties":"{\"properties\":[{\"type\":\"olm.gvk\",\"value\":{\"group\":\"operator.openshift.io\",\"kind\":\"KubeDescheduler\",\"version\":\"v1\"}},{\"type\":\"olm.package\",\"value\":{\"packageName\":\"cluster-kube-descheduler-operator\",\"version\":\"4.9.0-202112142229\"}}]}"} Name: clusterkubedescheduleroperator.4.9.0-202112142229 Source Name: qe-app-registry Source Namespace: openshift-marketplace Version: v1alpha1 Status: Created Resolving: clusterkubedescheduleroperator.4.9.0-202112142229 Resource: Group: apiextensions.k8s.io Kind: CustomResourceDefinition Manifest: {"kind":"ConfigMap","name":"c712c8a7a2f1000d041329479e8b118fd7b01ca1370e77cbf381c87a53c94b6","namespace":"openshift-marketplace","catalogSourceName":"qe-app-registry","catalogSourceNamespace":"openshift-marketplace","replaces":"clusterkubedescheduleroperator.4.8.0-202112141153","properties":"{\"properties\":[{\"type\":\"olm.gvk\",\"value\":{\"group\":\"operator.openshift.io\",\"kind\":\"KubeDescheduler\",\"version\":\"v1\"}},{\"type\":\"olm.package\",\"value\":{\"packageName\":\"cluster-kube-descheduler-operator\",\"version\":\"4.9.0-202112142229\"}}]}"} Name: kubedeschedulers.operator.openshift.io Source Name: qe-app-registry Source Namespace: openshift-marketplace Version: v1 Status: Unknown Resolving: clusterkubedescheduleroperator.4.9.0-202112142229 Resource: Group: Kind: ServiceAccount Manifest: {"kind":"ConfigMap","name":"c712c8a7a2f1000d041329479e8b118fd7b01ca1370e77cbf381c87a53c94b6","namespace":"openshift-marketplace","catalogSourceName":"qe-app-registry","catalogSourceNamespace":"openshift-marketplace","replaces":"clusterkubedescheduleroperator.4.8.0-202112141153","properties":"{\"properties\":[{\"type\":\"olm.gvk\",\"value\":{\"group\":\"operator.openshift.io\",\"kind\":\"KubeDescheduler\",\"version\":\"v1\"}},{\"type\":\"olm.package\",\"value\":{\"packageName\":\"cluster-kube-descheduler-operator\",\"version\":\"4.9.0-202112142229\"}}]}"} Name: openshift-descheduler Source Name: qe-app-registry Source Namespace: openshift-marketplace Version: v1 Status: Unknown Resolving: clusterkubedescheduleroperator.4.9.0-202112142229 Resource: Group: rbac.authorization.k8s.io Kind: ClusterRole Manifest: {"kind":"ConfigMap","name":"c712c8a7a2f1000d041329479e8b118fd7b01ca1370e77cbf381c87a53c94b6","namespace":"openshift-marketplace","catalogSourceName":"qe-app-registry","catalogSourceNamespace":"openshift-marketplace","replaces":"clusterkubedescheduleroperator.4.8.0-202112141153","properties":"{\"properties\":[{\"type\":\"olm.gvk\",\"value\":{\"group\":\"operator.openshift.io\",\"kind\":\"KubeDescheduler\",\"version\":\"v1\"}},{\"type\":\"olm.package\",\"value\":{\"packageName\":\"cluster-kube-descheduler-operator\",\"version\":\"4.9.0-202112142229\"}}]}"} Name: clusterkubedescheduleroperator.4.9.0-202112142229-54f457cf66 Source Name: qe-app-registry Source Namespace: openshift-marketplace Version: v1 Status: Unknown Resolving: clusterkubedescheduleroperator.4.9.0-202112142229 Resource: Group: rbac.authorization.k8s.io Kind: ClusterRoleBinding Manifest: {"kind":"ConfigMap","name":"c712c8a7a2f1000d041329479e8b118fd7b01ca1370e77cbf381c87a53c94b6","namespace":"openshift-marketplace","catalogSourceName":"qe-app-registry","catalogSourceNamespace":"openshift-marketplace","replaces":"clusterkubedescheduleroperator.4.8.0-202112141153","properties":"{\"properties\":[{\"type\":\"olm.gvk\",\"value\":{\"group\":\"operator.openshift.io\",\"kind\":\"KubeDescheduler\",\"version\":\"v1\"}},{\"type\":\"olm.package\",\"value\":{\"packageName\":\"cluster-kube-descheduler-operator\",\"version\":\"4.9.0-202112142229\"}}]}"} Name: clusterkubedescheduleroperator.4.9.0-202112142229-54f457cf66 Source Name: qe-app-registry Source Namespace: openshift-marketplace Version: v1 Status: Unknown Start Time: 2022-01-19T17:54:28Z Events: <none> Based on the above moving the bug to assigned state. I still have the cluster where this test is run incase some one would like to take a look at, thanks !!
Due to higher priority tasks I have been able to resolve this issue in time. Moving to the next sprint.
Similar issue is seen when upgrading descheduler form 4.7 to 4.9. Spec: Approval: Automatic Approved: true Cluster Service Version Names: clusterkubedescheduleroperator.4.9.0-202206271415 Generation: 2 Status: Bundle Lookups: Catalog Source Ref: Name: qe-app-registry Namespace: openshift-marketplace Identifier: clusterkubedescheduleroperator.4.9.0-202206271415 Path: brew.registry.redhat.io/rh-osbs/openshift-ose-cluster-kube-descheduler-operator-bundle@sha256:e0cae11ae498bbe7dfb3afe7a596cd11a15eeea0b334fd1181bc5dea6f92456d Properties: {"properties":[{"type":"olm.gvk","value":{"group":"operator.openshift.io","kind":"KubeDescheduler","version":"v1"}},{"type":"olm.package","value":{"packageName":"cluster-kube-descheduler-operator","version":"4.9.0-202206271415"}}]} Replaces: clusterkubedescheduleroperator.4.7.0-202207010816 Catalog Sources: Conditions: Last Transition Time: 2022-07-06T13:38:37Z Last Update Time: 2022-07-06T13:38:37Z Message: risk of data loss updating "kubedeschedulers.operator.openshift.io": new CRD removes version v1beta1 that is listed as a stored version on the existing CRD Reason: InstallComponentFailed Status: False Type: Installed Message: risk of data loss updating "kubedeschedulers.operator.openshift.io": new CRD removes version v1beta1 that is listed as a stored version on the existing CRD Phase: Failed Plan: Resolving: clusterkubedescheduleroperator.4.9.0-202206271415 Resource: Group: operators.coreos.com Kind: ClusterServiceVersion Manifest: {"kind":"ConfigMap","name":"a9b5a06ab809f987598ea5e5d9dfdd0c811ed48009097d2e16799b9b42fa78a","namespace":"openshift-marketplace","catalogSourceName":"qe-app-registry","catalogSourceNamespace":"openshift-marketplace","replaces":"clusterkubedescheduleroperator.4.7.0-202207010816","properties":"{\"properties\":[{\"type\":\"olm.gvk\",\"value\":{\"group\":\"operator.openshift.io\",\"kind\":\"KubeDescheduler\",\"version\":\"v1\"}},{\"type\":\"olm.package\",\"value\":{\"packageName\":\"cluster-kube-descheduler-operator\",\"version\":\"4.9.0-202206271415\"}}]}"} Name: clusterkubedescheduleroperator.4.9.0-202206271415 Source Name: qe-app-registry Source Namespace: openshift-marketplace Version: v1alpha1 Status: Created @jan i see the bug is in modified state, do we already have fix for this ? If yes could you please help check and move the bug to ON_QA?
The fix was merged in March. Moving to ON_QA > Similar issue is seen when upgrading descheduler form 4.7 to 4.9. 4.7 is out of support. We will not be fixing the upgrade path there.
Verified bug in the latest payload below and i see that upgrade from 4.8 to 4.9 works fine. [knarra@knarra ~]$ oc get csv -n openshift-kube-descheduler-operator NAME DISPLAY VERSION REPLACES PHASE clusterkubedescheduleroperator.4.9.0-202206271415 Kube Descheduler Operator 4.9.0-202206271415 clusterkubedescheduleroperator.4.8.0-202206281335 Succeeded [knarra@knarra ~]$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.9.0-0.nightly-2022-06-24-070308 True False 20m Cluster version is 4.9.0-0.nightly-2022-06-24-070308 [knarra@knarra ~]$ oc get ip -n openshift-kube-descheduler-operator NAME CSV APPROVAL APPROVED install-2rpv9 clusterkubedescheduleroperator.4.8.0-202206281335 Automatic true install-dmpbd clusterkubedescheduleroperator.4.9.0-202206271415 Automatic true [knarra@knarra ~]$ oc get csv -n openshift-kube-descheduler-operator NAME DISPLAY VERSION REPLACES PHASE clusterkubedescheduleroperator.4.9.0-202206271415 Kube Descheduler Operator 4.9.0-202206271415 clusterkubedescheduleroperator.4.8.0-202206281335 Succeeded Based on the above moving the bug to verified state.
@jan one question, what would we do with the customers who would want to upgrade from 4.7 to 4.9 as we know that it does not work ?
Uninstall the operator, upgrade to 4.9 and install the operator again. Since the option of upgrading the operator to 4.8 first is not applicable.
In cases this approach leads to errors, we will need to discuss the situation per case to provide better guidance. E.g. manually updating the CRD from v1beta1 to v1, migrating the CR data and update the CRD to server only v1.
(In reply to Jan Chaloupka from comment #36) > In cases this approach leads to errors, we will need to discuss the > situation per case to provide better guidance. E.g. manually updating the > CRD from v1beta1 to v1, migrating the CR data and update the CRD to server > only v1. Thanks for the explanation jan !!
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Container Platform 4.8.49 security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:6308