Bug 1991938
| Summary: | Upgrading descheduler operator from 4.8 to 4.9 or installing latest 4.8 operator fails | |||
|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | RamaKasturi <knarra> | |
| Component: | kube-scheduler | Assignee: | Jan Chaloupka <jchaloup> | |
| Status: | CLOSED ERRATA | QA Contact: | RamaKasturi <knarra> | |
| Severity: | high | Docs Contact: | ||
| Priority: | high | |||
| Version: | 4.8 | CC: | akanekar, jchaloup, maszulik, mfojtik, vdinh, vlaad, wking | |
| Target Milestone: | --- | |||
| Target Release: | 4.8.z | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | No Doc Update | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1992478 (view as bug list) | Environment: | ||
| Last Closed: | 2022-09-14 20:38:55 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 2012890 | |||
*** Bug 1992478 has been marked as a duplicate of this bug. *** Validation is required to test adding DevPreviewLongLifecycle to the kubedescheduler cluster object as currently it fails with the below error. Once this bug moves to ON_QA we will need to validate below as well. # kubedeschedulers.operator.openshift.io "cluster" was not valid: # * spec.profiles: Unsupported value: "DevPreviewLongLifecycle": supported values: "AffinityAndTaints", "TopologyAndDuplicates", "LifecycleAndUtilization Tried verifying the bug here but i still see the error below, @jan should i wait for next respin of descheduler operator ? Version i tried verifying is below. Could you please help confirm ? [knarra@knarra openshift-client-linux-4.6.0-0.nightly-2021-09-13-181051]$ oc get csv -n openshift-kube-descheduler-operator NAME DISPLAY VERSION REPLACES PHASE clusterkubedescheduleroperator.4.8.0-202109100832 Kube Descheduler Operator 4.8.0-202109100832 clusterkubedescheduleroperator.4.7.0-202108311835 Succeeded E0916 15:40:44.574486 1 target_config_reconciler.go:425] key failed with : storageversionmigrations.migration.k8s.io "operator-kubedescheduler-storage-version-migration" is forbidden: User "system:serviceaccount:openshift-kube-descheduler-operator:openshift-descheduler" cannot get resource "storageversionmigrations" in API group "migration.k8s.io" at the cluster scope Moving the bug ack as the PR below [1] still has not merged yet, clearing needinfo on jan as he has confirmed the same on slack. [1] https://github.com/openshift/cluster-kube-descheduler-operator/pull/221 setting blocker '?' as any customer who tries to install or upgrade descheduler on 4.8 would hit this bug. Tried to upgrade from 4.6 to 4.9, but after upgrading to 4.8 i could see below error in the descheduler logs. message E1004 11:14:41.092535 1 target_config_reconciler.go:425] key failed with : object *v1alpha1.StorageVersionMigration does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message E1004 11:24:41.093216 1 target_config_reconciler.go:425] key failed with : object *v1alpha1.StorageVersionMigration does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message E1004 11:25:07.156259 1 target_config_reconciler.go:425] key failed with : object *v1alpha1.StorageVersionMigration does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message E1004 11:34:41.101799 1 target_config_reconciler.go:425] key failed with : object *v1alpha1.StorageVersionMigration does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message E1004 11:41:47.164784 1 target_config_reconciler.go:425] key failed with : object *v1alpha1.StorageVersionMigration does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message E1004 11:44:41.094502 1 target_config_reconciler.go:425] key failed with : object *v1alpha1.StorageVersionMigration does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message E1004 11:54:41.095124 1 target_config_reconciler.go:425] key failed with : object *v1alpha1.StorageVersionMigration does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message E1004 11:58:27.174260 1 target_config_reconciler.go:425] key failed with : object *v1alpha1.StorageVersionMigration does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message E1004 12:04:41.094648 1 target_config_reconciler.go:425] key failed with : object *v1alpha1.StorageVersionMigration does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message E1004 12:14:41.095152 1 target_config_reconciler.go:425] key failed with : object *v1alpha1.StorageVersionMigration does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message E1004 12:15:07.182630 1 target_config_reconciler.go:425] key failed with : object *v1alpha1.StorageVersionMigration does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message E1004 12:24:41.096773 1 target_config_reconciler.go:425] key failed with : object *v1alpha1.StorageVersionMigration does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message E1004 12:31:47.192133 1 target_config_reconciler.go:425] key failed with : object *v1alpha1.StorageVersionMigration does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message E1004 12:34:41.102838 1 target_config_reconciler.go:425] key failed with : object *v1alpha1.StorageVersionMigration does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message E1004 12:44:41.104141 1 target_config_reconciler.go:425] key failed with : object *v1alpha1.StorageVersionMigration does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message E1004 12:48:27.202362 1 target_config_reconciler.go:425] key failed with : object *v1alpha1.StorageVersionMigration does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message E1004 12:54:41.106192 1 target_config_reconciler.go:425] key failed with : object *v1alpha1.StorageVersionMigration does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message Here is the 4.8.0 descheduler version, clusterkubedescheduleroperator.4.8.0-202109281622 Both of the linked bugs have merged. Bad bot. Moving to ON_QA Moving the bug back to assigned because i see that with the 4.8 fix in the descheduler operator now it starts to fail upgrading to 4.9 and below is the error i see.
1) upgrade from 4.6 -> 4.7 -> 4.8 -> 4.9
2) 4.8 version from where upgrade was done
[knarra@knarra openshift-client-linux-4.9.0-0.nightly-2021-10-28-150616]$ oc get csv -n openshift-kube-descheduler-operator
NAME DISPLAY VERSION REPLACES PHASE
clusterkubedescheduleroperator.4.8.0-202110131431 Kube Descheduler Operator 4.8.0-202110131431 clusterkubedescheduleroperator.4.7.0-202110121415 Replacing
clusterkubedescheduleroperator.4.9.0-202110182323 Kube Descheduler Operator 4.9.0-202110182323 clusterkubedescheduleroperator.4.8.0-202110131431 Pending
Install Modes:
Supported: true
Type: OwnNamespace
Supported: true
Type: SingleNamespace
Supported: false
Type: MultiNamespace
Supported: false
Type: AllNamespaces
Keywords:
scheduling
descheduler
workload
Labels:
Olm - Owner - Enterprise - App: cluster-kube-descheduler-operator
Olm - Status - Descriptors: cluster-kube-descheduler-operator.v4.9.0
Links:
Name: Source Code
URL: https://github.com/openshift/cluster-kube-descheduler-operator
Maintainers:
Email: support
Name: Red Hat
Maturity: beta
Min Kube Version: 1.22.0
Provider:
Name: Red Hat, Inc.
Related Images:
Image: registry.redhat.io/openshift4/ose-cluster-kube-descheduler-operator@sha256:0d2ff85df5662d937b4aa2d96ebcd3b0ecc8807d7170e0008965aa28f2aba1d1
Name: ose-cluster-kube-descheduler-operator
Image: registry.redhat.io/openshift4/ose-descheduler@sha256:05c2392601b177dd27ac0af9e40754f793b8befcf42afed393640b67f9c8cd11
Name: ose-descheduler
Replaces: clusterkubedescheduleroperator.4.8.0-202110131431
Version: 4.9.0-202110182323
Status:
Cleanup:
Conditions:
Last Transition Time: 2021-10-29T15:50:30Z
Last Update Time: 2021-10-29T15:50:30Z
Message: requirements not yet checked
Phase: Pending
Reason: RequirementsUnknown
Last Transition Time: 2021-10-29T15:50:30Z
Last Update Time: 2021-10-29T15:50:30Z
Message: one or more requirements couldn't be found
Phase: Pending
Reason: RequirementsNotMet
Last Transition Time: 2021-10-29T15:50:30Z
Last Update Time: 2021-10-29T15:50:30Z
Message: one or more requirements couldn't be found
Phase: Pending
Reason: RequirementsNotMet
Requirement Status:
Group: operators.coreos.com
Kind: ClusterServiceVersion
Message: CSV minKubeVersion (1.22.0) less than server version (v1.22.0-rc.0+a44d0f0)
Name: clusterkubedescheduleroperator.4.9.0-202110182323
Status: Present
Version: v1alpha1
Group: apiextensions.k8s.io
Kind: CustomResourceDefinition
Message: CRD installed alongside other CSV(s): clusterkubedescheduleroperator.4.8.0-202110131431
Name: kubedeschedulers.operator.openshift.io
Status: PresentNotSatisfied
Version: v1
Group:
Kind: ServiceAccount
Message: Service account is owned by another ClusterServiceVersion
Name: openshift-descheduler
Status: PresentNotSatisfied
Version: v1
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal RequirementsUnknown 23m operator-lifecycle-manager requirements not yet checked
Normal RequirementsNotMet 23m (x2 over 23m) operator-lifecycle-manager one or more requirements couldn't be found
must-gather could be found in the link below:
==============================================
http://virt-openshift-05.lab.eng.nay.redhat.com/knarra/1991938/
Setting blocker '?' as if any customer/user tries to upgrade descheduler from currently released version of 4.8 i.e 4.8.0-202110131431 to 4.9 the upgrade is going to fail.
Hey Rama, I tried to reproduce this bug on 4.9.0-0.ci-2021-11-01-163801 build and didn't encounter this issue. Admittedly, I installed only 4.8 and upgraded to 4.9 version of descheduler. It worked just fine. Are you about to reproduce this issue consistently? Vu Hello Vu Dinh, So i tried to reproduce again and i see that when we try to upgrade descheduler operator in a fresh installed 4.8 cluster i.e 4.8.0-202110131431 to 4.9 it does not fail, but when trying to upgrade descheduler operator from 4.6 to 4.9 it fails. when upgrading from 4.6 -> 4.7 -> 4.8 kubedescheduler CRD does not get updated to operator.openshift.io/v1 & remains at operator.openshift.io/v1beta1 in 4.8 and that might be the reason why there is a failure when upgrading from 4.8 -> 4.9 Operator logs at 4.8 contains below error logs. E1102 12:07:33.814476 1 target_config_reconciler.go:425] key failed with : customresourcedefinitions.apiextensions.k8s.io "kubedeschedulers.operator.openshift.io" is forbidden: User "system:serviceaccount:openshift-kube-descheduler-operator:openshift-descheduler" cannot get resource "customresourcedefinitions" in API group "apiextensions.k8s.io" at the cluster scope Hope this helps !! Please do let me know if you need more clarification, thanks !! Thanks kasturi Hey Rama, So the issue is here the CRD versions. In 4.7, CRD has v1beta1 version which is both served and storage. In 4.8, CRD has v1 and v1beta1 versions and v1 is now served and storage while v1beta1 is not served nor storage which is considered compatible by OLM in term of CRD upgrade. However, in 4.9, the CRD only has v1 as v1beta1 is removed. Because v1beta1 was previously storage version (if you install 4.7 version of the operator), this removal of storage version is considered incompatible due to potential data loss as OLM doesn't do storage version/CR migration. As a result, OLM is unable to upgrade the underlying CRD which eventually leads to the failed upgrade with descheduler 4.9. If you inspect the InstallPlan that has 4.9 version, you will notice the CRD step is listed as FAILED with the message "data loss risk". In the past, OLM would handle the storage version removal on its own in certain situation. However, this practice is considered too risky without CR migration and/or user's approval. Therefore, in the case of incompatible CRD upgrade (such as CRD storage version), OLM will simply fail to upgrade the operator and user will need to manually intervene. This is a known behavior of OLM. It is up to the team on how to handle this CRD upgrade process. I can see one of merged PRs which has something to do with storageversionmigrations.migration.k8s.io so I wonder if the team will handle this migration programmatically or manually. Thanks, Vu Hi Vu Dinh,
Thanks for the detailed explanation & with respect to storageversionmigrations a fix has already been merged and the latest issue was something to do with permissions as i see below errors in the operator logs. Also i see that mike has already raised a PR [1] to fix the issue. Hopefully that will fix the issue.
E1102 12:07:33.814476 1 target_config_reconciler.go:425] key failed with : customresourcedefinitions.apiextensions.k8s.io "kubedeschedulers.operator.openshift.io" is forbidden: User "system:serviceaccount:openshift-kube-descheduler-operator:openshift-descheduler" cannot get resource "customresourcedefinitions" in API group "apiextensions.k8s.io" at the cluster scope
[1] https://github.com/openshift/cluster-kube-descheduler-operator/pull/228
Thanks
kasturi
I tried to upgrade descheduler again from 4.6 -> 4.7 -> 4.8 -> 4.9 but i still see an issue, i do not see upgrade happening. Below is what i see from the installplan.
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-01-19-104449]$ oc get ip -n openshift-kube-descheduler-operator
NAME CSV APPROVAL APPROVED
install-hqfnt clusterkubedescheduleroperator.4.9.0-202112142229 Automatic true
install-jpz9q clusterkubedescheduleroperator.4.7.0-202111041636 Automatic true
install-kbpng clusterkubedescheduleroperator.4.8.0-202112141153 Automatic true
install-prn7s clusterkubedescheduleroperator.4.6.0-202111301700 Automatic true
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-01-19-104449]$ oc describe ip install-hqfnt -n openshift-kube-descheduler-operator
Name: install-hqfnt
Namespace: openshift-kube-descheduler-operator
Labels: operators.coreos.com/cluster-kube-descheduler-operator.openshift-kube-descheduler-op=
Annotations: <none>
API Version: operators.coreos.com/v1alpha1
Kind: InstallPlan
Metadata:
Creation Timestamp: 2022-01-19T17:54:12Z
Generate Name: install-
Generation: 1
Managed Fields:
API Version: operators.coreos.com/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:generateName:
f:ownerReferences:
.:
k:{"uid":"3dc9b9a3-fedd-428c-b245-752705c2d13e"}:
f:spec:
.:
f:approval:
f:approved:
f:clusterServiceVersionNames:
f:generation:
Manager: catalog
Operation: Update
Time: 2022-01-19T17:54:12Z
API Version: operators.coreos.com/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:labels:
.:
f:operators.coreos.com/cluster-kube-descheduler-operator.openshift-kube-descheduler-op:
Manager: olm
Operation: Update
Time: 2022-01-19T17:54:12Z
API Version: operators.coreos.com/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:bundleLookups:
f:catalogSources:
f:conditions:
f:message:
f:phase:
f:plan:
f:startTime:
Manager: catalog
Operation: Update
Subresource: status
Time: 2022-01-19T17:55:28Z
Owner References:
API Version: operators.coreos.com/v1alpha1
Block Owner Deletion: false
Controller: false
Kind: Subscription
Name: cluster-kube-descheduler-operator
UID: 3dc9b9a3-fedd-428c-b245-752705c2d13e
Resource Version: 212102
UID: ecfdcf87-8a55-430b-b801-d72225f30959
Spec:
Approval: Automatic
Approved: true
Cluster Service Version Names:
clusterkubedescheduleroperator.4.9.0-202112142229
Generation: 4
Status:
Bundle Lookups:
Catalog Source Ref:
Name: qe-app-registry
Namespace: openshift-marketplace
Identifier: clusterkubedescheduleroperator.4.9.0-202112142229
Path: registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-cluster-kube-descheduler-operator-bundle@sha256:cb84fbf8495438a637d95b21d8de981fb3ad7e37a553e24c18aab3f1b88765e2
Properties: {"properties":[{"type":"olm.gvk","value":{"group":"operator.openshift.io","kind":"KubeDescheduler","version":"v1"}},{"type":"olm.package","value":{"packageName":"cluster-kube-descheduler-operator","version":"4.9.0-202112142229"}}]}
Replaces: clusterkubedescheduleroperator.4.8.0-202112141153
Catalog Sources:
Conditions:
Last Transition Time: 2022-01-19T17:55:28Z
Last Update Time: 2022-01-19T17:55:28Z
Message: risk of data loss updating "kubedeschedulers.operator.openshift.io": new CRD removes version v1beta1 that is listed as a stored version on the existing CRD
Reason: InstallComponentFailed
Status: False
Type: Installed
Message: risk of data loss updating "kubedeschedulers.operator.openshift.io": new CRD removes version v1beta1 that is listed as a stored version on the existing CRD
Phase: Failed
Plan:
Resolving: clusterkubedescheduleroperator.4.9.0-202112142229
Resource:
Group: operators.coreos.com
Kind: ClusterServiceVersion
Manifest: {"kind":"ConfigMap","name":"c712c8a7a2f1000d041329479e8b118fd7b01ca1370e77cbf381c87a53c94b6","namespace":"openshift-marketplace","catalogSourceName":"qe-app-registry","catalogSourceNamespace":"openshift-marketplace","replaces":"clusterkubedescheduleroperator.4.8.0-202112141153","properties":"{\"properties\":[{\"type\":\"olm.gvk\",\"value\":{\"group\":\"operator.openshift.io\",\"kind\":\"KubeDescheduler\",\"version\":\"v1\"}},{\"type\":\"olm.package\",\"value\":{\"packageName\":\"cluster-kube-descheduler-operator\",\"version\":\"4.9.0-202112142229\"}}]}"}
Name: clusterkubedescheduleroperator.4.9.0-202112142229
Source Name: qe-app-registry
Source Namespace: openshift-marketplace
Version: v1alpha1
Status: Created
Resolving: clusterkubedescheduleroperator.4.9.0-202112142229
Resource:
Group: apiextensions.k8s.io
Kind: CustomResourceDefinition
Manifest: {"kind":"ConfigMap","name":"c712c8a7a2f1000d041329479e8b118fd7b01ca1370e77cbf381c87a53c94b6","namespace":"openshift-marketplace","catalogSourceName":"qe-app-registry","catalogSourceNamespace":"openshift-marketplace","replaces":"clusterkubedescheduleroperator.4.8.0-202112141153","properties":"{\"properties\":[{\"type\":\"olm.gvk\",\"value\":{\"group\":\"operator.openshift.io\",\"kind\":\"KubeDescheduler\",\"version\":\"v1\"}},{\"type\":\"olm.package\",\"value\":{\"packageName\":\"cluster-kube-descheduler-operator\",\"version\":\"4.9.0-202112142229\"}}]}"}
Name: kubedeschedulers.operator.openshift.io
Source Name: qe-app-registry
Source Namespace: openshift-marketplace
Version: v1
Status: Unknown
Resolving: clusterkubedescheduleroperator.4.9.0-202112142229
Resource:
Group:
Kind: ServiceAccount
Manifest: {"kind":"ConfigMap","name":"c712c8a7a2f1000d041329479e8b118fd7b01ca1370e77cbf381c87a53c94b6","namespace":"openshift-marketplace","catalogSourceName":"qe-app-registry","catalogSourceNamespace":"openshift-marketplace","replaces":"clusterkubedescheduleroperator.4.8.0-202112141153","properties":"{\"properties\":[{\"type\":\"olm.gvk\",\"value\":{\"group\":\"operator.openshift.io\",\"kind\":\"KubeDescheduler\",\"version\":\"v1\"}},{\"type\":\"olm.package\",\"value\":{\"packageName\":\"cluster-kube-descheduler-operator\",\"version\":\"4.9.0-202112142229\"}}]}"}
Name: openshift-descheduler
Source Name: qe-app-registry
Source Namespace: openshift-marketplace
Version: v1
Status: Unknown
Resolving: clusterkubedescheduleroperator.4.9.0-202112142229
Resource:
Group: rbac.authorization.k8s.io
Kind: ClusterRole
Manifest: {"kind":"ConfigMap","name":"c712c8a7a2f1000d041329479e8b118fd7b01ca1370e77cbf381c87a53c94b6","namespace":"openshift-marketplace","catalogSourceName":"qe-app-registry","catalogSourceNamespace":"openshift-marketplace","replaces":"clusterkubedescheduleroperator.4.8.0-202112141153","properties":"{\"properties\":[{\"type\":\"olm.gvk\",\"value\":{\"group\":\"operator.openshift.io\",\"kind\":\"KubeDescheduler\",\"version\":\"v1\"}},{\"type\":\"olm.package\",\"value\":{\"packageName\":\"cluster-kube-descheduler-operator\",\"version\":\"4.9.0-202112142229\"}}]}"}
Name: clusterkubedescheduleroperator.4.9.0-202112142229-54f457cf66
Source Name: qe-app-registry
Source Namespace: openshift-marketplace
Version: v1
Status: Unknown
Resolving: clusterkubedescheduleroperator.4.9.0-202112142229
Resource:
Group: rbac.authorization.k8s.io
Kind: ClusterRoleBinding
Manifest: {"kind":"ConfigMap","name":"c712c8a7a2f1000d041329479e8b118fd7b01ca1370e77cbf381c87a53c94b6","namespace":"openshift-marketplace","catalogSourceName":"qe-app-registry","catalogSourceNamespace":"openshift-marketplace","replaces":"clusterkubedescheduleroperator.4.8.0-202112141153","properties":"{\"properties\":[{\"type\":\"olm.gvk\",\"value\":{\"group\":\"operator.openshift.io\",\"kind\":\"KubeDescheduler\",\"version\":\"v1\"}},{\"type\":\"olm.package\",\"value\":{\"packageName\":\"cluster-kube-descheduler-operator\",\"version\":\"4.9.0-202112142229\"}}]}"}
Name: clusterkubedescheduleroperator.4.9.0-202112142229-54f457cf66
Source Name: qe-app-registry
Source Namespace: openshift-marketplace
Version: v1
Status: Unknown
Start Time: 2022-01-19T17:54:28Z
Events: <none>
Based on the above moving the bug to assigned state. I still have the cluster where this test is run incase some one would like to take a look at, thanks !!
Due to higher priority tasks I have been able to resolve this issue in time. Moving to the next sprint. Due to higher priority tasks I have been able to resolve this issue in time. Moving to the next sprint. Similar issue is seen when upgrading descheduler form 4.7 to 4.9.
Spec:
Approval: Automatic
Approved: true
Cluster Service Version Names:
clusterkubedescheduleroperator.4.9.0-202206271415
Generation: 2
Status:
Bundle Lookups:
Catalog Source Ref:
Name: qe-app-registry
Namespace: openshift-marketplace
Identifier: clusterkubedescheduleroperator.4.9.0-202206271415
Path: brew.registry.redhat.io/rh-osbs/openshift-ose-cluster-kube-descheduler-operator-bundle@sha256:e0cae11ae498bbe7dfb3afe7a596cd11a15eeea0b334fd1181bc5dea6f92456d
Properties: {"properties":[{"type":"olm.gvk","value":{"group":"operator.openshift.io","kind":"KubeDescheduler","version":"v1"}},{"type":"olm.package","value":{"packageName":"cluster-kube-descheduler-operator","version":"4.9.0-202206271415"}}]}
Replaces: clusterkubedescheduleroperator.4.7.0-202207010816
Catalog Sources:
Conditions:
Last Transition Time: 2022-07-06T13:38:37Z
Last Update Time: 2022-07-06T13:38:37Z
Message: risk of data loss updating "kubedeschedulers.operator.openshift.io": new CRD removes version v1beta1 that is listed as a stored version on the existing CRD
Reason: InstallComponentFailed
Status: False
Type: Installed
Message: risk of data loss updating "kubedeschedulers.operator.openshift.io": new CRD removes version v1beta1 that is listed as a stored version on the existing CRD
Phase: Failed
Plan:
Resolving: clusterkubedescheduleroperator.4.9.0-202206271415
Resource:
Group: operators.coreos.com
Kind: ClusterServiceVersion
Manifest: {"kind":"ConfigMap","name":"a9b5a06ab809f987598ea5e5d9dfdd0c811ed48009097d2e16799b9b42fa78a","namespace":"openshift-marketplace","catalogSourceName":"qe-app-registry","catalogSourceNamespace":"openshift-marketplace","replaces":"clusterkubedescheduleroperator.4.7.0-202207010816","properties":"{\"properties\":[{\"type\":\"olm.gvk\",\"value\":{\"group\":\"operator.openshift.io\",\"kind\":\"KubeDescheduler\",\"version\":\"v1\"}},{\"type\":\"olm.package\",\"value\":{\"packageName\":\"cluster-kube-descheduler-operator\",\"version\":\"4.9.0-202206271415\"}}]}"}
Name: clusterkubedescheduleroperator.4.9.0-202206271415
Source Name: qe-app-registry
Source Namespace: openshift-marketplace
Version: v1alpha1
Status: Created
@jan i see the bug is in modified state, do we already have fix for this ? If yes could you please help check and move the bug to ON_QA?
The fix was merged in March. Moving to ON_QA
> Similar issue is seen when upgrading descheduler form 4.7 to 4.9.
4.7 is out of support. We will not be fixing the upgrade path there.
Verified bug in the latest payload below and i see that upgrade from 4.8 to 4.9 works fine. [knarra@knarra ~]$ oc get csv -n openshift-kube-descheduler-operator NAME DISPLAY VERSION REPLACES PHASE clusterkubedescheduleroperator.4.9.0-202206271415 Kube Descheduler Operator 4.9.0-202206271415 clusterkubedescheduleroperator.4.8.0-202206281335 Succeeded [knarra@knarra ~]$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.9.0-0.nightly-2022-06-24-070308 True False 20m Cluster version is 4.9.0-0.nightly-2022-06-24-070308 [knarra@knarra ~]$ oc get ip -n openshift-kube-descheduler-operator NAME CSV APPROVAL APPROVED install-2rpv9 clusterkubedescheduleroperator.4.8.0-202206281335 Automatic true install-dmpbd clusterkubedescheduleroperator.4.9.0-202206271415 Automatic true [knarra@knarra ~]$ oc get csv -n openshift-kube-descheduler-operator NAME DISPLAY VERSION REPLACES PHASE clusterkubedescheduleroperator.4.9.0-202206271415 Kube Descheduler Operator 4.9.0-202206271415 clusterkubedescheduleroperator.4.8.0-202206281335 Succeeded Based on the above moving the bug to verified state. @jan one question, what would we do with the customers who would want to upgrade from 4.7 to 4.9 as we know that it does not work ? Uninstall the operator, upgrade to 4.9 and install the operator again. Since the option of upgrading the operator to 4.8 first is not applicable. In cases this approach leads to errors, we will need to discuss the situation per case to provide better guidance. E.g. manually updating the CRD from v1beta1 to v1, migrating the CR data and update the CRD to server only v1. (In reply to Jan Chaloupka from comment #36) > In cases this approach leads to errors, we will need to discuss the > situation per case to provide better guidance. E.g. manually updating the > CRD from v1beta1 to v1, migrating the CR data and update the CRD to server > only v1. Thanks for the explanation jan !! Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Container Platform 4.8.49 security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:6308 |
Description of problem: Upgrading descheduler operator from 4.8 to 4.9 fails with error “CRD removes version v1beta1 that is listed as a stored version on the existing CRD” Version-Release number of selected component (if applicable): clusterkubedescheduleroperator.4.9.0-202108050954 How reproducible: Hit once Steps to Reproduce: 1. Install 4.6 descheduler operator 2. Upgrade from 4.6 -> 4.7 -> 4.8 -> 4.9 3. Actual results: Upgrading from 4.8 to 4.9 fails [knarra@knarra ~]$ oc get csv -n openshift-kube-descheduler-operator NAME DISPLAY VERSION REPLACES PHASE clusterkubedescheduleroperator.4.8.0-202107291502 Kube Descheduler Operator 4.8.0-202107291502 clusterkubedescheduleroperator.4.7.0-202107292319 Replacing clusterkubedescheduleroperator.4.9.0-202108050954 Kube Descheduler Operator 4.9.0-202108050954 clusterkubedescheduleroperator.4.8.0-202107291502 Pending elasticsearch-operator.5.2.0-28 OpenShift Elasticsearch Operator 5.2.0-28 elasticsearch-operator.5.1.1-27 Succeeded status: bundleLookups: - catalogSourceRef: name: qe-app-registry namespace: openshift-marketplace identifier: clusterkubedescheduleroperator.4.9.0-202108050954 path: registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-cluster-kube-descheduler-operator-bundle@sha256:f2be2b8652b6e98693cafc4ed311c95d2ce6af9c458bedad41341703dae3e1d1 properties: '{"properties":[{"type":"olm.gvk","value":{"group":"operator.openshift.io","kind":"KubeDescheduler","version":"v1"}},{"type":"olm.package","value":{"packageName":"cluster-kube-descheduler-operator","version":"4.9.0-202108050954"}}]}' replaces: clusterkubedescheduleroperator.4.8.0-202107291502 catalogSources: [] conditions: - lastTransitionTime: "2021-08-10T10:03:01Z" lastUpdateTime: "2021-08-10T10:03:01Z" message: 'risk of data loss updating "kubedeschedulers.operator.openshift.io": new CRD removes version v1beta1 that is listed as a stored version on the existing CRD' reason: InstallComponentFailed status: "False" type: Installed message: 'risk of data loss updating "kubedeschedulers.operator.openshift.io": new CRD removes version v1beta1 that is listed as a stored version on the existing CRD' phase: Failed plan: - resolving: clusterkubedescheduleroperator.4.9.0-202108050954 resource: group: operators.coreos.com kind: ClusterServiceVersion Expected results: Upgrade should be successful Additional info: