Description of problem: Upgrading descheduler from 4.8 to 4.9 fails, i see that when channel & starting CSV is set i do not see any upgrade starting, jian zhang looked further and below is what he found. 1, Only remove the sub, not the csv, resubscribe the 4.9 one, get the below errors: - message: 'constraints not satisfiable: @existing/openshift-kube-descheduler-operator//clusterkubedescheduleroperator.4.8.0-202108312109 and qe-app-registry/openshift-marketplace/4.9/clusterkubedescheduleroperator.4.9.0-202109071344 originate from package cluster-kube-descheduler-operator, subscription cluster-kube-descheduler-operator requires qe-app-registry/openshift-marketplace/4.9/clusterkubedescheduleroperator.4.9.0-202109071344, subscription cluster-kube-descheduler-operator exists, clusterserviceversion clusterkubedescheduleroperator.4.8.0-202108312109 exists and is not referenced by a subscription' reason: ConstraintsNotSatisfiable status: "True" type: ResolutionFailed 2,, Remove the sub and csv, and then recreate it, it works well. [cloud-user@preserve-olm-env jian]$ cat sub-descheduler.yaml apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: cluster-kube-descheduler-operator namespace: openshift-kube-descheduler-operator spec: channel: "4.9" installPlanApproval: Automatic name: cluster-kube-descheduler-operator source: qe-app-registry sourceNamespace: openshift-marketplace startingCSV: clusterkubedescheduleroperator.4.9.0-202109071344 [cloud-user@preserve-olm-env jian]$ oc get sub NAME PACKAGE SOURCE CHANNEL cluster-kube-descheduler-operator cluster-kube-descheduler-operator qe-app-registry 4.9 [cloud-user@preserve-olm-env jian]$ oc get ip NAME CSV APPROVAL APPROVED install-4w9lm clusterkubedescheduleroperator.4.9.0-202109071344 Automatic true [cloud-user@preserve-olm-env jian]$ oc get csv NAME DISPLAY VERSION REPLACES PHASE clusterkubedescheduleroperator.4.9.0-202109071344 Kube Descheduler Operator 4.9.0-202109071344 Succeeded elasticsearch-operator.5.2.0-60 OpenShift Elasticsearch Operator 5.2.0-60 elasticsearch-operator.5.1.1-56 Succeeded 3, check the bundle content: clusterkubedescheduleroperator.4.8.0-202108312109 {"apiVersion":"opera {"apiVersion":"operators.coreos.com/v1alpha1","kin registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose- >=4.6.0 <4.8.0-202108312109 4.8.0-202108312109 clusterkubedescheduleroperator.4.9.0-202109071344 {"apiVersion":"opera {"apiVersion":"operators.coreos.com/v1alpha1","kin registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose- >=4.6.0 <4.9.0-202109071344 4.9.0-202109071344 Version-Release number of selected component (if applicable): [knarra@knarra ~]$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.9.0-0.nightly-2021-09-06-004132 True False 23h Cluster version is 4.9.0-0.nightly-2021-09-06-004132 Clusterkubedescheduleroperator.4.8.0-202108312109 How reproducible: Install 4.8 cluster Install 4.8 descheduler operator Upgrade system to 4.9 Now edit descheduler sub & set channel to 4.9, starting csv to clusterkubedescheduleroperator.4.9.0-202109071344 spec: channel: "4.9" installPlanApproval: Automatic name: cluster-kube-descheduler-operator source: qe-app-registry sourceNamespace: openshift-marketplace startingCSV: clusterkubedescheduleroperator.4.9.0-202109071344 Actual Results: Upgrade does not start at all Expected Results: Upgrade should work fine.
From the Descheduler side, we haven't seen an error like this before so we are not sure what the cause is. Hopefully someone from OLM can lend some insight, and if there are any changes needed from us please let us know
Hi, The issue here is the CSV for 4.8 is still present while you were trying to install another version of the same operator. If you uninstall an operator like you did in step 1, then you need to delete both Subscription as CSV. If you just delete the Subscription, the CSV is still there meaning the operator for 4.8 is still present. When you do a cross channel upgrade, then you simply need to update the channel info. That should trigger the upgrade. The upgrade process that you should do is: 1. Create a sub for 4.8 with channel and startingCSV information that you desire. 2. Ensure the operator is installed properly. 3. Upgrade the cluster to 4.9. 4. Update the Subscription channel info to the 4.9 channel that you desire. 5. Check for the upgrade to happen. Note: The head of channel on the new 4.9 channel must have a compatible upgrade path from the previous version in 4.8 (such as skipRange). Vu
Thanks Vu!
Hi Vu, > The issue here is the CSV for 4.8 is still present while you were trying to install another version of the same operator. If you uninstall an operator like you did in step 1, then you need to delete both Subscription as CSV. If you just delete the Subscription, the CSV is still there meaning the operator for 4.8 is still present. > 1, Only remove the sub, not the csv, resubscribe the 4.9 one, get the below errors: - message: 'constraints not satisfiable: @existing/openshift-kube-descheduler-operator//clusterkubedescheduleroperator.4.8.0-202108312109 and qe-app-registry/openshift-marketplace/4.9/clusterkubedescheduleroperator.4.9.0-202109071344 originate from package cluster-kube-descheduler-operator, subscription cluster-kube-descheduler-operator requires qe-app-registry/openshift-marketplace/4.9/clusterkubedescheduleroperator.4.9.0-202109071344, subscription cluster-kube-descheduler-operator exists, clusterserviceversion clusterkubedescheduleroperator.4.8.0-202108312109 exists and is not referenced by a subscription' reason: ConstraintsNotSatisfiable status: "True" type: ResolutionFailed For this error, that's my debug trying. There is no this error when the deschduler operator upgrading. > The upgrade process that you should do is: 1. Create a sub for 4.8 with channel and startingCSV information that you desire. 2. Ensure the operator is installed properly. 3. Upgrade the cluster to 4.9. 4. Update the Subscription channel info to the 4.9 channel that you desire. 5. Check for the upgrade to happen. I think the only difference is that they also update the `startingCSV` field. Does it master? The origin scenario as follows, as you can see, there is no InstallPlan generated for the 4.9 channel. And, I checked the CatalogSource, the 4.9 did there. [cloud-user@preserve-olm-env jian]$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.9.0-0.nightly-2021-09-06-004132 True False 22h Cluster version is 4.9.0-0.nightly-2021-09-06-004132 [cloud-user@preserve-olm-env jian]$ oc project openshift-kube-descheduler-operator Now using project "openshift-kube-descheduler-operator" on server "https://api.yangyang-share.qe.devcluster.openshift.com:6443". [cloud-user@preserve-olm-env jian]$ oc get sub NAME PACKAGE SOURCE CHANNEL cluster-kube-descheduler-operator cluster-kube-descheduler-operator qe-app-registry 4.9 [cloud-user@preserve-olm-env jian]$ oc get ip NAME CSV APPROVAL APPROVED install-78wtg clusterkubedescheduleroperator.4.8.0-202108312109 Automatic true [cloud-user@preserve-olm-env jian]$ oc get csv NAME DISPLAY VERSION REPLACES PHASE clusterkubedescheduleroperator.4.8.0-202108312109 Kube Descheduler Operator 4.8.0-202108312109 Succeeded [cloud-user@preserve-olm-env jian]$ oc get sub cluster-kube-descheduler-operator -o yaml apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: creationTimestamp: "2021-09-07T07:59:58Z" generation: 3 labels: operators.coreos.com/cluster-kube-descheduler-operator.openshift-kube-descheduler-op: "" name: cluster-kube-descheduler-operator namespace: openshift-kube-descheduler-operator resourceVersion: "1098745" uid: b2dbaea7-3945-4b8a-b7ba-df184132a01f spec: channel: "4.9" installPlanApproval: Automatic name: cluster-kube-descheduler-operator source: qe-app-registry sourceNamespace: openshift-marketplace startingCSV: clusterkubedescheduleroperator.4.9.0-202109071344 status: catalogHealth: - catalogSourceRef: apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource name: certified-operators namespace: openshift-marketplace resourceVersion: "1027588" uid: 419b602f-6f8a-4a79-b056-c095c655609b healthy: true lastUpdated: "2021-09-08T07:49:33Z" - catalogSourceRef: apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource name: community-operators namespace: openshift-marketplace resourceVersion: "1024335" uid: 28414a7f-ee4a-4f08-bc17-e0a49f87cba0 healthy: true lastUpdated: "2021-09-08T07:49:33Z" - catalogSourceRef: apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource name: file-integrity-operator namespace: openshift-marketplace resourceVersion: "1021345" uid: 3aa1359e-d848-475b-b610-f934c5a18636 healthy: true lastUpdated: "2021-09-08T07:49:33Z" - catalogSourceRef: apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource name: qe-app-registry namespace: openshift-marketplace resourceVersion: "1027283" uid: e981e4b1-80ff-4279-9693-72549926e693 healthy: true lastUpdated: "2021-09-08T07:49:33Z" - catalogSourceRef: apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource name: redhat-marketplace namespace: openshift-marketplace resourceVersion: "1023548" uid: f5e16d4e-54f8-452d-ad46-235c398288f2 healthy: true lastUpdated: "2021-09-08T07:49:33Z" - catalogSourceRef: apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource name: redhat-operators namespace: openshift-marketplace resourceVersion: "1022918" uid: 5af279d5-9cc0-47c4-83d6-62f0815fba38 healthy: true lastUpdated: "2021-09-08T07:49:33Z" conditions: - lastTransitionTime: "2021-09-08T07:49:33Z" message: all available catalogsources are healthy reason: AllCatalogSourcesHealthy status: "False" type: CatalogSourcesUnhealthy - status: "False" type: ResolutionFailed currentCSV: clusterkubedescheduleroperator.4.8.0-202108312109 installPlanGeneration: 1 installPlanRef: apiVersion: operators.coreos.com/v1alpha1 kind: InstallPlan name: install-78wtg namespace: openshift-kube-descheduler-operator resourceVersion: "79121" uid: 6e7041e0-b3b2-490c-81fe-774ac92aaba8 installedCSV: clusterkubedescheduleroperator.4.8.0-202108312109 installplan: apiVersion: operators.coreos.com/v1alpha1 kind: InstallPlan name: install-78wtg uuid: 6e7041e0-b3b2-490c-81fe-774ac92aaba8 lastUpdated: "2021-09-08T08:40:43Z" state: AtLatestKnown
Can we confirm that there is actually an upgrade edge that OLM is ignoring in this situation? How is the 4.9 catalog being built? As far as I'm aware, the 4.9 index is an older copy of the 4.8 index while the pipelines are still in the process of bootstrapping the 4.9 indexes.
Hi Jian, Can you do a `oc get event` to see what is the ResolutionFailed event is? Plus, can we have the yaml of the CSV which is the head of the 4.9 channel as well? Thanks, Vu
Hello Vu Dinh, Could you please help check the cluster today if possible as the cluster will be pruned after 30 hours, thanks !!
Hi, I haven't had a chance to check out the cluster yet and it seems it is cleaned-up. Can you spin up another cluster and grant me full access to the cluster if possible? I did try to reproduce this in a cluster-bot cluster but unfortunately due to some inaccessible bundle images, it didn't work. Vu
I also tried it without cluster upgrading. Met this issue too. As follows: 1, Install OCP 4.9 and subscribe to the cluster-kube-descheduler-operator. [cloud-user@preserve-olm-env jian]$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.9.0-0.nightly-2021-09-15-125245 True False 36m Cluster version is 4.9.0-0.nightly-2021-09-15-125245 [cloud-user@preserve-olm-env jian]$ oc adm new-project openshift-kube-descheduler-operator Created project openshift-kube-descheduler-operator [cloud-user@preserve-olm-env jian]$ oc project openshift-kube-descheduler-operator Now using project "openshift-kube-descheduler-operator" on server "https://api.bandrade43220.qe.devcluster.openshift.com:6443". [cloud-user@preserve-olm-env jian]$ oc get sub NAME PACKAGE SOURCE CHANNEL cluster-kube-descheduler-operator cluster-kube-descheduler-operator redhat-operators 4.8 [cloud-user@preserve-olm-env jian]$ oc get ip NAME CSV APPROVAL APPROVED install-5v6lm clusterkubedescheduleroperator.4.8.0-202107291502 Automatic true [cloud-user@preserve-olm-env jian]$ oc get csv NAME DISPLAY VERSION REPLACES PHASE clusterkubedescheduleroperator.4.8.0-202107291502 Kube Descheduler Operator 4.8.0-202107291502 Succeeded elasticsearch-operator.5.2.1-7 OpenShift Elasticsearch Operator 5.2.1-7 Succeeded 2, Switch it to the "qe-app-registry" CatalogSource, 4.9 channel manually. But, it failed to upgrade. As follows: [cloud-user@preserve-olm-env jian]$ oc edit sub cluster-kube-descheduler-operator subscription.operators.coreos.com/cluster-kube-descheduler-operator edited [cloud-user@preserve-olm-env jian]$ oc get sub NAME PACKAGE SOURCE CHANNEL cluster-kube-descheduler-operator cluster-kube-descheduler-operator qe-app-registry 4.9 [cloud-user@preserve-olm-env jian]$ oc get ip NAME CSV APPROVAL APPROVED install-5v6lm clusterkubedescheduleroperator.4.8.0-202107291502 Automatic true [cloud-user@preserve-olm-env jian]$ oc get csv NAME DISPLAY VERSION REPLACES PHASE clusterkubedescheduleroperator.4.8.0-202107291502 Kube Descheduler Operator 4.8.0-202107291502 Succeeded elasticsearch-operator.5.2.1-7 OpenShift Elasticsearch Operator 5.2.1-7 Succeeded [cloud-user@preserve-olm-env jian]$ oc get sub cluster-kube-descheduler-operator -o yaml apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: creationTimestamp: "2021-09-16T05:55:33Z" generation: 2 labels: operators.coreos.com/cluster-kube-descheduler-operator.openshift-kube-descheduler-op: "" name: cluster-kube-descheduler-operator namespace: openshift-kube-descheduler-operator resourceVersion: "40723" uid: c99d3f96-13b2-4f65-91b8-d6a4db724a3e spec: channel: "4.9" installPlanApproval: Automatic name: cluster-kube-descheduler-operator source: qe-app-registry sourceNamespace: openshift-marketplace startingCSV: clusterkubedescheduleroperator.4.9.0-202109132154 [cloud-user@preserve-olm-env jian]$ grpcurl -plaintext -d '{"name":"cluster-kube-descheduler-operator"}' localhost:50051 api.Registry/GetPackage { "name": "cluster-kube-descheduler-operator", "channels": [ { "name": "4.6", "csvName": "clusterkubedescheduleroperator.4.6.0-202109091425" }, { "name": "4.7", "csvName": "clusterkubedescheduleroperator.4.7.0-202109092011" }, { "name": "4.8", "csvName": "clusterkubedescheduleroperator.4.8.0-202109100832" }, { "name": "4.9", "csvName": "clusterkubedescheduleroperator.4.9.0-202109132154" } ], "defaultChannelName": "4.9" } I have another try that subscribes this operator from the same CatalogSource: qe-app-registry 1, Subscribe it to the 4.8 channel. [cloud-user@preserve-olm-env jian]$ cat sub-descheduler.yaml apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: cluster-kube-descheduler-operator namespace: openshift-kube-descheduler-operator spec: channel: "4.8" installPlanApproval: Automatic name: cluster-kube-descheduler-operator source: qe-app-registry sourceNamespace: openshift-marketplace startingCSV: clusterkubedescheduleroperator.4.8.0-202109100832 [cloud-user@preserve-olm-env jian]$ oc get sub NAME PACKAGE SOURCE CHANNEL cluster-kube-descheduler-operator cluster-kube-descheduler-operator qe-app-registry 4.8 [cloud-user@preserve-olm-env jian]$ oc get ip NAME CSV APPROVAL APPROVED install-v7l98 clusterkubedescheduleroperator.4.8.0-202109100832 Automatic true [cloud-user@preserve-olm-env jian]$ oc get csv NAME DISPLAY VERSION REPLACES PHASE elasticsearch-operator.5.2.1-7 OpenShift Elasticsearch Operator 5.2.1-7 Succeeded [cloud-user@preserve-olm-env jian]$ oc get csv NAME DISPLAY VERSION REPLACES PHASE clusterkubedescheduleroperator.4.8.0-202109100832 Kube Descheduler Operator 4.8.0-202109100832 Succeeded elasticsearch-operator.5.2.1-7 OpenShift Elasticsearch Operator 5.2.1-7 Succeeded 2, Switch it to the 4.9 channle, no the CatalogSource changes. Still failed to upgrade to the 4.9 version, as follows, [cloud-user@preserve-olm-env jian]$ oc edit sub cluster-kube-descheduler-operator subscription.operators.coreos.com/cluster-kube-descheduler-operator edited [cloud-user@preserve-olm-env jian]$ [cloud-user@preserve-olm-env jian]$ oc get sub NAME PACKAGE SOURCE CHANNEL cluster-kube-descheduler-operator cluster-kube-descheduler-operator qe-app-registry 4.9 [cloud-user@preserve-olm-env jian]$ oc get ip NAME CSV APPROVAL APPROVED install-v7l98 clusterkubedescheduleroperator.4.8.0-202109100832 Automatic true [cloud-user@preserve-olm-env jian]$ oc get csv NAME DISPLAY VERSION REPLACES PHASE clusterkubedescheduleroperator.4.8.0-202109100832 Kube Descheduler Operator 4.8.0-202109100832 Succeeded elasticsearch-operator.5.2.1-7 OpenShift Elasticsearch Operator 5.2.1-7 Succeeded
I also tried it on OCP 4.8, still met this issue. [cloud-user@preserve-olm-env jian]$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.8.0-0.nightly-2021-09-15-234929 True False 11m Cluster version is 4.8.0-0.nightly-2021-09-15-234929 1, subscribe to cluster-kube-descheduler-operator. 4,8 version. [cloud-user@preserve-olm-env jian]$ cat sub-descheduler.yaml apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: cluster-kube-descheduler-operator namespace: openshift-kube-descheduler-operator spec: channel: "4.8" installPlanApproval: Automatic name: cluster-kube-descheduler-operator source: qe-app-registry sourceNamespace: openshift-marketplace startingCSV: clusterkubedescheduleroperator.4.8.0-202109100832 [cloud-user@preserve-olm-env jian]$ oc project openshift-kube-descheduler-operator Now using project "openshift-kube-descheduler-operator" on server "https://api.jiazha16.qe.devcluster.openshift.com:6443". [cloud-user@preserve-olm-env jian]$ oc get sub NAME PACKAGE SOURCE CHANNEL cluster-kube-descheduler-operator cluster-kube-descheduler-operator qe-app-registry 4.8 [cloud-user@preserve-olm-env jian]$ oc get ip NAME CSV APPROVAL APPROVED install-7dc4n clusterkubedescheduleroperator.4.8.0-202109100832 Automatic true [cloud-user@preserve-olm-env jian]$ oc get csv NAME DISPLAY VERSION REPLACES PHASE clusterkubedescheduleroperator.4.8.0-202109100832 Kube Descheduler Operator 4.8.0-202109100832 Succeeded elasticsearch-operator.5.1.2-9 OpenShift Elasticsearch Operator 5.1.2-9 Succeeded 2, switch it to the 4.9 channel. failed to upgrade it to 4.9 version. As follows, [cloud-user@preserve-olm-env jian]$ oc edit sub cluster-kube-descheduler-operator subscription.operators.coreos.com/cluster-kube-descheduler-operator edited [cloud-user@preserve-olm-env jian]$ [cloud-user@preserve-olm-env jian]$ oc get sub NAME PACKAGE SOURCE CHANNEL cluster-kube-descheduler-operator cluster-kube-descheduler-operator qe-app-registry 4.9 [cloud-user@preserve-olm-env jian]$ oc get ip NAME CSV APPROVAL APPROVED install-7dc4n clusterkubedescheduleroperator.4.8.0-202109100832 Automatic true [cloud-user@preserve-olm-env jian]$ oc get csv NAME DISPLAY VERSION REPLACES PHASE clusterkubedescheduleroperator.4.8.0-202109100832 Kube Descheduler Operator 4.8.0-202109100832 Succeeded elasticsearch-operator.5.1.2-9 OpenShift Elasticsearch Operator 5.1.2-9 Succeeded workaround: Delete the pods of the "qe-app-registry" CatalogSource. The new 4.9 InstallPlan generated automatically. [cloud-user@preserve-olm-env jian]$ oc get pods NAME READY STATUS RESTARTS AGE 2c50b45b77048cf2e36b46e51086180dc097ed6f2d3f19c77df3764edcg5b5g 0/1 Completed 0 9m58s 3b70efc64794590b7929b08f32b13e8105fe7aca05881162e9fb145c89n6pnt 0/1 Completed 0 23m 5b197a688ab46afbf86cfe7e90d5f1c3f3f2c42092f5c92d5caf990393gj5bt 0/1 Completed 0 23m certified-operators-8vnvh 1/1 Running 0 51m community-operators-vcg7x 1/1 Running 0 51m marketplace-operator-559b85dd99-4lxp2 1/1 Running 0 60m qe-app-registry-4vcpc 1/1 Running 0 23m redhat-marketplace-6hlbs 1/1 Running 0 51m redhat-operators-c84lj 1/1 Running 0 51m [cloud-user@preserve-olm-env jian]$ oc delete pods qe-app-registry-4vcpc pod "qe-app-registry-4vcpc" deleted [cloud-user@preserve-olm-env jian]$ oc get sub -n openshift-kube-descheduler-operator NAME PACKAGE SOURCE CHANNEL cluster-kube-descheduler-operator cluster-kube-descheduler-operator qe-app-registry 4.9 [cloud-user@preserve-olm-env jian]$ oc get ip -n openshift-kube-descheduler-operator NAME CSV APPROVAL APPROVED install-7dc4n clusterkubedescheduleroperator.4.8.0-202109100832 Automatic true install-bqt8p clusterkubedescheduleroperator.4.9.0-202109132154 Automatic true [cloud-user@preserve-olm-env jian]$ oc get csv -n openshift-kube-descheduler-operator NAME DISPLAY VERSION REPLACES PHASE clusterkubedescheduleroperator.4.8.0-202109100832 Kube Descheduler Operator 4.8.0-202109100832 Replacing clusterkubedescheduleroperator.4.9.0-202109132154 Kube Descheduler Operator 4.9.0-202109132154 clusterkubedescheduleroperator.4.8.0-202109100832 Pending Note: the pending reason is as follows, not related to this issue. Message: CSV version requirement not met: minKubeVersion (1.22.0) > server version (1.21.1+4ebdf8e) Name: clusterkubedescheduleroperator.4.9.0-202109132154 Status: PresentNotSatisfied Version: v1alpha1
Also met this issue when subscribing to the 4.6 version during the minor version upgrading(4.6.0-202108202025 upgrade to 4.6.0-202109091425). After recreate the qe-app-registry CatalogSource pod, it works as expected. [cloud-user@preserve-olm-env jian]$ oc get pods NAME READY STATUS RESTARTS AGE 2804f5e5a8f80dcfe6d49b373a63b141277f26dff508355a90bee94248fdhsv 0/1 Completed 0 41m certified-operators-58gsw 1/1 Running 0 50m community-operators-q5b5z 1/1 Running 0 45m marketplace-operator-97588d7f9-cgnlp 1/1 Running 0 43m qe-app-registry-9g4bn 1/1 Running 0 49m redhat-marketplace-cncn7 1/1 Running 0 47m redhat-operators-njdmd 1/1 Running 0 50m [cloud-user@preserve-olm-env jian]$ oc delete pods qe-app-registry-9g4bn [cloud-user@preserve-olm-env jian]$ oc get pods NAME READY STATUS RESTARTS AGE 2804f5e5a8f80dcfe6d49b373a63b141277f26dff508355a90bee94248fdhsv 0/1 Completed 0 42m certified-operators-58gsw 1/1 Running 0 50m community-operators-q5b5z 1/1 Running 0 45m marketplace-operator-97588d7f9-cgnlp 1/1 Running 0 44m qe-app-registry-5nq8j 0/1 ContainerCreating 0 5s redhat-marketplace-cncn7 1/1 Running 0 48m redhat-operators-njdmd 1/1 Running 0 50m [cloud-user@preserve-olm-env jian]$ oc get sub -n openshift-kube-descheduler-operator NAME PACKAGE SOURCE CHANNEL cluster-kube-descheduler-operator cluster-kube-descheduler-operator qe-app-registry 4.6 [cloud-user@preserve-olm-env jian]$ oc get ip -n openshift-kube-descheduler-operator NAME CSV APPROVAL APPROVED install-jqfvw clusterkubedescheduleroperator.4.6.0-202109091425 Automatic true install-p8bh7 clusterkubedescheduleroperator.4.6.0-202108202025 Automatic true [cloud-user@preserve-olm-env jian]$ oc get csv -n openshift-kube-descheduler-operator NAME DISPLAY VERSION REPLACES PHASE clusterkubedescheduleroperator.4.6.0-202109091425 Kube Descheduler Operator 4.6.0-202109091425 clusterkubedescheduleroperator.4.6.0-202108202025 Succeeded
Hey folks, I just want to give a quick update that this issue is indeed reproducible in the QE cluster. It doesn't seem to be a resolver problem but instead it is some sort of cache/sync problem as I did notice some sync error on subscription syncer. catalog-operator seems not to take any action on the subscription change. However, if you want for the next sync cycle (around 15 mins), the operator will be upgraded. I would say if you install a new operator in the same namespace, that will trigger the descheduler upgrade as well. I'm not sure what is the root cause yet as I'm still investigating. Please keep the cluster around. Thanks, Vu
Hi, I reduce the severity and priority to HIGH as the upgrade will eventually happen (after 15 minutes or not on my estimate). The delay is certainly a problem and we will address that but the delay shouldn't be a blocker. Vu
Hey folks, I have determined that the upgrade delay is due to a condition check on OLM that is using some obsolete APIs that are no longer needed since the introduction of the new resolver. I have opened a PR on upstream to remove this logic. That should allow the upgrade to happen quicker. I want to reinstate that this is an upgrade delay, not a block as the upgrade will eventually happen so this shouldn't be a blocker for the upcoming release. Thanks, Vu
Hello vu Dinh, Today i have tried upgrading descheduler from 4.8 to 4.9 and i see that upgrade takes place but it does happen after sometime, do we need to mark this as known issue so that users or customers would not wonder on why the upgrade has still not happened yet ? WDYT ? Thanks kasturi
Hey, Yes, it is a known issue until we merge the fix and backport it back to prior versions. In the meantime, let people know that the upgrade will eventually happen but it takes a bit of time around 10 to 15 minutes. After the fix, it should much quicker. Vu
okay, thank you, one more last question... so do you add requires doc text flag and provide doc text so that documentation team can add it to the release notes or how does it work ?
Hi, There is no need for documentation on this. This is a known issue that will be addressed soon after the fix is merged and it will be backported. I said earlier that for the time being, if issue arises, simply being aware that this issue exists. Vu
(In reply to Vu Dinh from comment #22) > Hi, > > There is no need for documentation on this. This is a known issue that will > be addressed soon after the fix is merged and it will be backported. I said > earlier that for the time being, if issue arises, simply being aware that > this issue exists. > > Vu okay, thank you !!
*** Bug 2010602 has been marked as a duplicate of this bug. ***
*** Bug 2007778 has been marked as a duplicate of this bug. ***
*** Bug 2013811 has been marked as a duplicate of this bug. ***
verify: zhaoxia@xzha-mac bug-2002276 % oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.10.0-0.nightly-2021-11-02-121455 True False 64m Cluster version is 4.10.0-0.nightly-2021-11-02-121455 zhaoxia@xzha-mac bug-2002276 % oc adm release info registry.ci.openshift.org/ocp/release:4.10.0-0.nightly-2021-11-02-121455 --commits|grep operator-lifecycle-manager operator-lifecycle-manager https://github.com/openshift/operator-framework-olm 8970fc081f9c411112446d29ebaeb82d00ba7331 [root@preserve-olm-agent-test ~]# opm alpha list channels quay.io/openshift-qe-optional-operators/ocp4-index:latest cluster-kube-descheduler-operator PACKAGE CHANNEL HEAD cluster-kube-descheduler-operator 4.6 clusterkubedescheduleroperator.4.6.0-202110262229 cluster-kube-descheduler-operator 4.7 clusterkubedescheduleroperator.4.7.0-202111020909 cluster-kube-descheduler-operator 4.8 clusterkubedescheduleroperator.4.8.0-202111021756 cluster-kube-descheduler-operator 4.9 clusterkubedescheduleroperator.4.9.0-202111020858 cluster-kube-descheduler-operator stable clusterkubedescheduleroperator.4.9.0-202111020858 1, create project zhaoxia@xzha-mac bug-2002276 % oc adm new-project openshift-kube-descheduler-operator Created project openshift-kube-descheduler-operator zhaoxia@xzha-mac bug-2002276 % oc project openshift-kube-descheduler-operator Now using project "openshift-kube-descheduler-operator" on server "https://api.xzha-1103.qe.devcluster.openshift.com:6443". 2, create sub zhaoxia@xzha-mac bug-2002276 % cat sub.yaml apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: cluster-kube-descheduler-operator namespace: openshift-kube-descheduler-operator spec: channel: "4.8" installPlanApproval: Automatic name: cluster-kube-descheduler-operator source: qe-app-registry sourceNamespace: openshift-marketplace startingCSV: clusterkubedescheduleroperator.4.8.0-202111021756 zhaoxia@xzha-mac bug-2002276 % cat og.yaml kind: OperatorGroup apiVersion: operators.coreos.com/v1 metadata: name: og-single namespace: openshift-kube-descheduler-operator spec: targetNamespaces: - openshift-kube-descheduler-operator zhaoxia@xzha-mac bug-2002276 % oc apply -f og.yaml operatorgroup.operators.coreos.com/og-single created zhaoxia@xzha-mac bug-2002276 % oc apply -f sub.yaml subscription.operators.coreos.com/cluster-kube-descheduler-operator created 3, check ip/csv zhaoxia@xzha-mac bug-2002276 % oc get ip NAME CSV APPROVAL APPROVED install-kpn95 clusterkubedescheduleroperator.4.8.0-202111021756 Automatic true zhaoxia@xzha-mac bug-2002276 % oc get csv NAME DISPLAY VERSION REPLACES PHASE clusterkubedescheduleroperator.4.8.0-202111021756 Kube Descheduler Operator 4.8.0-202111021756 Succeeded elasticsearch-operator.5.2.3-14 OpenShift Elasticsearch Operator 5.2.3-14 Succeeded 4, edit sub to channel "4.9" zhaoxia@xzha-mac bug-2002276 % oc edit sub cluster-kube-descheduler-operator subscription.operators.coreos.com/cluster-kube-descheduler-operator edited 5, check ip/csv zhaoxia@xzha-mac bug-2002276 % oc get ip NAME CSV APPROVAL APPROVED install-kpn95 clusterkubedescheduleroperator.4.8.0-202111021756 Automatic true install-pf2q8 clusterkubedescheduleroperator.4.9.0-202111020858 Automatic true zhaoxia@xzha-mac bug-2002276 % oc get csv NAME DISPLAY VERSION REPLACES PHASE clusterkubedescheduleroperator.4.9.0-202111020858 Kube Descheduler Operator 4.9.0-202111020858 clusterkubedescheduleroperator.4.8.0-202111021756 Succeeded elasticsearch-operator.5.2.3-14 OpenShift Elasticsearch Operator 5.2.3-14 Succeeded LGTM. verified.
*** Bug 2026003 has been marked as a duplicate of this bug. ***
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0056