Bug 2002276 - OLM fails to upgrade operators immediately
Summary: OLM fails to upgrade operators immediately
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OLM
Version: 4.9
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.10.0
Assignee: Vu Dinh
QA Contact: xzha
URL:
Whiteboard:
: 2007778 2010602 2013811 2026003 (view as bug list)
Depends On:
Blocks: 2024048
TreeView+ depends on / blocked
 
Reported: 2021-09-08 12:05 UTC by RamaKasturi
Modified: 2022-10-11 07:12 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: Known Issue
Doc Text:
Cause: OLM uses a combination of timestamp check and obsolete API call (which doesn't work for skipRange upgrade) to determine if it is necessary to perform an upgrade a particular subscription. Consequence: For operators that use skipRange upgrade, there is a delay in upgrade process that may take up 15 minutes to resolve and potentially being blocked for longer time. Workaround (if any): The current workaround is to delete the catalog-operator pod in openshift-operator-lifecycle-manager namespace (the pod will be automatically recreated). Result: Recycling the catalog-operator pod will trigger the upgrade to happen.
Clone Of:
Environment:
Last Closed: 2022-03-10 16:08:32 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift operator-framework-olm pull 211 0 None open Bug 2002276: Remove oudated subscription update logic to improve resolution delay 2021-10-27 13:43:49 UTC
Red Hat Product Errata RHSA-2022:0056 0 None None None 2022-03-10 16:08:52 UTC

Description RamaKasturi 2021-09-08 12:05:24 UTC
Description of problem:
Upgrading descheduler from 4.8 to 4.9 fails, i see that when channel & starting CSV is set i do not see any upgrade starting, jian zhang looked further and below is what he found.

1, Only remove the sub, not the csv, resubscribe the 4.9 one, get the below errors:
  - message: 'constraints not satisfiable: @existing/openshift-kube-descheduler-operator//clusterkubedescheduleroperator.4.8.0-202108312109
      and qe-app-registry/openshift-marketplace/4.9/clusterkubedescheduleroperator.4.9.0-202109071344
      originate from package cluster-kube-descheduler-operator, subscription cluster-kube-descheduler-operator
      requires qe-app-registry/openshift-marketplace/4.9/clusterkubedescheduleroperator.4.9.0-202109071344,
      subscription cluster-kube-descheduler-operator exists, clusterserviceversion
      clusterkubedescheduleroperator.4.8.0-202108312109 exists and is not referenced
      by a subscription'
    reason: ConstraintsNotSatisfiable
    status: "True"
    type: ResolutionFailed



2,, Remove the sub and csv, and then recreate it, it works well.
[cloud-user@preserve-olm-env jian]$ cat sub-descheduler.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: cluster-kube-descheduler-operator
  namespace: openshift-kube-descheduler-operator
spec:
  channel: "4.9"
  installPlanApproval: Automatic
  name: cluster-kube-descheduler-operator
  source: qe-app-registry
  sourceNamespace: openshift-marketplace
  startingCSV: clusterkubedescheduleroperator.4.9.0-202109071344

[cloud-user@preserve-olm-env jian]$ oc get sub
NAME                                PACKAGE                             SOURCE            CHANNEL
cluster-kube-descheduler-operator   cluster-kube-descheduler-operator   qe-app-registry   4.9

[cloud-user@preserve-olm-env jian]$ oc get ip
NAME            CSV                                                 APPROVAL    APPROVED
install-4w9lm   clusterkubedescheduleroperator.4.9.0-202109071344   Automatic   true

[cloud-user@preserve-olm-env jian]$ oc get csv
NAME                                                DISPLAY                            VERSION              REPLACES                          PHASE
clusterkubedescheduleroperator.4.9.0-202109071344   Kube Descheduler Operator          4.9.0-202109071344                                     Succeeded
elasticsearch-operator.5.2.0-60                     OpenShift Elasticsearch Operator   5.2.0-60             elasticsearch-operator.5.1.1-56   Succeeded

3, check the bundle content:
clusterkubedescheduleroperator.4.8.0-202108312109   {"apiVersion":"opera  {"apiVersion":"operators.coreos.com/v1alpha1","kin  registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-  >=4.6.0 <4.8.0-202108312109     4.8.0-202108312109

clusterkubedescheduleroperator.4.9.0-202109071344   {"apiVersion":"opera  {"apiVersion":"operators.coreos.com/v1alpha1","kin  registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-  >=4.6.0 <4.9.0-202109071344     4.9.0-202109071344
Version-Release number of selected component (if applicable):

[knarra@knarra ~]$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.9.0-0.nightly-2021-09-06-004132   True        False         23h     Cluster version is 4.9.0-0.nightly-2021-09-06-004132

 Clusterkubedescheduleroperator.4.8.0-202108312109

How reproducible:
Install 4.8 cluster
Install 4.8 descheduler operator
Upgrade system to 4.9
Now edit descheduler sub & set channel to 4.9, starting csv to clusterkubedescheduleroperator.4.9.0-202109071344
spec:
  channel: "4.9"
  installPlanApproval: Automatic
  name: cluster-kube-descheduler-operator
  source: qe-app-registry
  sourceNamespace: openshift-marketplace
  startingCSV: clusterkubedescheduleroperator.4.9.0-202109071344


Actual Results:
 Upgrade does not start at all

Expected Results:
 Upgrade should work fine.

Comment 1 Mike Dame 2021-09-08 15:07:31 UTC
From the Descheduler side, we haven't seen an error like this before so we are not sure what the cause is. Hopefully someone from OLM can lend some insight, and if there are any changes needed from us please let us know

Comment 2 Vu Dinh 2021-09-08 18:57:37 UTC
Hi,

The issue here is the CSV for 4.8 is still present while you were trying to install another version of the same operator. If you uninstall an operator like you did in step 1, then you need to delete both Subscription as CSV. If you just delete the Subscription, the CSV is still there meaning the operator for 4.8 is still present.

When you do a cross channel upgrade, then you simply need to update the channel info. That should trigger the upgrade.

The upgrade process that you should do is:

1. Create a sub for 4.8 with channel and startingCSV information that you desire.
2. Ensure the operator is installed properly.
3. Upgrade the cluster to 4.9.
4. Update the Subscription channel info to the 4.9 channel that you desire.
5. Check for the upgrade to happen.

Note: The head of channel on the new 4.9 channel must have a compatible upgrade path from the previous version in 4.8 (such as skipRange).

Vu

Comment 3 Mike Dame 2021-09-08 19:52:59 UTC
Thanks Vu!

Comment 4 Jian Zhang 2021-09-09 06:38:40 UTC
Hi Vu,

> The issue here is the CSV for 4.8 is still present while you were trying to install another version of the same operator. If you uninstall an operator like you did in step 1, then you need to delete both Subscription as CSV. If you just delete the Subscription, the CSV is still there meaning the operator for 4.8 is still present.

> 1, Only remove the sub, not the csv, resubscribe the 4.9 one, get the below errors:
  - message: 'constraints not satisfiable: @existing/openshift-kube-descheduler-operator//clusterkubedescheduleroperator.4.8.0-202108312109
      and qe-app-registry/openshift-marketplace/4.9/clusterkubedescheduleroperator.4.9.0-202109071344
      originate from package cluster-kube-descheduler-operator, subscription cluster-kube-descheduler-operator
      requires qe-app-registry/openshift-marketplace/4.9/clusterkubedescheduleroperator.4.9.0-202109071344,
      subscription cluster-kube-descheduler-operator exists, clusterserviceversion
      clusterkubedescheduleroperator.4.8.0-202108312109 exists and is not referenced
      by a subscription'
    reason: ConstraintsNotSatisfiable
    status: "True"
    type: ResolutionFailed

For this error, that's my debug trying. There is no this error when the deschduler operator upgrading.


> The upgrade process that you should do is:

1. Create a sub for 4.8 with channel and startingCSV information that you desire.
2. Ensure the operator is installed properly.
3. Upgrade the cluster to 4.9.
4. Update the Subscription channel info to the 4.9 channel that you desire.
5. Check for the upgrade to happen.


I think the only difference is that they also update the `startingCSV` field. Does it master?
The origin scenario as follows, as you can see, there is no InstallPlan generated for the 4.9 channel.
And, I checked the CatalogSource, the 4.9 did there.

[cloud-user@preserve-olm-env jian]$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.9.0-0.nightly-2021-09-06-004132   True        False         22h     Cluster version is 4.9.0-0.nightly-2021-09-06-004132

[cloud-user@preserve-olm-env jian]$ oc project openshift-kube-descheduler-operator
Now using project "openshift-kube-descheduler-operator" on server "https://api.yangyang-share.qe.devcluster.openshift.com:6443".

[cloud-user@preserve-olm-env jian]$ oc get sub
NAME                                PACKAGE                             SOURCE            CHANNEL
cluster-kube-descheduler-operator   cluster-kube-descheduler-operator   qe-app-registry   4.9

[cloud-user@preserve-olm-env jian]$ oc get ip
NAME            CSV                                                 APPROVAL    APPROVED
install-78wtg   clusterkubedescheduleroperator.4.8.0-202108312109   Automatic   true

[cloud-user@preserve-olm-env jian]$ oc get csv
NAME                                                DISPLAY                            VERSION              REPLACES                          PHASE
clusterkubedescheduleroperator.4.8.0-202108312109   Kube Descheduler Operator          4.8.0-202108312109                                     Succeeded

[cloud-user@preserve-olm-env jian]$ oc get sub cluster-kube-descheduler-operator -o yaml
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  creationTimestamp: "2021-09-07T07:59:58Z"
  generation: 3
  labels:
    operators.coreos.com/cluster-kube-descheduler-operator.openshift-kube-descheduler-op: ""
  name: cluster-kube-descheduler-operator
  namespace: openshift-kube-descheduler-operator
  resourceVersion: "1098745"
  uid: b2dbaea7-3945-4b8a-b7ba-df184132a01f
spec:
  channel: "4.9"
  installPlanApproval: Automatic
  name: cluster-kube-descheduler-operator
  source: qe-app-registry
  sourceNamespace: openshift-marketplace
  startingCSV: clusterkubedescheduleroperator.4.9.0-202109071344
status:
  catalogHealth:
  - catalogSourceRef:
      apiVersion: operators.coreos.com/v1alpha1
      kind: CatalogSource
      name: certified-operators
      namespace: openshift-marketplace
      resourceVersion: "1027588"
      uid: 419b602f-6f8a-4a79-b056-c095c655609b
    healthy: true
    lastUpdated: "2021-09-08T07:49:33Z"
  - catalogSourceRef:
      apiVersion: operators.coreos.com/v1alpha1
      kind: CatalogSource
      name: community-operators
      namespace: openshift-marketplace
      resourceVersion: "1024335"
      uid: 28414a7f-ee4a-4f08-bc17-e0a49f87cba0
    healthy: true
    lastUpdated: "2021-09-08T07:49:33Z"
  - catalogSourceRef:
      apiVersion: operators.coreos.com/v1alpha1
      kind: CatalogSource
      name: file-integrity-operator
      namespace: openshift-marketplace
      resourceVersion: "1021345"
      uid: 3aa1359e-d848-475b-b610-f934c5a18636
    healthy: true
    lastUpdated: "2021-09-08T07:49:33Z"
  - catalogSourceRef:
      apiVersion: operators.coreos.com/v1alpha1
      kind: CatalogSource
      name: qe-app-registry
      namespace: openshift-marketplace
      resourceVersion: "1027283"
      uid: e981e4b1-80ff-4279-9693-72549926e693
    healthy: true
    lastUpdated: "2021-09-08T07:49:33Z"
  - catalogSourceRef:
      apiVersion: operators.coreos.com/v1alpha1
      kind: CatalogSource
      name: redhat-marketplace
      namespace: openshift-marketplace
      resourceVersion: "1023548"
      uid: f5e16d4e-54f8-452d-ad46-235c398288f2
    healthy: true
    lastUpdated: "2021-09-08T07:49:33Z"
  - catalogSourceRef:
      apiVersion: operators.coreos.com/v1alpha1
      kind: CatalogSource
      name: redhat-operators
      namespace: openshift-marketplace
      resourceVersion: "1022918"
      uid: 5af279d5-9cc0-47c4-83d6-62f0815fba38
    healthy: true
    lastUpdated: "2021-09-08T07:49:33Z"
  conditions:
  - lastTransitionTime: "2021-09-08T07:49:33Z"
    message: all available catalogsources are healthy
    reason: AllCatalogSourcesHealthy
    status: "False"
    type: CatalogSourcesUnhealthy
  - status: "False"
    type: ResolutionFailed
  currentCSV: clusterkubedescheduleroperator.4.8.0-202108312109
  installPlanGeneration: 1
  installPlanRef:
    apiVersion: operators.coreos.com/v1alpha1
    kind: InstallPlan
    name: install-78wtg
    namespace: openshift-kube-descheduler-operator
    resourceVersion: "79121"
    uid: 6e7041e0-b3b2-490c-81fe-774ac92aaba8
  installedCSV: clusterkubedescheduleroperator.4.8.0-202108312109
  installplan:
    apiVersion: operators.coreos.com/v1alpha1
    kind: InstallPlan
    name: install-78wtg
    uuid: 6e7041e0-b3b2-490c-81fe-774ac92aaba8
  lastUpdated: "2021-09-08T08:40:43Z"
  state: AtLatestKnown

Comment 5 Kevin Rizza 2021-09-09 13:06:38 UTC
Can we confirm that there is actually an upgrade edge that OLM is ignoring in this situation? How is the 4.9 catalog being built? As far as I'm aware, the 4.9 index is an older copy of the 4.8 index while the pipelines are still in the process of bootstrapping the 4.9 indexes.

Comment 6 Vu Dinh 2021-09-09 14:26:24 UTC
Hi Jian,

Can you do a `oc get event` to see what is the ResolutionFailed event is? Plus, can we have the yaml of the CSV which is the head of the 4.9 channel as well?

Thanks,
Vu

Comment 8 RamaKasturi 2021-09-14 18:30:20 UTC
Hello Vu Dinh,

   Could you please help check the cluster today if possible as the cluster will be pruned after 30 hours, thanks !!

Comment 9 Vu Dinh 2021-09-15 16:24:16 UTC
Hi,

I haven't had a chance to check out the cluster yet and it seems it is cleaned-up. Can you spin up another cluster and grant me full access to the cluster if possible?

I did try to reproduce this in a cluster-bot cluster but unfortunately due to some inaccessible bundle images, it didn't work.

Vu

Comment 11 Jian Zhang 2021-09-16 06:12:32 UTC
I also tried it without cluster upgrading. Met this issue too. As follows:

1, Install OCP 4.9 and subscribe to the cluster-kube-descheduler-operator.
[cloud-user@preserve-olm-env jian]$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.9.0-0.nightly-2021-09-15-125245   True        False         36m     Cluster version is 4.9.0-0.nightly-2021-09-15-125245

[cloud-user@preserve-olm-env jian]$ oc adm new-project openshift-kube-descheduler-operator
Created project openshift-kube-descheduler-operator

[cloud-user@preserve-olm-env jian]$ oc project  openshift-kube-descheduler-operator
Now using project "openshift-kube-descheduler-operator" on server "https://api.bandrade43220.qe.devcluster.openshift.com:6443".

[cloud-user@preserve-olm-env jian]$ oc get sub 
NAME                                PACKAGE                             SOURCE             CHANNEL
cluster-kube-descheduler-operator   cluster-kube-descheduler-operator   redhat-operators   4.8
[cloud-user@preserve-olm-env jian]$ oc get ip 
NAME            CSV                                                 APPROVAL    APPROVED
install-5v6lm   clusterkubedescheduleroperator.4.8.0-202107291502   Automatic   true
[cloud-user@preserve-olm-env jian]$ oc get csv
NAME                                                DISPLAY                            VERSION              REPLACES   PHASE
clusterkubedescheduleroperator.4.8.0-202107291502   Kube Descheduler Operator          4.8.0-202107291502              Succeeded
elasticsearch-operator.5.2.1-7                      OpenShift Elasticsearch Operator   5.2.1-7                         Succeeded

2, Switch it to the "qe-app-registry" CatalogSource, 4.9 channel manually. But, it failed to upgrade. As follows:

[cloud-user@preserve-olm-env jian]$ oc edit sub cluster-kube-descheduler-operator
subscription.operators.coreos.com/cluster-kube-descheduler-operator edited 
[cloud-user@preserve-olm-env jian]$ oc get sub
NAME                                PACKAGE                             SOURCE            CHANNEL
cluster-kube-descheduler-operator   cluster-kube-descheduler-operator   qe-app-registry   4.9
[cloud-user@preserve-olm-env jian]$ oc get ip
NAME            CSV                                                 APPROVAL    APPROVED
install-5v6lm   clusterkubedescheduleroperator.4.8.0-202107291502   Automatic   true
[cloud-user@preserve-olm-env jian]$ oc get csv
NAME                                                DISPLAY                            VERSION              REPLACES   PHASE
clusterkubedescheduleroperator.4.8.0-202107291502   Kube Descheduler Operator          4.8.0-202107291502              Succeeded
elasticsearch-operator.5.2.1-7                      OpenShift Elasticsearch Operator   5.2.1-7                         Succeeded

[cloud-user@preserve-olm-env jian]$ oc get sub cluster-kube-descheduler-operator -o yaml
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  creationTimestamp: "2021-09-16T05:55:33Z"
  generation: 2
  labels:
    operators.coreos.com/cluster-kube-descheduler-operator.openshift-kube-descheduler-op: ""
  name: cluster-kube-descheduler-operator
  namespace: openshift-kube-descheduler-operator
  resourceVersion: "40723"
  uid: c99d3f96-13b2-4f65-91b8-d6a4db724a3e
spec:
  channel: "4.9"
  installPlanApproval: Automatic
  name: cluster-kube-descheduler-operator
  source: qe-app-registry
  sourceNamespace: openshift-marketplace
  startingCSV: clusterkubedescheduleroperator.4.9.0-202109132154


[cloud-user@preserve-olm-env jian]$ grpcurl -plaintext -d '{"name":"cluster-kube-descheduler-operator"}' localhost:50051 api.Registry/GetPackage
{
  "name": "cluster-kube-descheduler-operator",
  "channels": [
    {
      "name": "4.6",
      "csvName": "clusterkubedescheduleroperator.4.6.0-202109091425"
    },
    {
      "name": "4.7",
      "csvName": "clusterkubedescheduleroperator.4.7.0-202109092011"
    },
    {
      "name": "4.8",
      "csvName": "clusterkubedescheduleroperator.4.8.0-202109100832"
    },
    {
      "name": "4.9",
      "csvName": "clusterkubedescheduleroperator.4.9.0-202109132154"
    }
  ],
  "defaultChannelName": "4.9"
}


I have another try that subscribes this operator from the same CatalogSource: qe-app-registry

1, Subscribe it to the 4.8 channel.
[cloud-user@preserve-olm-env jian]$ cat sub-descheduler.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: cluster-kube-descheduler-operator
  namespace: openshift-kube-descheduler-operator
spec:
  channel: "4.8"
  installPlanApproval: Automatic
  name: cluster-kube-descheduler-operator
  source: qe-app-registry
  sourceNamespace: openshift-marketplace
  startingCSV: clusterkubedescheduleroperator.4.8.0-202109100832

[cloud-user@preserve-olm-env jian]$ oc get sub
NAME                                PACKAGE                             SOURCE            CHANNEL
cluster-kube-descheduler-operator   cluster-kube-descheduler-operator   qe-app-registry   4.8
[cloud-user@preserve-olm-env jian]$ oc get ip
NAME            CSV                                                 APPROVAL    APPROVED
install-v7l98   clusterkubedescheduleroperator.4.8.0-202109100832   Automatic   true
[cloud-user@preserve-olm-env jian]$ oc get csv
NAME                             DISPLAY                            VERSION   REPLACES   PHASE
elasticsearch-operator.5.2.1-7   OpenShift Elasticsearch Operator   5.2.1-7              Succeeded
[cloud-user@preserve-olm-env jian]$ oc get csv
NAME                                                DISPLAY                            VERSION              REPLACES   PHASE
clusterkubedescheduleroperator.4.8.0-202109100832   Kube Descheduler Operator          4.8.0-202109100832              Succeeded
elasticsearch-operator.5.2.1-7                      OpenShift Elasticsearch Operator   5.2.1-7                         Succeeded

2, Switch it to the 4.9 channle, no the CatalogSource changes. Still failed to upgrade to the 4.9 version, as follows,
[cloud-user@preserve-olm-env jian]$ oc edit sub cluster-kube-descheduler-operator 
subscription.operators.coreos.com/cluster-kube-descheduler-operator edited
[cloud-user@preserve-olm-env jian]$ 
[cloud-user@preserve-olm-env jian]$ oc get sub 
NAME                                PACKAGE                             SOURCE            CHANNEL
cluster-kube-descheduler-operator   cluster-kube-descheduler-operator   qe-app-registry   4.9

[cloud-user@preserve-olm-env jian]$ oc get ip
NAME            CSV                                                 APPROVAL    APPROVED
install-v7l98   clusterkubedescheduleroperator.4.8.0-202109100832   Automatic   true
[cloud-user@preserve-olm-env jian]$ oc get csv
NAME                                                DISPLAY                            VERSION              REPLACES   PHASE
clusterkubedescheduleroperator.4.8.0-202109100832   Kube Descheduler Operator          4.8.0-202109100832              Succeeded
elasticsearch-operator.5.2.1-7                      OpenShift Elasticsearch Operator   5.2.1-7                         Succeeded

Comment 12 Jian Zhang 2021-09-16 07:29:21 UTC
I also tried it on OCP 4.8, still met this issue.
[cloud-user@preserve-olm-env jian]$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.nightly-2021-09-15-234929   True        False         11m     Cluster version is 4.8.0-0.nightly-2021-09-15-234929

1, subscribe to cluster-kube-descheduler-operator. 4,8 version.

[cloud-user@preserve-olm-env jian]$ cat sub-descheduler.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: cluster-kube-descheduler-operator
  namespace: openshift-kube-descheduler-operator
spec:
  channel: "4.8"
  installPlanApproval: Automatic
  name: cluster-kube-descheduler-operator
  source: qe-app-registry
  sourceNamespace: openshift-marketplace
  startingCSV: clusterkubedescheduleroperator.4.8.0-202109100832

[cloud-user@preserve-olm-env jian]$ oc project openshift-kube-descheduler-operator
Now using project "openshift-kube-descheduler-operator" on server "https://api.jiazha16.qe.devcluster.openshift.com:6443".
[cloud-user@preserve-olm-env jian]$ oc get sub
NAME                                PACKAGE                             SOURCE            CHANNEL
cluster-kube-descheduler-operator   cluster-kube-descheduler-operator   qe-app-registry   4.8
[cloud-user@preserve-olm-env jian]$ oc get ip
NAME            CSV                                                 APPROVAL    APPROVED
install-7dc4n   clusterkubedescheduleroperator.4.8.0-202109100832   Automatic   true
[cloud-user@preserve-olm-env jian]$ oc get csv
NAME                                                DISPLAY                            VERSION              REPLACES   PHASE
clusterkubedescheduleroperator.4.8.0-202109100832   Kube Descheduler Operator          4.8.0-202109100832              Succeeded
elasticsearch-operator.5.1.2-9                      OpenShift Elasticsearch Operator   5.1.2-9                         Succeeded


2, switch it to the 4.9 channel. failed to upgrade it to 4.9 version. As follows,
[cloud-user@preserve-olm-env jian]$ oc edit sub cluster-kube-descheduler-operator
subscription.operators.coreos.com/cluster-kube-descheduler-operator edited
[cloud-user@preserve-olm-env jian]$ 
[cloud-user@preserve-olm-env jian]$ oc get sub
NAME                                PACKAGE                             SOURCE            CHANNEL
cluster-kube-descheduler-operator   cluster-kube-descheduler-operator   qe-app-registry   4.9
[cloud-user@preserve-olm-env jian]$ oc get ip
NAME            CSV                                                 APPROVAL    APPROVED
install-7dc4n   clusterkubedescheduleroperator.4.8.0-202109100832   Automatic   true
[cloud-user@preserve-olm-env jian]$ oc get csv
NAME                                                DISPLAY                            VERSION              REPLACES   PHASE
clusterkubedescheduleroperator.4.8.0-202109100832   Kube Descheduler Operator          4.8.0-202109100832              Succeeded
elasticsearch-operator.5.1.2-9                      OpenShift Elasticsearch Operator   5.1.2-9                         Succeeded


workaround:

Delete the pods of the "qe-app-registry" CatalogSource. The new 4.9 InstallPlan generated automatically.

[cloud-user@preserve-olm-env jian]$ oc get pods
NAME                                                              READY   STATUS      RESTARTS   AGE
2c50b45b77048cf2e36b46e51086180dc097ed6f2d3f19c77df3764edcg5b5g   0/1     Completed   0          9m58s
3b70efc64794590b7929b08f32b13e8105fe7aca05881162e9fb145c89n6pnt   0/1     Completed   0          23m
5b197a688ab46afbf86cfe7e90d5f1c3f3f2c42092f5c92d5caf990393gj5bt   0/1     Completed   0          23m
certified-operators-8vnvh                                         1/1     Running     0          51m
community-operators-vcg7x                                         1/1     Running     0          51m
marketplace-operator-559b85dd99-4lxp2                             1/1     Running     0          60m
qe-app-registry-4vcpc                                             1/1     Running     0          23m
redhat-marketplace-6hlbs                                          1/1     Running     0          51m
redhat-operators-c84lj                                            1/1     Running     0          51m
[cloud-user@preserve-olm-env jian]$ oc delete pods qe-app-registry-4vcpc
pod "qe-app-registry-4vcpc" deleted

[cloud-user@preserve-olm-env jian]$ oc get sub -n openshift-kube-descheduler-operator
NAME                                PACKAGE                             SOURCE            CHANNEL
cluster-kube-descheduler-operator   cluster-kube-descheduler-operator   qe-app-registry   4.9
[cloud-user@preserve-olm-env jian]$ oc get ip -n openshift-kube-descheduler-operator
NAME            CSV                                                 APPROVAL    APPROVED
install-7dc4n   clusterkubedescheduleroperator.4.8.0-202109100832   Automatic   true
install-bqt8p   clusterkubedescheduleroperator.4.9.0-202109132154   Automatic   true

[cloud-user@preserve-olm-env jian]$ oc get csv -n openshift-kube-descheduler-operator
NAME                                                DISPLAY                            VERSION              REPLACES                                            PHASE
clusterkubedescheduleroperator.4.8.0-202109100832   Kube Descheduler Operator          4.8.0-202109100832                                                       Replacing
clusterkubedescheduleroperator.4.9.0-202109132154   Kube Descheduler Operator          4.9.0-202109132154   clusterkubedescheduleroperator.4.8.0-202109100832   Pending

Note: the pending reason is as follows, not related to this issue.
    Message:  CSV version requirement not met: minKubeVersion (1.22.0) > server version (1.21.1+4ebdf8e)
    Name:     clusterkubedescheduleroperator.4.9.0-202109132154
    Status:   PresentNotSatisfied
    Version:  v1alpha1

Comment 13 Jian Zhang 2021-09-16 10:31:43 UTC
Also met this issue when subscribing to the 4.6 version during the minor version upgrading(4.6.0-202108202025 upgrade to 4.6.0-202109091425).

After recreate the qe-app-registry CatalogSource pod, it works as expected.

[cloud-user@preserve-olm-env jian]$ oc get pods
NAME                                                              READY   STATUS      RESTARTS   AGE
2804f5e5a8f80dcfe6d49b373a63b141277f26dff508355a90bee94248fdhsv   0/1     Completed   0          41m
certified-operators-58gsw                                         1/1     Running     0          50m
community-operators-q5b5z                                         1/1     Running     0          45m
marketplace-operator-97588d7f9-cgnlp                              1/1     Running     0          43m
qe-app-registry-9g4bn                                             1/1     Running     0          49m
redhat-marketplace-cncn7                                          1/1     Running     0          47m
redhat-operators-njdmd                                            1/1     Running     0          50m
[cloud-user@preserve-olm-env jian]$ oc delete pods qe-app-registry-9g4bn 


[cloud-user@preserve-olm-env jian]$ oc get pods
NAME                                                              READY   STATUS              RESTARTS   AGE
2804f5e5a8f80dcfe6d49b373a63b141277f26dff508355a90bee94248fdhsv   0/1     Completed           0          42m
certified-operators-58gsw                                         1/1     Running             0          50m
community-operators-q5b5z                                         1/1     Running             0          45m
marketplace-operator-97588d7f9-cgnlp                              1/1     Running             0          44m
qe-app-registry-5nq8j                                             0/1     ContainerCreating   0          5s
redhat-marketplace-cncn7                                          1/1     Running             0          48m
redhat-operators-njdmd                                            1/1     Running             0          50m

[cloud-user@preserve-olm-env jian]$ oc get sub -n openshift-kube-descheduler-operator
NAME                                PACKAGE                             SOURCE            CHANNEL
cluster-kube-descheduler-operator   cluster-kube-descheduler-operator   qe-app-registry   4.6
[cloud-user@preserve-olm-env jian]$ oc get ip -n openshift-kube-descheduler-operator
NAME            CSV                                                 APPROVAL    APPROVED
install-jqfvw   clusterkubedescheduleroperator.4.6.0-202109091425   Automatic   true
install-p8bh7   clusterkubedescheduleroperator.4.6.0-202108202025   Automatic   true
[cloud-user@preserve-olm-env jian]$ oc get csv -n openshift-kube-descheduler-operator
NAME                                                DISPLAY                     VERSION              REPLACES                                            PHASE
clusterkubedescheduleroperator.4.6.0-202109091425   Kube Descheduler Operator   4.6.0-202109091425   clusterkubedescheduleroperator.4.6.0-202108202025   Succeeded

Comment 14 Vu Dinh 2021-09-16 20:48:50 UTC
Hey folks,

I just want to give a quick update that this issue is indeed reproducible in the QE cluster. It doesn't seem to be a resolver problem but instead it is some sort of cache/sync problem as I did notice some sync error on subscription syncer. catalog-operator seems not to take any action on the subscription change. However, if you want for the next sync cycle (around 15 mins), the operator will be upgraded. I would say if you install a new operator in the same namespace, that will trigger the descheduler upgrade as well. I'm not sure what is the root cause yet as I'm still investigating. Please keep the cluster around.

Thanks,
Vu

Comment 16 Vu Dinh 2021-09-17 15:58:52 UTC
Hi,

I reduce the severity and priority to HIGH as the upgrade will eventually happen (after 15 minutes or not on my estimate). The delay is certainly a problem and we will address that but the delay shouldn't be a blocker.

Vu

Comment 18 Vu Dinh 2021-09-21 21:09:03 UTC
Hey folks,

I have determined that the upgrade delay is due to a condition check on OLM that is using some obsolete APIs that are no longer needed since the introduction of the new resolver. I have opened a PR on upstream to remove this logic. That should allow the upgrade to happen quicker.

I want to reinstate that this is an upgrade delay, not a block as the upgrade will eventually happen so this shouldn't be a blocker for the upcoming release.

Thanks,
Vu

Comment 19 RamaKasturi 2021-09-22 13:29:49 UTC
Hello vu Dinh,

   Today i have tried upgrading descheduler from 4.8 to 4.9 and i see that upgrade takes place but it does happen after sometime, do we need to mark this as known issue so that users or customers would not wonder on why the upgrade has still not happened yet ? WDYT ?

Thanks
kasturi

Comment 20 Vu Dinh 2021-09-22 17:55:05 UTC
Hey,

Yes, it is a known issue until we merge the fix and backport it back to prior versions. In the meantime, let people know that the upgrade will eventually happen but it takes a bit of time around 10 to 15 minutes. After the fix, it should much quicker.

Vu

Comment 21 RamaKasturi 2021-09-23 05:20:56 UTC
okay, thank you, one more last question... so do you add requires doc text flag and provide doc text so that documentation team can add it to the release notes or how does it work ?

Comment 22 Vu Dinh 2021-09-23 14:35:34 UTC
Hi,

There is no need for documentation on this. This is a known issue that will be addressed soon after the fix is merged and it will be backported. I said earlier that for the time being, if issue arises, simply being aware that this issue exists.

Vu

Comment 23 RamaKasturi 2021-09-23 17:10:55 UTC
(In reply to Vu Dinh from comment #22)
> Hi,
> 
> There is no need for documentation on this. This is a known issue that will
> be addressed soon after the fix is merged and it will be backported. I said
> earlier that for the time being, if issue arises, simply being aware that
> this issue exists.
> 
> Vu

okay, thank you !!

Comment 24 Prashant Dhamdhere 2021-10-06 16:59:10 UTC
*** Bug 2010602 has been marked as a duplicate of this bug. ***

Comment 26 tflannag 2021-10-08 20:03:32 UTC
*** Bug 2007778 has been marked as a duplicate of this bug. ***

Comment 28 tflannag 2021-10-14 16:22:07 UTC
*** Bug 2013811 has been marked as a duplicate of this bug. ***

Comment 31 xzha 2021-11-03 02:45:54 UTC
verify:
zhaoxia@xzha-mac bug-2002276 % oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.10.0-0.nightly-2021-11-02-121455   True        False         64m     Cluster version is 4.10.0-0.nightly-2021-11-02-121455
zhaoxia@xzha-mac bug-2002276 % oc adm release info  registry.ci.openshift.org/ocp/release:4.10.0-0.nightly-2021-11-02-121455 --commits|grep operator-lifecycle-manager
  operator-lifecycle-manager                     https://github.com/openshift/operator-framework-olm                         8970fc081f9c411112446d29ebaeb82d00ba7331

[root@preserve-olm-agent-test ~]# opm alpha list channels quay.io/openshift-qe-optional-operators/ocp4-index:latest cluster-kube-descheduler-operator 
PACKAGE                            CHANNEL  HEAD
cluster-kube-descheduler-operator  4.6      clusterkubedescheduleroperator.4.6.0-202110262229
cluster-kube-descheduler-operator  4.7      clusterkubedescheduleroperator.4.7.0-202111020909
cluster-kube-descheduler-operator  4.8      clusterkubedescheduleroperator.4.8.0-202111021756
cluster-kube-descheduler-operator  4.9      clusterkubedescheduleroperator.4.9.0-202111020858
cluster-kube-descheduler-operator  stable   clusterkubedescheduleroperator.4.9.0-202111020858

1, create project
zhaoxia@xzha-mac bug-2002276 % oc adm new-project openshift-kube-descheduler-operator
Created project openshift-kube-descheduler-operator
zhaoxia@xzha-mac bug-2002276 % oc project openshift-kube-descheduler-operator
Now using project "openshift-kube-descheduler-operator" on server "https://api.xzha-1103.qe.devcluster.openshift.com:6443".

2, create sub
zhaoxia@xzha-mac bug-2002276 % cat sub.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: cluster-kube-descheduler-operator
  namespace: openshift-kube-descheduler-operator
spec:
  channel: "4.8"
  installPlanApproval: Automatic
  name: cluster-kube-descheduler-operator
  source: qe-app-registry
  sourceNamespace: openshift-marketplace
  startingCSV: clusterkubedescheduleroperator.4.8.0-202111021756

zhaoxia@xzha-mac bug-2002276 % cat og.yaml 
kind: OperatorGroup
apiVersion: operators.coreos.com/v1
metadata:
  name: og-single
  namespace: openshift-kube-descheduler-operator
spec:
  targetNamespaces:
  - openshift-kube-descheduler-operator

zhaoxia@xzha-mac bug-2002276 % oc apply -f og.yaml 
operatorgroup.operators.coreos.com/og-single created
zhaoxia@xzha-mac bug-2002276 % oc apply -f sub.yaml 
subscription.operators.coreos.com/cluster-kube-descheduler-operator created

3, check ip/csv
zhaoxia@xzha-mac bug-2002276 % oc get ip
NAME            CSV                                                 APPROVAL    APPROVED
install-kpn95   clusterkubedescheduleroperator.4.8.0-202111021756   Automatic   true
zhaoxia@xzha-mac bug-2002276 % oc get csv
NAME                                                DISPLAY                            VERSION              REPLACES   PHASE
clusterkubedescheduleroperator.4.8.0-202111021756   Kube Descheduler Operator          4.8.0-202111021756              Succeeded
elasticsearch-operator.5.2.3-14                     OpenShift Elasticsearch Operator   5.2.3-14                        Succeeded

4, edit sub to channel "4.9"
zhaoxia@xzha-mac bug-2002276 % oc edit sub cluster-kube-descheduler-operator
subscription.operators.coreos.com/cluster-kube-descheduler-operator edited

5, check ip/csv
zhaoxia@xzha-mac bug-2002276 % oc get ip 
NAME            CSV                                                 APPROVAL    APPROVED
install-kpn95   clusterkubedescheduleroperator.4.8.0-202111021756   Automatic   true
install-pf2q8   clusterkubedescheduleroperator.4.9.0-202111020858   Automatic   true

zhaoxia@xzha-mac bug-2002276 % oc get csv
NAME                                                DISPLAY                            VERSION              REPLACES                                            PHASE
clusterkubedescheduleroperator.4.9.0-202111020858   Kube Descheduler Operator          4.9.0-202111020858   clusterkubedescheduleroperator.4.8.0-202111021756   Succeeded
elasticsearch-operator.5.2.3-14                     OpenShift Elasticsearch Operator   5.2.3-14                                                                 Succeeded

LGTM. verified.

Comment 32 Oren Cohen 2021-11-24 09:10:26 UTC
*** Bug 2026003 has been marked as a duplicate of this bug. ***

Comment 36 errata-xmlrpc 2022-03-10 16:08:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056


Note You need to log in before you can comment on or make changes to this bug.