Bug 2024048 - OLM fails to upgrade operators immediately
Summary: OLM fails to upgrade operators immediately
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OLM
Version: 4.9
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.9.z
Assignee: Vu Dinh
QA Contact: xzha
URL:
Whiteboard:
Depends On: 2002276
Blocks: 2030489
TreeView+ depends on / blocked
 
Reported: 2021-11-17 06:02 UTC by Vu Dinh
Modified: 2021-12-13 12:06 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-12-13 12:06:24 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift operator-framework-olm pull 215 0 None open Bug 2024048: Remove outdated subscription update logic to improve resolution delay 2021-11-17 06:03:20 UTC
Red Hat Product Errata RHBA-2021:5003 0 None None None 2021-12-13 12:06:38 UTC

Description Vu Dinh 2021-11-17 06:02:24 UTC
This bug was initially created as a copy of Bug #2002276

I am copying this bug because: 
Backporting

Description of problem:
Upgrading descheduler from 4.8 to 4.9 fails, i see that when channel & starting CSV is set i do not see any upgrade starting, jian zhang looked further and below is what he found.

1, Only remove the sub, not the csv, resubscribe the 4.9 one, get the below errors:
  - message: 'constraints not satisfiable: @existing/openshift-kube-descheduler-operator//clusterkubedescheduleroperator.4.8.0-202108312109
      and qe-app-registry/openshift-marketplace/4.9/clusterkubedescheduleroperator.4.9.0-202109071344
      originate from package cluster-kube-descheduler-operator, subscription cluster-kube-descheduler-operator
      requires qe-app-registry/openshift-marketplace/4.9/clusterkubedescheduleroperator.4.9.0-202109071344,
      subscription cluster-kube-descheduler-operator exists, clusterserviceversion
      clusterkubedescheduleroperator.4.8.0-202108312109 exists and is not referenced
      by a subscription'
    reason: ConstraintsNotSatisfiable
    status: "True"
    type: ResolutionFailed



2,, Remove the sub and csv, and then recreate it, it works well.
[cloud-user@preserve-olm-env jian]$ cat sub-descheduler.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: cluster-kube-descheduler-operator
  namespace: openshift-kube-descheduler-operator
spec:
  channel: "4.9"
  installPlanApproval: Automatic
  name: cluster-kube-descheduler-operator
  source: qe-app-registry
  sourceNamespace: openshift-marketplace
  startingCSV: clusterkubedescheduleroperator.4.9.0-202109071344

[cloud-user@preserve-olm-env jian]$ oc get sub
NAME                                PACKAGE                             SOURCE            CHANNEL
cluster-kube-descheduler-operator   cluster-kube-descheduler-operator   qe-app-registry   4.9

[cloud-user@preserve-olm-env jian]$ oc get ip
NAME            CSV                                                 APPROVAL    APPROVED
install-4w9lm   clusterkubedescheduleroperator.4.9.0-202109071344   Automatic   true

[cloud-user@preserve-olm-env jian]$ oc get csv
NAME                                                DISPLAY                            VERSION              REPLACES                          PHASE
clusterkubedescheduleroperator.4.9.0-202109071344   Kube Descheduler Operator          4.9.0-202109071344                                     Succeeded
elasticsearch-operator.5.2.0-60                     OpenShift Elasticsearch Operator   5.2.0-60             elasticsearch-operator.5.1.1-56   Succeeded

3, check the bundle content:
clusterkubedescheduleroperator.4.8.0-202108312109   {"apiVersion":"opera  {"apiVersion":"operators.coreos.com/v1alpha1","kin  registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-  >=4.6.0 <4.8.0-202108312109     4.8.0-202108312109

clusterkubedescheduleroperator.4.9.0-202109071344   {"apiVersion":"opera  {"apiVersion":"operators.coreos.com/v1alpha1","kin  registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-  >=4.6.0 <4.9.0-202109071344     4.9.0-202109071344
Version-Release number of selected component (if applicable):

[knarra@knarra ~]$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.9.0-0.nightly-2021-09-06-004132   True        False         23h     Cluster version is 4.9.0-0.nightly-2021-09-06-004132

 Clusterkubedescheduleroperator.4.8.0-202108312109

How reproducible:
Install 4.8 cluster
Install 4.8 descheduler operator
Upgrade system to 4.9
Now edit descheduler sub & set channel to 4.9, starting csv to clusterkubedescheduleroperator.4.9.0-202109071344
spec:
  channel: "4.9"
  installPlanApproval: Automatic
  name: cluster-kube-descheduler-operator
  source: qe-app-registry
  sourceNamespace: openshift-marketplace
  startingCSV: clusterkubedescheduleroperator.4.9.0-202109071344


Actual Results:
 Upgrade does not start at all

Expected Results:
 Upgrade should work fine.

Comment 3 xzha 2021-12-03 07:20:24 UTC
verify:
zhaoxia@xzha-mac bug-2002276 % oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.9.0-0.nightly-2021-12-03-045106   True        False         24m     Cluster version is 4.9.0-0.nightly-2021-12-03-045106

zhaoxia@xzha-mac bug-2002276 % oc adm release info registry.ci.openshift.org/ocp/release:4.9.0-0.nightly-2021-12-03-045106 --commits|grep operator-lifecycle-manager
  operator-lifecycle-manager                     https://github.com/openshift/operator-framework-olm                         5fc4c78bb48e66a252672aae137c8dcd8a6e43a8

zhaoxia@xzha-mac src % opm alpha list channels quay.io/openshift-qe-optional-operators/ocp4-index:latest cluster-kube-descheduler-operator 
WARN[0015] DEPRECATION NOTICE:
Sqlite-based catalogs and their related subcommands are deprecated. Support for
them will be removed in a future release. Please migrate your catalog workflows
to the new file-based catalog format. 
PACKAGE                            CHANNEL  HEAD
cluster-kube-descheduler-operator  4.6      clusterkubedescheduleroperator.4.6.0-202112020740
cluster-kube-descheduler-operator  4.7      clusterkubedescheduleroperator.4.7.0-202112020740
cluster-kube-descheduler-operator  4.8      clusterkubedescheduleroperator.4.8.0-202112020740
cluster-kube-descheduler-operator  4.9      clusterkubedescheduleroperator.4.9.0-202111300333
cluster-kube-descheduler-operator  stable   clusterkubedescheduleroperator.4.9.0-202111300333

1, create project
zhaoxia@xzha-mac src % oc adm new-project openshift-kube-descheduler-operator
Created project openshift-kube-descheduler-operator
zhaoxia@xzha-mac src % oc project openshift-kube-descheduler-operator
Now using project "openshift-kube-descheduler-operator" on server "https://api.xzha-1203.qe.devcluster.openshift.com:6443".

2, install operator
zhaoxia@xzha-mac bug-2002276 % cat sub.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: cluster-kube-descheduler-operator
  namespace: openshift-kube-descheduler-operator
spec:
  channel: "4.8"
  installPlanApproval: Automatic
  name: cluster-kube-descheduler-operator
  source: qe-app-registry
  sourceNamespace: openshift-marketplace

zhaoxia@xzha-mac bug-2002276 % cat og.yaml 
kind: OperatorGroup
apiVersion: operators.coreos.com/v1
metadata:
  name: og-single
  namespace: openshift-kube-descheduler-operator
spec:
  targetNamespaces:
  - openshift-kube-descheduler-operator
zhaoxia@xzha-mac bug-2002276 % oc apply -f og.yaml 
operatorgroup.operators.coreos.com/og-single created
zhaoxia@xzha-mac bug-2002276 % oc apply -f sub.yaml 
subscription.operators.coreos.com/cluster-kube-descheduler-operator created

3, check ip/csv

zhaoxia@xzha-mac bug-2002276 % oc get ip
NAME            CSV                                                 APPROVAL    APPROVED
install-875wl   clusterkubedescheduleroperator.4.8.0-202112020740   Automatic   true
zhaoxia@xzha-mac bug-2002276 % oc get csv
NAME                                                DISPLAY                            VERSION              REPLACES   PHASE
clusterkubedescheduleroperator.4.8.0-202112020740   Kube Descheduler Operator          4.8.0-202112020740              Installing
elasticsearch-operator.5.2.3-47                     OpenShift Elasticsearch Operator   5.2.3-47                        Succeeded

4, edit sub to channel "4.9"
zhaoxia@xzha-mac bug-2002276 % oc edit sub cluster-kube-descheduler-operator 
subscription.operators.coreos.com/cluster-kube-descheduler-operator edited

5, check ip/csv
zhaoxia@xzha-mac bug-2002276 % oc get ip
NAME            CSV                                                 APPROVAL    APPROVED
install-78rph   clusterkubedescheduleroperator.4.9.0-202111300333   Automatic   true
install-875wl   clusterkubedescheduleroperator.4.8.0-202112020740   Automatic   true

zhaoxia@xzha-mac bug-2002276 % oc get csv
NAME                                                DISPLAY                            VERSION              REPLACES                                            PHASE
clusterkubedescheduleroperator.4.8.0-202112020740   Kube Descheduler Operator          4.8.0-202112020740                                                       Replacing
clusterkubedescheduleroperator.4.9.0-202111300333   Kube Descheduler Operator          4.9.0-202111300333   clusterkubedescheduleroperator.4.8.0-202112020740   Installing
elasticsearch-operator.5.2.3-47                     OpenShift Elasticsearch Operator   5.2.3-47                                                                 Succeeded

zhaoxia@xzha-mac bug-2002276 % oc get csv
NAME                                                DISPLAY                            VERSION              REPLACES                                            PHASE
clusterkubedescheduleroperator.4.9.0-202111300333   Kube Descheduler Operator          4.9.0-202111300333   clusterkubedescheduleroperator.4.8.0-202112020740   Succeeded
elasticsearch-operator.5.2.3-47                     OpenShift Elasticsearch Operator   5.2.3-47                                                                 Succeeded


LGTM, verified.

Comment 6 errata-xmlrpc 2021-12-13 12:06:24 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.9.11 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:5003


Note You need to log in before you can comment on or make changes to this bug.