Bug 1827822 - Operator update is failing due to missing replace field in Operator CSV
Summary: Operator update is failing due to missing replace field in Operator CSV
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OLM
Version: 4.3.z
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.3.z
Assignee: Alexander Greene
QA Contact: Jian Zhang
URL:
Whiteboard:
Depends On: 1827821
Blocks: 1836447
TreeView+ depends on / blocked
 
Reported: 2020-04-24 21:20 UTC by Alexander Greene
Modified: 2023-09-07 22:57 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: If an operator is being upgraded that provides a required API whose GVK has not changed since the previous version of the operator and the operator that depends on the API uses a skipRange instead of the Spec.Replaces field, OLM fails to generate the "upgraded CSV" with the correct replaces field. Specifically, OLM would: 1. Add the new operator to the generation, and marking the APIs it provides as "present". 2. Remove the old operator from the generation, marking the APIs it provides as "absent", despite being provided by the new version of the operator. 3. Attempt to resolve the "missing" apis, overwriting the the new version of the operator with a copy that does not have its Spec.Replaces field set. Consequence: Certain operators would fail to upgrade to new versions. Fix: OLM was updated to remove the old operator from the current generation before adding the new operator to the generation. Result: The upgrade will succeed as expected.
Clone Of:
Environment:
Last Closed: 2020-05-27 17:00:45 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github operator-framework operator-lifecycle-manager pull 1485 0 None closed Bug 1827822: Generation bug 4.3 2020-07-30 08:56:02 UTC
Red Hat Product Errata RHBA-2020:2184 0 None None None 2020-05-27 17:00:58 UTC

Description Alexander Greene 2020-04-24 21:20:58 UTC
This bug was initially created as a copy of Bug #1827821

I am copying this bug because: 

It required a backport

This bug was initially created as a copy of Bug #1818788

I am copying this bug because: 



Description of problem:

Updating to OpenShift Container Platform 4.3.8 triggered also `elasticsearch-operator` from being updated. This update though failed and got stuck because the CSV was not correctly rolled and therefore the CSV for the older version was active as well as the CSV for the new version.

This caused a ownership conflict which could only be resolved by manually removing the CSV from the old `elasticsearch` operator version

Version-Release number of selected component (if applicable):

 - OpenShift Container Platform 4.3.8


How reproducible:

 - N/A


Steps to Reproduce:
1. N/A

Actual results:

Update of `elasticsearch-operator` was stuck, impacting additional operators from being able to get installed

Expected results:

Update to work and to avoid one failing part to impact the entire operator installation and update capabilities

Additional info:

Comment 4 Jian Zhang 2020-05-15 11:08:32 UTC
Cluster version is 4.3.0-0.nightly-2020-05-14-230729
[root@preserve-olm-env ~]# oc -n openshift-operator-lifecycle-manager  exec catalog-operator-7fbf559bc8-gt4lj -- olm --version
OLM version: 0.13.0
git commit: 3740fa3a8aa571ba4ee1f048561d674d4fc6c2d3

1, installed a catsrc that pointed to the catalog image that only contained the 1.0.0 versions of the operator
[root@preserve-olm-env ~]# oc create -f cs-bug.yaml 
catalogsource.operators.coreos.com/agreene-operators created

[root@preserve-olm-env ~]# cat cs-bug.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: agreene-operators
  namespace: openshift-marketplace
spec:
  displayName: Agreene Operators
  image: quay.io/agreene/busybox-dependencies:old
  sourceType: grpc

[root@preserve-olm-env ~]# oc project openshift-marketplace
Now using project "openshift-marketplace" on server "https://api.ci-ln-yj67hgt-d5d6b.origin-ci-int-aws.dev.rhcloud.com:6443".
[root@preserve-olm-env ~]# oc get catalogsource
NAME                  DISPLAY               TYPE   PUBLISHER   AGE
agreene-operators     Agreene Operators     grpc               57s
certified-operators   Certified Operators   grpc   Red Hat     17m
community-operators   Community Operators   grpc   Red Hat     17m
redhat-operators      Red Hat Operators     grpc   Red Hat     17m
[root@preserve-olm-env ~]# oc get pods
NAME                                   READY   STATUS    RESTARTS   AGE
agreene-operators-nszm4                1/1     Running   0          67s
certified-operators-ffdbd8db9-fldwv    1/1     Running   0          17m
community-operators-6c4989f48b-x9ljq   1/1     Running   0          17m
marketplace-operator-768558b86-gk69b   1/1     Running   0          18m
redhat-operators-78bd964694-5z2p6      1/1     Running   0          17m
[root@preserve-olm-env ~]# oc get packagemanifest|grep busy
busybox                                      Agreene Operators     75s
busybox-dependency                           Agreene Operators     75s

2, created an OperatorGroup and a subscription 
[root@preserve-olm-env ~]# oc create -f og.yaml 
operatorgroup.operators.coreos.com/test-og created
[root@preserve-olm-env ~]# cat og.yaml 
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: test-og
  namespace: openshift-marketplace
spec:
  targetNamespaces:
  - openshift-marketplace

[root@preserve-olm-env ~]# oc create -f sub-bug.yaml 
subscription.operators.coreos.com/busybox created
[root@preserve-olm-env ~]# cat sub-bug.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: busybox
  namespace: openshift-marketplace
spec:
  channel: "alpha"
  installPlanApproval: Automatic
  name: busybox
  source: agreene-operators
  sourceNamespace: openshift-marketplace
  startingCSV: busybox.v1.0.0

[root@preserve-olm-env ~]# oc get sub
NAME                                                               PACKAGE              SOURCE              CHANNEL
busybox                                                            busybox              agreene-operators   alpha
busybox-dependency-alpha-agreene-operators-openshift-marketplace   busybox-dependency   agreene-operators   alpha
[root@preserve-olm-env ~]# oc get csv
NAME                        DISPLAY              VERSION   REPLACES   PHASE
busybox-dependency.v1.0.0   busybox-dependency   1.0.0                Succeeded
busybox.v1.0.0              busybox              1.0.0                Succeeded

3, Update this CatalogSource image(quay.io/agreene/busybox-dependencies:old) to the new one: quay.io/agreene/busybox-dependencies:new(contains 2.0.0 version)

[root@preserve-olm-env ~]# oc edit catalogsource agreene-operators
catalogsource.operators.coreos.com/agreene-operators edited

[root@preserve-olm-env ~]# oc get csv
NAME                        DISPLAY              VERSION   REPLACES         PHASE
busybox-dependency.v1.0.0   busybox-dependency   1.0.0                      Succeeded
busybox.v1.0.0              busybox              1.0.0                      Replacing
busybox.v2.0.0              busybox              2.0.0     busybox.v1.0.0   InstallReady
[root@preserve-olm-env ~]# oc get csv
NAME                        DISPLAY              VERSION   REPLACES                    PHASE
busybox-dependency.v1.0.0   busybox-dependency   1.0.0                                 Deleting
busybox-dependency.v2.0.0   busybox-dependency   2.0.0     busybox-dependency.v1.0.0   Succeeded
busybox.v2.0.0              busybox              2.0.0     busybox.v1.0.0              Succeeded

[root@preserve-olm-env ~]# oc get csv
NAME                        DISPLAY              VERSION   REPLACES                    PHASE
busybox-dependency.v2.0.0   busybox-dependency   2.0.0     busybox-dependency.v1.0.0   Succeeded
busybox.v2.0.0              busybox              2.0.0     busybox.v1.0.0              Succeeded
[root@preserve-olm-env ~]# oc get pods
NAME                                   READY   STATUS    RESTARTS   AGE
agreene-operators-j4hp9                1/1     Running   0          74s
busybox-8598cc9bcb-mjwqz               1/1     Running   0          55s
busybox-dependency-76f7c74648-6mnzw    1/1     Running   0          51s
certified-operators-ffdbd8db9-fldwv    1/1     Running   0          22m
community-operators-6c4989f48b-x9ljq   1/1     Running   0          22m
marketplace-operator-768558b86-gk69b   1/1     Running   0          23m
redhat-operators-78bd964694-5z2p6      1/1     Running   0          22m


The operator has been upgraded to "2.0.0" version successfully. LGTM, verify it.

Comment 6 errata-xmlrpc 2020-05-27 17:00:45 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2184


Note You need to log in before you can comment on or make changes to this bug.