Bug 1827822

Summary:	Operator update is failing due to missing replace field in Operator CSV
Product:	OpenShift Container Platform	Reporter:	Alexander Greene <agreene>
Component:	OLM	Assignee:	Alexander Greene <agreene>
OLM sub component:	OLM	QA Contact:	Jian Zhang <jiazha>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	high
Priority:	high	CC:	ecordell
Version:	4.3.z
Target Milestone:	---
Target Release:	4.3.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	Cause: If an operator is being upgraded that provides a required API whose GVK has not changed since the previous version of the operator and the operator that depends on the API uses a skipRange instead of the Spec.Replaces field, OLM fails to generate the "upgraded CSV" with the correct replaces field. Specifically, OLM would: 1. Add the new operator to the generation, and marking the APIs it provides as "present". 2. Remove the old operator from the generation, marking the APIs it provides as "absent", despite being provided by the new version of the operator. 3. Attempt to resolve the "missing" apis, overwriting the the new version of the operator with a copy that does not have its Spec.Replaces field set. Consequence: Certain operators would fail to upgrade to new versions. Fix: OLM was updated to remove the old operator from the current generation before adding the new operator to the generation. Result: The upgrade will succeed as expected.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-05-27 17:00:45 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1827821
Bug Blocks:	1836447

Description Alexander Greene 2020-04-24 21:20:58 UTC

This bug was initially created as a copy of Bug #1827821

I am copying this bug because: 

It required a backport

This bug was initially created as a copy of Bug #1818788

I am copying this bug because: 



Description of problem:

Updating to OpenShift Container Platform 4.3.8 triggered also `elasticsearch-operator` from being updated. This update though failed and got stuck because the CSV was not correctly rolled and therefore the CSV for the older version was active as well as the CSV for the new version.

This caused a ownership conflict which could only be resolved by manually removing the CSV from the old `elasticsearch` operator version

Version-Release number of selected component (if applicable):

 - OpenShift Container Platform 4.3.8


How reproducible:

 - N/A


Steps to Reproduce:
1. N/A

Actual results:

Update of `elasticsearch-operator` was stuck, impacting additional operators from being able to get installed

Expected results:

Update to work and to avoid one failing part to impact the entire operator installation and update capabilities

Additional info:

Comment 4 Jian Zhang 2020-05-15 11:08:32 UTC

Cluster version is 4.3.0-0.nightly-2020-05-14-230729
[root@preserve-olm-env ~]# oc -n openshift-operator-lifecycle-manager  exec catalog-operator-7fbf559bc8-gt4lj -- olm --version
OLM version: 0.13.0
git commit: 3740fa3a8aa571ba4ee1f048561d674d4fc6c2d3

1, installed a catsrc that pointed to the catalog image that only contained the 1.0.0 versions of the operator
[root@preserve-olm-env ~]# oc create -f cs-bug.yaml 
catalogsource.operators.coreos.com/agreene-operators created

[root@preserve-olm-env ~]# cat cs-bug.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: agreene-operators
  namespace: openshift-marketplace
spec:
  displayName: Agreene Operators
  image: quay.io/agreene/busybox-dependencies:old
  sourceType: grpc

[root@preserve-olm-env ~]# oc project openshift-marketplace
Now using project "openshift-marketplace" on server "https://api.ci-ln-yj67hgt-d5d6b.origin-ci-int-aws.dev.rhcloud.com:6443".
[root@preserve-olm-env ~]# oc get catalogsource
NAME                  DISPLAY               TYPE   PUBLISHER   AGE
agreene-operators     Agreene Operators     grpc               57s
certified-operators   Certified Operators   grpc   Red Hat     17m
community-operators   Community Operators   grpc   Red Hat     17m
redhat-operators      Red Hat Operators     grpc   Red Hat     17m
[root@preserve-olm-env ~]# oc get pods
NAME                                   READY   STATUS    RESTARTS   AGE
agreene-operators-nszm4                1/1     Running   0          67s
certified-operators-ffdbd8db9-fldwv    1/1     Running   0          17m
community-operators-6c4989f48b-x9ljq   1/1     Running   0          17m
marketplace-operator-768558b86-gk69b   1/1     Running   0          18m
redhat-operators-78bd964694-5z2p6      1/1     Running   0          17m
[root@preserve-olm-env ~]# oc get packagemanifest|grep busy
busybox                                      Agreene Operators     75s
busybox-dependency                           Agreene Operators     75s

2, created an OperatorGroup and a subscription 
[root@preserve-olm-env ~]# oc create -f og.yaml 
operatorgroup.operators.coreos.com/test-og created
[root@preserve-olm-env ~]# cat og.yaml 
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: test-og
  namespace: openshift-marketplace
spec:
  targetNamespaces:
  - openshift-marketplace

[root@preserve-olm-env ~]# oc create -f sub-bug.yaml 
subscription.operators.coreos.com/busybox created
[root@preserve-olm-env ~]# cat sub-bug.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: busybox
  namespace: openshift-marketplace
spec:
  channel: "alpha"
  installPlanApproval: Automatic
  name: busybox
  source: agreene-operators
  sourceNamespace: openshift-marketplace
  startingCSV: busybox.v1.0.0

[root@preserve-olm-env ~]# oc get sub
NAME                                                               PACKAGE              SOURCE              CHANNEL
busybox                                                            busybox              agreene-operators   alpha
busybox-dependency-alpha-agreene-operators-openshift-marketplace   busybox-dependency   agreene-operators   alpha
[root@preserve-olm-env ~]# oc get csv
NAME                        DISPLAY              VERSION   REPLACES   PHASE
busybox-dependency.v1.0.0   busybox-dependency   1.0.0                Succeeded
busybox.v1.0.0              busybox              1.0.0                Succeeded

3, Update this CatalogSource image(quay.io/agreene/busybox-dependencies:old) to the new one: quay.io/agreene/busybox-dependencies:new(contains 2.0.0 version)

[root@preserve-olm-env ~]# oc edit catalogsource agreene-operators
catalogsource.operators.coreos.com/agreene-operators edited

[root@preserve-olm-env ~]# oc get csv
NAME                        DISPLAY              VERSION   REPLACES         PHASE
busybox-dependency.v1.0.0   busybox-dependency   1.0.0                      Succeeded
busybox.v1.0.0              busybox              1.0.0                      Replacing
busybox.v2.0.0              busybox              2.0.0     busybox.v1.0.0   InstallReady
[root@preserve-olm-env ~]# oc get csv
NAME                        DISPLAY              VERSION   REPLACES                    PHASE
busybox-dependency.v1.0.0   busybox-dependency   1.0.0                                 Deleting
busybox-dependency.v2.0.0   busybox-dependency   2.0.0     busybox-dependency.v1.0.0   Succeeded
busybox.v2.0.0              busybox              2.0.0     busybox.v1.0.0              Succeeded

[root@preserve-olm-env ~]# oc get csv
NAME                        DISPLAY              VERSION   REPLACES                    PHASE
busybox-dependency.v2.0.0   busybox-dependency   2.0.0     busybox-dependency.v1.0.0   Succeeded
busybox.v2.0.0              busybox              2.0.0     busybox.v1.0.0              Succeeded
[root@preserve-olm-env ~]# oc get pods
NAME                                   READY   STATUS    RESTARTS   AGE
agreene-operators-j4hp9                1/1     Running   0          74s
busybox-8598cc9bcb-mjwqz               1/1     Running   0          55s
busybox-dependency-76f7c74648-6mnzw    1/1     Running   0          51s
certified-operators-ffdbd8db9-fldwv    1/1     Running   0          22m
community-operators-6c4989f48b-x9ljq   1/1     Running   0          22m
marketplace-operator-768558b86-gk69b   1/1     Running   0          23m
redhat-operators-78bd964694-5z2p6      1/1     Running   0          22m


The operator has been upgraded to "2.0.0" version successfully. LGTM, verify it.

Comment 6 errata-xmlrpc 2020-05-27 17:00:45 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2184