Bug 1860185

Summary:	Operator installation failed - `Subscription` object stays in `UpgradePending` but the `InstallPlan` it associated with isn’t installing
Product:	OpenShift Container Platform	Reporter:	tony.wu
Component:	OLM	Assignee:	Nick Hale <nhale>
OLM sub component:	OLM	QA Contact:	kuiwang
Status:	CLOSED ERRATA	Docs Contact:
Severity:	medium
Priority:	medium	CC:	afield, assingh, dmoessne, jaharrin, jiazha, mharri, nhale, rdomnu, rkshirsa, scuppett, vdinh
Version:	4.6	Keywords:	UpcomingSprint
Target Milestone:	---
Target Release:	4.6.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1864121 (view as bug list)		Environment:
Last Closed:	2020-10-27 16:16:56 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1864121

Description tony.wu 2020-07-23 22:46:36 UTC

Description of problem:
Ran into this "Operator installation failed" when installing an Operator into a single NS, uninstall it, and then install another Operator to the same NS.
(please see detailed reproduce steps below)

This issue is quite consistent and always reproducible in OCP 4.6.

Version-Release number of selected component (if applicable):
OCP 4.6

How reproducible:
100% (3/3)

Steps to Reproduce:
0. Log in to OCP Console and goto 'OperatorHub' page

1. Install Operator A in own NS mode to a NS
--> e.g. install Strimzi to NS “tw” (installPlan ‘install-mxvv9’ created)

2. Install Operator B in own NS mode to the same NS
--> e.g. install Portworx to NS “tw” (installPlan ‘install-plmll’ created)

3. On the 'Installed Operators' page, uninstall Operator B using Console
(the Subscription of Operator B has been removed)

4. Go back to 'OperatorHub' page, install Operator C in own NS mode to the same NS
--> e.g. install Couchbase to NS “tw”
(Subscription object is associated to the same installPlan ‘install-plmll’ as the removed Operator B)

5. Operator C’s Subscription object stays in `UpgradePending` , while installPlan it associated with isn’t installing nor reporting any errors

Actual results:
In above Step 5, the Operator C’s Subscription object stays in `UpgradePending` , while installPlan it associated with isn’t installing nor reporting any errors

Expected results:
Successfully install Operator C in above Step 5.

Additional info:

Comment 1 tony.wu 2020-07-23 22:48:35 UTC

OpenShift Version:
4.6.0-0.ci-2020-07-21-114552

Comment 3 Nick Hale 2020-07-27 14:21:53 UTC

Tony, thanks for the report.

Does this happen with any combination of operators that support "own namespace"? 

> while installPlan it associated with isn’t installing nor reporting any errors

Does the InstallPlan exist? Could you include it in this BZ?

> uninstall Operator B using Console 

Could you list the resources the Console actually deletes?

Comment 4 tony.wu 2020-07-27 20:20:14 UTC

The cluster was gone, so the logs would need to be recaptured again.

Since this is easily reproducible on any cluster from 4.5.x to 4.6 (confirmed by Daniel Messer as well), can any OLM devs reproduce this and troubleshoot the issue with any logs/resources needed following the repo steps? 

thanks

Comment 6 Nick Hale 2020-07-28 15:42:35 UTC

*** Bug 1860352 has been marked as a duplicate of this bug. ***

Comment 10 kuiwang 2020-07-30 06:58:33 UTC

verify it on 4.6. LGTM

---
kuiwang@Kuis-MacBook-Pro 1860185 % olmo=`oc get pod -n openshift-operator-lifecycle-manager|grep olm-operator|cut -d" " -f1`;oc exec $olmo  -n openshift-operator-lifecycle-manager -- olm --version

OLM version: 0.16.0
git commit: 69478e6938b1737585df8cfbd3b4eaf06401b3d2


kuiwang@Kuis-MacBook-Pro 1860185 % cat ns.yaml 
apiVersion: v1
kind: Namespace
metadata:
  name: verifybug

kuiwang@Kuis-MacBook-Pro 1860185 % oc apply -f ns.yaml 
namespace/verifybug created

kuiwang@Kuis-MacBook-Pro 1860185 % cat og.yaml 
kind: OperatorGroup
apiVersion: operators.coreos.com/v1
metadata:
  name: og-single
  namespace: verifybug
spec:
  targetNamespaces:
  - verifybug


kuiwang@Kuis-MacBook-Pro 1860185 % oc apply -f og.yaml 
operatorgroup.operators.coreos.com/og-single created

kuiwang@Kuis-MacBook-Pro 1860185 % oc project verifybug
Now using project "verifybug" on server "https://api.kuiwang20200730t132822.qe.devcluster.openshift.com:6443".

kuiwang@Kuis-MacBook-Pro 1860185 % cat Strimzi.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: strimzi
  namespace: verifybug
spec:
  channel: stable
  installPlanApproval: Automatic
  name: strimzi-kafka-operator
  source: community-operators
  sourceNamespace: openshift-marketplace
  startingCSV: strimzi-cluster-operator.v0.19.0
kuiwang@Kuis-MacBook-Pro 1860185 % oc apply -f Strimzi.yaml
subscription.operators.coreos.com/strimzi created
kuiwang@Kuis-MacBook-Pro 1860185 % oc get ip
NAME            CSV                                APPROVAL    APPROVED
install-gkhwh   strimzi-cluster-operator.v0.19.0   Automatic   true
kuiwang@Kuis-MacBook-Pro 1860185 % oc get csv
NAME                               DISPLAY   VERSION   REPLACES   PHASE
strimzi-cluster-operator.v0.19.0   Strimzi   0.19.0               Installing
kuiwang@Kuis-MacBook-Pro 1860185 % oc get csv
NAME                               DISPLAY   VERSION   REPLACES   PHASE
strimzi-cluster-operator.v0.19.0   Strimzi   0.19.0               Succeeded


kuiwang@Kuis-MacBook-Pro 1860185 % cat Portworx.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: portworx
  namespace: verifybug
spec:
  channel: stable
  installPlanApproval: Automatic
  name: portworx-certified
  source: certified-operators
  sourceNamespace: openshift-marketplace
  startingCSV: portworx-operator.v1.3.2
kuiwang@Kuis-MacBook-Pro 1860185 % oc apply -f Portworx.yaml 
subscription.operators.coreos.com/portworx created
kuiwang@Kuis-MacBook-Pro 1860185 % oc get csv
NAME                               DISPLAY               VERSION   REPLACES   PHASE
portworx-operator.v1.3.2           Portworx Enterprise   1.3.2                Installing
strimzi-cluster-operator.v0.19.0   Strimzi               0.19.0               Succeeded
kuiwang@Kuis-MacBook-Pro 1860185 % oc get csv
NAME                               DISPLAY               VERSION   REPLACES   PHASE
portworx-operator.v1.3.2           Portworx Enterprise   1.3.2                Succeeded
strimzi-cluster-operator.v0.19.0   Strimzi               0.19.0               Succeeded
kuiwang@Kuis-MacBook-Pro 1860185 % oc get ip 
NAME            CSV                                APPROVAL    APPROVED
install-gkhwh   strimzi-cluster-operator.v0.19.0   Automatic   true
install-tchlv   portworx-operator.v1.3.2           Automatic   true


kuiwang@Kuis-MacBook-Pro 1860185 % oc delete sub portworx
subscription.operators.coreos.com "portworx" deleted
kuiwang@Kuis-MacBook-Pro 1860185 % oc delete csv portworx-operator.v1.3.2
clusterserviceversion.operators.coreos.com "portworx-operator.v1.3.2" deleted
kuiwang@Kuis-MacBook-Pro 1860185 % oc get ip 
NAME            CSV                                APPROVAL    APPROVED
install-gkhwh   strimzi-cluster-operator.v0.19.0   Automatic   true
install-tchlv   portworx-operator.v1.3.2           Automatic   true


kuiwang@Kuis-MacBook-Pro 1860185 % cat Couchbase.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: couchbase
  namespace: verifybug
spec:
  channel: stable
  installPlanApproval: Automatic
  name: couchbase-enterprise-certified
  source: certified-operators
  sourceNamespace: openshift-marketplace
  startingCSV: couchbase-operator.v2.0.1
kuiwang@Kuis-MacBook-Pro 1860185 % oc apply -f Couchbase.yaml 
subscription.operators.coreos.com/couchbase created
kuiwang@Kuis-MacBook-Pro 1860185 % oc get ip
NAME            CSV                                APPROVAL    APPROVED
install-gkhwh   strimzi-cluster-operator.v0.19.0   Automatic   true
install-pcrb9   couchbase-operator.v2.0.1          Automatic   true
install-tchlv   portworx-operator.v1.3.2           Automatic   true
kuiwang@Kuis-MacBook-Pro 1860185 % oc get csv
NAME                               DISPLAY              VERSION   REPLACES   PHASE
couchbase-operator.v2.0.1          Couchbase Operator   2.0.1                Succeeded
strimzi-cluster-operator.v0.19.0   Strimzi              0.19.0               Succeeded



---

Comment 11 Vu Dinh 2020-08-20 14:03:25 UTC

*** Bug 1868253 has been marked as a duplicate of this bug. ***

Comment 16 errata-xmlrpc 2020-10-27 16:16:56 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196

Comment 17 Kevin Rizza 2020-11-09 21:43:55 UTC

*** Bug 1896102 has been marked as a duplicate of this bug. ***