Bug 1860185 - Operator installation failed - `Subscription` object stays in `UpgradePending` but the `InstallPlan` it associated with isn’t installing
Summary: Operator installation failed - `Subscription` object stays in `UpgradePending...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OLM
Version: 4.6
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.6.0
Assignee: Nick Hale
QA Contact: kuiwang
URL:
Whiteboard:
: 1860352 1868253 1896102 (view as bug list)
Depends On:
Blocks: 1864121
TreeView+ depends on / blocked
 
Reported: 2020-07-23 22:46 UTC by tony.wu
Modified: 2021-04-27 05:59 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1864121 (view as bug list)
Environment:
Last Closed: 2020-10-27 16:16:56 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github operator-framework operator-lifecycle-manager pull 1684 0 None closed Bug 1860185: fix(install): account for installplans when calculating generation 2021-02-08 10:57:54 UTC
Red Hat Knowledge Base (Solution) 5319681 0 None None None 2020-08-20 14:03:24 UTC
Red Hat Product Errata RHBA-2020:4196 0 None None None 2020-10-27 16:17:14 UTC

Description tony.wu 2020-07-23 22:46:36 UTC
Description of problem:
Ran into this "Operator installation failed" when installing an Operator into a single NS, uninstall it, and then install another Operator to the same NS.
(please see detailed reproduce steps below)

This issue is quite consistent and always reproducible in OCP 4.6.


Version-Release number of selected component (if applicable):
OCP 4.6

How reproducible:
100% (3/3)

Steps to Reproduce:
0. Log in to OCP Console and goto 'OperatorHub' page

1. Install Operator A in own NS mode to a NS 
   --> e.g. install Strimzi to NS “tw” (installPlan ‘install-mxvv9’ created)

2. Install Operator B in own NS mode to the same NS 
   --> e.g. install Portworx to NS “tw” (installPlan ‘install-plmll’ created)

3. On the 'Installed Operators' page, uninstall Operator B using Console 
   (the Subscription of Operator B has been removed)

4. Go back to 'OperatorHub' page, install Operator C in own NS mode to the same NS 
   --> e.g. install Couchbase to NS “tw” 
  (Subscription object is associated to the same installPlan ‘install-plmll’ as the removed Operator B)

5. Operator C’s Subscription object stays in `UpgradePending` , while installPlan it associated with isn’t installing nor reporting any errors


Actual results:
In above Step 5, the Operator C’s Subscription object stays in `UpgradePending` , while installPlan it associated with isn’t installing nor reporting any errors

Expected results:
Successfully install Operator C in above Step 5.

Additional info:

Comment 1 tony.wu 2020-07-23 22:48:35 UTC
OpenShift Version:
4.6.0-0.ci-2020-07-21-114552

Comment 3 Nick Hale 2020-07-27 14:21:53 UTC
Tony, thanks for the report.

Does this happen with any combination of operators that support "own namespace"? 

> while installPlan it associated with isn’t installing nor reporting any errors

Does the InstallPlan exist? Could you include it in this BZ?

> uninstall Operator B using Console 

Could you list the resources the Console actually deletes?

Comment 4 tony.wu 2020-07-27 20:20:14 UTC
The cluster was gone, so the logs would need to be recaptured again.

Since this is easily reproducible on any cluster from 4.5.x to 4.6 (confirmed by Daniel Messer as well), can any OLM devs reproduce this and troubleshoot the issue with any logs/resources needed following the repo steps? 

thanks

Comment 6 Nick Hale 2020-07-28 15:42:35 UTC
*** Bug 1860352 has been marked as a duplicate of this bug. ***

Comment 10 kuiwang 2020-07-30 06:58:33 UTC
verify it on 4.6. LGTM

---
kuiwang@Kuis-MacBook-Pro 1860185 % olmo=`oc get pod -n openshift-operator-lifecycle-manager|grep olm-operator|cut -d" " -f1`;oc exec $olmo  -n openshift-operator-lifecycle-manager -- olm --version

OLM version: 0.16.0
git commit: 69478e6938b1737585df8cfbd3b4eaf06401b3d2


kuiwang@Kuis-MacBook-Pro 1860185 % cat ns.yaml 
apiVersion: v1
kind: Namespace
metadata:
  name: verifybug

kuiwang@Kuis-MacBook-Pro 1860185 % oc apply -f ns.yaml 
namespace/verifybug created

kuiwang@Kuis-MacBook-Pro 1860185 % cat og.yaml 
kind: OperatorGroup
apiVersion: operators.coreos.com/v1
metadata:
  name: og-single
  namespace: verifybug
spec:
  targetNamespaces:
  - verifybug


kuiwang@Kuis-MacBook-Pro 1860185 % oc apply -f og.yaml 
operatorgroup.operators.coreos.com/og-single created

kuiwang@Kuis-MacBook-Pro 1860185 % oc project verifybug
Now using project "verifybug" on server "https://api.kuiwang20200730t132822.qe.devcluster.openshift.com:6443".

kuiwang@Kuis-MacBook-Pro 1860185 % cat Strimzi.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: strimzi
  namespace: verifybug
spec:
  channel: stable
  installPlanApproval: Automatic
  name: strimzi-kafka-operator
  source: community-operators
  sourceNamespace: openshift-marketplace
  startingCSV: strimzi-cluster-operator.v0.19.0
kuiwang@Kuis-MacBook-Pro 1860185 % oc apply -f Strimzi.yaml
subscription.operators.coreos.com/strimzi created
kuiwang@Kuis-MacBook-Pro 1860185 % oc get ip
NAME            CSV                                APPROVAL    APPROVED
install-gkhwh   strimzi-cluster-operator.v0.19.0   Automatic   true
kuiwang@Kuis-MacBook-Pro 1860185 % oc get csv
NAME                               DISPLAY   VERSION   REPLACES   PHASE
strimzi-cluster-operator.v0.19.0   Strimzi   0.19.0               Installing
kuiwang@Kuis-MacBook-Pro 1860185 % oc get csv
NAME                               DISPLAY   VERSION   REPLACES   PHASE
strimzi-cluster-operator.v0.19.0   Strimzi   0.19.0               Succeeded


kuiwang@Kuis-MacBook-Pro 1860185 % cat Portworx.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: portworx
  namespace: verifybug
spec:
  channel: stable
  installPlanApproval: Automatic
  name: portworx-certified
  source: certified-operators
  sourceNamespace: openshift-marketplace
  startingCSV: portworx-operator.v1.3.2
kuiwang@Kuis-MacBook-Pro 1860185 % oc apply -f Portworx.yaml 
subscription.operators.coreos.com/portworx created
kuiwang@Kuis-MacBook-Pro 1860185 % oc get csv
NAME                               DISPLAY               VERSION   REPLACES   PHASE
portworx-operator.v1.3.2           Portworx Enterprise   1.3.2                Installing
strimzi-cluster-operator.v0.19.0   Strimzi               0.19.0               Succeeded
kuiwang@Kuis-MacBook-Pro 1860185 % oc get csv
NAME                               DISPLAY               VERSION   REPLACES   PHASE
portworx-operator.v1.3.2           Portworx Enterprise   1.3.2                Succeeded
strimzi-cluster-operator.v0.19.0   Strimzi               0.19.0               Succeeded
kuiwang@Kuis-MacBook-Pro 1860185 % oc get ip 
NAME            CSV                                APPROVAL    APPROVED
install-gkhwh   strimzi-cluster-operator.v0.19.0   Automatic   true
install-tchlv   portworx-operator.v1.3.2           Automatic   true


kuiwang@Kuis-MacBook-Pro 1860185 % oc delete sub portworx
subscription.operators.coreos.com "portworx" deleted
kuiwang@Kuis-MacBook-Pro 1860185 % oc delete csv portworx-operator.v1.3.2
clusterserviceversion.operators.coreos.com "portworx-operator.v1.3.2" deleted
kuiwang@Kuis-MacBook-Pro 1860185 % oc get ip 
NAME            CSV                                APPROVAL    APPROVED
install-gkhwh   strimzi-cluster-operator.v0.19.0   Automatic   true
install-tchlv   portworx-operator.v1.3.2           Automatic   true


kuiwang@Kuis-MacBook-Pro 1860185 % cat Couchbase.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: couchbase
  namespace: verifybug
spec:
  channel: stable
  installPlanApproval: Automatic
  name: couchbase-enterprise-certified
  source: certified-operators
  sourceNamespace: openshift-marketplace
  startingCSV: couchbase-operator.v2.0.1
kuiwang@Kuis-MacBook-Pro 1860185 % oc apply -f Couchbase.yaml 
subscription.operators.coreos.com/couchbase created
kuiwang@Kuis-MacBook-Pro 1860185 % oc get ip
NAME            CSV                                APPROVAL    APPROVED
install-gkhwh   strimzi-cluster-operator.v0.19.0   Automatic   true
install-pcrb9   couchbase-operator.v2.0.1          Automatic   true
install-tchlv   portworx-operator.v1.3.2           Automatic   true
kuiwang@Kuis-MacBook-Pro 1860185 % oc get csv
NAME                               DISPLAY              VERSION   REPLACES   PHASE
couchbase-operator.v2.0.1          Couchbase Operator   2.0.1                Succeeded
strimzi-cluster-operator.v0.19.0   Strimzi              0.19.0               Succeeded



---

Comment 11 Vu Dinh 2020-08-20 14:03:25 UTC
*** Bug 1868253 has been marked as a duplicate of this bug. ***

Comment 16 errata-xmlrpc 2020-10-27 16:16:56 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196

Comment 17 Kevin Rizza 2020-11-09 21:43:55 UTC
*** Bug 1896102 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.