Bug 1869717

Summary:	Operator installation failed - `Subscription` object stays in `UpgradePending` but the `InstallPlan` it associated with isn’t installing
Product:	OpenShift Container Platform	Reporter:	Vu Dinh <vdinh>
Component:	OLM	Assignee:	Nick Hale <nhale>
OLM sub component:	OLM	QA Contact:	yhui
Status:	CLOSED ERRATA	Docs Contact:
Severity:	urgent
Priority:	urgent	CC:	assingh, dmoessne, jiazha, krizza, rkshirsa, vdinh
Version:	4.6
Target Milestone:	---
Target Release:	4.4.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1864121	Environment:
Last Closed:	2020-09-15 17:32:44 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1864121
Bug Blocks:

Comment 1 Kevin Rizza 2020-08-21 18:56:21 UTC

Waiting for 4.5.z bz to be verified, backport pr is up and ready to merge. Marking for upcoming sprint.

Comment 2 Nick Hale 2020-09-08 17:42:35 UTC

The backport PR has merged and is awaiting QE verification.

Jian, could you attempt to verify when you get the chance?

Comment 5 yhui 2020-09-09 09:00:48 UTC

Version:
[hui@localhost verification-tests]$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.0-0.nightly-2020-09-08-111845   True        False         44m     Cluster version is 4.4.0-0.nightly-2020-09-08-111845
[hui@localhost verification-tests]$ oc exec olm-operator-878755cd6-5jlp9 -n openshift-operator-lifecycle-manager -- olm --version
OLM version: 0.14.2
git commit: f2e7b9f63fc54268c4e14ba7900e09f1ebc929d2

Steps to test:
0. Login to OCP web Console and go to 'OperatorHub' page

1. Install Operator A in own NS mode to a NS 
   --> e.g. install Strimzi to NS “tw” (installPlan ‘install-mxvv9’ created)

2. Install Operator B in own NS mode to the same NS 
   --> e.g. install Portworx to NS “tw” (installPlan ‘install-plmll’ created)

[hui@localhost verification-tests]$ oc get sub -n tw
NAME                     PACKAGE                  SOURCE                CHANNEL
portworx-essentials      portworx-essentials      community-operators   stable
strimzi-kafka-operator   strimzi-kafka-operator   community-operators   stable
[hui@localhost verification-tests]$ oc get csv -n tw
NAME                               DISPLAY               VERSION   REPLACES   PHASE
portworx-essentials.v1.3.4         Portworx Essentials   1.3.4                Succeeded
strimzi-cluster-operator.v0.19.0   Strimzi               0.19.0               Succeeded
[hui@localhost verification-tests]$ oc get ip 
NAME            CSV                                APPROVAL    APPROVED
install-5cvwh   strimzi-cluster-operator.v0.19.0   Automatic   true
install-mwhkm   portworx-essentials.v1.3.4         Automatic   true


3. On the 'Installed Operators' page, uninstall Operator B using Console 
   (the Subscription of Operator B has been removed)

[hui@localhost verification-tests]$ oc get sub -n tw
NAME                     PACKAGE                  SOURCE                CHANNEL
strimzi-kafka-operator   strimzi-kafka-operator   community-operators   stable


4. Go back to 'OperatorHub' page, install Operator C in own NS mode to the same NS 
   --> e.g. install Couchbase to NS “tw” 

[hui@localhost verification-tests]$ oc get sub
NAME                             PACKAGE                          SOURCE                CHANNEL
couchbase-enterprise-certified   couchbase-enterprise-certified   certified-operators   stable
strimzi-kafka-operator           strimzi-kafka-operator           community-operators   stable
[hui@localhost verification-tests]$ oc get csv
NAME                               DISPLAY              VERSION   REPLACES   PHASE
couchbase-operator.v2.0.2          Couchbase Operator   2.0.2                Succeeded
strimzi-cluster-operator.v0.19.0   Strimzi              0.19.0               Succeeded
[hui@localhost verification-tests]$ oc get ip
NAME            CSV                                APPROVAL    APPROVED
install-5cvwh   strimzi-cluster-operator.v0.19.0   Automatic   true
install-mwhkm   portworx-essentials.v1.3.4         Automatic   true
install-z48x7   couchbase-operator.v2.0.2          Automatic   true
[hui@localhost verification-tests]$ oc get pods
NAME                                                READY   STATUS    RESTARTS   AGE
couchbase-operator-689d6b4694-wdwlc                 1/1     Running   0          5m23s
strimzi-cluster-operator-v0.19.0-6b99547f97-kggj7   1/1     Running   0          23m


  The new sub  and a new ip are created successfully. And the csv and pod are created successfully.

5. Operator C’s Subscription object is succeeded.
[root@preserve-olm-env foo-operator]# oc get sub couchbase-enterprise-certified -o yaml
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
...
status:
  catalogHealth:
  - catalogSourceRef:
      apiVersion: operators.coreos.com/v1alpha1
      kind: CatalogSource
      name: certified-operators
      namespace: openshift-marketplace
      resourceVersion: "26422"
      uid: adf2e07c-45c3-4961-9a26-d2003648983b
    healthy: true
    lastUpdated: "2020-09-09T08:53:15Z"
  - catalogSourceRef:
      apiVersion: operators.coreos.com/v1alpha1
      kind: CatalogSource
      name: community-operators
      namespace: openshift-marketplace
      resourceVersion: "26421"
      uid: 18d394d0-3cb5-4d85-888a-9b9556b4e375
    healthy: true
    lastUpdated: "2020-09-09T08:53:15Z"
  - catalogSourceRef:
      apiVersion: operators.coreos.com/v1alpha1
      kind: CatalogSource
      name: qe-app-registry
      namespace: openshift-marketplace
      resourceVersion: "26428"
      uid: cffa580f-e2fa-4e3f-9da4-43f72758d002
    healthy: true
    lastUpdated: "2020-09-09T08:53:15Z"
  - catalogSourceRef:
      apiVersion: operators.coreos.com/v1alpha1
      kind: CatalogSource
      name: redhat-marketplace
      namespace: openshift-marketplace
      resourceVersion: "26423"
      uid: 3c9d62c6-be6e-415e-8d6a-603ad881b6d9
    healthy: true
    lastUpdated: "2020-09-09T08:53:15Z"
  - catalogSourceRef:
      apiVersion: operators.coreos.com/v1alpha1
      kind: CatalogSource
      name: redhat-operators
      namespace: openshift-marketplace
      resourceVersion: "26425"
      uid: 05af87c3-a07f-48c8-aaf5-58233e6a2a39
    healthy: true
    lastUpdated: "2020-09-09T08:53:15Z"
  conditions:
  - lastTransitionTime: "2020-09-09T08:53:15Z"
    message: all available catalogsources are healthy
    reason: AllCatalogSourcesHealthy
    status: "False"
    type: CatalogSourcesUnhealthy
  currentCSV: couchbase-operator.v2.0.2
  installPlanGeneration: 3
  installPlanRef:
    apiVersion: operators.coreos.com/v1alpha1
    kind: InstallPlan
    name: install-z48x7
    namespace: tw
    resourceVersion: "35383"
    uid: e22f0226-ef79-4a1e-9c1d-a04332a6e6fb
  installedCSV: couchbase-operator.v2.0.2
  installplan:
    apiVersion: operators.coreos.com/v1alpha1
    kind: InstallPlan
    name: install-z48x7
    uuid: e22f0226-ef79-4a1e-9c1d-a04332a6e6fb
  lastUpdated: "2020-09-09T08:53:20Z"
  state: AtLatestKnown


The bug is verified on 4.4.

Comment 7 errata-xmlrpc 2020-09-15 17:32:44 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.4.21 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:3605