Bug 1840709

Summary: Elasticsearch Operator upgrade to 4.4 is not kicked after subscription channel update.
Product: OpenShift Container Platform Reporter: Nicolas Nosenzo <nnosenzo>
Component: OLMAssignee: Evan Cordell <ecordell>
OLM sub component: OLM QA Contact: Jian Zhang <jiazha>
Status: CLOSED DUPLICATE Docs Contact:
Severity: high    
Priority: high CC: aos-bugs, bsong, ecordell, ikarpukh, jesusr, kpelc, krizza, nhale
Version: 4.4   
Target Milestone: ---   
Target Release: 4.6.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-06-25 14:11:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
deployment info
none
olm-operator logs
none
catalog operator logs
none
csv info
none
CSV and InstallPlans list
none
subscriptions manifests none

Description Nicolas Nosenzo 2020-05-27 13:13:09 UTC
Description of problem:

- Logging upgrade path 4.3 -> 4.4.
- Elasticsearch Operator is not upgraded even though the subscription has been set to the 4.4 channel:

~~~
$ oc get subscriptions
NAME              PACKAGE           SOURCE             CHANNEL
cluster-logging   cluster-logging   redhat-operators   4.4

[nnosenzo@localhost OCP4]$ oc get csv
NAME                                         DISPLAY                          VERSION               REPLACES                     PHASE
clusterlogging.4.4.0-202005121717            Cluster Logging                  4.4.0-202005121717                                 Succeeded
elasticsearch-operator.4.3.19-202005041055   Elasticsearch Operator           4.3.19-202005041055                                Succeeded
[...]
~~~
Version-Release number of selected component (if applicable):
4.4

How reproducible:
100%

Steps to Reproduce:
1. Deploy logging in 4.3
2. Upgrade cluster to latest 4.4
3. Follow the steps in [0] to upgrade Logging

[0] https://docs.openshift.com/container-platform/4.4/logging/cluster-logging-upgrading.html

Actual results:
EO remains in version elasticsearch-operator.4.3.19

Expected results:
EO should upgrade to 4.4

Additional info:

Comment 2 Anping Li 2020-05-28 01:14:06 UTC
Could you provide the output.
#oc version
#oc get packagemanifest elasticsearch-operator -o json
#oc get sub elasticsearch-operator -o json -n openshift-operators-redhat

Comment 3 Nicolas Nosenzo 2020-05-28 08:00:39 UTC
Created attachment 1692962 [details]
deployment info

Just fyi, service mesh was already deployed on this cluster, so the EO Subscription resides within the openshift-operators namespace.

Comment 5 Jeff Cantrill 2020-06-01 15:06:55 UTC
Moving to OLM team to investigate as far as I understand the logging csv et al is correct

Comment 7 Evan Cordell 2020-06-01 17:53:02 UTC
Could you please provide more information about this cluster and the upgrade?

- What is current OCP version? (z-stream)
- Logs of the catalog-operator pod in openshift-operator-lifecycle-manager
- Logs of the olm-operator pod in openshift-operator-lifecycle-manager
- Status of the CSV objects listed (elasticsearch and cluster-logging)

Comment 8 Nicolas Nosenzo 2020-06-02 08:11:46 UTC
Created attachment 1694345 [details]
olm-operator logs


- What is current OCP version? (z-stream)
$ oc version
Client Version: 4.5.0-0.nightly-2020-05-13-092431
Server Version: 4.4.4
Kubernetes Version: v1.17.1

- Logs of the catalog-operator pod in openshift-operator-lifecycle-manager
Attached.

- Logs of the olm-operator pod in openshift-operator-lifecycle-manager
Attached.

- Status of the CSV objects listed (elasticsearch and cluster-logging)
Attached, as well as the objects manifest.

Comment 9 Nicolas Nosenzo 2020-06-02 08:12:33 UTC
Created attachment 1694346 [details]
catalog operator logs

Comment 10 Nicolas Nosenzo 2020-06-02 08:14:49 UTC
Created attachment 1694347 [details]
csv info

Comment 11 Nick Hale 2020-06-02 13:25:44 UTC
Hi Nicolas,

Could you also grab the CSVs and InstallPlans from the openshift-operators namespace?

Thanks

Comment 12 Nicolas Nosenzo 2020-06-02 13:33:23 UTC
Created attachment 1694469 [details]
CSV and InstallPlans list

Hi Nick,
Info attached.

Comment 14 Evan Cordell 2020-06-10 13:23:52 UTC
The logs and objects provided do not indicate any errors.

From the first report:

$ oc get subscriptions
NAME              PACKAGE           SOURCE             CHANNEL
cluster-logging   cluster-logging   redhat-operators   4.4


There should also be a subscription for elasticsearch. Can you find that subscription and ensure that the channel has been set to 4.4 there as well?

If that does not appear to be the issue, can you share the full contents of all subscriptions on the cluster?

Comment 15 Nicolas Nosenzo 2020-06-11 07:11:26 UTC
Evan, as mentioned in the issue description the EO subscription was set to the correct channel, even though the Operator was not upgraded:

$ oc get subscriptions --all-namespaces
NAMESPACE             NAME                     PACKAGE                  SOURCE             CHANNEL
openshift-logging     cluster-logging          cluster-logging          redhat-operators   4.4
openshift-operators   elasticsearch-operator   elasticsearch-operator   redhat-operators   4.4
openshift-operators   jaeger-product           jaeger-product           redhat-operators   stable
openshift-operators   kiali-ossm               kiali-ossm               redhat-operators   stable
openshift-operators   servicemeshoperator      servicemeshoperator      redhat-operators   stable


Attaching all the subscription manifests.

Comment 16 Nicolas Nosenzo 2020-06-11 07:15:03 UTC
Created attachment 1696673 [details]
subscriptions manifests

May the below be the culprit ?

    conditions:
    - lastTransitionTime: "2020-05-21T10:59:07Z"
      message: all available catalogsources are healthy
      reason: AllCatalogSourcesHealthy
      status: "False"
      type: CatalogSourcesUnhealthy