Bug 1816611 - elasticsearch-operator issues upgrading to a new version
Summary: elasticsearch-operator issues upgrading to a new version
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 4.2.z
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.5.0
Assignee: Jeff Cantrill
QA Contact: Anping Li
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-03-24 11:24 UTC by oarribas
Modified: 2023-10-06 19:29 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-08-04 18:06:50 UTC
Target Upstream Version:
Embargoed:
jcantril: needinfo-


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift elasticsearch-operator issues 203 0 None closed Upgrade stuck 2021-02-18 17:15:00 UTC
Red Hat Issue Tracker MAISTRA-1044 0 Major Closed elasticsearch-operator failed in automatic upgrade to elasticsearch-operator.4.2.4-201911050122 2020-12-17 01:33:07 UTC
Red Hat Product Errata RHBA-2020:2409 0 None None None 2020-08-04 18:06:51 UTC

Description oarribas 2020-03-24 11:24:17 UTC
Description of problem:

In a Service Mesh operators installation, the update to a newer version of the elasticsearch-operator is not working.

When the update is applied, the new version fails with error "Failed: conflicting CRD owner in namespace message".

A workaround for that is to remove the old CSV. Doing that, the new CSV is installed, but the deployments (and pods) images are not updated to the new ones.

Also, in some namespaces, both versions of the CSV still installed/copied



Version-Release number of selected component (if applicable):
OCP 4.2.18
elasticsearch-operator 4.2.15-202001171551 and 4.2.21-202002240343, but other versions have the same behaviour


How reproducible:


Steps to Reproduce:

1. Install an old version of elasticsearch-operator (in my case, as part of the ServiceMesh installation), with installPlanApproval set to manual. Use a subscription similar to:

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:

  name: elasticsearch
  namespace: openshift-operators
spec:
  channel: "4.2"
  installPlanApproval: Manual
  name: elasticsearch-operator
  source: ${SOURCE}
  sourceNamespace: openshift-operators
  startingCSV: elasticsearch-operator.4.2.15-202001171551


2. Install the other ServiceMesh operators.


3. Create a ServiceMeshControlPlane in namespace istio-system.


4. Check the images in the elasticsearch deployments:

  oc get deployment elasticsearch-cdm-istiosystemjaeger-1 -n istio-system -o yaml | grep "image:"


5. Accept the installPlan to update the elasticsearch-operator


6. Check the upgrade status

  oc get csv -n openshift-operators


7. Remove the previous csv

  oc delete csv elasticsearch-operator.4.2.15-202001171551 -n openshift-operators


8. Wait for the new csv installed

  oc get csv -n openshift-operators


9. Check the elasticsearch deployment image

  oc get deployment elasticsearch-cdm-istiosystemjaeger-1 -n istio-system -o yaml | grep "image:"


10. Check if the older operator remains in some namespace:

  oc get csv --all-namespaces | grep "elasticsearch-operator.4.2.15"



Actual results:

It's needed to remove the previous CSV from the installed namespace (openshift-operators in the example).


The old operator version and the new one appears as "Success" in some namespaces.

    oc get csv --all-namespaces | grep "elasticsearch-operator.4.2.15"                                                                   
    caas-monitoring                                         elasticsearch-operator.4.2.15-202001171551   Elasticsearch Operator           4.2.15-202001171551                                        Succeeded
    openshift-logging                                       elasticsearch-operator.4.2.15-202001171551   Elasticsearch Operator           4.2.15-202001171551                                        Succeeded
    openshift-monitoring                                    elasticsearch-operator.4.2.15-202001171551   Elasticsearch Operator           4.2.15-202001171551                                        Succeeded
    openshift-operator-lifecycle-manager                    elasticsearch-operator.4.2.15-202001171551   Elasticsearch Operator           4.2.15-202001171551                                        Succeeded


The image used in the deployment is the same before and after the update:

  registry.redhat.io/openshift4/ose-logging-elasticsearch5@sha256:4e32ebcc73eb83e5a55df258df81ccac44dd564fdb1a815fccff80caad21b3d8



Expected results:

Operator installed without manually delete the old CSV.

Only new version of the operator in all the namespaces.

Deployment (and pods) updated with the new image.

  registry.redhat.io/openshift4/ose-logging-elasticsearch5@sha256:f4c62752ce87d358c9084a2a4e3d39e3ca7184ac6a1d27fb4c24b8fa268eb6de

Comment 1 Kevin Rizza 2020-03-24 17:05:19 UTC
I want to ask a few clarifying questions here:

This step is confusing to me: "1. Install an old version of elasticsearch-operator (in my case, as part of the ServiceMesh installation), with installPlanApproval set to manual. Use a subscription similar to:"

Are you saying that the elasticsearch-operator is installed due to a dependency defined in the ServiceMesh operator's CSV? And then separately you are creating a subscription to the elasticsearch-operator to install a specific version?

FYI this is not possible, the error you are seeing is (if my assumptions are true from what I read above) because OLM can't create two separate subscriptions to install the same package in the same namespace. The CRD conflict exists because both installs are attempting to take ownership over the CRDs in the same namespace and OLM doesn't allow it. There isn't currently a concept of configuring a dependency at this level, you would need to set the subscription of the parent operator to manual and then allow each individual installplan for the dependencies.

Comment 2 oarribas 2020-03-24 18:01:35 UTC
Hi Kevin,

Sorry, I will try to explain it better.

I'm installing all the components of ServiceMesh using subscriptions, because I need to install a specific version of each one, and don't want automatic updates. I think it's not possible to install an older version from the console. So only one elasticsearch-operator is installed.

The "conflicting CRD owner" error appears when try to update the elasticsearch-operator, approving the installplan for a newer version. 

You can see more info in https://issues.redhat.com/browse/MAISTRA-1044 and in https://github.com/openshift/elasticsearch-operator/issues/203


When using the workaround of deleting the old CSV, the new version of the elasticsearch-operator is installed, but the images in the deployments are not updated.
Also, in some (few) namespaces, the old CSV remains copied.

Comment 7 Jeff Cantrill 2020-04-14 19:01:07 UTC
@Gary,

Assigning to myself and looking for you to provide assistance to validate the modification to service mesh due April 7 will resolve this.  We do not have any testing around Service Mesh

Comment 16 errata-xmlrpc 2020-08-04 18:06:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.5 image release advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409


Note You need to log in before you can comment on or make changes to this bug.