Bug 2276565

Summary: [Provider mode] odf-csi-addons-operator did not upgrade automatically after upgrading ocs-client-operator
Product: [Red Hat Storage] Red Hat OpenShift Data Foundation Reporter: Jilju Joy <jijoy>
Component: ocs-client-operatorAssignee: Rohan Gupta <rohgupta>
Status: CLOSED ERRATA QA Contact: Jilju Joy <jijoy>
Severity: high Docs Contact:
Priority: high    
Version: 4.15CC: lgangava, mrajanna, muagarwa, nberry, odf-bz-bot, omitrani, resoni, rohgupta, sheggodu
Target Milestone: ---   
Target Release: ODF 4.15.5   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: isf-provider
Fixed In Version: 4.15.5-5 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2024-07-15 14:30:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jilju Joy 2024-04-23 04:35:26 UTC 
Description of problem:
After upgrading ocs-client-operator from 4.14.6 to 4.15.2, odf-csi-addons-operator  did not upgrade to 4.15.2. odf-csi-addons-operator remained in 4.14.6 version. This problem is in native client in the clients on hosted clusters.

Version-Release number of selected component (if applicable):
$ oc get csv -n openshift-storage-client
NAME                                    DISPLAY                            VERSION               REPLACES                                PHASE
metallb-operator.v4.15.0-202403220332   MetalLB Operator                   4.15.0-202403220332   metallb-operator.v4.14.0-202402161909   Succeeded
ocs-client-operator.v4.15.2-rhodf       OpenShift Data Foundation Client   4.15.2-rhodf          ocs-client-operator.v4.14.6-rhodf       Succeeded
odf-csi-addons-operator.v4.14.6-rhodf   CSI Addons                         4.14.6-rhodf          odf-csi-addons-operator.v4.14.5-rhodf   Succeeded


$ oc get csv -n openshift-storage
NAME                                    DISPLAY                       VERSION               REPLACES                                PHASE
mcg-operator.v4.15.2-rhodf              NooBaa Operator               4.15.2-rhodf          mcg-operator.v4.14.6-rhodf              Succeeded
metallb-operator.v4.15.0-202403220332   MetalLB Operator              4.15.0-202403220332   metallb-operator.v4.14.0-202402161909   Succeeded
ocs-operator.v4.15.2-rhodf              OpenShift Container Storage   4.15.2-rhodf          ocs-operator.v4.14.6-rhodf              Succeeded
odf-csi-addons-operator.v4.15.2-rhodf   CSI Addons                    4.15.2-rhodf          odf-csi-addons-operator.v4.14.6-rhodf   Succeeded
odf-operator.v4.15.2-rhodf              OpenShift Data Foundation     4.15.2-rhodf          odf-operator.v4.14.6-rhodf              Succeeded


$ oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.16.0-0.nightly-2024-04-15-115534   True        False         5d16h   Cluster version is 4.16.0-0.nightly-2024-04-15-115534

Note: OCP was upgraded from 4.14.18 to 4.16 directly to verify OCP upgrade bug https://issues.redhat.com/browse/OCPBUGS-27853. OpenShift Virtualization is 4.15.2-124 because of issues with 4.14 and 4.16 versions. ODF and client upgrade from 4.14.6 to 4.15.2 was done when OCP is at 4.16.
===================================================================
How reproducible:
Issue found in 1 native client and 2 hosted clusters. Provider is same.
=========================================================================
Steps to Reproduce:

1. Create provider -client setup with 1 or more hosted cluster clients. ODF and client version 4.14
2. Upgrade provider cluster to ODF 4.15
3. Verify that the provider is upgraded.
$ oc get csv -n openshift-storage
NAME                                    DISPLAY                       VERSION               REPLACES                                PHASE
mcg-operator.v4.15.2-rhodf              NooBaa Operator               4.15.2-rhodf          mcg-operator.v4.14.6-rhodf              Succeeded
metallb-operator.v4.15.0-202403220332   MetalLB Operator              4.15.0-202403220332   metallb-operator.v4.14.0-202402161909   Succeeded
ocs-operator.v4.15.2-rhodf              OpenShift Container Storage   4.15.2-rhodf          ocs-operator.v4.14.6-rhodf              Succeeded
odf-csi-addons-operator.v4.15.2-rhodf   CSI Addons                    4.15.2-rhodf          odf-csi-addons-operator.v4.14.6-rhodf   Succeeded
odf-operator.v4.15.2-rhodf              OpenShift Data Foundation     4.15.2-rhodf          odf-operator.v4.14.6-rhodf              Succeeded

4. Upgrade native client on provider cluster to 4.15
5. Verify that the client operator is upgraded and the dependency csi-addons is also upgraded to 4.15
$ oc get csv -n openshift-storage-client
NAME                                    DISPLAY                            VERSION               REPLACES                                PHASE
metallb-operator.v4.15.0-202403220332   MetalLB Operator                   4.15.0-202403220332   metallb-operator.v4.14.0-202402161909   Succeeded
ocs-client-operator.v4.15.2-rhodf       OpenShift Data Foundation Client   4.15.2-rhodf          ocs-client-operator.v4.14.6-rhodf       Succeeded
odf-csi-addons-operator.v4.14.6-rhodf   CSI Addons                         4.14.6-rhodf          odf-csi-addons-operator.v4.14.5-rhodf   Succeeded

Repeat step 4 and 5 on clients in hosted clusters.

========================================================
Actual results:
Step 4 and 5 was not successful. odf-csi-addons-operator did not upgrade automatically to 4.15.2.

========================================================
Expected results:
odf-csi-addons-operator should upgrade automatically to the upgraded version (4.15.2 in this case) of ocs-client-operator.


Additional info:
Logs will be shared in the next comment.
Comment 7 Rohan Gupta 2024-05-16 09:44:37 UTC 
It was discussed that we update the subscription of CSI addons in 4.15
Comment 8 Rewant 2024-05-27 11:25:52 UTC 
As we have a bug for it targeted for 4.16: https://bugzilla.redhat.com/show_bug.cgi?id=2280820, moving it for 4.15 and providing dev ack.
Comment 15 errata-xmlrpc 2024-07-15 14:30:27 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenShift Data Foundation 4.15.5 Bug Fix Update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2024:4538