Bug 2183520

Summary: [Fusion-aaS] csi-cephfs-plugin pods are not created after installing ocs-client-operator
Product: [Red Hat Storage] Red Hat OpenShift Data Foundation Reporter: Jilju Joy <jijoy>
Component: ocs-client-operatorAssignee: Ohad <omitrani>
Status: CLOSED ERRATA QA Contact: Neha Berry <nberry>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.13CC: branto, muagarwa, ocs-bugs, odf-bz-bot, omitrani
Target Milestone: ---Keywords: AutomationTriaged, TestBlocker
Target Release: ODF 4.13.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of:
: 2187801 (view as bug list) Environment:
Last Closed: 2023-06-21 15:25:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2187801    

Description Jilju Joy 2023-03-31 13:36:26 UTC
Description of problem:
csi-cephfs-plugin pods are missing in the namespace where the operator "ocs-client-operator" in a consumer cluster.

$ oc describe Daemonset csi-cephfsplugin

.......
Events:
  Type     Reason        Age                 From                  Message
  ----     ------        ----                ----                  -------
  Warning  FailedCreate  29s (x22 over 28m)  daemonset-controller  Error creating: pods "csi-cephfsplugin-" is forbidden: error looking up service account odf-storage/ocs-client-operator-csi-cephfs-plugin-sa: serviceaccount "ocs-client-operator-csi-cephfs-plugin-sa" not found


A service account with the name "ocs-client-operator-csi-cephfs-plugin-sa
" is not given in the csv "ocs-client-operator.v4.13.0-121.stable". 


$ oc get pods -o wide -n odf-storage
NAME                                                      READY   STATUS    RESTARTS   AGE   IP            NODE                                        NOMINATED NODE   READINESS GATES
csi-addons-controller-manager-99d79b666-zp54h             2/2     Running   0          21m   10.128.2.14   ip-10-0-23-174.us-east-2.compute.internal   <none>           <none>
csi-cephfsplugin-provisioner-5bb6cd88-9ftgv               5/5     Running   0          21m   10.128.2.15   ip-10-0-23-174.us-east-2.compute.internal   <none>           <none>
csi-cephfsplugin-provisioner-5bb6cd88-nv7sr               5/5     Running   0          21m   10.130.0.7    ip-10-0-18-234.us-east-2.compute.internal   <none>           <none>
csi-rbdplugin-8ttmz                                       3/3     Running   0          21m   10.0.18.234   ip-10-0-18-234.us-east-2.compute.internal   <none>           <none>
csi-rbdplugin-f7rwh                                       3/3     Running   0          21m   10.0.12.54    ip-10-0-12-54.us-east-2.compute.internal    <none>           <none>
csi-rbdplugin-provisioner-b665ffd5f-87wdj                 5/5     Running   0          21m   10.128.2.16   ip-10-0-23-174.us-east-2.compute.internal   <none>           <none>
csi-rbdplugin-provisioner-b665ffd5f-dptcx                 5/5     Running   0          21m   10.129.2.23   ip-10-0-12-54.us-east-2.compute.internal    <none>           <none>
csi-rbdplugin-v9rm7                                       3/3     Running   0          21m   10.0.23.174   ip-10-0-23-174.us-east-2.compute.internal   <none>           <none>
ocs-client-operator-controller-manager-75d97fb7b5-ptqht   2/2     Running   0          21m   10.128.2.13   ip-10-0-23-174.us-east-2.compute.internal   <none>           <none>


$ oc get csv ocs-client-operator.v4.13.0-121.stable -n odf-storage
NAME                                     DISPLAY                            VERSION             REPLACES   PHASE
ocs-client-operator.v4.13.0-121.stable   OpenShift Data Foundation Client   4.13.0-121.stable              Succeeded



$ oc get Daemonset -n odf-storage 
NAME               DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
csi-cephfsplugin   3         0         0       0            0           <none>          27m
csi-rbdplugin      3         3         3       3            3           <none>          27m


$ oc describe Daemonset csi-cephfsplugin

.......
Events:
  Type     Reason        Age                 From                  Message
  ----     ------        ----                ----                  -------
  Warning  FailedCreate  29s (x22 over 28m)  daemonset-controller  Error creating: pods "csi-cephfsplugin-" is forbidden: error looking up service account odf-storage/ocs-client-operator-csi-cephfs-plugin-sa: serviceaccount "ocs-client-operator-csi-cephfs-plugin-sa" not found



The serviceaccount "ocs-client-operator-csi-cephfs-plugin-sa" is not created.

$ oc get serviceaccount -n odf-storage
NAME                                            SECRETS   AGE
builder                                         1         70m
csi-addons-controller-manager                   1         30m
default                                         1         70m
deployer                                        1         70m
ocs-client-operator-controller-manager          1         30m
ocs-client-operator-csi-cephfs-provisioner-sa   1         30m
ocs-client-operator-csi-rbd-plugin-sa           1         30m
ocs-client-operator-csi-rbd-provisioner-sa      1         30m
ocs-client-operator-status-reporter             1         30m


A service account with the name "ocs-client-operator-csi-cephfs-plugin-sa
" is not given in the csv "ocs-client-operator.v4.13.0-121.stable"

$ oc get csv ocs-client-operator.v4.13.0-121.stable -o yaml | grep ocs-client-operator-csi-cephfs-plugin-sa



$ oc get subscription -n odf-storage
NAME                                                                          PACKAGE                   SOURCE              CHANNEL
ocs-client-operator                                                           ocs-client-operator       ocs-catalogsource   stable-4.13
odf-csi-addons-operator-stable-4.13-ocs-catalogsource-openshift-marketplace   odf-csi-addons-operator   ocs-catalogsource   stable-4.13

ocs-client-operator was installed from UI.


========================================================================

Version-Release number of selected component (if applicable):
$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.12.8    True        False         133m    Cluster version is 4.12.8

$ oc get csv
NAME                                               DISPLAY                            VERSION             REPLACES                                           PHASE
configure-alertmanager-operator.v0.1.518-ec4c3cd   configure-alertmanager-operator    0.1.518-ec4c3cd     configure-alertmanager-operator.v0.1.516-bdea4ea   Succeeded
observability-operator.v0.0.20                     Observability Operator             0.0.20              observability-operator.v0.0.19                     Succeeded
ocs-client-operator.v4.13.0-121.stable             OpenShift Data Foundation Client   4.13.0-121.stable                                                      Succeeded
odf-csi-addons-operator.v4.13.0-121.stable         CSI Addons                         4.13.0-121.stable                                                      Succeeded
route-monitor-operator.v0.1.493-a866e7c            Route Monitor Operator             0.1.493-a866e7c     route-monitor-operator.v0.1.489-7d9fe90            Succeeded


=======================================================================

How reproducible:
1/1

Steps to Reproduce:
1. Create cluster on ROSA and install ocs-client-operator in a namespace. 
eg: odf-storage namespace
[This will be installed in a consumer cluster but the entire provider-consumer configuration on ROSA is not required to reproduce this bug] 

2. Verify the pods in the namespace where the operator ocs-client-operator is installed.

3. Verify that csi-cephfs-plugin pods are present.
===========================================
Actual results:
csi-cephfs-plugin pods are not present. 

Expected results:
csi-cephfs-plugin pods should be present.

Additional info:
Relevant command outputs are attached.

Comment 13 Jilju Joy 2023-04-10 11:50:03 UTC
Verified in version:

$ oc get csv 
NAME                                               DISPLAY                            VERSION             REPLACES                                           PHASE
configure-alertmanager-operator.v0.1.518-ec4c3cd   configure-alertmanager-operator    0.1.518-ec4c3cd     configure-alertmanager-operator.v0.1.516-bdea4ea   Succeeded
observability-operator.v0.0.20                     Observability Operator             0.0.20              observability-operator.v0.0.19                     Succeeded
ocs-client-operator.v4.13.0-130.stable             OpenShift Data Foundation Client   4.13.0-130.stable                                                      Succeeded
odf-csi-addons-operator.v4.13.0-130.stable         CSI Addons                         4.13.0-130.stable                                                      Succeeded
route-monitor-operator.v0.1.493-a866e7c            Route Monitor Operator             0.1.493-a866e7c     route-monitor-operator.v0.1.489-7d9fe90            Succeeded


$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.12.9    True        False         3h22m   Cluster version is 4.12.9

------------------------------------------------------------
csi-cephfsplugin pods are created after installing ocs-client-operator operator.
$ oc get pods -o wide
NAME                                                      READY   STATUS    RESTARTS   AGE   IP            NODE                                        NOMINATED NODE   READINESS GATES
csi-addons-controller-manager-6c54f6d9d8-62gfq            2/2     Running   0          22m   10.128.2.20   ip-10-0-23-171.us-east-2.compute.internal   <none>           <none>
csi-cephfsplugin-kv24q                                    2/2     Running   0          22m   10.0.15.221   ip-10-0-15-221.us-east-2.compute.internal   <none>           <none>
csi-cephfsplugin-provisioner-74dc84865-7z922              5/5     Running   0          22m   10.128.2.23   ip-10-0-23-171.us-east-2.compute.internal   <none>           <none>
csi-cephfsplugin-provisioner-74dc84865-mww7f              5/5     Running   0          22m   10.129.2.17   ip-10-0-16-242.us-east-2.compute.internal   <none>           <none>
csi-cephfsplugin-ps9wg                                    2/2     Running   0          22m   10.0.16.242   ip-10-0-16-242.us-east-2.compute.internal   <none>           <none>
csi-cephfsplugin-xs7fw                                    2/2     Running   0          22m   10.0.23.171   ip-10-0-23-171.us-east-2.compute.internal   <none>           <none>
csi-rbdplugin-cn5jl                                       3/3     Running   0          22m   10.0.23.171   ip-10-0-23-171.us-east-2.compute.internal   <none>           <none>
csi-rbdplugin-ncgmh                                       3/3     Running   0          22m   10.0.16.242   ip-10-0-16-242.us-east-2.compute.internal   <none>           <none>
csi-rbdplugin-provisioner-c94458946-h9qv6                 5/5     Running   0          22m   10.128.2.24   ip-10-0-23-171.us-east-2.compute.internal   <none>           <none>
csi-rbdplugin-provisioner-c94458946-kjw8h                 5/5     Running   0          22m   10.131.0.5    ip-10-0-15-221.us-east-2.compute.internal   <none>           <none>
csi-rbdplugin-wprgs                                       3/3     Running   0          22m   10.0.15.221   ip-10-0-15-221.us-east-2.compute.internal   <none>           <none>
ocs-client-operator-controller-manager-74f9895fb4-5zmx9   2/2     Running   0          22m   10.128.2.22   ip-10-0-23-171.us-east-2.compute.internal   <none>           <none>

Service account ocs-client-operator-csi-cephfs-plugin-sa is present.

$ oc get serviceaccount ocs-client-operator-csi-cephfs-plugin-sa
NAME                                       SECRETS   AGE
ocs-client-operator-csi-cephfs-plugin-sa   1         45m


$ oc get csv ocs-client-operator.v4.13.0-130.stable -o yaml | grep ocs-client-operator-csi-cephfs-plugin-sa
        serviceAccountName: ocs-client-operator-csi-cephfs-plugin-sa
    name: ocs-client-operator-csi-cephfs-plugin-sa

Comment 17 errata-xmlrpc 2023-06-21 15:25:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenShift Data Foundation 4.13.0 enhancement and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:3742