Bug 2187801

Summary: [Fusion-aaS][Backport to 4.12.3] csi-cephfs-plugin pods are not created after installing ocs-client-operator
Product: [Red Hat Storage] Red Hat OpenShift Data Foundation Reporter: Neha Berry <nberry>
Component: ocs-client-operatorAssignee: Madhu Rajanna <mrajanna>
Status: CLOSED CURRENTRELEASE QA Contact: Neha Berry <nberry>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.13CC: branto, jijoy, mrajanna, muagarwa, ocs-bugs, odf-bz-bot, omitrani, rcyriac
Target Milestone: ---Keywords: TestBlocker
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 2183520 Environment:
Last Closed: 2023-04-20 16:33:23 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2183520    
Bug Blocks:    

Description Neha Berry 2023-04-18 17:19:50 UTC 
+++ This bug was initially created as a clone of Bug #2183520 +++

Description of problem:
csi-cephfs-plugin pods are missing in the namespace where the operator "ocs-client-operator" in a consumer cluster.

$ oc describe Daemonset csi-cephfsplugin

.......
Events:
  Type     Reason        Age                 From                  Message
  ----     ------        ----                ----                  -------
  Warning  FailedCreate  29s (x22 over 28m)  daemonset-controller  Error creating: pods "csi-cephfsplugin-" is forbidden: error looking up service account odf-storage/ocs-client-operator-csi-cephfs-plugin-sa: serviceaccount "ocs-client-operator-csi-cephfs-plugin-sa" not found


A service account with the name "ocs-client-operator-csi-cephfs-plugin-sa
" is not given in the csv "ocs-client-operator.v4.13.0-121.stable". 


$ oc get pods -o wide -n odf-storage
NAME                                                      READY   STATUS    RESTARTS   AGE   IP            NODE                                        NOMINATED NODE   READINESS GATES
csi-addons-controller-manager-99d79b666-zp54h             2/2     Running   0          21m   10.128.2.14   ip-10-0-23-174.us-east-2.compute.internal   <none>           <none>
csi-cephfsplugin-provisioner-5bb6cd88-9ftgv               5/5     Running   0          21m   10.128.2.15   ip-10-0-23-174.us-east-2.compute.internal   <none>           <none>
csi-cephfsplugin-provisioner-5bb6cd88-nv7sr               5/5     Running   0          21m   10.130.0.7    ip-10-0-18-234.us-east-2.compute.internal   <none>           <none>
csi-rbdplugin-8ttmz                                       3/3     Running   0          21m   10.0.18.234   ip-10-0-18-234.us-east-2.compute.internal   <none>           <none>
csi-rbdplugin-f7rwh                                       3/3     Running   0          21m   10.0.12.54    ip-10-0-12-54.us-east-2.compute.internal    <none>           <none>
csi-rbdplugin-provisioner-b665ffd5f-87wdj                 5/5     Running   0          21m   10.128.2.16   ip-10-0-23-174.us-east-2.compute.internal   <none>           <none>
csi-rbdplugin-provisioner-b665ffd5f-dptcx                 5/5     Running   0          21m   10.129.2.23   ip-10-0-12-54.us-east-2.compute.internal    <none>           <none>
csi-rbdplugin-v9rm7                                       3/3     Running   0          21m   10.0.23.174   ip-10-0-23-174.us-east-2.compute.internal   <none>           <none>
ocs-client-operator-controller-manager-75d97fb7b5-ptqht   2/2     Running   0          21m   10.128.2.13   ip-10-0-23-174.us-east-2.compute.internal   <none>           <none>


$ oc get csv ocs-client-operator.v4.13.0-121.stable -n odf-storage
NAME                                     DISPLAY                            VERSION             REPLACES   PHASE
ocs-client-operator.v4.13.0-121.stable   OpenShift Data Foundation Client   4.13.0-121.stable              Succeeded



$ oc get Daemonset -n odf-storage 
NAME               DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
csi-cephfsplugin   3         0         0       0            0           <none>          27m
csi-rbdplugin      3         3         3       3            3           <none>          27m


$ oc describe Daemonset csi-cephfsplugin

.......
Events:
  Type     Reason        Age                 From                  Message
  ----     ------        ----                ----                  -------
  Warning  FailedCreate  29s (x22 over 28m)  daemonset-controller  Error creating: pods "csi-cephfsplugin-" is forbidden: error looking up service account odf-storage/ocs-client-operator-csi-cephfs-plugin-sa: serviceaccount "ocs-client-operator-csi-cephfs-plugin-sa" not found



The serviceaccount "ocs-client-operator-csi-cephfs-plugin-sa" is not created.

$ oc get serviceaccount -n odf-storage
NAME                                            SECRETS   AGE
builder                                         1         70m
csi-addons-controller-manager                   1         30m
default                                         1         70m
deployer                                        1         70m
ocs-client-operator-controller-manager          1         30m
ocs-client-operator-csi-cephfs-provisioner-sa   1         30m
ocs-client-operator-csi-rbd-plugin-sa           1         30m
ocs-client-operator-csi-rbd-provisioner-sa      1         30m
ocs-client-operator-status-reporter             1         30m


A service account with the name "ocs-client-operator-csi-cephfs-plugin-sa
" is not given in the csv "ocs-client-operator.v4.13.0-121.stable"

$ oc get csv ocs-client-operator.v4.13.0-121.stable -o yaml | grep ocs-client-operator-csi-cephfs-plugin-sa



$ oc get subscription -n odf-storage
NAME                                                                          PACKAGE                   SOURCE              CHANNEL
ocs-client-operator                                                           ocs-client-operator       ocs-catalogsource   stable-4.13
odf-csi-addons-operator-stable-4.13-ocs-catalogsource-openshift-marketplace   odf-csi-addons-operator   ocs-catalogsource   stable-4.13

ocs-client-operator was installed from UI.


========================================================================

Version-Release number of selected component (if applicable):
$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.12.8    True        False         133m    Cluster version is 4.12.8

$ oc get csv
NAME                                               DISPLAY                            VERSION             REPLACES                                           PHASE
configure-alertmanager-operator.v0.1.518-ec4c3cd   configure-alertmanager-operator    0.1.518-ec4c3cd     configure-alertmanager-operator.v0.1.516-bdea4ea   Succeeded
observability-operator.v0.0.20                     Observability Operator             0.0.20              observability-operator.v0.0.19                     Succeeded
ocs-client-operator.v4.13.0-121.stable             OpenShift Data Foundation Client   4.13.0-121.stable                                                      Succeeded
odf-csi-addons-operator.v4.13.0-121.stable         CSI Addons                         4.13.0-121.stable                                                      Succeeded
route-monitor-operator.v0.1.493-a866e7c            Route Monitor Operator             0.1.493-a866e7c     route-monitor-operator.v0.1.489-7d9fe90            Succeeded


=======================================================================

How reproducible:
1/1

Steps to Reproduce:
1. Create cluster on ROSA and install ocs-client-operator in a namespace. 
eg: odf-storage namespace
[This will be installed in a consumer cluster but the entire provider-consumer configuration on ROSA is not required to reproduce this bug] 

2. Verify the pods in the namespace where the operator ocs-client-operator is installed.

3. Verify that csi-cephfs-plugin pods are present.
===========================================
Actual results:
csi-cephfs-plugin pods are not present. 

Expected results:
csi-cephfs-plugin pods should be present.

Additional info:
Relevant command outputs are attached.

--- Additional comment from Jilju Joy on 2023-03-31 13:36:26 UTC ---

Output of the commands given below are present in the attached file yaml_outputs.txt

oc get csv ocs-client-operator.v4.13.0-121.stable -n odf-storage -o yaml
oc get subscription ocs-client-operator -n odf-storage -o yaml
oc get subscription odf-csi-addons-operator-stable-4.13-ocs-catalogsource-openshift-marketplace -n odf-storage -o yaml
oc get Daemonset csi-cephfsplugin -n odf-storage -o yaml
oc describe Daemonset csi-cephfsplugin -n odf-storage

--- Additional comment from Jilju Joy on 2023-03-31 13:37:46 UTC ---



--- Additional comment from Jilju Joy on 2023-03-31 13:43:21 UTC ---



--- Additional comment from RHEL Program Management on 2023-03-31 14:21:47 UTC ---

This bug having no release flag set previously, is now set with release flag 'odf‑4.13.0' to '?', and so is being proposed to be fixed at the ODF 4.13.0 release. Note that the 3 Acks (pm_ack, devel_ack, qa_ack), if any previously set while release flag was missing, have now been reset since the Acks are to be set against a release flag.

--- Additional comment from RHEL Program Management on 2023-03-31 14:21:47 UTC ---

This bug report has Keywords: Regression or TestBlocker.
Since no regressions or test blockers are allowed between releases, it is also being proposed as a blocker for this release. Please resolve ASAP.

--- Additional comment from Madhu Rajanna on 2023-04-03 07:26:31 UTC ---

Note:-

ocs-client-operator-csi-cephfs-plugin-sa doesn't have any clusterRole and clusterRolebinding associated with it, and we hit this problem only downstream not in upstream.
Not sure what the difference is we have downstream. @Boris are we hitting similar to https://github.com/rook/rook/issues/10141 downstream?

--- Additional comment from Madhu Rajanna on 2023-04-03 08:08:07 UTC ---

Moving the BZ to managed service as we dont have new component for ocs-client-operator

--- Additional comment from Madhu Rajanna on 2023-04-03 08:13:23 UTC ---

removing NI on Boris

--- Additional comment from Madhu Rajanna on 2023-04-04 14:49:07 UTC ---

https://github.com/red-hat-storage/ocs-client-operator/pull/27 is merged should be available in next ODF 4.13 build

--- Additional comment from RHEL Program Management on 2023-04-04 14:52:03 UTC ---

This BZ is being approved for ODF 4.13.0 release, upon receipt of the 3 ACKs (PM,Devel,QA) for the release flag 'odf‑4.13.0

--- Additional comment from RHEL Program Management on 2023-04-04 14:52:03 UTC ---

Since this bug has been approved for ODF 4.13.0 release, through release flag 'odf-4.13.0+', the Target Release is being set to 'ODF 4.13.0

--- Additional comment from Mudit Agarwal on 2023-04-07 05:07:54 UTC ---

Fixed in version: 4.13.0-130

--- Additional comment from Jilju Joy on 2023-04-10 11:50:03 UTC ---

Verified in version:

$ oc get csv 
NAME                                               DISPLAY                            VERSION             REPLACES                                           PHASE
configure-alertmanager-operator.v0.1.518-ec4c3cd   configure-alertmanager-operator    0.1.518-ec4c3cd     configure-alertmanager-operator.v0.1.516-bdea4ea   Succeeded
observability-operator.v0.0.20                     Observability Operator             0.0.20              observability-operator.v0.0.19                     Succeeded
ocs-client-operator.v4.13.0-130.stable             OpenShift Data Foundation Client   4.13.0-130.stable                                                      Succeeded
odf-csi-addons-operator.v4.13.0-130.stable         CSI Addons                         4.13.0-130.stable                                                      Succeeded
route-monitor-operator.v0.1.493-a866e7c            Route Monitor Operator             0.1.493-a866e7c     route-monitor-operator.v0.1.489-7d9fe90            Succeeded


$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.12.9    True        False         3h22m   Cluster version is 4.12.9

------------------------------------------------------------
csi-cephfsplugin pods are created after installing ocs-client-operator operator.
$ oc get pods -o wide
NAME                                                      READY   STATUS    RESTARTS   AGE   IP            NODE                                        NOMINATED NODE   READINESS GATES
csi-addons-controller-manager-6c54f6d9d8-62gfq            2/2     Running   0          22m   10.128.2.20   ip-10-0-23-171.us-east-2.compute.internal   <none>           <none>
csi-cephfsplugin-kv24q                                    2/2     Running   0          22m   10.0.15.221   ip-10-0-15-221.us-east-2.compute.internal   <none>           <none>
csi-cephfsplugin-provisioner-74dc84865-7z922              5/5     Running   0          22m   10.128.2.23   ip-10-0-23-171.us-east-2.compute.internal   <none>           <none>
csi-cephfsplugin-provisioner-74dc84865-mww7f              5/5     Running   0          22m   10.129.2.17   ip-10-0-16-242.us-east-2.compute.internal   <none>           <none>
csi-cephfsplugin-ps9wg                                    2/2     Running   0          22m   10.0.16.242   ip-10-0-16-242.us-east-2.compute.internal   <none>           <none>
csi-cephfsplugin-xs7fw                                    2/2     Running   0          22m   10.0.23.171   ip-10-0-23-171.us-east-2.compute.internal   <none>           <none>
csi-rbdplugin-cn5jl                                       3/3     Running   0          22m   10.0.23.171   ip-10-0-23-171.us-east-2.compute.internal   <none>           <none>
csi-rbdplugin-ncgmh                                       3/3     Running   0          22m   10.0.16.242   ip-10-0-16-242.us-east-2.compute.internal   <none>           <none>
csi-rbdplugin-provisioner-c94458946-h9qv6                 5/5     Running   0          22m   10.128.2.24   ip-10-0-23-171.us-east-2.compute.internal   <none>           <none>
csi-rbdplugin-provisioner-c94458946-kjw8h                 5/5     Running   0          22m   10.131.0.5    ip-10-0-15-221.us-east-2.compute.internal   <none>           <none>
csi-rbdplugin-wprgs                                       3/3     Running   0          22m   10.0.15.221   ip-10-0-15-221.us-east-2.compute.internal   <none>           <none>
ocs-client-operator-controller-manager-74f9895fb4-5zmx9   2/2     Running   0          22m   10.128.2.22   ip-10-0-23-171.us-east-2.compute.internal   <none>           <none>

Service account ocs-client-operator-csi-cephfs-plugin-sa is present.

$ oc get serviceaccount ocs-client-operator-csi-cephfs-plugin-sa
NAME                                       SECRETS   AGE
ocs-client-operator-csi-cephfs-plugin-sa   1         45m


$ oc get csv ocs-client-operator.v4.13.0-130.stable -o yaml | grep ocs-client-operator-csi-cephfs-plugin-sa
        serviceAccountName: ocs-client-operator-csi-cephfs-plugin-sa
    name: ocs-client-operator-csi-cephfs-plugin-sa

--- Additional comment from errata-xmlrpc on 2023-04-17 13:27:26 UTC ---

This bug has been added to advisory RHBA-2023:108078 by Boris Ranto (branto)