Bug 2183520 - [Fusion-aaS] csi-cephfs-plugin pods are not created after installing ocs-client-operator
Summary: [Fusion-aaS] csi-cephfs-plugin pods are not created after installing ocs-clie...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenShift Data Foundation
Classification: Red Hat Storage
Component: ocs-client-operator
Version: 4.13
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: ODF 4.13.0
Assignee: Ohad
QA Contact: Neha Berry
URL:
Whiteboard:
Depends On:
Blocks: 2187801
TreeView+ depends on / blocked
 
Reported: 2023-03-31 13:36 UTC by Jilju Joy
Modified: 2023-08-09 17:00 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
: 2187801 (view as bug list)
Environment:
Last Closed: 2023-06-21 15:25:02 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github red-hat-storage ocs-client-operator pull 27 0 None open Fix cephfs-plugin service acccount creation in CSV 2023-04-03 08:08:37 UTC
Red Hat Product Errata RHBA-2023:3742 0 None None None 2023-06-21 15:25:32 UTC

Description Jilju Joy 2023-03-31 13:36:26 UTC
Description of problem:
csi-cephfs-plugin pods are missing in the namespace where the operator "ocs-client-operator" in a consumer cluster.

$ oc describe Daemonset csi-cephfsplugin

.......
Events:
  Type     Reason        Age                 From                  Message
  ----     ------        ----                ----                  -------
  Warning  FailedCreate  29s (x22 over 28m)  daemonset-controller  Error creating: pods "csi-cephfsplugin-" is forbidden: error looking up service account odf-storage/ocs-client-operator-csi-cephfs-plugin-sa: serviceaccount "ocs-client-operator-csi-cephfs-plugin-sa" not found


A service account with the name "ocs-client-operator-csi-cephfs-plugin-sa
" is not given in the csv "ocs-client-operator.v4.13.0-121.stable". 


$ oc get pods -o wide -n odf-storage
NAME                                                      READY   STATUS    RESTARTS   AGE   IP            NODE                                        NOMINATED NODE   READINESS GATES
csi-addons-controller-manager-99d79b666-zp54h             2/2     Running   0          21m   10.128.2.14   ip-10-0-23-174.us-east-2.compute.internal   <none>           <none>
csi-cephfsplugin-provisioner-5bb6cd88-9ftgv               5/5     Running   0          21m   10.128.2.15   ip-10-0-23-174.us-east-2.compute.internal   <none>           <none>
csi-cephfsplugin-provisioner-5bb6cd88-nv7sr               5/5     Running   0          21m   10.130.0.7    ip-10-0-18-234.us-east-2.compute.internal   <none>           <none>
csi-rbdplugin-8ttmz                                       3/3     Running   0          21m   10.0.18.234   ip-10-0-18-234.us-east-2.compute.internal   <none>           <none>
csi-rbdplugin-f7rwh                                       3/3     Running   0          21m   10.0.12.54    ip-10-0-12-54.us-east-2.compute.internal    <none>           <none>
csi-rbdplugin-provisioner-b665ffd5f-87wdj                 5/5     Running   0          21m   10.128.2.16   ip-10-0-23-174.us-east-2.compute.internal   <none>           <none>
csi-rbdplugin-provisioner-b665ffd5f-dptcx                 5/5     Running   0          21m   10.129.2.23   ip-10-0-12-54.us-east-2.compute.internal    <none>           <none>
csi-rbdplugin-v9rm7                                       3/3     Running   0          21m   10.0.23.174   ip-10-0-23-174.us-east-2.compute.internal   <none>           <none>
ocs-client-operator-controller-manager-75d97fb7b5-ptqht   2/2     Running   0          21m   10.128.2.13   ip-10-0-23-174.us-east-2.compute.internal   <none>           <none>


$ oc get csv ocs-client-operator.v4.13.0-121.stable -n odf-storage
NAME                                     DISPLAY                            VERSION             REPLACES   PHASE
ocs-client-operator.v4.13.0-121.stable   OpenShift Data Foundation Client   4.13.0-121.stable              Succeeded



$ oc get Daemonset -n odf-storage 
NAME               DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
csi-cephfsplugin   3         0         0       0            0           <none>          27m
csi-rbdplugin      3         3         3       3            3           <none>          27m


$ oc describe Daemonset csi-cephfsplugin

.......
Events:
  Type     Reason        Age                 From                  Message
  ----     ------        ----                ----                  -------
  Warning  FailedCreate  29s (x22 over 28m)  daemonset-controller  Error creating: pods "csi-cephfsplugin-" is forbidden: error looking up service account odf-storage/ocs-client-operator-csi-cephfs-plugin-sa: serviceaccount "ocs-client-operator-csi-cephfs-plugin-sa" not found



The serviceaccount "ocs-client-operator-csi-cephfs-plugin-sa" is not created.

$ oc get serviceaccount -n odf-storage
NAME                                            SECRETS   AGE
builder                                         1         70m
csi-addons-controller-manager                   1         30m
default                                         1         70m
deployer                                        1         70m
ocs-client-operator-controller-manager          1         30m
ocs-client-operator-csi-cephfs-provisioner-sa   1         30m
ocs-client-operator-csi-rbd-plugin-sa           1         30m
ocs-client-operator-csi-rbd-provisioner-sa      1         30m
ocs-client-operator-status-reporter             1         30m


A service account with the name "ocs-client-operator-csi-cephfs-plugin-sa
" is not given in the csv "ocs-client-operator.v4.13.0-121.stable"

$ oc get csv ocs-client-operator.v4.13.0-121.stable -o yaml | grep ocs-client-operator-csi-cephfs-plugin-sa



$ oc get subscription -n odf-storage
NAME                                                                          PACKAGE                   SOURCE              CHANNEL
ocs-client-operator                                                           ocs-client-operator       ocs-catalogsource   stable-4.13
odf-csi-addons-operator-stable-4.13-ocs-catalogsource-openshift-marketplace   odf-csi-addons-operator   ocs-catalogsource   stable-4.13

ocs-client-operator was installed from UI.


========================================================================

Version-Release number of selected component (if applicable):
$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.12.8    True        False         133m    Cluster version is 4.12.8

$ oc get csv
NAME                                               DISPLAY                            VERSION             REPLACES                                           PHASE
configure-alertmanager-operator.v0.1.518-ec4c3cd   configure-alertmanager-operator    0.1.518-ec4c3cd     configure-alertmanager-operator.v0.1.516-bdea4ea   Succeeded
observability-operator.v0.0.20                     Observability Operator             0.0.20              observability-operator.v0.0.19                     Succeeded
ocs-client-operator.v4.13.0-121.stable             OpenShift Data Foundation Client   4.13.0-121.stable                                                      Succeeded
odf-csi-addons-operator.v4.13.0-121.stable         CSI Addons                         4.13.0-121.stable                                                      Succeeded
route-monitor-operator.v0.1.493-a866e7c            Route Monitor Operator             0.1.493-a866e7c     route-monitor-operator.v0.1.489-7d9fe90            Succeeded


=======================================================================

How reproducible:
1/1

Steps to Reproduce:
1. Create cluster on ROSA and install ocs-client-operator in a namespace. 
eg: odf-storage namespace
[This will be installed in a consumer cluster but the entire provider-consumer configuration on ROSA is not required to reproduce this bug] 

2. Verify the pods in the namespace where the operator ocs-client-operator is installed.

3. Verify that csi-cephfs-plugin pods are present.
===========================================
Actual results:
csi-cephfs-plugin pods are not present. 

Expected results:
csi-cephfs-plugin pods should be present.

Additional info:
Relevant command outputs are attached.

Comment 13 Jilju Joy 2023-04-10 11:50:03 UTC
Verified in version:

$ oc get csv 
NAME                                               DISPLAY                            VERSION             REPLACES                                           PHASE
configure-alertmanager-operator.v0.1.518-ec4c3cd   configure-alertmanager-operator    0.1.518-ec4c3cd     configure-alertmanager-operator.v0.1.516-bdea4ea   Succeeded
observability-operator.v0.0.20                     Observability Operator             0.0.20              observability-operator.v0.0.19                     Succeeded
ocs-client-operator.v4.13.0-130.stable             OpenShift Data Foundation Client   4.13.0-130.stable                                                      Succeeded
odf-csi-addons-operator.v4.13.0-130.stable         CSI Addons                         4.13.0-130.stable                                                      Succeeded
route-monitor-operator.v0.1.493-a866e7c            Route Monitor Operator             0.1.493-a866e7c     route-monitor-operator.v0.1.489-7d9fe90            Succeeded


$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.12.9    True        False         3h22m   Cluster version is 4.12.9

------------------------------------------------------------
csi-cephfsplugin pods are created after installing ocs-client-operator operator.
$ oc get pods -o wide
NAME                                                      READY   STATUS    RESTARTS   AGE   IP            NODE                                        NOMINATED NODE   READINESS GATES
csi-addons-controller-manager-6c54f6d9d8-62gfq            2/2     Running   0          22m   10.128.2.20   ip-10-0-23-171.us-east-2.compute.internal   <none>           <none>
csi-cephfsplugin-kv24q                                    2/2     Running   0          22m   10.0.15.221   ip-10-0-15-221.us-east-2.compute.internal   <none>           <none>
csi-cephfsplugin-provisioner-74dc84865-7z922              5/5     Running   0          22m   10.128.2.23   ip-10-0-23-171.us-east-2.compute.internal   <none>           <none>
csi-cephfsplugin-provisioner-74dc84865-mww7f              5/5     Running   0          22m   10.129.2.17   ip-10-0-16-242.us-east-2.compute.internal   <none>           <none>
csi-cephfsplugin-ps9wg                                    2/2     Running   0          22m   10.0.16.242   ip-10-0-16-242.us-east-2.compute.internal   <none>           <none>
csi-cephfsplugin-xs7fw                                    2/2     Running   0          22m   10.0.23.171   ip-10-0-23-171.us-east-2.compute.internal   <none>           <none>
csi-rbdplugin-cn5jl                                       3/3     Running   0          22m   10.0.23.171   ip-10-0-23-171.us-east-2.compute.internal   <none>           <none>
csi-rbdplugin-ncgmh                                       3/3     Running   0          22m   10.0.16.242   ip-10-0-16-242.us-east-2.compute.internal   <none>           <none>
csi-rbdplugin-provisioner-c94458946-h9qv6                 5/5     Running   0          22m   10.128.2.24   ip-10-0-23-171.us-east-2.compute.internal   <none>           <none>
csi-rbdplugin-provisioner-c94458946-kjw8h                 5/5     Running   0          22m   10.131.0.5    ip-10-0-15-221.us-east-2.compute.internal   <none>           <none>
csi-rbdplugin-wprgs                                       3/3     Running   0          22m   10.0.15.221   ip-10-0-15-221.us-east-2.compute.internal   <none>           <none>
ocs-client-operator-controller-manager-74f9895fb4-5zmx9   2/2     Running   0          22m   10.128.2.22   ip-10-0-23-171.us-east-2.compute.internal   <none>           <none>

Service account ocs-client-operator-csi-cephfs-plugin-sa is present.

$ oc get serviceaccount ocs-client-operator-csi-cephfs-plugin-sa
NAME                                       SECRETS   AGE
ocs-client-operator-csi-cephfs-plugin-sa   1         45m


$ oc get csv ocs-client-operator.v4.13.0-130.stable -o yaml | grep ocs-client-operator-csi-cephfs-plugin-sa
        serviceAccountName: ocs-client-operator-csi-cephfs-plugin-sa
    name: ocs-client-operator-csi-cephfs-plugin-sa

Comment 17 errata-xmlrpc 2023-06-21 15:25:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenShift Data Foundation 4.13.0 enhancement and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:3742


Note You need to log in before you can comment on or make changes to this bug.