+++ This bug was initially created as a clone of Bug #2183520 +++ Description of problem: csi-cephfs-plugin pods are missing in the namespace where the operator "ocs-client-operator" in a consumer cluster. $ oc describe Daemonset csi-cephfsplugin ....... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedCreate 29s (x22 over 28m) daemonset-controller Error creating: pods "csi-cephfsplugin-" is forbidden: error looking up service account odf-storage/ocs-client-operator-csi-cephfs-plugin-sa: serviceaccount "ocs-client-operator-csi-cephfs-plugin-sa" not found A service account with the name "ocs-client-operator-csi-cephfs-plugin-sa " is not given in the csv "ocs-client-operator.v4.13.0-121.stable". $ oc get pods -o wide -n odf-storage NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES csi-addons-controller-manager-99d79b666-zp54h 2/2 Running 0 21m 10.128.2.14 ip-10-0-23-174.us-east-2.compute.internal <none> <none> csi-cephfsplugin-provisioner-5bb6cd88-9ftgv 5/5 Running 0 21m 10.128.2.15 ip-10-0-23-174.us-east-2.compute.internal <none> <none> csi-cephfsplugin-provisioner-5bb6cd88-nv7sr 5/5 Running 0 21m 10.130.0.7 ip-10-0-18-234.us-east-2.compute.internal <none> <none> csi-rbdplugin-8ttmz 3/3 Running 0 21m 10.0.18.234 ip-10-0-18-234.us-east-2.compute.internal <none> <none> csi-rbdplugin-f7rwh 3/3 Running 0 21m 10.0.12.54 ip-10-0-12-54.us-east-2.compute.internal <none> <none> csi-rbdplugin-provisioner-b665ffd5f-87wdj 5/5 Running 0 21m 10.128.2.16 ip-10-0-23-174.us-east-2.compute.internal <none> <none> csi-rbdplugin-provisioner-b665ffd5f-dptcx 5/5 Running 0 21m 10.129.2.23 ip-10-0-12-54.us-east-2.compute.internal <none> <none> csi-rbdplugin-v9rm7 3/3 Running 0 21m 10.0.23.174 ip-10-0-23-174.us-east-2.compute.internal <none> <none> ocs-client-operator-controller-manager-75d97fb7b5-ptqht 2/2 Running 0 21m 10.128.2.13 ip-10-0-23-174.us-east-2.compute.internal <none> <none> $ oc get csv ocs-client-operator.v4.13.0-121.stable -n odf-storage NAME DISPLAY VERSION REPLACES PHASE ocs-client-operator.v4.13.0-121.stable OpenShift Data Foundation Client 4.13.0-121.stable Succeeded $ oc get Daemonset -n odf-storage NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE csi-cephfsplugin 3 0 0 0 0 <none> 27m csi-rbdplugin 3 3 3 3 3 <none> 27m $ oc describe Daemonset csi-cephfsplugin ....... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedCreate 29s (x22 over 28m) daemonset-controller Error creating: pods "csi-cephfsplugin-" is forbidden: error looking up service account odf-storage/ocs-client-operator-csi-cephfs-plugin-sa: serviceaccount "ocs-client-operator-csi-cephfs-plugin-sa" not found The serviceaccount "ocs-client-operator-csi-cephfs-plugin-sa" is not created. $ oc get serviceaccount -n odf-storage NAME SECRETS AGE builder 1 70m csi-addons-controller-manager 1 30m default 1 70m deployer 1 70m ocs-client-operator-controller-manager 1 30m ocs-client-operator-csi-cephfs-provisioner-sa 1 30m ocs-client-operator-csi-rbd-plugin-sa 1 30m ocs-client-operator-csi-rbd-provisioner-sa 1 30m ocs-client-operator-status-reporter 1 30m A service account with the name "ocs-client-operator-csi-cephfs-plugin-sa " is not given in the csv "ocs-client-operator.v4.13.0-121.stable" $ oc get csv ocs-client-operator.v4.13.0-121.stable -o yaml | grep ocs-client-operator-csi-cephfs-plugin-sa $ oc get subscription -n odf-storage NAME PACKAGE SOURCE CHANNEL ocs-client-operator ocs-client-operator ocs-catalogsource stable-4.13 odf-csi-addons-operator-stable-4.13-ocs-catalogsource-openshift-marketplace odf-csi-addons-operator ocs-catalogsource stable-4.13 ocs-client-operator was installed from UI. ======================================================================== Version-Release number of selected component (if applicable): $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.12.8 True False 133m Cluster version is 4.12.8 $ oc get csv NAME DISPLAY VERSION REPLACES PHASE configure-alertmanager-operator.v0.1.518-ec4c3cd configure-alertmanager-operator 0.1.518-ec4c3cd configure-alertmanager-operator.v0.1.516-bdea4ea Succeeded observability-operator.v0.0.20 Observability Operator 0.0.20 observability-operator.v0.0.19 Succeeded ocs-client-operator.v4.13.0-121.stable OpenShift Data Foundation Client 4.13.0-121.stable Succeeded odf-csi-addons-operator.v4.13.0-121.stable CSI Addons 4.13.0-121.stable Succeeded route-monitor-operator.v0.1.493-a866e7c Route Monitor Operator 0.1.493-a866e7c route-monitor-operator.v0.1.489-7d9fe90 Succeeded ======================================================================= How reproducible: 1/1 Steps to Reproduce: 1. Create cluster on ROSA and install ocs-client-operator in a namespace. eg: odf-storage namespace [This will be installed in a consumer cluster but the entire provider-consumer configuration on ROSA is not required to reproduce this bug] 2. Verify the pods in the namespace where the operator ocs-client-operator is installed. 3. Verify that csi-cephfs-plugin pods are present. =========================================== Actual results: csi-cephfs-plugin pods are not present. Expected results: csi-cephfs-plugin pods should be present. Additional info: Relevant command outputs are attached. --- Additional comment from Jilju Joy on 2023-03-31 13:36:26 UTC --- Output of the commands given below are present in the attached file yaml_outputs.txt oc get csv ocs-client-operator.v4.13.0-121.stable -n odf-storage -o yaml oc get subscription ocs-client-operator -n odf-storage -o yaml oc get subscription odf-csi-addons-operator-stable-4.13-ocs-catalogsource-openshift-marketplace -n odf-storage -o yaml oc get Daemonset csi-cephfsplugin -n odf-storage -o yaml oc describe Daemonset csi-cephfsplugin -n odf-storage --- Additional comment from Jilju Joy on 2023-03-31 13:37:46 UTC --- --- Additional comment from Jilju Joy on 2023-03-31 13:43:21 UTC --- --- Additional comment from RHEL Program Management on 2023-03-31 14:21:47 UTC --- This bug having no release flag set previously, is now set with release flag 'odf‑4.13.0' to '?', and so is being proposed to be fixed at the ODF 4.13.0 release. Note that the 3 Acks (pm_ack, devel_ack, qa_ack), if any previously set while release flag was missing, have now been reset since the Acks are to be set against a release flag. --- Additional comment from RHEL Program Management on 2023-03-31 14:21:47 UTC --- This bug report has Keywords: Regression or TestBlocker. Since no regressions or test blockers are allowed between releases, it is also being proposed as a blocker for this release. Please resolve ASAP. --- Additional comment from Madhu Rajanna on 2023-04-03 07:26:31 UTC --- Note:- ocs-client-operator-csi-cephfs-plugin-sa doesn't have any clusterRole and clusterRolebinding associated with it, and we hit this problem only downstream not in upstream. Not sure what the difference is we have downstream. @Boris are we hitting similar to https://github.com/rook/rook/issues/10141 downstream? --- Additional comment from Madhu Rajanna on 2023-04-03 08:08:07 UTC --- Moving the BZ to managed service as we dont have new component for ocs-client-operator --- Additional comment from Madhu Rajanna on 2023-04-03 08:13:23 UTC --- removing NI on Boris --- Additional comment from Madhu Rajanna on 2023-04-04 14:49:07 UTC --- https://github.com/red-hat-storage/ocs-client-operator/pull/27 is merged should be available in next ODF 4.13 build --- Additional comment from RHEL Program Management on 2023-04-04 14:52:03 UTC --- This BZ is being approved for ODF 4.13.0 release, upon receipt of the 3 ACKs (PM,Devel,QA) for the release flag 'odf‑4.13.0 --- Additional comment from RHEL Program Management on 2023-04-04 14:52:03 UTC --- Since this bug has been approved for ODF 4.13.0 release, through release flag 'odf-4.13.0+', the Target Release is being set to 'ODF 4.13.0 --- Additional comment from Mudit Agarwal on 2023-04-07 05:07:54 UTC --- Fixed in version: 4.13.0-130 --- Additional comment from Jilju Joy on 2023-04-10 11:50:03 UTC --- Verified in version: $ oc get csv NAME DISPLAY VERSION REPLACES PHASE configure-alertmanager-operator.v0.1.518-ec4c3cd configure-alertmanager-operator 0.1.518-ec4c3cd configure-alertmanager-operator.v0.1.516-bdea4ea Succeeded observability-operator.v0.0.20 Observability Operator 0.0.20 observability-operator.v0.0.19 Succeeded ocs-client-operator.v4.13.0-130.stable OpenShift Data Foundation Client 4.13.0-130.stable Succeeded odf-csi-addons-operator.v4.13.0-130.stable CSI Addons 4.13.0-130.stable Succeeded route-monitor-operator.v0.1.493-a866e7c Route Monitor Operator 0.1.493-a866e7c route-monitor-operator.v0.1.489-7d9fe90 Succeeded $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.12.9 True False 3h22m Cluster version is 4.12.9 ------------------------------------------------------------ csi-cephfsplugin pods are created after installing ocs-client-operator operator. $ oc get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES csi-addons-controller-manager-6c54f6d9d8-62gfq 2/2 Running 0 22m 10.128.2.20 ip-10-0-23-171.us-east-2.compute.internal <none> <none> csi-cephfsplugin-kv24q 2/2 Running 0 22m 10.0.15.221 ip-10-0-15-221.us-east-2.compute.internal <none> <none> csi-cephfsplugin-provisioner-74dc84865-7z922 5/5 Running 0 22m 10.128.2.23 ip-10-0-23-171.us-east-2.compute.internal <none> <none> csi-cephfsplugin-provisioner-74dc84865-mww7f 5/5 Running 0 22m 10.129.2.17 ip-10-0-16-242.us-east-2.compute.internal <none> <none> csi-cephfsplugin-ps9wg 2/2 Running 0 22m 10.0.16.242 ip-10-0-16-242.us-east-2.compute.internal <none> <none> csi-cephfsplugin-xs7fw 2/2 Running 0 22m 10.0.23.171 ip-10-0-23-171.us-east-2.compute.internal <none> <none> csi-rbdplugin-cn5jl 3/3 Running 0 22m 10.0.23.171 ip-10-0-23-171.us-east-2.compute.internal <none> <none> csi-rbdplugin-ncgmh 3/3 Running 0 22m 10.0.16.242 ip-10-0-16-242.us-east-2.compute.internal <none> <none> csi-rbdplugin-provisioner-c94458946-h9qv6 5/5 Running 0 22m 10.128.2.24 ip-10-0-23-171.us-east-2.compute.internal <none> <none> csi-rbdplugin-provisioner-c94458946-kjw8h 5/5 Running 0 22m 10.131.0.5 ip-10-0-15-221.us-east-2.compute.internal <none> <none> csi-rbdplugin-wprgs 3/3 Running 0 22m 10.0.15.221 ip-10-0-15-221.us-east-2.compute.internal <none> <none> ocs-client-operator-controller-manager-74f9895fb4-5zmx9 2/2 Running 0 22m 10.128.2.22 ip-10-0-23-171.us-east-2.compute.internal <none> <none> Service account ocs-client-operator-csi-cephfs-plugin-sa is present. $ oc get serviceaccount ocs-client-operator-csi-cephfs-plugin-sa NAME SECRETS AGE ocs-client-operator-csi-cephfs-plugin-sa 1 45m $ oc get csv ocs-client-operator.v4.13.0-130.stable -o yaml | grep ocs-client-operator-csi-cephfs-plugin-sa serviceAccountName: ocs-client-operator-csi-cephfs-plugin-sa name: ocs-client-operator-csi-cephfs-plugin-sa --- Additional comment from errata-xmlrpc on 2023-04-17 13:27:26 UTC --- This bug has been added to advisory RHBA-2023:108078 by Boris Ranto (branto)