The behavior to update the mon list on the cephcluster exists and is handled inside the product (ocs-operator). 
I don't see a point to have this bug on the odf-managed-service component. 
Can we move it, please?

The bug does not reproduce on another cluster. [same version]
move severity to high

(In reply to Oded from comment #6)
> The bug does not reproduce on another cluster. [same version]
> move severity to high

thanks Oded.

This issue is reproduced again:
SetUp:
ODF

Test Process:
1.Get rook-ceph-mon-endpoints on C1 and C2
C1:
$ oc get cm rook-ceph-mon-endpoints -oyaml -n openshift-storage
apiVersion: v1
data:
  data: b=10.0.141.80:6789

C2:
$ oc get cm rook-ceph-mon-endpoints -oyaml -n openshift-storage
apiVersion: v1
data:
  data: b=10.0.141.80:6789


2.Replace node 10.0.141.80 on provider cluster [ip-10-0-141-80.ec2.internal]
$ oc get pods -o wide | grep mon-b
rook-ceph-mon-b-5b5944464b-fmmcs                                  2/2     Running     0              175m   10.0.141.80    ip-10-0-141-80.ec2.internal    <none>           <none>

3. Create PVC [FS+RBD]
PVC_FS - moved to Bound state
PVC_RBD - Stuck on Pending state

$ oc get pvc -n oded
NAME         STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
simple-pvc   Pending                                                                        ocs-storagecluster-ceph-rbd   32m
test2        Bound     pvc-694766e8-ec4e-4e83-92a7-571a177962a8   2Gi        RWO            ocs-storagecluster-cephfs     4m54s
test3        Pending                                                                        ocs-storagecluster-ceph-rbd   4m8s

Events:
  Type     Reason                Age                     From                                                                                                                Message
  ----     ------                ----                    ----                                                                                                                -------
  Warning  ProvisioningFailed    7m9s                    openshift-storage.rbd.csi.ceph.com_csi-rbdplugin-provisioner-6455fd4867-ccpx5_bcd9ac5d-a6ed-43ae-a908-88e3178a6ff6  failed to provision volume with StorageClass "ocs-storagecluster-ceph-rbd": rpc error: code = DeadlineExceeded desc = context deadline exceeded
  Normal   ExternalProvisioning  3m26s (x26 over 9m39s)  persistentvolume-controller                                                                                         waiting for a volume to be created, either by external provisioner "openshift-storage.rbd.csi.ceph.com" or manually created by system administrator
  Normal   Provisioning          113s (x11 over 9m39s)   openshift-storage.rbd.csi.ceph.com_csi-rbdplugin-provisioner-6455fd4867-ccpx5_bcd9ac5d-a6ed-43ae-a908-88e3178a6ff6  External provisioner is provisioning volume for claim "oded/simple-pvc"
  Warning  ProvisioningFailed    113s (x10 over 7m8s)    openshift-storage.rbd.csi.ceph.com_csi-rbdplugin-provisioner-6455fd4867-ccpx5_bcd9ac5d-a6ed-43ae-a908-88e3178a6ff6  failed to provision volume with StorageClass "ocs-storagecluster-ceph-rbd": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-cf937385-0e7e-447b-b7cb-504588e568a7 already exists


4.Check rook-ceph-mon-endpoints on C1 and C2:

C1:
$ oc get cm rook-ceph-mon-endpoints -oyaml -n openshift-storage
apiVersion: v1
data:
  data: b=10.0.141.80:6789

C2:
$ oc get cm rook-ceph-mon-endpoints -oyaml -n openshift-storage
apiVersion: v1
data:
  data: b=10.0.141.80:6789

5. Run WA on C1: 
a. Respin ocs-operator
observation - rook-ceph-mon-endpoint configmap got updated with new IP but CSI-rook-config stayed unupdated

b. Then respinned the rook-ceph-operator and csi-configmap also got updated with new IPs

7.Check rook-ceph-mon-endpoints on C1 and C2:
$ oc get cm rook-ceph-mon-endpoints -oyaml -n openshift-storage
apiVersion: v1
data:
  data: a=10.0.212.251:6789

$ oc get cm rook-ceph-mon-endpoints -oyaml -n openshift-storage
apiVersion: v1
data:
  data: b=10.0.141.80:6789


8. PVCs on C1 moved to bound state [RBD+FS]
$ oc get pvc -n oded
NAME          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
test55        Bound    pvc-bb290310-1cee-44ae-8a6d-88812020c383   2Gi        RWO            ocs-storagecluster-cephfs     101m
test6         Bound    pvc-118e3689-dc84-4d85-9023-81507a13d578   2Gi        RWO            ocs-storagecluster-ceph-rbd   93m
test9         Bound    pvc-561b2021-54ce-416a-8bda-a3143946d49e   1Gi        RWX            ocs-storagecluster-ceph-rbd   92m


Logs:
Logs from all the 3 clusters - http://rhsqe-repo.lab.eng.blr.redhat.com/OCS/ocs-qe-bugs/PVC-issue-mon-endpoint-2076457/reproduce-node-repl/

zipped:
http://rhsqe-repo.lab.eng.blr.redhat.com/OCS/ocs-qe-bugs/PVC-issue-mon-endpoint-2076457/reproduce-node-repl.zip

Logs from C1 after the Workaround:
http://rhsqe-repo.lab.eng.blr.redhat.com/OCS/ocs-qe-bugs/PVC-issue-mon-endpoint-2076457/reproduce-node-repl/C1-WA-applied-few-logs/

We had a call yesterday b/w me, Seb, Ohad, Parth and we decided to provide a custom build of API server which will return all mon endpoints instead of 1. We verified that the custom build of the API server is working and providing all endpoints. But we are still not able to create a file PVCs on the consumer even though we have all 3 correct endpoints in the `rook-ceph-mon-endpoints` configmap. Ceph Cluster state is also healthy.


Chat thread for more info:
https://chat.google.com/room/AAAASHA9vWs/TJcGGyOuBJU


@shan Can you pls take a look at the setup again?


We also realized that the `rook-ceph-csi-config` configmap still has one old mon endpoints `10.0.141.80` which is not available anymore.


$ oc get cm rook-ceph-csi-config rook-ceph-mon-endpoints -o yaml
apiVersion: v1
items:
- apiVersion: v1
  data:
    csi-cluster-config-json: '[{"clusterID":"b7658893757ca59147f3d74393e36215","monitors":["10.0.141.80:6789"],"cephFS":{"subvolumeGroup":"cephfilesystemsubvolumegroup-storageconsumer-d63b8440-f498-4d72-ae05-1306d6e6cc83"}},{"clusterID":"openshift-storage","monitors":["10.0.144.250:6789","10.0.171.89:6789","10.0.197.112:6789"]}]'
  kind: ConfigMap
  metadata:
    creationTimestamp: "2022-04-20T08:07:59Z"
    name: rook-ceph-csi-config
    namespace: openshift-storage
    ownerReferences:
    - apiVersion: apps/v1
      blockOwnerDeletion: false
      controller: true
      kind: Deployment
      name: rook-ceph-operator
      uid: e3268a48-8f99-45d8-aaee-abc0e268d363
    resourceVersion: "1420666"
    uid: ba502d8e-a5c8-4810-905c-33a1d85adf68
- apiVersion: v1
  data:
    data: c=10.0.171.89:6789,e=10.0.197.112:6789,d=10.0.144.250:6789
    mapping: '{}'
    maxMonId: "0"
  kind: ConfigMap
  metadata:
    creationTimestamp: "2022-04-20T08:07:57Z"
    finalizers:
    - ceph.rook.io/disaster-protection
    name: rook-ceph-mon-endpoints
    namespace: openshift-storage
    ownerReferences:
    - apiVersion: ceph.rook.io/v1
      blockOwnerDeletion: true
      controller: true
      kind: CephCluster
      name: ocs-storagecluster-cephcluster
      uid: c1c61bd0-1da1-48b5-b174-96323a6274d1
    resourceVersion: "1404875"
    uid: 66f3c208-45d2-49bf-919f-7685b43f43fe
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""


@paarora Pls add if I am missing something.

Discussed with Madhu and we need an update to rook and the csi driver to handle the csi configmap differently. The update is tracked here: https://github.com/rook/rook/issues/10126

PR is in review now for updating the mon endpoints when there is a mon failover.

Fix is merged downstream now with https://github.com/red-hat-storage/rook/pull/372

Do we have a bug tracking for backport to 4.10?

Created Clone BZ, removing NI on me.

For all mons IP being not updated in the rook-ceph-mon-endpoints cm 
We need to have a change in OCS-operator, to override it correctly while reconciling
https://github.com/red-hat-storage/ocs-operator/pull/1673

Bug fixed

SetUp Provider:
OCP Version: 4.10.18
ODF Version: 4.11.0-104

SetUp Consumer:
OCP Version: 4.10.18
ODF Version: 4.11.0-105

for more info https://docs.google.com/document/d/11s__tQ_mxSGRl-I4Z6Y4L-lwjPcf1aGv5CtfkQvZXGE/edit

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat OpenShift Data Foundation 4.11.0 security, enhancement, & bugfix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:6156