Bug 2308091

Description of problem (please be detailed as possible and provide log
snippests):
CephFS networkfence CIDR ip remains in blocklisted state in ceph for long time post unfencing

Version of all relevant components (if applicable):
OCP 4.17.0-0.nightly-2024-08-19-165854
ODF 4.17.0-80.stable provided by Red Hat


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
NA

Is there any workaround available to the best of your knowledge?
Wait for more than an hour for the ip to get unblocked from ceph osd blocklist ls

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
1

Can this issue reproducible?
Yes

Can this issue reproduce from the UI?
NA

If this is a regression, please provide more details to justify this:
NA

Steps to Reproduce:
1. Install ODF 4.17.0-80
2. Create a deployment pod, on my test setup i created logwriter-ceph pod
3. Shutdown the node on which CephFS RWO pod is deployed
3.Once the node is down, add taint
```oc  taint nodes <node-name> node.kubernetes.io/out-of-service=nodeshutdown:NoExecute ```
Wait for some time(if the application pod and rook operator are on the same node wait for bit logger) then check the networkFence cr status and make sure its state is fenced 
4. Check "ceph osd blocklist ls" to see CIDR ip of the tainted node to be part of the blocklist
4. Un taint the node with command ```oc  taint nodes <node-name> node.kubernetes.io/out-of-service=nodeshutdown:NoExecute- ```
5. The CIDR ip for CephFS does not get unbblocked from "ceph osd blocklist ls" atleast for an hour post unfencing/untainting the node


Actual results:
The CIDR ip for CephFS does not get unbblocked from "ceph osd blocklist ls" atleast for an hour post unfencing/untainting the node


rook ceph operator log with timestamp of unfenced operation:
2024-08-27 05:51:46.897288 I | ceph-cluster-controller: successfully unfenced "cephfs" network fence cr "jopinto-ibm2-bp4nw-worker-2-c2ppg-cephfs-openshift-storage", proceeding with deletion
2024-08-27 05:51:46.909484 I | ceph-cluster-controller: successfully deleted network fence CR jopinto-ibm2-bp4nw-worker-2-c2ppg-cephfs-openshift-storage

ceph osd blocklist ls output:
sh-5.1$ date
Tue Aug 27 07:26:20 UTC 2024
sh-5.1$ ceph osd blocklist ls|grep 100
listed 38 entries
100.64.0.7:0/0 2029-08-26T19:36:37.137293+0000
sh-5.1$ 

Expected results:
The CIDR ip should be unblocked immedietely or within few minutes post unfencing

Additional info: