Bug 2308091 - CephFS networkfence CIDR ip remains in blocklisted state in ceph for long time post unfencing
Summary: CephFS networkfence CIDR ip remains in blocklisted state in ceph for long tim...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenShift Data Foundation
Classification: Red Hat Storage
Component: csi-driver
Version: 4.17
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: ODF 4.17.0
Assignee: Niraj Yadav
QA Contact: krishnaram Karthick
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2024-08-27 07:32 UTC by Joy John Pinto
Modified: 2025-02-28 04:25 UTC (History)
6 users (show)

Fixed In Version: 4.17.0-96
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2024-10-30 14:32:11 UTC
Embargoed:
khiremat: needinfo-


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github red-hat-storage ceph-csi pull 373 0 None open Bug 2308091: [release-4.17] cephfs: Fix Removal of IPs from blocklist 2024-09-09 14:13:42 UTC
Red Hat Issue Tracker OCSBZM-8863 0 None None None 2024-08-27 07:33:08 UTC
Red Hat Product Errata RHSA-2024:8676 0 None None None 2024-10-30 14:32:14 UTC

Description Joy John Pinto 2024-08-27 07:32:40 UTC
Description of problem (please be detailed as possible and provide log
snippests):
CephFS networkfence CIDR ip remains in blocklisted state in ceph for long time post unfencing

Version of all relevant components (if applicable):
OCP 4.17.0-0.nightly-2024-08-19-165854
ODF 4.17.0-80.stable provided by Red Hat


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
NA

Is there any workaround available to the best of your knowledge?
Wait for more than an hour for the ip to get unblocked from ceph osd blocklist ls

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
1

Can this issue reproducible?
Yes

Can this issue reproduce from the UI?
NA

If this is a regression, please provide more details to justify this:
NA

Steps to Reproduce:
1. Install ODF 4.17.0-80
2. Create a deployment pod, on my test setup i created logwriter-ceph pod
3. Shutdown the node on which CephFS RWO pod is deployed
3.Once the node is down, add taint
```oc  taint nodes <node-name> node.kubernetes.io/out-of-service=nodeshutdown:NoExecute ```
Wait for some time(if the application pod and rook operator are on the same node wait for bit logger) then check the networkFence cr status and make sure its state is fenced 
4. Check "ceph osd blocklist ls" to see CIDR ip of the tainted node to be part of the blocklist
4. Un taint the node with command ```oc  taint nodes <node-name> node.kubernetes.io/out-of-service=nodeshutdown:NoExecute- ```
5. The CIDR ip for CephFS does not get unbblocked from "ceph osd blocklist ls" atleast for an hour post unfencing/untainting the node


Actual results:
The CIDR ip for CephFS does not get unbblocked from "ceph osd blocklist ls" atleast for an hour post unfencing/untainting the node


rook ceph operator log with timestamp of unfenced operation:
2024-08-27 05:51:46.897288 I | ceph-cluster-controller: successfully unfenced "cephfs" network fence cr "jopinto-ibm2-bp4nw-worker-2-c2ppg-cephfs-openshift-storage", proceeding with deletion
2024-08-27 05:51:46.909484 I | ceph-cluster-controller: successfully deleted network fence CR jopinto-ibm2-bp4nw-worker-2-c2ppg-cephfs-openshift-storage

ceph osd blocklist ls output:
sh-5.1$ date
Tue Aug 27 07:26:20 UTC 2024
sh-5.1$ ceph osd blocklist ls|grep 100
listed 38 entries
100.64.0.7:0/0 2029-08-26T19:36:37.137293+0000
sh-5.1$ 

Expected results:
The CIDR ip should be unblocked immedietely or within few minutes post unfencing

Additional info:

Comment 16 Sunil Kumar Acharya 2024-09-18 12:06:54 UTC
Please update the RDT flag/text appropriately.

Comment 17 errata-xmlrpc 2024-10-30 14:32:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat OpenShift Data Foundation 4.17.0 Security, Enhancement, & Bug Fix Update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2024:8676

Comment 18 Red Hat Bugzilla 2025-02-28 04:25:28 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days


Note You need to log in before you can comment on or make changes to this bug.