Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Created attachment 1695447[details]
'pcs cluster report' taken after the problem occured
Description of problem:
We have automated test that does the following:
1) install rhel
2) configure wwo node cluster with fence_scsi and auto tie-breaker
3) cause network split using iptables
4) second node is evicted from the cluster and have its scsi key removed
5) verify that second node cannot write to shared storage
6) second is rebooted manually
7) wait for second node to rejoin cluster
8) verify that the second node keys is re-added to shared storage
9) verify that second node can write again to the shared storage
In rare case the node it happens that victim node unfencing fails it's scsi key
is not added to the shared disk and it is unable to to write to shared storage
even though it makes part of the cluster.
The problem happens in about one run out of hundred tests.
Version-Release number of selected component (if applicable):
pacemaker-2.0.3-5.el8_2.1.x86_64
corosync-3.0.3-2.el8.x86_64
fence-agents-all-4.2.1-41.el8.x86_64
How reproducible: very rare, ~2% of runs
Steps to Reproduce:
1. see description of the problem
Actual results: node not unfenced after it rejoins the cluster
Expected results: node unfenced after re-joining the cluster after being evicted
Additional info:
I never saw the problem happening on cluster start it always happened at point
8) after the node tries to rejoin the cluster.
This is Bug 1797579 -- a failed meta-data command can leave the cluster not realizing a fence device is capable of unfencing.
When the problem occurs, virt-213 is DC and schedules virt-214 for unfencing. virt-213's fencer queries both nodes to find out what devices they have that are capable of it. However virt-214's meta-data command when registering fence_scsi had failed:
Jun 04 16:08:37 virt-214.cluster-qe.lab.eng.brq.redhat.com pacemaker-fenced [1535] (action_synced_wait) warning: fence_scsi_metadata_1:1894 - timed out after 5000ms
Therefore, virt-214 did not know fence_scsi could unfence, and did not reply with any devices. virt-213 did have the meta-data, and since the meta-data requires unfencing to be performed on the target, it could not perform the unfencing either.
Per the original bz, we will need a better log message when the meta-data command fails, and hopefully we can also retry failed meta-data commands when meta-data is needed later.
Why fence_scsi timed out for a meta-data command is another question, outside of pacemaker. There's no indication in the logs that anything was going wrong on that node at the time.
*** This bug has been marked as a duplicate of bug 1797579 ***
Created attachment 1695447 [details] 'pcs cluster report' taken after the problem occured Description of problem: We have automated test that does the following: 1) install rhel 2) configure wwo node cluster with fence_scsi and auto tie-breaker 3) cause network split using iptables 4) second node is evicted from the cluster and have its scsi key removed 5) verify that second node cannot write to shared storage 6) second is rebooted manually 7) wait for second node to rejoin cluster 8) verify that the second node keys is re-added to shared storage 9) verify that second node can write again to the shared storage In rare case the node it happens that victim node unfencing fails it's scsi key is not added to the shared disk and it is unable to to write to shared storage even though it makes part of the cluster. The problem happens in about one run out of hundred tests. Version-Release number of selected component (if applicable): pacemaker-2.0.3-5.el8_2.1.x86_64 corosync-3.0.3-2.el8.x86_64 fence-agents-all-4.2.1-41.el8.x86_64 How reproducible: very rare, ~2% of runs Steps to Reproduce: 1. see description of the problem Actual results: node not unfenced after it rejoins the cluster Expected results: node unfenced after re-joining the cluster after being evicted Additional info: I never saw the problem happening on cluster start it always happened at point 8) after the node tries to rejoin the cluster.