Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Description of problem:
When the DC and another node that isn't running resources are killed at the same time, the node not hosting any resources won't be fenced - it will remain OFFLINE until manually rebooted.
Example:
[root@virt-550 ~]# pcs status
Cluster name: STSRHTS20109
Status of pacemakerd: 'Pacemaker is running' (last updated 2023-04-11 14:07:41 +02:00)
Cluster Summary:
* Stack: corosync
* Current DC: virt-560 (version 2.1.5-7.el9-a3f44794f94) - partition with quorum
* Last updated: Tue Apr 11 14:07:42 2023
* Last change: Tue Apr 11 12:53:54 2023 by root via cibadmin on virt-560
* 5 nodes configured
* 1 resource instance configured
Node List:
* Online: [ virt-510 virt-549 virt-550 virt-552 virt-560 ]
Full List of Resources:
* fencer (stonith:fence_xvm): Started virt-549
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
[root@virt-550 ~]# timeout -s kill -k 3 3 ssh root@virt-560 halt -fin & timeout -s kill -k 3 3 ssh root@virt-552 halt -fin
[root@virt-550 ~]# pcs status
Cluster name: STSRHTS20109
Status of pacemakerd: 'Pacemaker is running' (last updated 2023-04-11 14:08:40 +02:00)
Cluster Summary:
* Stack: corosync
* Current DC: virt-549 (version 2.1.5-7.el9-a3f44794f94) - partition with quorum
* Last updated: Tue Apr 11 14:08:40 2023
* Last change: Tue Apr 11 12:53:54 2023 by root via cibadmin on virt-560
* 5 nodes configured
* 1 resource instance configured
Node List:
* Online: [ virt-510 virt-549 virt-550 ]
* OFFLINE: [ virt-552 virt-560 ]
Full List of Resources:
* fencer (stonith:fence_xvm): Started virt-549
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
[root@virt-550 ~]# pcs status
Cluster name: STSRHTS20109
Status of pacemakerd: 'Pacemaker is running' (last updated 2023-04-11 14:21:01 +02:00)
Cluster Summary:
* Stack: corosync
* Current DC: virt-549 (version 2.1.5-7.el9-a3f44794f94) - partition with quorum
* Last updated: Tue Apr 11 14:21:01 2023
* Last change: Tue Apr 11 12:53:54 2023 by root via cibadmin on virt-560
* 5 nodes configured
* 1 resource instance configured
Node List:
* Online: [ virt-510 virt-549 virt-550 virt-560 ]
* OFFLINE: [ virt-552 ]
Full List of Resources:
* fencer (stonith:fence_xvm): Started virt-549
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
Version-Release number of selected component (if applicable):
pacemaker-2.1.5-7.el9.x86_64
How reproducible:
Reliably
Steps to Reproduce:
1. Create and start a cluster with for example five nodes (so that the cluster keeps quorum when 2 nodes are killed).
2. At least one of the nodes must not be hosting any resources.
3. Ensure no processes/resources are running that could interfere with the reproducer by submitting fence requests (eg. dlm).
4. Simultaneously kill both the DC node and another node that does not host any resource, e.g. run command `timeout -s kill -k 3 3 ssh root@${dc_node} halt -fin & timeout -s kill -k 3 3 ssh root@${idle_node} halt -fin` and check if both nodes are offline in `pcs status`.
5. Wait some time and then check if both nodes are back in cluster.
Actual results:
DC node is back in cluster, but the other is not.
Expected results:
Both nodes should be back in cluster.
Additional info:
A workaround would be to define a dummy clone that runs on all nodes
Comment 4RHEL Program Management
2023-09-22 20:14:23 UTC
Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.
Comment 5RHEL Program Management
2023-09-22 20:15:32 UTC
This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.
Due to differences in account names between systems, some fields were not replicated. Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.
To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer. You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:
"Bugzilla Bug" = 1234567
In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.
Description of problem: When the DC and another node that isn't running resources are killed at the same time, the node not hosting any resources won't be fenced - it will remain OFFLINE until manually rebooted. Example: [root@virt-550 ~]# pcs status Cluster name: STSRHTS20109 Status of pacemakerd: 'Pacemaker is running' (last updated 2023-04-11 14:07:41 +02:00) Cluster Summary: * Stack: corosync * Current DC: virt-560 (version 2.1.5-7.el9-a3f44794f94) - partition with quorum * Last updated: Tue Apr 11 14:07:42 2023 * Last change: Tue Apr 11 12:53:54 2023 by root via cibadmin on virt-560 * 5 nodes configured * 1 resource instance configured Node List: * Online: [ virt-510 virt-549 virt-550 virt-552 virt-560 ] Full List of Resources: * fencer (stonith:fence_xvm): Started virt-549 Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled [root@virt-550 ~]# timeout -s kill -k 3 3 ssh root@virt-560 halt -fin & timeout -s kill -k 3 3 ssh root@virt-552 halt -fin [root@virt-550 ~]# pcs status Cluster name: STSRHTS20109 Status of pacemakerd: 'Pacemaker is running' (last updated 2023-04-11 14:08:40 +02:00) Cluster Summary: * Stack: corosync * Current DC: virt-549 (version 2.1.5-7.el9-a3f44794f94) - partition with quorum * Last updated: Tue Apr 11 14:08:40 2023 * Last change: Tue Apr 11 12:53:54 2023 by root via cibadmin on virt-560 * 5 nodes configured * 1 resource instance configured Node List: * Online: [ virt-510 virt-549 virt-550 ] * OFFLINE: [ virt-552 virt-560 ] Full List of Resources: * fencer (stonith:fence_xvm): Started virt-549 Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled [root@virt-550 ~]# pcs status Cluster name: STSRHTS20109 Status of pacemakerd: 'Pacemaker is running' (last updated 2023-04-11 14:21:01 +02:00) Cluster Summary: * Stack: corosync * Current DC: virt-549 (version 2.1.5-7.el9-a3f44794f94) - partition with quorum * Last updated: Tue Apr 11 14:21:01 2023 * Last change: Tue Apr 11 12:53:54 2023 by root via cibadmin on virt-560 * 5 nodes configured * 1 resource instance configured Node List: * Online: [ virt-510 virt-549 virt-550 virt-560 ] * OFFLINE: [ virt-552 ] Full List of Resources: * fencer (stonith:fence_xvm): Started virt-549 Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled Version-Release number of selected component (if applicable): pacemaker-2.1.5-7.el9.x86_64 How reproducible: Reliably Steps to Reproduce: 1. Create and start a cluster with for example five nodes (so that the cluster keeps quorum when 2 nodes are killed). 2. At least one of the nodes must not be hosting any resources. 3. Ensure no processes/resources are running that could interfere with the reproducer by submitting fence requests (eg. dlm). 4. Simultaneously kill both the DC node and another node that does not host any resource, e.g. run command `timeout -s kill -k 3 3 ssh root@${dc_node} halt -fin & timeout -s kill -k 3 3 ssh root@${idle_node} halt -fin` and check if both nodes are offline in `pcs status`. 5. Wait some time and then check if both nodes are back in cluster. Actual results: DC node is back in cluster, but the other is not. Expected results: Both nodes should be back in cluster. Additional info: