Bug 2186464
| Summary: | Node without resources not fenced after it and the DC uncleanly leave cluster | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 9 | Reporter: | jrehova |
| Component: | pacemaker | Assignee: | Ken Gaillot <kgaillot> |
| Status: | NEW --- | QA Contact: | cluster-qe <cluster-qe> |
| Severity: | high | Docs Contact: | |
| Priority: | medium | ||
| Version: | 9.2 | CC: | cluster-maint |
| Target Milestone: | rc | Keywords: | Triaged |
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | Type: | Bug | |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Hi, Thanks for the report. This is a known issue, but there is no bz for it yet, so we can use this one. A workaround would be to define a dummy clone that runs on all nodes |
Description of problem: When the DC and another node that isn't running resources are killed at the same time, the node not hosting any resources won't be fenced - it will remain OFFLINE until manually rebooted. Example: [root@virt-550 ~]# pcs status Cluster name: STSRHTS20109 Status of pacemakerd: 'Pacemaker is running' (last updated 2023-04-11 14:07:41 +02:00) Cluster Summary: * Stack: corosync * Current DC: virt-560 (version 2.1.5-7.el9-a3f44794f94) - partition with quorum * Last updated: Tue Apr 11 14:07:42 2023 * Last change: Tue Apr 11 12:53:54 2023 by root via cibadmin on virt-560 * 5 nodes configured * 1 resource instance configured Node List: * Online: [ virt-510 virt-549 virt-550 virt-552 virt-560 ] Full List of Resources: * fencer (stonith:fence_xvm): Started virt-549 Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled [root@virt-550 ~]# timeout -s kill -k 3 3 ssh root@virt-560 halt -fin & timeout -s kill -k 3 3 ssh root@virt-552 halt -fin [root@virt-550 ~]# pcs status Cluster name: STSRHTS20109 Status of pacemakerd: 'Pacemaker is running' (last updated 2023-04-11 14:08:40 +02:00) Cluster Summary: * Stack: corosync * Current DC: virt-549 (version 2.1.5-7.el9-a3f44794f94) - partition with quorum * Last updated: Tue Apr 11 14:08:40 2023 * Last change: Tue Apr 11 12:53:54 2023 by root via cibadmin on virt-560 * 5 nodes configured * 1 resource instance configured Node List: * Online: [ virt-510 virt-549 virt-550 ] * OFFLINE: [ virt-552 virt-560 ] Full List of Resources: * fencer (stonith:fence_xvm): Started virt-549 Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled [root@virt-550 ~]# pcs status Cluster name: STSRHTS20109 Status of pacemakerd: 'Pacemaker is running' (last updated 2023-04-11 14:21:01 +02:00) Cluster Summary: * Stack: corosync * Current DC: virt-549 (version 2.1.5-7.el9-a3f44794f94) - partition with quorum * Last updated: Tue Apr 11 14:21:01 2023 * Last change: Tue Apr 11 12:53:54 2023 by root via cibadmin on virt-560 * 5 nodes configured * 1 resource instance configured Node List: * Online: [ virt-510 virt-549 virt-550 virt-560 ] * OFFLINE: [ virt-552 ] Full List of Resources: * fencer (stonith:fence_xvm): Started virt-549 Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled Version-Release number of selected component (if applicable): pacemaker-2.1.5-7.el9.x86_64 How reproducible: Reliably Steps to Reproduce: 1. Create and start a cluster with for example five nodes (so that the cluster keeps quorum when 2 nodes are killed). 2. At least one of the nodes must not be hosting any resources. 3. Ensure no processes/resources are running that could interfere with the reproducer by submitting fence requests (eg. dlm). 4. Simultaneously kill both the DC node and another node that does not host any resource, e.g. run command `timeout -s kill -k 3 3 ssh root@${dc_node} halt -fin & timeout -s kill -k 3 3 ssh root@${idle_node} halt -fin` and check if both nodes are offline in `pcs status`. 5. Wait some time and then check if both nodes are back in cluster. Actual results: DC node is back in cluster, but the other is not. Expected results: Both nodes should be back in cluster. Additional info: