Bug 2039399
| Summary: | Removing pacemaker_remote node ends up in fence action on the remote node | |||
|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 9 | Reporter: | Michal Mazourek <mmazoure> | |
| Component: | pacemaker | Assignee: | Ken Gaillot <kgaillot> | |
| Status: | CLOSED ERRATA | QA Contact: | cluster-qe <cluster-qe> | |
| Severity: | medium | Docs Contact: | ||
| Priority: | high | |||
| Version: | 9.0 | CC: | cluster-maint, iromanos, msmazova | |
| Target Milestone: | rc | Keywords: | Regression, Triaged | |
| Target Release: | 9.0 | Flags: | pm-rhel:
mirror+
|
|
| Hardware: | All | |||
| OS: | All | |||
| Whiteboard: | ||||
| Fixed In Version: | pacemaker-2.1.2-4.el9 | Doc Type: | No Doc Update | |
| Doc Text: |
This issue was not in a released build
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 2046446 (view as bug list) | Environment: | ||
| Last Closed: | 2022-05-17 12:20:40 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 2046446 | |||
Hi, can you attach a pcs cluster report from around the time of the issue? I confirmed that the issue was introduced between the upstream 2.1.1 and 2.1.2 releases. Pacemaker is not properly detecting that the remote node was intentionally shut down. More investigation will be needed to determine the cause and a fix. The fix turned out to be straightforward, so we're going to get this into 9.0 Fixed upstream by commit 16928cfc69 [root@virt-258 ~]# rpm -qa pacemaker pacemaker-2.1.2-4.el9.x86_64 [root@virt-258 ~]# pcs status Cluster name: STSRHTS21045 Cluster Summary: * Stack: corosync * Current DC: virt-260 (version 2.1.2-4.el9-ada5c3b36e2) - partition with quorum * Last updated: Thu Feb 24 15:06:04 2022 * Last change: Thu Feb 24 15:01:45 2022 by root via cibadmin on virt-258 * 5 nodes configured * 5 resource instances configured Node List: * Online: [ virt-258 virt-259 virt-260 virt-261 virt-262 ] Full List of Resources: * fence-virt-258 (stonith:fence_xvm): Started virt-258 * fence-virt-259 (stonith:fence_xvm): Started virt-259 * fence-virt-260 (stonith:fence_xvm): Started virt-260 * fence-virt-261 (stonith:fence_xvm): Started virt-261 * fence-virt-262 (stonith:fence_xvm): Started virt-262 Daemon Status: corosync: active/disabled pacemaker: active/disabled pcsd: active/enabled [root@virt-258 ~]# pcs cluster node remove virt-262 Destroying cluster on hosts: 'virt-262'... virt-262: Successfully destroyed cluster Sending updated corosync.conf to nodes... virt-260: Succeeded virt-259: Succeeded virt-261: Succeeded virt-258: Succeeded virt-258: Corosync configuration reloaded [root@virt-258 ~]# pcs cluster node add-remote virt-262 No addresses specified for host 'virt-262', using 'virt-262' Sending 'pacemaker authkey' to 'virt-262' virt-262: successful distribution of the file 'pacemaker authkey' Requesting 'pacemaker_remote enable', 'pacemaker_remote start' on 'virt-262' virt-262: successful run of 'pacemaker_remote enable' virt-262: successful run of 'pacemaker_remote start' [root@virt-258 ~]# pcs status | grep "Node List" -A 2 Node List: * Online: [ virt-258 virt-259 virt-260 virt-261 ] * RemoteOnline: [ virt-262 ] [root@virt-258 ~]# pcs cluster node remove-remote virt-262 Requesting 'pacemaker_remote disable', 'pacemaker_remote stop' on 'virt-262' virt-262: successful run of 'pacemaker_remote disable' virt-262: successful run of 'pacemaker_remote stop' Requesting remove 'pacemaker authkey' from 'virt-262' virt-262: successful removal of the file 'pacemaker authkey' Deleting Resource - virt-262 [root@virt-258 ~]# echo $? 0 [root@virt-258 ~]# sleep 120 && pcs stonith history 0 events found Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (new packages: pacemaker), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:2293 |
Description of problem: When removing pacemaker_remote node from cluster configuration (via 'pcs cluster node remove-remote'), fence action occurs on the remote node. Version-Release number of selected component (if applicable): pacemaker-2.1.2-1.el9.x86_64 How reproducible: always Steps to Reproduce: [root@virt-255 ~]# pcs status Cluster name: STSRHTS22418 Cluster Summary: * Stack: corosync * Current DC: virt-256 (version 2.1.2-1.el9-ada5c3b36e2) - partition with quorum * Last updated: Tue Jan 11 15:29:56 2022 * Last change: Tue Jan 11 15:29:01 2022 by root via cibadmin on virt-255 * 4 nodes configured * 4 resource instances configured Node List: * Online: [ virt-255 virt-256 virt-261 virt-262 ] Full List of Resources: * fence-virt-255 (stonith:fence_xvm): Started virt-255 * fence-virt-256 (stonith:fence_xvm): Started virt-256 * fence-virt-261 (stonith:fence_xvm): Started virt-261 * fence-virt-262 (stonith:fence_xvm): Started virt-262 Daemon Status: corosync: active/disabled pacemaker: active/disabled pcsd: active/enabled [root@virt-255 ~]# pcs cluster node remove virt-262 Destroying cluster on hosts: 'virt-262'... virt-262: Successfully destroyed cluster Sending updated corosync.conf to nodes... virt-261: Succeeded virt-255: Succeeded virt-256: Succeeded virt-255: Corosync configuration reloaded [root@virt-255 ~]# pcs cluster node add-remote virt-262 No addresses specified for host 'virt-262', using 'virt-262' Sending 'pacemaker authkey' to 'virt-262' virt-262: successful distribution of the file 'pacemaker authkey' Requesting 'pacemaker_remote enable', 'pacemaker_remote start' on 'virt-262' virt-262: successful run of 'pacemaker_remote enable' virt-262: successful run of 'pacemaker_remote start' [root@virt-255 ~]# pcs status | grep "Node List" -A 2 Node List: * Online: [ virt-255 virt-256 virt-261 ] * RemoteOnline: [ virt-262 ] ## removing the remote node [root@virt-255 ~]# pcs cluster node remove-remote virt-262 Requesting 'pacemaker_remote disable', 'pacemaker_remote stop' on 'virt-262' virt-262: successful run of 'pacemaker_remote disable' virt-262: successful run of 'pacemaker_remote stop' Requesting remove 'pacemaker authkey' from 'virt-262' virt-262: successful removal of the file 'pacemaker authkey' Deleting Resource - virt-262 [root@virt-255 ~]# echo $? 0 > This will stuck on 'Deleting Resource - virt-262' line for a few minutes, fence of the remote node will happen after that. Actual results: [root@virt-255 ~]# pcs stonith history reboot of virt-262 successful: delegate=virt-261, client=pacemaker-controld.439906, origin=virt-256, completed='1970-01-05 01:18:46 +01:00' 1 event found Expected results: No fence action will occur Additional info: The same issue is present also in pacemaker-2.1.2-2, both on RHEL8 and RHEL9. The issue is not present in version pacemaker-2.1.0-11.el9 and lower.