Bug 2163450

Summary: Removing remote node results in a fence action
Product: Red Hat Enterprise Linux 9 Reporter: Michal Mazourek <mmazoure>
Component: pacemakerAssignee: Ken Gaillot <kgaillot>
Status: CLOSED ERRATA QA Contact: cluster-qe <cluster-qe>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 9.2CC: cluster-maint, jrehova, msmazova
Target Milestone: rcKeywords: Regression, Triaged
Target Release: 9.2   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: pacemaker-2.1.5-5.el9 Doc Type: No Doc Update
Doc Text:
This was not in a released build
Story Points: ---
Clone Of:
: 2163567 (view as bug list) Environment:
Last Closed: 2023-05-09 07:18:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2163567    

Description Michal Mazourek 2023-01-23 14:30:06 UTC
The same issue as described in bz2039399 is present again:

(bz2039399 comment #0)
> Description of problem:
> When removing pacemaker_remote node from cluster configuration (via 'pcs
> cluster node remove-remote'), fence action occurs on the remote node.
> 
> 
> How reproducible:
> always
> 
> 
> Steps to Reproduce:
> [root@virt-255 ~]# pcs status
> Cluster name: STSRHTS22418
> Cluster Summary:
>   * Stack: corosync
>   * Current DC: virt-256 (version 2.1.2-1.el9-ada5c3b36e2) - partition with
> quorum
>   * Last updated: Tue Jan 11 15:29:56 2022
>   * Last change:  Tue Jan 11 15:29:01 2022 by root via cibadmin on virt-255
>   * 4 nodes configured
>   * 4 resource instances configured
> 
> Node List:
>   * Online: [ virt-255 virt-256 virt-261 virt-262 ]
> 
> Full List of Resources:
>   * fence-virt-255	(stonith:fence_xvm):	 Started virt-255
>   * fence-virt-256	(stonith:fence_xvm):	 Started virt-256
>   * fence-virt-261	(stonith:fence_xvm):	 Started virt-261
>   * fence-virt-262	(stonith:fence_xvm):	 Started virt-262
> 
> Daemon Status:
>   corosync: active/disabled
>   pacemaker: active/disabled
>   pcsd: active/enabled
> 
> [root@virt-255 ~]# pcs cluster node remove virt-262
> Destroying cluster on hosts: 'virt-262'...
> virt-262: Successfully destroyed cluster
> Sending updated corosync.conf to nodes...
> virt-261: Succeeded
> virt-255: Succeeded
> virt-256: Succeeded
> virt-255: Corosync configuration reloaded
> 
> [root@virt-255 ~]# pcs cluster node add-remote virt-262
> No addresses specified for host 'virt-262', using 'virt-262'
> Sending 'pacemaker authkey' to 'virt-262'
> virt-262: successful distribution of the file 'pacemaker authkey'
> Requesting 'pacemaker_remote enable', 'pacemaker_remote start' on 'virt-262'
> virt-262: successful run of 'pacemaker_remote enable'
> virt-262: successful run of 'pacemaker_remote start'
> 
> [root@virt-255 ~]# pcs status | grep "Node List" -A 2
> Node List:
>   * Online: [ virt-255 virt-256 virt-261 ]
>   * RemoteOnline: [ virt-262 ]
> 
> ## removing the remote node
> 
> [root@virt-255 ~]# pcs cluster node remove-remote virt-262
> Requesting 'pacemaker_remote disable', 'pacemaker_remote stop' on 'virt-262'
> virt-262: successful run of 'pacemaker_remote disable'
> virt-262: successful run of 'pacemaker_remote stop'
> Requesting remove 'pacemaker authkey' from 'virt-262'
> virt-262: successful removal of the file 'pacemaker authkey'
> Deleting Resource - virt-262
> [root@virt-255 ~]# echo $?
> 0
> 
> > This will stuck on 'Deleting Resource - virt-262' line for a few minutes, fence of the remote node will happen after that.
> 
> 
> Actual results:
> [root@virt-255 ~]# pcs stonith history
> reboot of virt-262 successful: delegate=virt-261,
> client=pacemaker-controld.439906, origin=virt-256, completed='1970-01-05
> 01:18:46 +01:00'
> 1 event found
> 
> 
> Expected results:
> No fence action will occur


Note:
After running "pcs cluster node remove-remote {node}", sometimes the command ends up in a timeout error (probably when the fence happens before the command ends) and sometimes it finishes successfully, however the fence action on the remote node happens every time.

Versions:
Found in pacemaker-2.1.5-4 (both on RHEL8 and RHEL9).
After examination, the issue is first present in pacemaker-2.1.5-1 (it is working fine in pacemaker-2.1.4-5).

Comment 1 Ken Gaillot 2023-01-23 22:23:33 UTC
The root cause is different from bz2039399 and was introduced by upstream commit d61494347, which was based on an incorrect analysis of bz1804259 and thus simply needs to be reverted.

Comment 5 jrehova 2023-01-31 14:00:11 UTC
* 4 nodes cluster

Version of pacemaker:

> [root@virt-313 ~]# rpm -qa pacemaker
> pacemaker-2.1.5-5.el9.x86_64

Status of cluster:

> [root@virt-313 ~]# pcs status
> Cluster name: STSRHTS23870
> Status of pacemakerd: 'Pacemaker is running' (last updated 2023-01-31 14:22:55 +01:00)
> Cluster Summary:
>   * Stack: corosync
>   * Current DC: virt-315 (version 2.1.5-5.el9-a3f44794f94) - partition with quorum
>   * Last updated: Tue Jan 31 14:22:56 2023
>   * Last change:  Tue Jan 31 14:00:48 2023 by root via cibadmin on virt-313
>   * 4 nodes configured
>   * 4 resource instances configured
> 
> Node List:
>   * Online: [ virt-313 virt-314 virt-315 virt-316 ]
> 
> Full List of Resources:
>   * fence-virt-313	(stonith:fence_xvm):	 Started virt-313
>   * fence-virt-314	(stonith:fence_xvm):	 Started virt-314
>   * fence-virt-315	(stonith:fence_xvm):	 Started virt-315
>   * fence-virt-316	(stonith:fence_xvm):	 Started virt-316
> 
> Daemon Status:
>   corosync: active/disabled
>   pacemaker: active/disabled
>   pcsd: active/enabled

Remove node virt-316:

> [root@virt-313 ~]# pcs cluster node remove virt-316
> Destroying cluster on hosts: 'virt-316'...
> virt-316: Successfully destroyed cluster
> Sending updated corosync.conf to nodes...
> virt-313: Succeeded
> virt-315: Succeeded
> virt-314: Succeeded
> virt-315: Corosync configuration reloaded

Install package pacemaker-remote on node, that is chosen to be remote:

> [root@virt-316 ~]# yum install pacemaker-remote

Create remote node:

> [root@virt-313 ~]# pcs cluster node add-remote virt-316
> No addresses specified for host 'virt-316', using 'virt-316'
> Sending 'pacemaker authkey' to 'virt-316'
> virt-316: successful distribution of the file 'pacemaker authkey'
> Requesting 'pacemaker_remote enable', 'pacemaker_remote start' on 'virt-316'
> virt-316: successful run of 'pacemaker_remote enable'
> virt-316: successful run of 'pacemaker_remote start'

Status of cluster:

> [root@virt-313 ~]# pcs status | grep "Node List" -A 2
> Node List:
>   * Online: [ virt-313 virt-314 virt-315 ]
>   * RemoteOnline: [ virt-316 ]

Remove remote node:

> [root@virt-313 ~]# pcs cluster node remove-remote virt-316
> Requesting 'pacemaker_remote disable', 'pacemaker_remote stop' on 'virt-316'
> virt-316: successful run of 'pacemaker_remote disable'
> virt-316: successful run of 'pacemaker_remote stop'
> Requesting remove 'pacemaker authkey' from 'virt-316'
> virt-316: successful removal of the file 'pacemaker authkey'
> Deleting Resource - virt-316

> [root@virt-313 ~]# echo $?
> 0

> [root@virt-313 ~]# pcs stonith history
> 0 events found

Result: No fence action occured.

Comment 7 errata-xmlrpc 2023-05-09 07:18:17 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (pacemaker bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:2150