RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2163450 - Removing remote node results in a fence action
Summary: Removing remote node results in a fence action
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: pacemaker
Version: 9.2
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: rc
: 9.2
Assignee: Ken Gaillot
QA Contact: cluster-qe@redhat.com
URL:
Whiteboard:
Depends On:
Blocks: 2163567
TreeView+ depends on / blocked
 
Reported: 2023-01-23 14:30 UTC by Michal Mazourek
Modified: 2023-05-09 07:38 UTC (History)
3 users (show)

Fixed In Version: pacemaker-2.1.5-5.el9
Doc Type: No Doc Update
Doc Text:
This was not in a released build
Clone Of:
: 2163567 (view as bug list)
Environment:
Last Closed: 2023-05-09 07:18:17 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker CLUSTERQE-6364 0 None None None 2023-01-24 09:07:13 UTC
Red Hat Issue Tracker RHELPLAN-146018 0 None None None 2023-01-23 14:33:41 UTC
Red Hat Product Errata RHBA-2023:2150 0 None None None 2023-05-09 07:18:40 UTC

Description Michal Mazourek 2023-01-23 14:30:06 UTC
The same issue as described in bz2039399 is present again:

(bz2039399 comment #0)
> Description of problem:
> When removing pacemaker_remote node from cluster configuration (via 'pcs
> cluster node remove-remote'), fence action occurs on the remote node.
> 
> 
> How reproducible:
> always
> 
> 
> Steps to Reproduce:
> [root@virt-255 ~]# pcs status
> Cluster name: STSRHTS22418
> Cluster Summary:
>   * Stack: corosync
>   * Current DC: virt-256 (version 2.1.2-1.el9-ada5c3b36e2) - partition with
> quorum
>   * Last updated: Tue Jan 11 15:29:56 2022
>   * Last change:  Tue Jan 11 15:29:01 2022 by root via cibadmin on virt-255
>   * 4 nodes configured
>   * 4 resource instances configured
> 
> Node List:
>   * Online: [ virt-255 virt-256 virt-261 virt-262 ]
> 
> Full List of Resources:
>   * fence-virt-255	(stonith:fence_xvm):	 Started virt-255
>   * fence-virt-256	(stonith:fence_xvm):	 Started virt-256
>   * fence-virt-261	(stonith:fence_xvm):	 Started virt-261
>   * fence-virt-262	(stonith:fence_xvm):	 Started virt-262
> 
> Daemon Status:
>   corosync: active/disabled
>   pacemaker: active/disabled
>   pcsd: active/enabled
> 
> [root@virt-255 ~]# pcs cluster node remove virt-262
> Destroying cluster on hosts: 'virt-262'...
> virt-262: Successfully destroyed cluster
> Sending updated corosync.conf to nodes...
> virt-261: Succeeded
> virt-255: Succeeded
> virt-256: Succeeded
> virt-255: Corosync configuration reloaded
> 
> [root@virt-255 ~]# pcs cluster node add-remote virt-262
> No addresses specified for host 'virt-262', using 'virt-262'
> Sending 'pacemaker authkey' to 'virt-262'
> virt-262: successful distribution of the file 'pacemaker authkey'
> Requesting 'pacemaker_remote enable', 'pacemaker_remote start' on 'virt-262'
> virt-262: successful run of 'pacemaker_remote enable'
> virt-262: successful run of 'pacemaker_remote start'
> 
> [root@virt-255 ~]# pcs status | grep "Node List" -A 2
> Node List:
>   * Online: [ virt-255 virt-256 virt-261 ]
>   * RemoteOnline: [ virt-262 ]
> 
> ## removing the remote node
> 
> [root@virt-255 ~]# pcs cluster node remove-remote virt-262
> Requesting 'pacemaker_remote disable', 'pacemaker_remote stop' on 'virt-262'
> virt-262: successful run of 'pacemaker_remote disable'
> virt-262: successful run of 'pacemaker_remote stop'
> Requesting remove 'pacemaker authkey' from 'virt-262'
> virt-262: successful removal of the file 'pacemaker authkey'
> Deleting Resource - virt-262
> [root@virt-255 ~]# echo $?
> 0
> 
> > This will stuck on 'Deleting Resource - virt-262' line for a few minutes, fence of the remote node will happen after that.
> 
> 
> Actual results:
> [root@virt-255 ~]# pcs stonith history
> reboot of virt-262 successful: delegate=virt-261,
> client=pacemaker-controld.439906, origin=virt-256, completed='1970-01-05
> 01:18:46 +01:00'
> 1 event found
> 
> 
> Expected results:
> No fence action will occur


Note:
After running "pcs cluster node remove-remote {node}", sometimes the command ends up in a timeout error (probably when the fence happens before the command ends) and sometimes it finishes successfully, however the fence action on the remote node happens every time.

Versions:
Found in pacemaker-2.1.5-4 (both on RHEL8 and RHEL9).
After examination, the issue is first present in pacemaker-2.1.5-1 (it is working fine in pacemaker-2.1.4-5).

Comment 1 Ken Gaillot 2023-01-23 22:23:33 UTC
The root cause is different from bz2039399 and was introduced by upstream commit d61494347, which was based on an incorrect analysis of bz1804259 and thus simply needs to be reverted.

Comment 5 jrehova 2023-01-31 14:00:11 UTC
* 4 nodes cluster

Version of pacemaker:

> [root@virt-313 ~]# rpm -qa pacemaker
> pacemaker-2.1.5-5.el9.x86_64

Status of cluster:

> [root@virt-313 ~]# pcs status
> Cluster name: STSRHTS23870
> Status of pacemakerd: 'Pacemaker is running' (last updated 2023-01-31 14:22:55 +01:00)
> Cluster Summary:
>   * Stack: corosync
>   * Current DC: virt-315 (version 2.1.5-5.el9-a3f44794f94) - partition with quorum
>   * Last updated: Tue Jan 31 14:22:56 2023
>   * Last change:  Tue Jan 31 14:00:48 2023 by root via cibadmin on virt-313
>   * 4 nodes configured
>   * 4 resource instances configured
> 
> Node List:
>   * Online: [ virt-313 virt-314 virt-315 virt-316 ]
> 
> Full List of Resources:
>   * fence-virt-313	(stonith:fence_xvm):	 Started virt-313
>   * fence-virt-314	(stonith:fence_xvm):	 Started virt-314
>   * fence-virt-315	(stonith:fence_xvm):	 Started virt-315
>   * fence-virt-316	(stonith:fence_xvm):	 Started virt-316
> 
> Daemon Status:
>   corosync: active/disabled
>   pacemaker: active/disabled
>   pcsd: active/enabled

Remove node virt-316:

> [root@virt-313 ~]# pcs cluster node remove virt-316
> Destroying cluster on hosts: 'virt-316'...
> virt-316: Successfully destroyed cluster
> Sending updated corosync.conf to nodes...
> virt-313: Succeeded
> virt-315: Succeeded
> virt-314: Succeeded
> virt-315: Corosync configuration reloaded

Install package pacemaker-remote on node, that is chosen to be remote:

> [root@virt-316 ~]# yum install pacemaker-remote

Create remote node:

> [root@virt-313 ~]# pcs cluster node add-remote virt-316
> No addresses specified for host 'virt-316', using 'virt-316'
> Sending 'pacemaker authkey' to 'virt-316'
> virt-316: successful distribution of the file 'pacemaker authkey'
> Requesting 'pacemaker_remote enable', 'pacemaker_remote start' on 'virt-316'
> virt-316: successful run of 'pacemaker_remote enable'
> virt-316: successful run of 'pacemaker_remote start'

Status of cluster:

> [root@virt-313 ~]# pcs status | grep "Node List" -A 2
> Node List:
>   * Online: [ virt-313 virt-314 virt-315 ]
>   * RemoteOnline: [ virt-316 ]

Remove remote node:

> [root@virt-313 ~]# pcs cluster node remove-remote virt-316
> Requesting 'pacemaker_remote disable', 'pacemaker_remote stop' on 'virt-316'
> virt-316: successful run of 'pacemaker_remote disable'
> virt-316: successful run of 'pacemaker_remote stop'
> Requesting remove 'pacemaker authkey' from 'virt-316'
> virt-316: successful removal of the file 'pacemaker authkey'
> Deleting Resource - virt-316

> [root@virt-313 ~]# echo $?
> 0

> [root@virt-313 ~]# pcs stonith history
> 0 events found

Result: No fence action occured.

Comment 7 errata-xmlrpc 2023-05-09 07:18:17 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (pacemaker bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:2150


Note You need to log in before you can comment on or make changes to this bug.