Bug 1390609 - Deleting of pacemaker remote resource leads to fencing of the remote node
Summary: Deleting of pacemaker remote resource leads to fencing of the remote node
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: pcs
Version: 7.3
Hardware: x86_64
OS: Linux
high
unspecified
Target Milestone: rc
: 7.4
Assignee: Ivan Devat
QA Contact: cluster-qe@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-11-01 14:08 UTC by Martin Juricek
Modified: 2017-08-01 18:24 UTC (History)
9 users (show)

Fixed In Version: pcs-0.9.158-2.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-08-01 18:24:40 UTC
Target Upstream Version:


Attachments (Terms of Use)
proposed fix (2.55 KB, patch)
2017-05-23 11:23 UTC, Ivan Devat
no flags Details | Diff


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:1958 normal SHIPPED_LIVE pcs bug fix and enhancement update 2017-08-01 18:09:47 UTC

Description Martin Juricek 2016-11-01 14:08:03 UTC
Description of problem:
Deleting of ocf:pacemaker:remote resource leads to fencing of the remote node sometimes.


Version-Release number of selected component (if applicable):
pacemaker-1.1.15-11.el7_3.2.x86_64
pacemaker-remote-1.1.15-11.el7_3.2.x86_64


How reproducible:
~ 20%

Steps to Reproduce:
1. Setup ocf:pacemaker:remote resource on a remote node
2. Delete the pacemaker remote resource


Actual results:
Remote node is fenced after deleting of the resource.

Expected results:
Resource is deleted without fencing of the remote node.


Additional info:

# pcs status
Cluster name: STSRHTS2682
Stack: corosync
Current DC: virt-005 (version 1.1.15-11.el7-e174ec8) - partition with quorum
Last updated: Tue Nov  1 14:34:24 2016          Last change: Tue Nov  1 14:26:46 2016 by root via cibadmin on virt-004

3 nodes and 10 resources configured

Online: [ virt-004 virt-005 ]
RemoteOnline: [ virt-009 ]

Full list of resources:

 fence-virt-004 (stonith:fence_xvm):    Started virt-005
 fence-virt-005 (stonith:fence_xvm):    Started virt-004
 fence-virt-009 (stonith:fence_xvm):    Started virt-005
 Clone Set: dlm-clone [dlm]
     Started: [ virt-004 virt-005 ]
     Stopped: [ virt-009 ]
 Clone Set: clvmd-clone [clvmd]
     Started: [ virt-004 virt-005 ]
     Stopped: [ virt-009 ]
 virt-009       (ocf::pacemaker:remote):        Started virt-004

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled


[root@virt-004 /]# pcs resource show virt-009
 Resource: virt-009 (class=ocf provider=pacemaker type=remote)
  Operations: start interval=0s timeout=60 (virt-009-start-interval-0s)
              stop interval=0s timeout=60 (virt-009-stop-interval-0s)
              monitor interval=60s timeout=30 (virt-009-monitor-interval-60s)


[root@virt-004 /]# pcs resource delete virt-009
Attempting to stop: virt-009...Stopped

<...remote node virt-009 is fenced...>

Comment 4 Ken Gaillot 2016-11-01 17:39:03 UTC
As of RHEL 7.3, pcs now executes a "crm_node --force --remove" after removing a remote node resource. That's a good idea, but since pcs runs the command faster than a person on the command line, it has exposed a race condition.

When the resource is deleted, the resource's operation history on each node will be deleted from the CIB.

When "crm_node --force --remove" is run, the remote node's state will be deleted from the CIB.

If the cluster DC processes the CIB change from the crm_node command before the CIB changes from the resource deletion, then it will consider the remote node to be "orphaned" with an unknown state, and thus in need of fencing.

I'm not sure whether this can be fixed on the pacemaker side, because once we've lost the node state, we might have no choice but to fence. This is why the upstream documentation for Pacemaker Remote has a warning about running the crm_node command, "Be absolutely sure that the node’s resource has been deleted from the configuration first."

I'll investigate whether it's safe to make any assumptions about the node state in this situation. If not, I'll reassign this to pcs, which can avoid the problem by ensuring that there are no remaining references to the remote resource in the CIB before running the crm_node command (doing "crm_resource --wait" should be sufficient, or the CIB could be scanned for <lrm_resources> references).

Comment 5 Ken Gaillot 2016-11-01 21:40:59 UTC
I don't think this can be handled reliably on the pacemaker side, so reassigning to pcs.

The easiest fix in pcs would be to do a "crm_resource --wait" between removing the resource and running crm_node.

However, if "--wait" is not passed to pcs, you may not want to do that. So an alternative would be to loop (with some timeout) until this command returns nonzero exit status (replacing RSC with the remote node name):

  cibadmin -Q --xpath="/cib/status/node_state/lrm/lrm_resources/lrm_resource[@id='RSC']"

Comment 6 Andrew Beekhof 2016-11-02 01:18:03 UTC
(In reply to Ken Gaillot from comment #5)
> I don't think this can be handled reliably on the pacemaker side, so
> reassigning to pcs.
> 
> The easiest fix in pcs would be to do a "crm_resource --wait" between
> removing the resource and running crm_node.
> 
> However, if "--wait" is not passed to pcs, you may not want to do that. So
> an alternative would be to loop (with some timeout) until this command
> returns nonzero exit status (replacing RSC with the remote node name):
> 
>   cibadmin -Q
> --xpath="/cib/status/node_state/lrm/lrm_resources/lrm_resource[@id='RSC']"

Dunno - if you're going to wait anyway, you might as well do it with the same code everything else uses :-/

Comment 7 Tomas Jelinek 2016-11-02 08:50:52 UTC
(In reply to Andrew Beekhof from comment #6)
> Dunno - if you're going to wait anyway, you might as well do it with the
> same code everything else uses :-/

Agreed.

Comment 9 Ivan Devat 2017-05-23 11:23:09 UTC
Created attachment 1281468 [details]
proposed fix

Comment 10 Tomas Jelinek 2017-05-26 11:29:04 UTC
After fix:

[root@rh73-node1:~]# rpm -q pcs
pcs-0.9.158-2.el7.x86_64

[root@rh73-node1:~]# pcs resource 
 dummy  (ocf::pacemaker:Dummy): Started rh73-node3
 rh73-node3     (ocf::pacemaker:remote):        Started rh73-node1
[root@rh73-node1:~]# pcs resource delete rh73-node3 --debug
{...snip...}
Running: /usr/sbin/crm_resource --wait
Return Value: 0
--Debug Output Start--
--Debug Output End--

Running: /usr/sbin/crm_node --force --remove rh73-node3
Return Value: 0
--Debug Output Start--
--Debug Output End--

Comment 13 errata-xmlrpc 2017-08-01 18:24:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1958


Note You need to log in before you can comment on or make changes to this bug.