As of RHEL 7.3, pcs now executes a "crm_node --force --remove" after removing a remote node resource. That's a good idea, but since pcs runs the command faster than a person on the command line, it has exposed a race condition.

When the resource is deleted, the resource's operation history on each node will be deleted from the CIB.

When "crm_node --force --remove" is run, the remote node's state will be deleted from the CIB.

If the cluster DC processes the CIB change from the crm_node command before the CIB changes from the resource deletion, then it will consider the remote node to be "orphaned" with an unknown state, and thus in need of fencing.

I'm not sure whether this can be fixed on the pacemaker side, because once we've lost the node state, we might have no choice but to fence. This is why the upstream documentation for Pacemaker Remote has a warning about running the crm_node command, "Be absolutely sure that the node’s resource has been deleted from the configuration first."

I'll investigate whether it's safe to make any assumptions about the node state in this situation. If not, I'll reassign this to pcs, which can avoid the problem by ensuring that there are no remaining references to the remote resource in the CIB before running the crm_node command (doing "crm_resource --wait" should be sufficient, or the CIB could be scanned for <lrm_resources> references).

I don't think this can be handled reliably on the pacemaker side, so reassigning to pcs.

The easiest fix in pcs would be to do a "crm_resource --wait" between removing the resource and running crm_node.

However, if "--wait" is not passed to pcs, you may not want to do that. So an alternative would be to loop (with some timeout) until this command returns nonzero exit status (replacing RSC with the remote node name):

  cibadmin -Q --xpath="/cib/status/node_state/lrm/lrm_resources/lrm_resource[@id='RSC']"

(In reply to Ken Gaillot from comment #5)
> I don't think this can be handled reliably on the pacemaker side, so
> reassigning to pcs.
> 
> The easiest fix in pcs would be to do a "crm_resource --wait" between
> removing the resource and running crm_node.
> 
> However, if "--wait" is not passed to pcs, you may not want to do that. So
> an alternative would be to loop (with some timeout) until this command
> returns nonzero exit status (replacing RSC with the remote node name):
> 
>   cibadmin -Q
> --xpath="/cib/status/node_state/lrm/lrm_resources/lrm_resource[@id='RSC']"

Dunno - if you're going to wait anyway, you might as well do it with the same code everything else uses :-/

(In reply to Andrew Beekhof from comment #6)
> Dunno - if you're going to wait anyway, you might as well do it with the
> same code everything else uses :-/

Agreed.

Created attachment 1281468 [details]
proposed fix

After fix:

[root@rh73-node1:~]# rpm -q pcs
pcs-0.9.158-2.el7.x86_64

[root@rh73-node1:~]# pcs resource 
 dummy  (ocf::pacemaker:Dummy): Started rh73-node3
 rh73-node3     (ocf::pacemaker:remote):        Started rh73-node1
[root@rh73-node1:~]# pcs resource delete rh73-node3 --debug
{...snip...}
Running: /usr/sbin/crm_resource --wait
Return Value: 0
--Debug Output Start--
--Debug Output End--

Running: /usr/sbin/crm_node --force --remove rh73-node3
Return Value: 0
--Debug Output Start--
--Debug Output End--

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1958