Bug 1390609
Summary: | Deleting of pacemaker remote resource leads to fencing of the remote node | ||||||
---|---|---|---|---|---|---|---|
Product: | Red Hat Enterprise Linux 7 | Reporter: | Martin Juricek <mjuricek> | ||||
Component: | pcs | Assignee: | Ivan Devat <idevat> | ||||
Status: | CLOSED ERRATA | QA Contact: | cluster-qe <cluster-qe> | ||||
Severity: | unspecified | Docs Contact: | |||||
Priority: | high | ||||||
Version: | 7.3 | CC: | abeekhof, cfeist, cluster-maint, fdinitto, idevat, omular, rsteiger, tlavigne, tojeline | ||||
Target Milestone: | rc | ||||||
Target Release: | 7.4 | ||||||
Hardware: | x86_64 | ||||||
OS: | Linux | ||||||
Whiteboard: | |||||||
Fixed In Version: | pcs-0.9.158-2.el7 | Doc Type: | If docs needed, set a value | ||||
Doc Text: | Story Points: | --- | |||||
Clone Of: | Environment: | ||||||
Last Closed: | 2017-08-01 18:24:40 UTC | Type: | Bug | ||||
Regression: | --- | Mount Type: | --- | ||||
Documentation: | --- | CRM: | |||||
Verified Versions: | Category: | --- | |||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
Cloudforms Team: | --- | Target Upstream Version: | |||||
Embargoed: | |||||||
Attachments: |
|
Description
Martin Juricek
2016-11-01 14:08:03 UTC
As of RHEL 7.3, pcs now executes a "crm_node --force --remove" after removing a remote node resource. That's a good idea, but since pcs runs the command faster than a person on the command line, it has exposed a race condition. When the resource is deleted, the resource's operation history on each node will be deleted from the CIB. When "crm_node --force --remove" is run, the remote node's state will be deleted from the CIB. If the cluster DC processes the CIB change from the crm_node command before the CIB changes from the resource deletion, then it will consider the remote node to be "orphaned" with an unknown state, and thus in need of fencing. I'm not sure whether this can be fixed on the pacemaker side, because once we've lost the node state, we might have no choice but to fence. This is why the upstream documentation for Pacemaker Remote has a warning about running the crm_node command, "Be absolutely sure that the node’s resource has been deleted from the configuration first." I'll investigate whether it's safe to make any assumptions about the node state in this situation. If not, I'll reassign this to pcs, which can avoid the problem by ensuring that there are no remaining references to the remote resource in the CIB before running the crm_node command (doing "crm_resource --wait" should be sufficient, or the CIB could be scanned for <lrm_resources> references). I don't think this can be handled reliably on the pacemaker side, so reassigning to pcs. The easiest fix in pcs would be to do a "crm_resource --wait" between removing the resource and running crm_node. However, if "--wait" is not passed to pcs, you may not want to do that. So an alternative would be to loop (with some timeout) until this command returns nonzero exit status (replacing RSC with the remote node name): cibadmin -Q --xpath="/cib/status/node_state/lrm/lrm_resources/lrm_resource[@id='RSC']" (In reply to Ken Gaillot from comment #5) > I don't think this can be handled reliably on the pacemaker side, so > reassigning to pcs. > > The easiest fix in pcs would be to do a "crm_resource --wait" between > removing the resource and running crm_node. > > However, if "--wait" is not passed to pcs, you may not want to do that. So > an alternative would be to loop (with some timeout) until this command > returns nonzero exit status (replacing RSC with the remote node name): > > cibadmin -Q > --xpath="/cib/status/node_state/lrm/lrm_resources/lrm_resource[@id='RSC']" Dunno - if you're going to wait anyway, you might as well do it with the same code everything else uses :-/ (In reply to Andrew Beekhof from comment #6) > Dunno - if you're going to wait anyway, you might as well do it with the > same code everything else uses :-/ Agreed. Created attachment 1281468 [details]
proposed fix
After fix: [root@rh73-node1:~]# rpm -q pcs pcs-0.9.158-2.el7.x86_64 [root@rh73-node1:~]# pcs resource dummy (ocf::pacemaker:Dummy): Started rh73-node3 rh73-node3 (ocf::pacemaker:remote): Started rh73-node1 [root@rh73-node1:~]# pcs resource delete rh73-node3 --debug {...snip...} Running: /usr/sbin/crm_resource --wait Return Value: 0 --Debug Output Start-- --Debug Output End-- Running: /usr/sbin/crm_node --force --remove rh73-node3 Return Value: 0 --Debug Output Start-- --Debug Output End-- Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:1958 |