Bug 1180390

Summary: issue deleting resources with constraint
Product: Red Hat Enterprise Linux 7 Reporter: Fabio Massimo Di Nitto <fdinitto>
Component: pcsAssignee: Tomas Jelinek <tojeline>
Status: CLOSED ERRATA QA Contact: cluster-qe <cluster-qe>
Severity: high Docs Contact:
Priority: high    
Version: 7.1CC: cluster-maint, lmiksik, rsteiger, tojeline
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: pcs-0.9.137-10.el7 Doc Type: Bug Fix
Doc Text:
Cause: User deletes a resource. Consequence: Pcs removes constraints referencing the resource, stops the resource and removes it. This can cause the resource to be moved to other cluster node by pacemaker. Fix: Stop the resource before removing the constraints. Result: Resource is not moved to other cluster nodes.
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-03-05 09:20:55 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
proposed fix none

Description Fabio Massimo Di Nitto 2015-01-09 06:04:39 UTC
Description of problem:

root@rhos6-node1 ~(keystone_admin)]$ pcs resource delete nova-compute
Removing Constraint - colocation-nova-compute-clone-ceilometer-compute-clone-INFINITY
Removing Constraint - order-ceilometer-compute-clone-nova-compute-clone-mandatory
Removing Constraint - order-nova-conductor-clone-nova-compute-clone-mandatory
Attempting to stop: nova-compute...Stopped
Deleting Resource - nova-compute

pcs resource delete operation ordering is dangerous.

Removing the constraints before stopping the resource has a race condition where the resource will be started on other nodes even for a few seconds, where things could go potentially very wrong.

The ideal sequence should be:

pcs resource delete nova-compute
Attempting to stop: nova-compute...Stopped
Removing Constraint - colocation-nova-compute-clone-ceilometer-compute-clone-INFINITY
Removing Constraint - order-ceilometer-compute-clone-nova-compute-clone-mandatory
Removing Constraint - order-nova-conductor-clone-nova-compute-clone-mandatory
Deleting Resource - nova-compute

Comment 1 Tomas Jelinek 2015-01-09 12:47:34 UTC
Created attachment 978135 [details]
proposed fix

Comment 2 Tomas Jelinek 2015-01-09 16:01:59 UTC
Before fix:
[root@rh70-node1:~]# rpm -q pcs
pcs-0.9.137-9.el7.x86_64
[root@rh70-node1:~]# pcs resource create dummy1 dummy --clone
[root@rh70-node1:~]# pcs resource create dummy2 dummy --clone
[root@rh70-node1:~]# pcs resource create dummy3 dummy --clone
[root@rh70-node1:~]# pcs constraint colocation add dummy1-clone dummy2-clone
[root@rh70-node1:~]# pcs constraint order dummy2-clone then dummy1-clone
Adding dummy2-clone dummy1-clone (kind: Mandatory) (Options: first-action=start then-action=start)
[root@rh70-node1:~]# pcs constraint order dummy3-clone then dummy1-clone
Adding dummy3-clone dummy1-clone (kind: Mandatory) (Options: first-action=start then-action=start)
[root@rh70-node1:~]# pcs resource delete dummy1
Removing Constraint - colocation-dummy1-clone-dummy2-clone-INFINITY
Removing Constraint - order-dummy2-clone-dummy1-clone-mandatory
Removing Constraint - order-dummy3-clone-dummy1-clone-mandatory
Attempting to stop: dummy1...Stopped
Deleting Resource - dummy1

After fix:
[root@rh70-node1:~]# rpm -q pcs
pcs-0.9.137-10.el7.x86_64
[root@rh70-node1:~]# pcs resource create dummy1 dummy --clone
[root@rh70-node1:~]# pcs resource create dummy2 dummy --clone
[root@rh70-node1:~]# pcs resource create dummy3 dummy --clone
[root@rh70-node1:~]# pcs constraint colocation add dummy1-clone dummy2-clone
[root@rh70-node1:~]# pcs constraint order dummy2-clone then dummy1-clone
Adding dummy2-clone dummy1-clone (kind: Mandatory) (Options: first-action=start then-action=start)
[root@rh70-node1:~]# pcs constraint order dummy3-clone then dummy1-clone
Adding dummy3-clone dummy1-clone (kind: Mandatory) (Options: first-action=start then-action=start)
[root@rh70-node1:~]# pcs resource delete dummy1
Attempting to stop: dummy1...Stopped
Removing Constraint - colocation-dummy1-clone-dummy2-clone-INFINITY
Removing Constraint - order-dummy2-clone-dummy1-clone-mandatory
Removing Constraint - order-dummy3-clone-dummy1-clone-mandatory
Deleting Resource - dummy1

Comment 6 errata-xmlrpc 2015-03-05 09:20:55 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-0415.html