There is currently an error in the way pacemaker start/stop ordering constraint are expressed. This can potentially lead to a cluster meltdown when issuing: pcs cluster stop --all because some services will fail to stop. The service will try to contact another API to notify the shutdown, but the VIP for the API is already down at that stage. Workaround: pcs cluster disable keystone wait for keystone to be Stopped pcs cluster stop --all on start: pcs cluster start --all pcs enable keystone This specific sequence affects only the process to put all controller nodes in shutdown at once. It does NOT affect reboot or shutdown of one node at a time (for upgrade purposes for example), hence the medium severity. I am currently working on a new constraint set that should prevent this problem, in the meantime this should be documented in the release notes for GA.
Reference arch: https://docs.google.com/a/redhat.com/document/d/1iO41-wcY81xKn46UDkjZ-HGFR80ARXwEElRqd2HtDI8/edit# is now updated with the new constraint order in v0.5 For expert users: - in previous setups the VIPs and lb-haproxy-clone start order was expressed as: pcs constraint order start lb-haproxy-clone then vip-... - this needs to be reversed by: pcs constraint delete order-lb-haproxy-clone-vip-...-mandatory pcs constraint order start vip-... then lb-haproxy-clone Replace "..." with the current name of the vip- service and it has to be done for all vip. This change can be applied on a live cluster (no need to stop any service to perform this change).
> pcs constraint order start vip-... then lb-haproxy-clone > > Replace "..." with the current name of the vip- service and it has to be > done for all vip. > > This change can be applied on a live cluster (no need to stop any service to > perform this change). this turns out not to be the correct command. The correct command is: pcs constraint order start vip-... then lb-haproxy-clone kind=Optional (note the extra option) This change is required to avoid a chain of events in start/stop all services when a VIP needs to move from node to another.