Bug 1084808
| Summary: | Policy engine doesn't restore all services to original status | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 6 | Reporter: | Fabio Massimo Di Nitto <fdinitto> | ||||
| Component: | pacemaker | Assignee: | Andrew Beekhof <abeekhof> | ||||
| Status: | CLOSED NOTABUG | QA Contact: | Cluster QE <mspqa-list> | ||||
| Severity: | medium | Docs Contact: | |||||
| Priority: | medium | ||||||
| Version: | 6.7 | CC: | cluster-maint, dvossel | ||||
| Target Milestone: | rc | ||||||
| Target Release: | --- | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2014-10-23 03:55:54 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
(In reply to Fabio Massimo Di Nitto from comment #0) Apr 6 20:43:22 rhel6-ha-node1 pengine[2129]: notice: LogActions: Start dummy2#011(rhel6-ha-node4) Apr 6 20:43:22 rhel6-ha-node1 pengine[2129]: notice: LogActions: Start dummy3#011(rhel6-ha-node3) Apr 6 20:43:22 rhel6-ha-node1 pengine[2129]: notice: LogActions: Start dummy4#011(rhel6-ha-node1) Apr 6 20:43:22 rhel6-ha-node1 pengine[2129]: notice: LogActions: Start dummy5#011(rhel6-ha-node2) Apr 6 20:43:22 rhel6-ha-node1 pengine[2129]: notice: LogActions: Start virt-fencing#011(rhel6-ha-node1) > on node4: > pcs cluster stop > > wait -> at this stage at lease one resource cannot run (expected) two actually, since you have 5 resources and 3 nodes Apr 6 20:44:04 rhel6-ha-node1 pengine[2129]: notice: LogActions: Stop dummy2#011(rhel6-ha-node4) Apr 6 20:44:08 rhel6-ha-node1 pengine[2129]: notice: stage6: Scheduling Node rhel6-ha-node4 for shutdown > > pcs cluster start > > dummy1 was in a "Stopped" condition, even tho there are nodes to start it. > (unexpected). no there's not :) node4 comes back but dummy2 is put back there. Apr 6 20:44:29 rhel6-ha-node1 pengine[2129]: notice: LogActions: Start dummy2#011(rhel6-ha-node4) So actually we restored it exactly how it was prior to the shutdown :) > > My expectations are, that once there are all nodes back, and space to start > the resources, those should migrate to free space to start dummy1. There is no reason for pacemaker to attempt this, you've not told it that dummy1 is more important. Funny thing: dummy1 is not allocated because of all the colocation constraints. In order to know where it can't go, we need to follow all the constraints and find a place for dummy{2,3,4,5} first - leaving nowhere for dummy1. -ENOBUG :) (In reply to Andrew Beekhof from comment #1) > (In reply to Fabio Massimo Di Nitto from comment #0) > > Apr 6 20:43:22 rhel6-ha-node1 pengine[2129]: notice: LogActions: Start > dummy2#011(rhel6-ha-node4) > Apr 6 20:43:22 rhel6-ha-node1 pengine[2129]: notice: LogActions: Start > dummy3#011(rhel6-ha-node3) > Apr 6 20:43:22 rhel6-ha-node1 pengine[2129]: notice: LogActions: Start > dummy4#011(rhel6-ha-node1) > Apr 6 20:43:22 rhel6-ha-node1 pengine[2129]: notice: LogActions: Start > dummy5#011(rhel6-ha-node2) > Apr 6 20:43:22 rhel6-ha-node1 pengine[2129]: notice: LogActions: Start > virt-fencing#011(rhel6-ha-node1) > > > on node4: > > pcs cluster stop > > > > wait -> at this stage at lease one resource cannot run (expected) > > two actually, since you have 5 resources and 3 nodes > > Apr 6 20:44:04 rhel6-ha-node1 pengine[2129]: notice: LogActions: Stop > dummy2#011(rhel6-ha-node4) > Apr 6 20:44:08 rhel6-ha-node1 pengine[2129]: notice: stage6: Scheduling > Node rhel6-ha-node4 for shutdown > > > > > pcs cluster start > > > > dummy1 was in a "Stopped" condition, even tho there are nodes to start it. > > (unexpected). > > no there's not :) > node4 comes back but dummy2 is put back there. > > Apr 6 20:44:29 rhel6-ha-node1 pengine[2129]: notice: LogActions: Start > dummy2#011(rhel6-ha-node4) > > So actually we restored it exactly how it was prior to the shutdown :) ehm.. no. prior to shutdown all resource were running :) > > > > > My expectations are, that once there are all nodes back, and space to start > > the resources, those should migrate to free space to start dummy1. > > There is no reason for pacemaker to attempt this, you've not told it that > dummy1 is more important. So ok, it's a possible configuration error. How do we solve it from a user perspective? > > Funny thing: dummy1 is not allocated because of all the colocation > constraints. > In order to know where it can't go, we need to follow all the constraints > and find a place for dummy{2,3,4,5} first - leaving nowhere for dummy1. > > -ENOBUG :) but what I don't understand is: - prior to shutdown, all resources were running - stop one node (clearly this trigger some resource NOT to run) - start the node (now theoretically there is space for dummy1, by shuffling things around) - dummy1 can't find space due to lack of shuffling. Am I missing something? In some other run of this test case, I noticed that dummy1 was not always the one being hit. Happy to close this as non-bug of course. I just need to understand what's happening, why and how eventually a customer come out of this condition. PS there is a use case where this might be necessary in the short future. (In reply to Fabio Massimo Di Nitto from comment #2) > (In reply to Andrew Beekhof from comment #1) > > So actually we restored it exactly how it was prior to the shutdown :) > > ehm.. no. prior to shutdown all resource were running :) not according to the logs (In reply to Andrew Beekhof from comment #3) > (In reply to Fabio Massimo Di Nitto from comment #2) > > (In reply to Andrew Beekhof from comment #1) > > > > So actually we restored it exactly how it was prior to the shutdown :) > > > > ehm.. no. prior to shutdown all resource were running :) > > not according to the logs Ok i´ll reproduce and report again. pcs status did report dummy1 running before shutdown. (In reply to Fabio Massimo Di Nitto from comment #4) > (In reply to Andrew Beekhof from comment #3) > > (In reply to Fabio Massimo Di Nitto from comment #2) > > > (In reply to Andrew Beekhof from comment #1) > > > > > > So actually we restored it exactly how it was prior to the shutdown :) > > > > > > ehm.. no. prior to shutdown all resource were running :) > > > > not according to the logs > > Ok i´ll reproduce and report again. pcs status did report dummy1 running > before shutdown. _That_ would be a bug :) If there are 5 resources (none of which can run together) and only 4 nodes, then clearly one must have been in a stopped state (In reply to Andrew Beekhof from comment #5) > (In reply to Fabio Massimo Di Nitto from comment #4) > > (In reply to Andrew Beekhof from comment #3) > > > (In reply to Fabio Massimo Di Nitto from comment #2) > > > > (In reply to Andrew Beekhof from comment #1) > > > > > > > > So actually we restored it exactly how it was prior to the shutdown :) > > > > > > > > ehm.. no. prior to shutdown all resource were running :) > > > > > > not according to the logs > > > > Ok i´ll reproduce and report again. pcs status did report dummy1 running > > before shutdown. > > _That_ would be a bug :) > > If there are 5 resources (none of which can run together) and only 4 nodes, > then clearly one must have been in a stopped state That's not the case. There are 5 resources and only 3 cannot run together with the others. 1 must run alone 2 must run alone 3 must run alone 4/5 can run together pcs resource create dummy1 ocf:heartbeat:Dummy pcs resource create dummy2 ocf:heartbeat:Dummy pcs resource create dummy3 ocf:heartbeat:Dummy pcs resource create dummy4 ocf:heartbeat:Dummy pcs resource create dummy5 ocf:heartbeat:Dummy # dummy1 should always run on a node without other dummy services pcs constraint colocation add dummy1 with dummy2 -INFINITY pcs constraint colocation add dummy1 with dummy3 -INFINITY pcs constraint colocation add dummy1 with dummy4 -INFINITY pcs constraint colocation add dummy1 with dummy5 -INFINITY # dummy2 should always run on a node without other dummy services #pcs constraint colocation add dummy2 with dummy1 -INFINITY pcs constraint colocation add dummy2 with dummy3 -INFINITY pcs constraint colocation add dummy2 with dummy4 -INFINITY pcs constraint colocation add dummy2 with dummy5 -INFINITY # dummy3 should always run on a node without other dummy services #pcs constraint colocation add dummy3 with dummy1 -INFINITY #pcs constraint colocation add dummy3 with dummy2 -INFINITY pcs constraint colocation add dummy3 with dummy4 -INFINITY pcs constraint colocation add dummy3 with dummy5 -INFINITY try this on a node4 cluster and shut down one node. I am currently using my cluster to solve another more important issue and wont be able to look at it for a while. Just looked at the logs again. At no point was dummy1 running. Ok, so classic backpacker problem... the human brain can easily see that dummy 4/5 should run on the same node which would make room for dummy1 but this is not at all obvious to the cluster. 'pcs constraint colocation add dummy4 with dummy5 1' would have helped though. Closing. The PE can only infer so much and sometimes needs additional hints. |
Created attachment 883370 [details] crm_report This is a rather odd corner case setup (found while demo'ing some complex conditions). How to reproduce: pcs cluster setup --name demo rhel6-ha-node1 rhel6-ha-node2 rhel6-ha-node3 rhel6-ha-node4 pcs cluster start # wait ~30 sec pcs status # show resources will start only after fencing is configured # setup fencing (from one node) pcs stonith create virt-fencing fence_xvm op monitor interval=10s pcs resource create dummy1 ocf:heartbeat:Dummy pcs resource create dummy2 ocf:heartbeat:Dummy pcs resource create dummy3 ocf:heartbeat:Dummy pcs resource create dummy4 ocf:heartbeat:Dummy pcs resource create dummy5 ocf:heartbeat:Dummy # dummy1 should always run on a node without other dummy services pcs constraint colocation add dummy1 with dummy2 -INFINITY pcs constraint colocation add dummy1 with dummy3 -INFINITY pcs constraint colocation add dummy1 with dummy4 -INFINITY pcs constraint colocation add dummy1 with dummy5 -INFINITY # dummy2 should always run on a node without other dummy services #pcs constraint colocation add dummy2 with dummy1 -INFINITY pcs constraint colocation add dummy2 with dummy3 -INFINITY pcs constraint colocation add dummy2 with dummy4 -INFINITY pcs constraint colocation add dummy2 with dummy5 -INFINITY ## NOTE: a bit extreme for a 4 node cluster # dummy3 should always run on a node without other dummy services #pcs constraint colocation add dummy3 with dummy1 -INFINITY #pcs constraint colocation add dummy3 with dummy2 -INFINITY pcs constraint colocation add dummy3 with dummy4 -INFINITY pcs constraint colocation add dummy3 with dummy5 -INFINITY on node4: pcs cluster stop wait -> at this stage at lease one resource cannot run (expected) pcs cluster start dummy1 was in a "Stopped" condition, even tho there are nodes to start it. (unexpected). My expectations are, that once there are all nodes back, and space to start the resources, those should migrate to free space to start dummy1. It's entirely possible that my expectations are wrong, but I couldn't find a way to recover from this status. pcs resource enable dummy1 (not part of this crm_report) didn't work.