Description of problem: In general pacemaker resource can be stopped for various reasons i.e. manually stopped, failed to start, failed during normal operation, has nowhere to run due to constraint restriction (allowed node is down etc.). RFE: Pacemaker should track the reason why resource is stopped in order to be able to tell that information to pcs which would then either display it in `pcs status` output or would be able to filter out resources stopped by specific action. Some customers would like to have possibility to not display resources which are in stopped state because they have currently nowhere to run due to constraint limitations. The main reason behind this request is to make `pcs status` output more readable in case there are lot of resources (i.e. openstack clusters with lot of cloned resources etc.) I already discussed this with pcs developers and apparently pcs currently doesn't have a way to figure out the reason why resource is stopped so it needs to come from pacemaker first. Then pcs could for example implement parameter to hide constrained resource from the output. EXAMPLE: ======== For cloned cluster resources there are situations in which they cannot run on some of the nodes due to constraints configured in cluster. However the output of 'pcs status' is showing them as 'Stopped' on those nodes which makes the output unclear when there is larger number of nodes. Using 'pcs status --hide-inactive' doesn't solve this problem as it hides all stopped resources regardless of reason why they are stopped. We would like to have option that would hide the 'Stopped' nodes only if the current constraints are clearly prohibiting that node from running the resource. == Environment: Pacemaker cluster. # pcs property list ... Node Attributes: cnodep01: osp-controler=false cnodep02: osp-controler=false cnodep03: osp-controler=false cnodep04: osp-controler=false osconp01: osp-controler=true osconp02: osp-controler=true osconp03: osp-controler=true ... # pcs constraint list ... Resource: nova-compute-clone Constraint: location-nova-compute-clone-1 Rule: score=0 Expression: #kind eq remote Constraint: location-nova-compute-clone (resource-discovery=exclusive) Rule: score=0 Expression: osp-controler eq false ... == Current state: # pcs status ... Clone Set: nova-compute-clone [nova-compute] Started: [ cnodep01 cnodep02 cnodep03 cnodep04 ] Stopped: [ osconp01 osconp02 osconp03 ] ... == Expected results: ## when all nodes are OK ## when some nodes that are controllers(osp-controler=true) are down/faulty # pcs status --hide-stopped-constrained ... Clone Set: nova-compute-clone [nova-compute] Started: [ cnodep01 cnodep02 cnodep03 cnodep04 ] ... ## when some nodes that are not controllers(osp-controler=false) are down/faulty ## when some nodes both controllers and non-controllers are down/faulty # pcs status --hide-stopped-constrained ... Clone Set: nova-compute-clone [nova-compute] Started: [ cnodep01 cnodep04 ] Stopped: [ cnodep02 ] FAILED: [ cnodep03 ] ... ============
Will be investigated in 7.5 timeframe
Due to a short time frame and limited capacity, this will not make 7.5.
A bit of further detail: Pacemaker already has a way to figure out why resources are stopped, for simple cases, and this information is already displayed in crm_mon (pcs status) and available to users and tools via crm_resource --why (or automatically displayed after a crm_resource --cleanup etc.). I see two areas that could be covered by this bz: * enhancing the "why" detection to cover more cases * adding an option to crm_mon to hide banned resources I do not think it is feasible for pcs to parse "why" reasons to implement filters there, nor to have wholly encompassing "why" detection, because (much like AI inputs and outputs) there are often multiple, complex, interacting factors that go into a stop decision, that simply aren't translatable into any easily followed natural language. Looking outside this bz to more general concerns, some other areas that could be addressed: * Static analysis of user configuration (conceptually an expanded version of crm_verify): This would intended to detect configuration issues that are likely the result of user mistakes. There is an upstream bug CLBZ#5277 asking for detection of constraint loops that would fit into this idea. A similar issue would be conflicting constraints that prevent a resource from starting. This would cover a significant part of the problem space this bz is intended to address. * Greatly enhanced use of crm_mon interactive mode (and/or crm_mon HTML output, and/or higher-level GUI tools) to greatly reduce the amount of information displayed, with ability to expand items on demand. For example, the initial display might show just a few lines (roughly one for each display section today); selecting a line would expand it into a list (with various modifiers available in a menu to filter the list), and selecting a list item would give detail on that item.
(In reply to Ken Gaillot from comment #7) > * enhancing the "why" detection to cover more cases I definitely agree that in many cases there isn't gonna be a simple, single and easy to describe reason. The current 'why'-implementation checks for 3 cases as far as I understand it. When extending this I'm afraid of running into reimplementation of the pengine finally (both implementations then aren't totally insync on top probably). That was why I rather had the idea of using pengine itself like having it calculate a transition noting some additional info on a side-track (can be conveyed with the transisiton, possibly switchable to reduce impact on a non-interactive run). > * adding an option to crm_mon to hide banned resources Neglecting the intial text a little and just going for the example a simple search for negative node-weights in placement-constraints might indeed be enough to cope with that example. This should be easy to do in crm_mon with the effects directly visible in pcs.
Looking more deeply at this, I think this BZ definitely needs to focus on the new crm_mon option, to have any chance of getting into 7.7. Adding more cases to the general "why" detection would have to be a separate bz, if we want to pursue that (note that a completely general, human-friendly, machine-parseable "why" as originally requested will never be possible due to the complexity of factors that can be involved in a decision). For the interface, I'm thinking of expanding crm_mon's current --inactive argument to take an optional value, going from the current: -r, --inactive Display inactive resources to: -r, --inactive[=value] Whether to show inactive resources 0=do not show inactive (default if not specified) 1=show inactive unless banned 2=show all inactive (default if specified without value) (Note that the default for crm_mon and pcs is swapped -- crm_mon does not show inactive resources by default, but pcs does. pcs currently calls crm_mon --inactive; it would use --inactive=1 for this BZ's purpose.) Unfortunately the implementation looks more complicated than initially expected. It partly goes back to the same complexity -- when a constraint is applied, pacemaker doesn't necessarily know whether the resource will end up active or not (a ban can be outweighed by some other consideration), and by the end of resource allocation, pacemaker knows whether a resource is active, but the "reason" is a cumulative score summed up from many considerations, not a list of individual considerations.
Due to this project's unexpected complexity and time constraints, this will not make 7.7. I am moving this to RHEL 8 because RHEL 7 will not get new features after 7.7.
Filed an upstream bug that can be used for tracking if the stale rule kicks in again