Bug 1721603

Description of problem: It is possible for a particular fence device to be usable for fencing only if a normal (non-fencing) cluster resource is active. (This is acceptable only if the fence device is configured such that it cannot target any nodes allowed to run the normal resource.)

OpenStack is an example where this can occur: if the controller nodes are pacemaker cluster nodes, and the compute nodes are pacemaker remote nodes, then unfencing the remote nodes with fence_compute requires access to the keystone authentication IP running as a pacemaker resource on a controller node.

Currently, this can result in a deadlock if both a dependent target node and the node running the normal resource must be fenced in the same cluster transition, the dependent resource is not functional, and the cluster serializes the normal resource's node fencing after the dependent target fencing (which can happen if the concurrent-fencing cluster property is false, or the resource's node is the DC, which implies it is scheduling itself for fencing for some reason other than complete node loss, such as a failed resource stop).

The proposed solution is a new constraint syntax that would specify the fencing device and the normal resource it depends on.

The most straightforward syntax would be to use pacemaker's current "rsc_order" constraint, with "first" set to the normal resource, "then" set to the fencing resource, and "then-action" set to the new value "fence". (Alternatively "then-action" could be set to "on", "off", or "reboot", but it seems more likely all fence actions would have the same requirement.)

Steps to Reproduce:
1. Configure a cluster such that 2 nodes can be fenced at the same time (e.g. 5 cluster nodes, or 2 cluster nodes plus a remote node).

2. Configure real fencing for all nodes.

3. Configure a normal resource, constrained to a single node (-INFINITY constraints on all other nodes).

4. Configure a fence device that fails if the normal resource is not active. (Dummy agents can be modified for this purpose, or fence_compute can be configured with the keystone IP as the normal resource.) The fence device should be configured in a topology with the real fencing to target some particular node other than the one that runs the normal resource.

5. Set concurrent-fencing=false for ease of testing.

6. Cause the normal resource to be nonfunction, and both the node running the normal resource and the node targeted by the fence device to require fencing (e.g. kill power on both simultaneously).

Actual results: If the cluster serializes the normal resource's node fencing last, the cluster will get stuck in a loop with the dependent fencing action repeatedly failing, and be unable to recover.

Expected results: The cluster eventually recovers properly.