Can you attach a policy engine file from after the upgrade? One of the older nodes will have messages from around that time like "Calculated transition ..., saving inputs in ..." with the filename. Any such file during the time the wait was hanging should be sufficient.

Created attachment 1393523 [details]
pe-input from Master node

'pengine-node113.tar.gz' archive with pe-input* files from node 113

Created attachment 1393524 [details]
outputs from cluster with timestamps reproducing the issue

As noted earlier in the case: in pacemaker-1.1.16-12.el7_4.4.src.rpm, file 089-unfencing.patch, "[PATCH 02/12] Support unfencing of remote nodes", I see some changes made to the StopRsc and StartRsc functions in native.c that may be relevant. Let me know if I'm barking up the wrong tree. For example:

diff --git a/pengine/native.c b/pengine/native.c
index dd5ff18..bc59405 100644
--- a/pengine/native.c
+++ b/pengine/native.c
@@ -1355,7 +1355,7 @@ native_internal_constraints(resource_t * rsc, pe_working_set_t * data_set)

         g_hash_table_iter_init(&iter, rsc->allowed_nodes);
         while (g_hash_table_iter_next(&iter, NULL, (void **)&node)) {
-            action_t *unfence = pe_fence_op(node, "on", TRUE, data_set);
+            action_t *unfence = pe_fence_op(node, "on", TRUE, __FUNCTION__, data_set);

             crm_debug("Ordering any stops of %s before %s, and any starts after",
                       rsc->id, unfence->uuid);
@@ -2455,6 +2455,16 @@ StopRsc(resource_t * rsc, node_t * next, gboolean optional, pe_working_set_t * d
         if (is_set(data_set->flags, pe_flag_remove_after_stop)) {
             DeleteRsc(rsc, current, optional, data_set);
         }
+
+        if(is_set(rsc->flags, pe_rsc_needs_unfencing)) {
+            action_t *unfence = pe_fence_op(current, "on", TRUE, __FUNCTION__, data_set);
+            const char *unfenced = g_hash_table_lookup(current->details->attrs, XML_NODE_IS_UNFENCED);
+
+            order_actions(stop, unfence, pe_order_implies_first);
+            if (unfenced == NULL || safe_str_eq("0", unfenced)) {
+                pe_proc_err("Stopping %s until %s can be unfenced", rsc->id, current->details->uname);
+            }
+        }
     }

     return TRUE;
@@ -2468,9 +2478,25 @@ StartRsc(resource_t * rsc, node_t * next, gboolean optional, pe_working_set_t *
     CRM_ASSERT(rsc);
     pe_rsc_trace(rsc, "%s on %s %d", rsc->id, next ? next->details->uname : "N/A", optional);
     start = start_action(rsc, next, TRUE);
+
+    if(is_set(rsc->flags, pe_rsc_needs_unfencing)) {
+        action_t *unfence = pe_fence_op(next, "on", TRUE, __FUNCTION__, data_set);
+        const char *unfenced = g_hash_table_lookup(next->details->attrs, XML_NODE_IS_UNFENCED);
+
+        order_actions(unfence, start, pe_order_implies_then);
+
+        if (unfenced == NULL || safe_str_eq("0", unfenced)) {
+            char *reason = crm_strdup_printf("Required by %s", rsc->id);
+            trigger_unfencing(NULL, next, reason, NULL, data_set);
+            free(reason);
+        }
+    }
+
     if (is_set(start->flags, pe_action_runnable) && optional == FALSE) {
         update_action_flags(start, pe_action_optional | pe_action_clear, __FUNCTION__, __LINE__);
     }
+
+
     return TRUE;
 }

@@ -2820,14 +2846,9 @@ native_create_probe(resource_t * rsc, node_t * node, action_t * complete,
      * probed, we know the state of all resources that require
      * unfencing and that unfencing occurred.
      */
-    if(is_set(rsc->flags, pe_rsc_fence_device) && is_set(data_set->flags, pe_flag_enable_unfencing)) {
-        trigger_unfencing(NULL, node, "node discovery", probe, data_set);
-        probe->priority = INFINITY; /* Ensure this runs if unfencing succeeds */
-
-    } else if(is_set(rsc->flags, pe_rsc_needs_unfencing)) {
-        action_t *unfence = pe_fence_op(node, "on", TRUE, data_set);
-
-        order_actions(probe, unfence, pe_order_optional);
+    if(is_set(rsc->flags, pe_rsc_needs_unfencing)) {
+        action_t *unfence = pe_fence_op(node, "on", TRUE, __FUNCTION__, data_set);
+        order_actions(unfence, probe, pe_order_optional);
     }

     /*

(In reply to Reid Wahl from comment #5)
> As noted earlier in the case: in pacemaker-1.1.16-12.el7_4.4.src.rpm, file
> 089-unfencing.patch, "[PATCH 02/12] Support unfencing of remote nodes", I
> see some changes made to the StopRsc and StartRsc functions in native.c that
> may be relevant. Let me know if I'm barking up the wrong tree.

It's indirectly relevant.

Pacemaker's "policy engine" is the code that decides what (if anything) needs to be done at the moment. The node with the longest uptime (always the older node in the attached policy engine files) will be elected the "DC", which runs the policy engine as needed.

"crm_resource --wait" also runs the policy engine on the current configuration, and returns once no more actions are seen to be necessary.

Based on the output in the Description, "crm_resource --wait" is being run on the upgraded node.

The older policy engine (run by the DC to determine actual actions) is generating a different list of required actions than the newer policy engine run by crm_resource. Thus, crm_resource is waiting for additional actions that the older DC will never do.

This raises an interesting (and so far unaddressed) question of whether "crm_resource --wait" can be supported on nodes with a different pacemaker version than the DC. It's likely impossible to guarantee any reliability of that.

I think the best we can do is make "crm_resource --wait" print a warning about the possibility if it is run in that situation. We can also update any documentation around upgrades to mention the issue.

Separately, I'm not sure the actions chosen by the newer version are ideal. I'm going to take a closer look at that as well.

A possible full solution would be to reimplement the crm_resource --wait code as a new crmd operation, which the local crmd would pass to the DC. That's obviously a bigger project though, and could have undesirable side effects (additional load on the DC, special handling needed if the DC election changes during the wait, etc.), so I'm not sure how feasible it is.

(In reply to Ken Gaillot from comment #6)
> Separately, I'm not sure the actions chosen by the newer version are ideal.
> I'm going to take a closer look at that as well.

The new actions are fine. The newer code utilizes a new node attribute to record the time of last unfencing. Since the configuration doesn't have that attribute, it will decide to unfence both nodes. That would be an expected side effect of an upgrade across those versions.

Warning implemented by upstream commit 8978473

QA: Test procedure:

* Configure a cluster with at least two nodes using 7.4 pacemaker packages.
* Upgrade pacemaker on one of the nodes (to 7.5 before fix, 7.6 after fix).
* Run "crm_resource --wait" on the upgraded node.

The 7.5 pacemaker packages will not give any indication that anything is wrong. The 7.6 packages will print a warning about mixed-version clusters.

before:
=======

Both cluster nodes running version 1.1.16-12.el7, one node upgraded to 1.1.18-10.el7 (7.5 packages).

> [root@virt-253 ~]# pcs status
> Cluster name: bzzt
> Stack: corosync
> Current DC: virt-254.cluster-qe.lab.eng.brq.redhat.com (version 1.1.16-12.el7-94ff4df) - partition with quorum
> Last updated: Mon Aug 20 14:40:07 2018
> Last change: Mon Aug 20 14:40:04 2018 by root via cibadmin on virt-253.cluster-qe.lab.eng.brq.redhat.com
> 
> 2 nodes configured
> 1 resource configured
> 
> Online: [ virt-253.cluster-qe.lab.eng.brq.redhat.com virt-254.cluster-qe.lab.eng.brq.redhat.com ]
> 
> Full list of resources:
> 
>  dummy	(ocf::pacemaker:Dummy):	Started virt-253.cluster-qe.lab.eng.brq.redhat.com
> 
> Daemon Status:
>   corosync: active/enabled
>   pacemaker: active/enabled
>   pcsd: active/enabled
> [root@virt-253 ~]# pcs node standby virt-254.cluster-qe.lab.eng.brq.redhat.com --wait
> [root@virt-254 ~]# yum install -y /mnt/redhat/brewroot/packages/pacemaker/1.1.18/10.el7/x86_64/pacemaker{,-cli,-cluster-libs,-libs}-1.1.18-10.el7*.rpm
> [root@virt-253 ~]# pcs node unstandby virt-254.cluster-qe.lab.eng.brq.redhat.com --wait
> [root@virt-254 ~]# crm_resource --wait
> [root@virt-254 ~]# echo $?
> 0

After upgrading one of the nodes to 7.5 pacemaker packages, "crm_resource --wait" does not print any warning when invoked from the upgraded node. Depending on the cluster configuration, the command may exit cleanly or wait for a cluster transition that will never happen.


after:
======

Both cluster nodes running version 1.1.16-12.el7, one node upgraded to 1.1.19-7.el7 (7.6 packages).

> [root@virt-253 ~]# pcs node standby virt-254.cluster-qe.lab.eng.brq.redhat.com --wait
> [root@virt-254 ~]# yum install -y /mnt/redhat/brewroot/packages/pacemaker/1.1.19/7.el7/x86_64/pacemaker{,-cli,-cluster-libs,-libs}-1.1.19-7.el7*.rpm
> [root@virt-253 ~]# pcs node unstandby virt-254.cluster-qe.lab.eng.brq.redhat.com --wait
> [root@virt-254 ~]# crm_resource --wait
> warning: --wait command may not work properly in mixed-version cluster
> [root@virt-254 ~]# echo $?
> 0

With 7.6 packages, the "crm_resource --wait" command works the same way as before (ie. might wait indefinitely) but a new warning is printed upon every invocation.

Marking verified in 1.1.19-7.el7.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3055