Description of problem:
galera being a M/S resource, the resource agent decides when and how
to promote a resource replica (i.e. start a galera process locally)
based on the current state of the entire galera cluster:

  . If there's no galera cluster, the replica is promoted as the
    bootstrap node: it will start a galera process that will bootstrap
    a new galera cluster.

  . If there's a running galera cluster: the replica is promoted as a
    joiner node: it will join the running galera cluster, and will
    never try to bootstrap a new cluster.

When one changes a property of a pacemaker resource, pacemaker must restart
the resource on all the nodes. For instance for galera:

[root@controller-0 ~]# pcs resource show galera
 Resource: galera (class=ocf provider=heartbeat type=galera)
  Attributes: additional_parameters=--open-files-limit=16384 cluster_host_map=controller-0:controller-0.internalapi.redhat.local;controller-1:controller-1.internalapi.redhat.local;controller-2:controller-2.internalapi.redhat.local enable_creation=true log=/var/log/mysql/mysqld.log wsrep_cluster_address=gcomm://controller-0.internalapi.redhat.local,controller-1.internalapi.redhat.local,controller-2.internalapi.redhat.local
  Meta Attrs: container-attribute-target=host master-max=3 ordered=true
  Operations: demote interval=0s timeout=120s (galera-demote-interval-0s)
              monitor interval=20s timeout=30s (galera-monitor-interval-20s)
              monitor interval=10s role=Master timeout=30s (galera-monitor-interval-10s)
              monitor interval=30s role=Slave timeout=30s (galera-monitor-interval-30s)
              promote interval=0s on-fail=block timeout=300s (galera-promote-interval-0s)
              start interval=0s timeout=120s (galera-start-interval-0s)
              stop interval=0s timeout=120s (galera-stop-interval-0s)

[root@controller-0 ~]# pcs resource update galera additional_parameters=--open-files-limit=20000

For all galera replicas, pacemaker will trigger a demote, the the resource agent
will eventually request a promote operation, and configure the local node as a
bootstrap node or a joiner node depending on the state of the galera cluster
when the agent was called.

During such a rolling restart, on galera node can request a promotion
as a joiner node because when the agent was called some replicas were
still running as Master.

However, ther be some time between the moment when a node is promoted and
when the promote operation effectively takes place. So if a node is
promoted for joining a cluster, all the running galera nodes are
stopped before the promote operation start, the joining node won't be
able to join the cluster, and it can't bootstrap a new one either
because it doesn't have the most recent copy of the DB.

This promotion window makes the resource agent fail its promotion, and
blocks the replica on this node until a manual "pcs resource cleanup galera"
is executed.


Version-Release number of selected component (if applicable):
resource-agents-4.1.1-61.el7.x86_64

How reproducible:
Timing-dependent, but happens almost always in OpenStack HA control plane

Steps to Reproduce:
1. Deploy an HA overcloud
2. On a controller node, change a resource parameter as shown above

Actual results:
One replica failed to restart due to all master being stopped before
the promotion effectively takes place:

* galera_promote_0 on galera-bundle-2 'unknown error' (1): call=998, status=complete, exitreason='Failure, Attempted to promote Master instance of galera before bootstrap node has been detected.',
    last-rc-change='Mon Sep 21 14:30:34 2020', queued=0ms, exec=1535ms


Expected results:
Promotion should no longer be attempted because Master are gone, this shouldn't be fatal.