Created attachment 1507690 [details]
crm_reports taken after running the idiom

Description of problem:

In OpenStack we have an idiomatic way to force a resource parameter
update without restarting the resource. This is used when we know we
don't require a immediate service restart and we want to avoid service
disruption.

We used to do the following high level step, e.g. for galera:
  . unmanage the resource
  . update the resource's property
  . refresh the resource to force pacemaker to reprobe the
    resource's state and forget about past operations
  . manage the resource

Currently in RHEL7.6, this does not prevent the resource from restarting
any longer. This causes unwanted resource restarts in our workload for
various scenarios (config update, cluster node replacement...)


Version-Release number of selected component (if applicable):
pacemaker-1.1.19-8.el7_6.1.x86_64

How reproducible:
Always

Steps to Reproduce:
0. create a 3-node cluster on three hosts called ra1, ra2, ra3

1. install the galera config provided in attachement on all cluster nodes
mkdir /etc/my.cnf.d
# copy galera.cnf attachment in /etc/my.cnf.d/galera.cnf

2. create a bundled galera master/slave resource on a cluster
pcs resource bundle create galera-bundle container docker image=docker.io/tripleoqueens/centos-binary-mariadb:current-tripleo-rdo network=host options="--user=root --log-driver=journald" replicas=3 masters=3 run-command="/usr/sbin/pacemaker_remoted" network control-port=3123 storage-map id=map0 source-dir=/dev/log target-dir=/dev/log storage-map id=map1 source-dir=/dev/zero target-dir=/etc/libqb/force-filesystem-sockets options=ro storage-map id=map2 source-dir=/etc/hosts target-dir=/etc/hosts options=ro storage-map id=map3 source-dir=/etc/localtime target-dir=/etc/localtime options=ro storage-map id=map4 source-dir=/etc/my.cnf.d target-dir=/etc/my.cnf.d options=ro storage-map id=map5 source-dir=/var/lib/mysql target-dir=/var/lib/mysql options=rw storage-map id=map6 source-dir=/var/log/mysql target-dir=/var/log/mysql options=rw storage-map id=pcmk1 source-dir=/var/log/pacemaker target-dir=/var/log/pacemaker options=rw --disabled

pcs resource create galera ocf:heartbeat:galera wsrep_cluster_address='gcomm://ra1,ra2,ra3' op promote timeout=60 on-fail=block meta container-attribute-target=host notify=true bundle galera-bundle

pcs resource enable galera-bundle

3. run the resource update idiom
pcs resource unmanage galera-bundle
pcs resource update galera additional_parameters=--open-files-limit=2048
pcs resource refresh galera-bundle
pcs resource manage galera-bundle

Actual results:
After the refresh, the three galera replicas are restarted

Expected results:
The galera replicas should not restart

Additional info:
quoting beekhof after he had a quick look at the attached crm_reports:
"""
<beekhof> the basic problem appears to be that start_mainloop() is not actually starting mainloop
<beekhof> allowing crm_resource to terminate early and causing the crmd to drop any commands it sent
<beekhof> maybe crmd_replies_needed needs to be declared volitile
<beekhof> because the code looks correct
"""

In addition to the crm_reports grabbed after running the update idiom,
I'm attaching the verbose logs captured while running:
 '/usr/sbin/crm_resource --refresh --resource galera-bundle -VVVVVV'