Red Hat Bugzilla – Bug 1451097
First galera cluster bootstrap may fail if cluster has no data
Last modified: 2017-08-01 11:00:11 EDT
Description of problem: The resource agent selects the first node to go into Master state based on the biggest commit version found on each node. If case no data were written yet into the galera cluster, the current node is selected as a "fallback" node to bootstrap the cluster. The way the "fallback" node is selected is wrong because every node takes a different decision, and this can yield to several nodes bootstrapping their own cluster instead of all nodes forming a single cluster. Version-Release number of selected component (if applicable): resource-agents-3.9.5-82.el7_3.10.x86_64 How reproducible: Non-deterministic Steps to Reproduce: 1. Deploy an HA overcloud with composable HA, galera resource on 3 dedicated nodes Actual results: Sometimes more than one node can be selected as the bootstrap node, causing the deploy to fail Expected results: A single node should be elected as bootstrap Additional info:
Here [1] the sosreports of two different deploys failed the same way. [1] http://file.rdu.redhat.com/~rscarazz/BZ1451097/
Created attachment 1279321 [details] commit a09b17f79ef3dcde8843ee25bb52b506b6948e8e for downstream
One need to apply in sequence the following commits from upstream: a09b17f79ef3dcde8843ee25bb52b506b6948e8e (downstream version attached) 1fe6d7e9aca159a04e984d693dfcc47580bf5def 0265bcebb24427f5c76cfaa58c281c3aaead7d52
Build with patches from comment #5.
Instruction for testing: 1. create a 3-node pacemaker cluster pcs cluster setup --name foo centos1 centos2 centos3 --force pcs cluster start --all 2. on all nodes, start from a clean mysql database in /var/lib/mysql rm -rf /var/lib/mysql mkdir /var/lib/mysql chown mysql. /var/lib/mysql restorecon /var/lib/mysql 3. create a galera resource, don't start it yet pcs resource create galera galera enable_creation=true wsrep_cluster_address='gcomm://centos1,centos2,centos3' meta master-max=3 --master --disable 4. monitor the cluster after the resource is enabled crm_mon -RrA pcs resource enable galera The last-commit attribute from all nodes will be set to -1 because no WSREP commit has been integrated yet. With the fix, on such start condition, the 3 nodes will always chose centos3 as a bootstrap node, as expected.
Additional comment, if the test is being run on a OpenStack HA overcloud, one should run the additional step on all nodes: 2b. let the resource agent use the default user for polling state rm /etc/sysconfig/clustercheck
Verified on resource-agents-3.9.5-99.el7.x86_64 with steps from comment #8
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:1844