Bug 1499677

Description of problem:
/var/lib/mysql/gvwstate.dat gets corrupted on one of the controller nodes after rebooting them post upgrade.

Version-Release number of selected component (if applicable):
mariadb-galera-server-5.5.42-6.el7ost.x86_64
mariadb-galera-common-5.5.42-6.el7ost.x86_64
galera-25.3.16-3.el7ost.x86_64

How reproducible:
This is not always reproducible - I've observed it on 1/3 attempts

Steps to Reproduce:
1. Deploy OSP11 with 3 controllers, 2 computes, 3 ceph nodes
2. Upgrade to OSP12
3. Force reboot controller-2, wait for all PCS services to come back up
4. Force reboot controller-0, wait for all PCS services to come back up
5. Force reboot controller-1

Actual results:
Galera cannot start on controller-1:

 Docker container set: galera-bundle [192.168.24.1:8787/rhosp12/openstack-mariadb-docker:pcmklatest]
   galera-bundle-0	(ocf::heartbeat:galera):	FAILED Master controller-1 (blocked)
   galera-bundle-1	(ocf::heartbeat:galera):	Master controller-2
   galera-bundle-2	(ocf::heartbeat:galera):	Master controller-0

Checking the logs inside the containers we can see:

171009 00:23:30 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
171009 00:23:30 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.P1xlGa' --pid-file='/var/lib/mysql/controller-1-recover.pid'
171009  0:23:30 [Warning] option 'open_files_limit': unsigned value 18446744073709551615 adjusted to 4294967295
171009 00:23:32 mysqld_safe WSREP: Recovered position 894400c0-ac6d-11e7-9bee-86888296944d:25272
171009  0:23:32 [Warning] option 'open_files_limit': unsigned value 18446744073709551615 adjusted to 4294967295
171009  0:23:32 [Note] WSREP: wsrep_start_position var submitted: '894400c0-ac6d-11e7-9bee-86888296944d:25272'
171009  0:23:32 [Note] WSREP: Read nil XID from storage engines, skipping position init
171009  0:23:32 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera/libgalera_smm.so'
171009  0:23:32 [Note] WSREP: wsrep_load(): Galera 3.16(r5c765eb) by Codership Oy <info> loaded successfully.
171009  0:23:32 [Note] WSREP: CRC-32C: using hardware acceleration.
171009  0:23:32 [Note] WSREP: Found saved state: 894400c0-ac6d-11e7-9bee-86888296944d:-1
171009  0:23:32 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = controller-1; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = tcp://172.17.1.14:4567; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; p
171009  0:23:32 [Note] WSREP: Service thread queue flushed.
171009  0:23:32 [Note] WSREP: Assign initial position for certification: 25272, protocol version: -1
171009  0:23:32 [Note] WSREP: wsrep_sst_grab()
171009  0:23:32 [Note] WSREP: Start replication
171009  0:23:32 [Note] WSREP: Setting initial position to 894400c0-ac6d-11e7-9bee-86888296944d:25272
171009  0:23:32 [Note] WSREP: protonet asio version 0
171009  0:23:32 [Note] WSREP: Using CRC-32C for message checksums.
171009  0:23:32 [Note] WSREP: backend: asio
171009  0:23:32 [Note] WSREP: gcomm thread scheduling priority set to other:0 
171009  0:23:32 [Note] WSREP: restore pc from disk successfully
171009  0:23:32 [Note] WSREP: GMCast version 0
171009  0:23:32 [Note] WSREP: (00000000, 'tcp://172.17.1.14:4567') listening at tcp://172.17.1.14:4567
171009  0:23:32 [Note] WSREP: (00000000, 'tcp://172.17.1.14:4567') multicast: , ttl: 1
171009  0:23:32 [ERROR] WSREP: failed to open gcomm backend connection: 131: invalid UUID: 00000000 (FATAL)
	 at gcomm/src/pc.cpp:PC():271
171009  0:23:32 [ERROR] WSREP: gcs/src/gcs_core.cpp:gcs_core_open():208: Failed to open backend connection: -131 (State not recoverable)
171009  0:23:32 [ERROR] WSREP: gcs/src/gcs.cpp:gcs_open():1380: Failed to open channel 'galera_cluster' at 'gcomm://controller-0.internalapi.localdomain,controller-1.internalapi.localdomain,controller-2.internalapi.localdomain': -131 (State not recoverable)
171009  0:23:32 [ERROR] WSREP: gcs connect failed: State not recoverable
171009  0:23:32 [ERROR] WSREP: wsrep::connect() failed: 7
171009  0:23:32 [ERROR] Aborting

171009  0:23:32 [Note] WSREP: Service disconnected.
171009  0:23:33 [Note] WSREP: Some threads may fail to exit.
171009  0:23:33 [Note] /usr/libexec/mysqld: Shutdown complete

Checking /var/lib/mysql/gvwstate.dat on the controller nodes we can see that on controller-1 it's empty:

(undercloud) [stack@undercloud-0 ~]$ ssh -q heat-admin.24.10 'cat /etc/hostname; sudo wc -c /var/lib/mysql/gvwstate.dat'
controller-0
217 /var/lib/mysql/gvwstate.dat
(undercloud) [stack@undercloud-0 ~]$ ssh -q heat-admin.24.7 'cat /etc/hostname; sudo wc -c /var/lib/mysql/gvwstate.dat'
controller-1
0 /var/lib/mysql/gvwstate.dat
(undercloud) [stack@undercloud-0 ~]$ ssh -q heat-admin.24.21 'cat /etc/hostname; sudo wc -c /var/lib/mysql/gvwstate.dat'
controller-2
217 /var/lib/mysql/gvwstate.dat

Expected results:
Reboots can be performed cleanly without Galera getting blocked.

Additional info:

Workaround: rm -f /var/lib/mysql/gvwstate.dat; then reboot the node