Bug 1283463 - MariaDB Galera Cluster tries to use SSL during IST even when SSL is disabled. [NEEDINFO]
MariaDB Galera Cluster tries to use SSL during IST even when SSL is disabled.
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-puppet-modules (Show other bugs)
7.0 (Kilo)
Unspecified Unspecified
high Severity high
: z3
: 7.0 (Kilo)
Assigned To: Ivan Chavero
Leonid Natapov
: Reopened, Triaged, ZStream
Depends On: 1194411
Blocks:
  Show dependency treegraph
 
Reported: 2015-11-18 22:07 EST by Ivan Chavero
Modified: 2015-12-21 12:12 EST (History)
14 users (show)

See Also:
Fixed In Version: openstack-puppet-modules-2015.1.8-30.el7ost
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1194411
Environment:
Last Closed: 2015-12-21 12:12:09 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
lnatapov: needinfo? (ichavero)


Attachments (Terms of Use)

  None (edit)
Comment 4 Leonid Natapov 2015-12-21 07:40:43 EST
Ivan,

tested according instructions to QA:

mysql fails fails to start.

Pls,advice id the test described in instructions to QA is relevant:



galera.cnf file:

[mysqld]
basedir = /usr
bind-address = 10.35.169.13
binlog_format = ROW
datadir = /var/lib/mysql
default-storage-engine = innodb
expire_logs_days = 10
innodb_autoinc_lock_mode = 2
innodb_locks_unsafe_for_binlog = 1
key_buffer_size = 16M
log-error = /var/log/mariadb/mariadb.log
max_allowed_packet = 16M
max_binlog_size = 100M
max_connections = 4096
myisam_recover = BACKUP
open_files_limit = -1
pid-file = /var/run/mariadb/mariadb.pid
port = 3306
query_cache_limit = 1M
query_cache_size = 0
query_cache_type = 0
skip-external-locking
skip-name-resolve = 1
socket = /var/lib/mysql/mysql.sock
ssl = false
ssl-ca = /etc/mysql/cacert.pem
ssl-cert = /etc/mysql/server-cert.pem
ssl-key = /etc/mysql/server-key.pem
thread_cache_size = 8
thread_stack = 256K
tmpdir = /tmp
user = mysql
wsrep_auto_increment_control = 1
wsrep_causal_reads = 0
wsrep_certify_nonPK = 1
wsrep_cluster_name = galera_cluster
wsrep_convert_LOCK_to_trx = 0
wsrep_debug = 0
wsrep_drupal_282555_workaround = 0
wsrep_max_ws_rows = 131072
wsrep_max_ws_size = 1073741824
wsrep_provider = /usr/lib64/galera/libgalera_smm.so
wsrep_retry_autocommit = 1
wsrep_slave_threads = 1
wsrep_sst_method = rsync
wsrep_provider_options="socket.ssl=false; socket.ssl_cert=/etc/pki/galera/galera.crt; socket.ssl_key=/etc/pki/galera/galera.key"


---------

mysql.log file

151221 07:33:21 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
151221 07:33:21 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.YWyo1G' --pid-file='/var/lib/mysql/overcloud-controller-1.localdomain-recover.pid'
151221  7:33:21 [Warning] option 'open_files_limit': unsigned value 18446744073709551615 adjusted to 4294967295
151221 07:33:23 mysqld_safe WSREP: Recovered position e68adf00-a3ee-11e5-9689-aa25c5e9f8f8:1264209
151221  7:33:23 [Warning] option 'open_files_limit': unsigned value 18446744073709551615 adjusted to 4294967295
151221  7:33:23 [Note] WSREP: wsrep_start_position var submitted: 'e68adf00-a3ee-11e5-9689-aa25c5e9f8f8:1264209'
151221  7:33:23 [Note] WSREP: Read nil XID from storage engines, skipping position init
151221  7:33:23 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera/libgalera_smm.so'
151221  7:33:23 [Note] WSREP: wsrep_load(): Galera 3.5(rXXXX) by Codership Oy <info@codership.com> loaded successfully.
151221  7:33:23 [Note] WSREP: CRC-32C: using hardware acceleration.
151221  7:33:23 [Note] WSREP: Found saved state: e68adf00-a3ee-11e5-9689-aa25c5e9f8f8:1264209
151221  7:33:23 [Note] WSREP: Passing config to GCS: base_host = 10.35.169.13; base_port = 4567; cert.log_conflicts = no; debug = no; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 1; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = P30S; pc.weight = 1; protone
151221  7:33:23 [Note] WSREP: Service thread queue flushed.
151221  7:33:23 [Note] WSREP: Assign initial position for certification: 1264209, protocol version: -1
151221  7:33:23 [Note] WSREP: wsrep_sst_grab()
151221  7:33:23 [Note] WSREP: Start replication
151221  7:33:23 [Note] WSREP: Setting initial position to e68adf00-a3ee-11e5-9689-aa25c5e9f8f8:1264209
151221  7:33:23 [Note] WSREP: protonet asio version 0
151221  7:33:23 [Note] WSREP: Using CRC-32C (optimized) for message checksums.
151221  7:33:23 [Note] WSREP: backend: asio
151221  7:33:23 [Note] WSREP: GMCast version 0
151221  7:33:23 [Note] WSREP: (0683c47e-a7df-11e5-927f-0fc9f35c1e50, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
151221  7:33:23 [Note] WSREP: (0683c47e-a7df-11e5-927f-0fc9f35c1e50, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
151221  7:33:23 [Note] WSREP: EVS version 0
151221  7:33:23 [Note] WSREP: PC version 0
151221  7:33:23 [Note] WSREP: gcomm: connecting to group 'galera_cluster', peer 'overcloud-controller-0:,overcloud-controller-1:,overcloud-controller-2:'
151221  7:33:23 [Warning] WSREP: (0683c47e-a7df-11e5-927f-0fc9f35c1e50, 'tcp://0.0.0.0:4567') address 'tcp://10.35.169.13:4567' points to own listening address, blacklisting
151221  7:33:23 [Note] WSREP: (0683c47e-a7df-11e5-927f-0fc9f35c1e50, 'tcp://0.0.0.0:4567') address 'tcp://10.35.169.13:4567' pointing to uuid 0683c47e-a7df-11e5-927f-0fc9f35c1e50 is blacklisted, skipping
151221  7:33:23 [Note] WSREP: (0683c47e-a7df-11e5-927f-0fc9f35c1e50, 'tcp://0.0.0.0:4567') address 'tcp://10.35.169.13:4567' pointing to uuid 0683c47e-a7df-11e5-927f-0fc9f35c1e50 is blacklisted, skipping
151221  7:33:23 [Note] WSREP: (0683c47e-a7df-11e5-927f-0fc9f35c1e50, 'tcp://0.0.0.0:4567') address 'tcp://10.35.169.13:4567' pointing to uuid 0683c47e-a7df-11e5-927f-0fc9f35c1e50 is blacklisted, skipping
151221  7:33:23 [Note] WSREP: (0683c47e-a7df-11e5-927f-0fc9f35c1e50, 'tcp://0.0.0.0:4567') address 'tcp://10.35.169.13:4567' pointing to uuid 0683c47e-a7df-11e5-927f-0fc9f35c1e50 is blacklisted, skipping
151221  7:33:23 [Note] WSREP: declaring 9461a07f-a7cb-11e5-8c0f-c6065cd4d4b2 stable
151221  7:33:23 [Note] WSREP: declaring 98b09d9a-a7cb-11e5-bcc4-eb4bc3071137 stable
151221  7:33:23 [Note] WSREP: Node 9461a07f-a7cb-11e5-8c0f-c6065cd4d4b2 state prim
151221  7:33:23 [Note] WSREP: view(view_id(PRIM,0683c47e-a7df-11e5-927f-0fc9f35c1e50,5) memb {
	0683c47e-a7df-11e5-927f-0fc9f35c1e50,0
	9461a07f-a7cb-11e5-8c0f-c6065cd4d4b2,0
	98b09d9a-a7cb-11e5-bcc4-eb4bc3071137,0
} joined {
} left {
} partitioned {
})
151221  7:33:23 [Note] WSREP: gcomm: connected
151221  7:33:23 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
151221  7:33:23 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
151221  7:33:23 [Note] WSREP: Opened channel 'galera_cluster'
151221  7:33:23 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3
151221  7:33:23 [Note] WSREP: Waiting for SST to complete.
151221  7:33:23 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 06d07e10-a7df-11e5-8b58-1aec02418f54
151221  7:33:23 [Note] WSREP: STATE EXCHANGE: sent state msg: 06d07e10-a7df-11e5-8b58-1aec02418f54
151221  7:33:23 [Note] WSREP: STATE EXCHANGE: got state msg: 06d07e10-a7df-11e5-8b58-1aec02418f54 from 0 (overcloud-controller-1.localdomain)
151221  7:33:23 [Note] WSREP: STATE EXCHANGE: got state msg: 06d07e10-a7df-11e5-8b58-1aec02418f54 from 1 (overcloud-controller-2.localdomain)
151221  7:33:23 [Note] WSREP: STATE EXCHANGE: got state msg: 06d07e10-a7df-11e5-8b58-1aec02418f54 from 2 (overcloud-controller-0.localdomain)
151221  7:33:23 [Note] WSREP: Quorum results:
	version    = 3,
	component  = PRIMARY,
	conf_id    = 4,
	members    = 2/3 (joined/total),
	act_id     = 1264393,
	last_appl. = -1,
	protocols  = 0/5/3 (gcs/repl/appl),
	group UUID = e68adf00-a3ee-11e5-9689-aa25c5e9f8f8
151221  7:33:23 [Note] WSREP: Flow-control interval: [28, 28]
151221  7:33:23 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 1264393)
151221  7:33:23 [Note] WSREP: State transfer required: 
	Group state: e68adf00-a3ee-11e5-9689-aa25c5e9f8f8:1264393
	Local state: e68adf00-a3ee-11e5-9689-aa25c5e9f8f8:1264209
151221  7:33:23 [Note] WSREP: New cluster view: global state: e68adf00-a3ee-11e5-9689-aa25c5e9f8f8:1264393, view# 5: Primary, number of nodes: 3, my index: 0, protocol version 3
151221  7:33:23 [Warning] WSREP: Gap in state sequence. Need state transfer.
151221  7:33:23 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '10.35.169.13' --auth '' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --parent '37395''
151221  7:33:23 [Note] WSREP: Prepared SST request: rsync|10.35.169.13:4444/rsync_sst
151221  7:33:23 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
151221  7:33:23 [Note] WSREP: REPL Protocols: 5 (3, 1)
151221  7:33:23 [Note] WSREP: Service thread queue flushed.
151221  7:33:23 [Note] WSREP: Assign initial position for certification: 1264393, protocol version: 3
151221  7:33:23 [Note] WSREP: Service thread queue flushed.
151221  7:33:23 [Note] WSREP: IST receiver using ssl
151221  7:33:23 [Note] WSREP: Prepared IST receiver, listening at: ssl://10.35.169.13:4568
151221  7:33:23 [Note] WSREP: Member 0.0 (overcloud-controller-1.localdomain) requested state transfer from '*any*'. Selected 1.0 (overcloud-controller-2.localdomain)(SYNCED) as donor.
151221  7:33:23 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 1264393)
151221  7:33:23 [Note] WSREP: Requesting state transfer: success, donor: 1
151221  7:33:23 [Note] WSREP: declaring 98b09d9a-a7cb-11e5-bcc4-eb4bc3071137 stable
151221  7:33:23 [Note] WSREP: (0683c47e-a7df-11e5-927f-0fc9f35c1e50, 'tcp://0.0.0.0:4567') address 'tcp://10.35.169.13:4567' pointing to uuid 0683c47e-a7df-11e5-927f-0fc9f35c1e50 is blacklisted, skipping
151221  7:33:23 [Note] WSREP: (0683c47e-a7df-11e5-927f-0fc9f35c1e50, 'tcp://0.0.0.0:4567') address 'tcp://10.35.169.13:4567' pointing to uuid 0683c47e-a7df-11e5-927f-0fc9f35c1e50 is blacklisted, skipping
151221  7:33:23 [Note] WSREP: (0683c47e-a7df-11e5-927f-0fc9f35c1e50, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.35.169.14:4567 
151221  7:33:23 [Note] WSREP: Node 0683c47e-a7df-11e5-927f-0fc9f35c1e50 state prim
151221  7:33:23 [Note] WSREP: view(view_id(PRIM,0683c47e-a7df-11e5-927f-0fc9f35c1e50,6) memb {
	0683c47e-a7df-11e5-927f-0fc9f35c1e50,0
	98b09d9a-a7cb-11e5-bcc4-eb4bc3071137,0
} joined {
} left {
} partitioned {
	9461a07f-a7cb-11e5-8c0f-c6065cd4d4b2,0
})
151221  7:33:23 [Note] WSREP: forgetting 9461a07f-a7cb-11e5-8c0f-c6065cd4d4b2 (tcp://10.35.169.14:4567)
151221  7:33:23 [Note] WSREP: (0683c47e-a7df-11e5-927f-0fc9f35c1e50, 'tcp://0.0.0.0:4567') address 'tcp://10.35.169.13:4567' pointing to uuid 0683c47e-a7df-11e5-927f-0fc9f35c1e50 is blacklisted, skipping
151221  7:33:23 [Note] WSREP: (0683c47e-a7df-11e5-927f-0fc9f35c1e50, 'tcp://0.0.0.0:4567') turning message relay requesting off
151221  7:33:23 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
151221  7:33:23 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 06dc4e17-a7df-11e5-b446-c7eed2b57770
151221  7:33:23 [Note] WSREP: STATE EXCHANGE: sent state msg: 06dc4e17-a7df-11e5-b446-c7eed2b57770
151221  7:33:23 [Note] WSREP: STATE EXCHANGE: got state msg: 06dc4e17-a7df-11e5-b446-c7eed2b57770 from 0 (overcloud-controller-1.localdomain)
151221  7:33:23 [Note] WSREP: STATE EXCHANGE: got state msg: 06dc4e17-a7df-11e5-b446-c7eed2b57770 from 1 (overcloud-controller-0.localdomain)
151221  7:33:23 [Note] WSREP: Quorum results:
	version    = 3,
	component  = PRIMARY,
	conf_id    = 5,
	members    = 1/2 (joined/total),
	act_id     = 1264393,
	last_appl. = 0,
	protocols  = 0/5/3 (gcs/repl/appl),
	group UUID = e68adf00-a3ee-11e5-9689-aa25c5e9f8f8
151221  7:33:23 [Warning] WSREP: Donor 9461a07f-a7cb-11e5-8c0f-c6065cd4d4b2 is no longer in the group. State transfer cannot be completed, need to abort. Aborting...
151221  7:33:23 [Note] WSREP: /usr/libexec/mysqld: Terminated.
151221 07:33:23 mysqld_safe mysqld from pid file /var/run/mysql/mysqld.pid ended
WSREP_SST: [ERROR] Parent mysqld process (PID:37395) terminated unexpectedly. (20151221 07:33:23.984)
WSREP_SST: [INFO] Joiner cleanup. (20151221 07:33:23.986)
WSREP_SST: [INFO] Joiner cleanup done. (20151221 07:33:24.492)
Comment 5 Leonid Natapov 2015-12-21 08:18:42 EST
        group UUID = e68adf00-a3ee-11e5-9689-aa25c5e9f8f8
151221  4:38:26 [Note] WSREP: Flow-control interval: [28, 28]
151221  4:38:26 [Note] WSREP: New cluster view: global state: e68adf00-a3ee-11e5-9689-aa25c5e9f8f8:1234773, view# 104: Primary, number of nodes: 3, my index: 0, protocol version 3
151221  4:38:26 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
151221  4:38:26 [Note] WSREP: REPL Protocols: 5 (3, 1)
151221  4:38:26 [Note] WSREP: Service thread queue flushed.
151221  4:38:26 [Note] WSREP: Assign initial position for certification: 1234773, protocol version: 3
151221  4:38:26 [Note] WSREP: Service thread queue flushed.
151221  4:38:26 [Warning] WSREP: Releasing seqno 1234773 before 1234774 was assigned.
151221  4:38:26 [Note] WSREP: Member 2.0 (overcloud-controller-1.localdomain) requested state transfer from '*any*'. Selected 0.0 (overcloud-controller-0.localdomain)(SYNCED) as donor.
151221  4:38:26 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 1234775)
151221  4:38:26 [Note] WSREP: IST request: e68adf00-a3ee-11e5-9689-aa25c5e9f8f8:1162912-1234773|ssl://10.35.169.13:4568
151221  4:38:26 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
151221  4:38:26 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'donor' --address '10.35.169.13:4444/rsync_sst' --auth '(null)' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --gtid 'e68adf0
0-a3ee-11e5-9689-aa25c5e9f8f8:1162912' --bypass'
151221  4:38:26 [Note] WSREP: sst_donor_thread signaled with 0
151221  4:38:26 [Note] WSREP: IST sender using ssl

151221  4:38:26 [ERROR] WSREP: non-standard exception

151221  4:38:26 [Note] WSREP: applier thread exiting (code:8)
151221  4:38:26 [Note] WSREP: starting shutdown
151221  4:38:26 [Note] /usr/libexec/mysqld: Normal shutdown
Comment 7 errata-xmlrpc 2015-12-21 12:12:09 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2015:2677

Note You need to log in before you can comment on or make changes to this bug.