Bug 1283463
| Summary: | MariaDB Galera Cluster tries to use SSL during IST even when SSL is disabled. | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Ivan Chavero <ichavero> |
| Component: | openstack-puppet-modules | Assignee: | Ivan Chavero <ichavero> |
| Status: | CLOSED ERRATA | QA Contact: | Leonid Natapov <lnatapov> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 7.0 (Kilo) | CC: | cwolfe, dgurtner, dnavale, ichavero, lbezdick, mbayer, mburns, mmagr, morazi, ohochman, rhos-maint, sclewis, yeylon |
| Target Milestone: | z3 | Keywords: | Reopened, Triaged, ZStream |
| Target Release: | 7.0 (Kilo) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | openstack-puppet-modules-2015.1.8-30.el7ost | Doc Type: | Bug Fix |
| Doc Text: | Story Points: | --- | |
| Clone Of: | 1194411 | Environment: | |
| Last Closed: | 2015-12-21 17:12:09 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1194411 | ||
| Bug Blocks: | |||
group UUID = e68adf00-a3ee-11e5-9689-aa25c5e9f8f8 151221 4:38:26 [Note] WSREP: Flow-control interval: [28, 28] 151221 4:38:26 [Note] WSREP: New cluster view: global state: e68adf00-a3ee-11e5-9689-aa25c5e9f8f8:1234773, view# 104: Primary, number of nodes: 3, my index: 0, protocol version 3 151221 4:38:26 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 151221 4:38:26 [Note] WSREP: REPL Protocols: 5 (3, 1) 151221 4:38:26 [Note] WSREP: Service thread queue flushed. 151221 4:38:26 [Note] WSREP: Assign initial position for certification: 1234773, protocol version: 3 151221 4:38:26 [Note] WSREP: Service thread queue flushed. 151221 4:38:26 [Warning] WSREP: Releasing seqno 1234773 before 1234774 was assigned. 151221 4:38:26 [Note] WSREP: Member 2.0 (overcloud-controller-1.localdomain) requested state transfer from '*any*'. Selected 0.0 (overcloud-controller-0.localdomain)(SYNCED) as donor. 151221 4:38:26 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 1234775) 151221 4:38:26 [Note] WSREP: IST request: e68adf00-a3ee-11e5-9689-aa25c5e9f8f8:1162912-1234773|ssl://10.35.169.13:4568 151221 4:38:26 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 151221 4:38:26 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'donor' --address '10.35.169.13:4444/rsync_sst' --auth '(null)' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --gtid 'e68adf0 0-a3ee-11e5-9689-aa25c5e9f8f8:1162912' --bypass' 151221 4:38:26 [Note] WSREP: sst_donor_thread signaled with 0 151221 4:38:26 [Note] WSREP: IST sender using ssl 151221 4:38:26 [ERROR] WSREP: non-standard exception 151221 4:38:26 [Note] WSREP: applier thread exiting (code:8) 151221 4:38:26 [Note] WSREP: starting shutdown 151221 4:38:26 [Note] /usr/libexec/mysqld: Normal shutdown Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2015:2677 The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days |
Ivan, tested according instructions to QA: mysql fails fails to start. Pls,advice id the test described in instructions to QA is relevant: galera.cnf file: [mysqld] basedir = /usr bind-address = 10.35.169.13 binlog_format = ROW datadir = /var/lib/mysql default-storage-engine = innodb expire_logs_days = 10 innodb_autoinc_lock_mode = 2 innodb_locks_unsafe_for_binlog = 1 key_buffer_size = 16M log-error = /var/log/mariadb/mariadb.log max_allowed_packet = 16M max_binlog_size = 100M max_connections = 4096 myisam_recover = BACKUP open_files_limit = -1 pid-file = /var/run/mariadb/mariadb.pid port = 3306 query_cache_limit = 1M query_cache_size = 0 query_cache_type = 0 skip-external-locking skip-name-resolve = 1 socket = /var/lib/mysql/mysql.sock ssl = false ssl-ca = /etc/mysql/cacert.pem ssl-cert = /etc/mysql/server-cert.pem ssl-key = /etc/mysql/server-key.pem thread_cache_size = 8 thread_stack = 256K tmpdir = /tmp user = mysql wsrep_auto_increment_control = 1 wsrep_causal_reads = 0 wsrep_certify_nonPK = 1 wsrep_cluster_name = galera_cluster wsrep_convert_LOCK_to_trx = 0 wsrep_debug = 0 wsrep_drupal_282555_workaround = 0 wsrep_max_ws_rows = 131072 wsrep_max_ws_size = 1073741824 wsrep_provider = /usr/lib64/galera/libgalera_smm.so wsrep_retry_autocommit = 1 wsrep_slave_threads = 1 wsrep_sst_method = rsync wsrep_provider_options="socket.ssl=false; socket.ssl_cert=/etc/pki/galera/galera.crt; socket.ssl_key=/etc/pki/galera/galera.key" --------- mysql.log file 151221 07:33:21 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql 151221 07:33:21 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.YWyo1G' --pid-file='/var/lib/mysql/overcloud-controller-1.localdomain-recover.pid' 151221 7:33:21 [Warning] option 'open_files_limit': unsigned value 18446744073709551615 adjusted to 4294967295 151221 07:33:23 mysqld_safe WSREP: Recovered position e68adf00-a3ee-11e5-9689-aa25c5e9f8f8:1264209 151221 7:33:23 [Warning] option 'open_files_limit': unsigned value 18446744073709551615 adjusted to 4294967295 151221 7:33:23 [Note] WSREP: wsrep_start_position var submitted: 'e68adf00-a3ee-11e5-9689-aa25c5e9f8f8:1264209' 151221 7:33:23 [Note] WSREP: Read nil XID from storage engines, skipping position init 151221 7:33:23 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera/libgalera_smm.so' 151221 7:33:23 [Note] WSREP: wsrep_load(): Galera 3.5(rXXXX) by Codership Oy <info> loaded successfully. 151221 7:33:23 [Note] WSREP: CRC-32C: using hardware acceleration. 151221 7:33:23 [Note] WSREP: Found saved state: e68adf00-a3ee-11e5-9689-aa25c5e9f8f8:1264209 151221 7:33:23 [Note] WSREP: Passing config to GCS: base_host = 10.35.169.13; base_port = 4567; cert.log_conflicts = no; debug = no; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 1; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = P30S; pc.weight = 1; protone 151221 7:33:23 [Note] WSREP: Service thread queue flushed. 151221 7:33:23 [Note] WSREP: Assign initial position for certification: 1264209, protocol version: -1 151221 7:33:23 [Note] WSREP: wsrep_sst_grab() 151221 7:33:23 [Note] WSREP: Start replication 151221 7:33:23 [Note] WSREP: Setting initial position to e68adf00-a3ee-11e5-9689-aa25c5e9f8f8:1264209 151221 7:33:23 [Note] WSREP: protonet asio version 0 151221 7:33:23 [Note] WSREP: Using CRC-32C (optimized) for message checksums. 151221 7:33:23 [Note] WSREP: backend: asio 151221 7:33:23 [Note] WSREP: GMCast version 0 151221 7:33:23 [Note] WSREP: (0683c47e-a7df-11e5-927f-0fc9f35c1e50, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567 151221 7:33:23 [Note] WSREP: (0683c47e-a7df-11e5-927f-0fc9f35c1e50, 'tcp://0.0.0.0:4567') multicast: , ttl: 1 151221 7:33:23 [Note] WSREP: EVS version 0 151221 7:33:23 [Note] WSREP: PC version 0 151221 7:33:23 [Note] WSREP: gcomm: connecting to group 'galera_cluster', peer 'overcloud-controller-0:,overcloud-controller-1:,overcloud-controller-2:' 151221 7:33:23 [Warning] WSREP: (0683c47e-a7df-11e5-927f-0fc9f35c1e50, 'tcp://0.0.0.0:4567') address 'tcp://10.35.169.13:4567' points to own listening address, blacklisting 151221 7:33:23 [Note] WSREP: (0683c47e-a7df-11e5-927f-0fc9f35c1e50, 'tcp://0.0.0.0:4567') address 'tcp://10.35.169.13:4567' pointing to uuid 0683c47e-a7df-11e5-927f-0fc9f35c1e50 is blacklisted, skipping 151221 7:33:23 [Note] WSREP: (0683c47e-a7df-11e5-927f-0fc9f35c1e50, 'tcp://0.0.0.0:4567') address 'tcp://10.35.169.13:4567' pointing to uuid 0683c47e-a7df-11e5-927f-0fc9f35c1e50 is blacklisted, skipping 151221 7:33:23 [Note] WSREP: (0683c47e-a7df-11e5-927f-0fc9f35c1e50, 'tcp://0.0.0.0:4567') address 'tcp://10.35.169.13:4567' pointing to uuid 0683c47e-a7df-11e5-927f-0fc9f35c1e50 is blacklisted, skipping 151221 7:33:23 [Note] WSREP: (0683c47e-a7df-11e5-927f-0fc9f35c1e50, 'tcp://0.0.0.0:4567') address 'tcp://10.35.169.13:4567' pointing to uuid 0683c47e-a7df-11e5-927f-0fc9f35c1e50 is blacklisted, skipping 151221 7:33:23 [Note] WSREP: declaring 9461a07f-a7cb-11e5-8c0f-c6065cd4d4b2 stable 151221 7:33:23 [Note] WSREP: declaring 98b09d9a-a7cb-11e5-bcc4-eb4bc3071137 stable 151221 7:33:23 [Note] WSREP: Node 9461a07f-a7cb-11e5-8c0f-c6065cd4d4b2 state prim 151221 7:33:23 [Note] WSREP: view(view_id(PRIM,0683c47e-a7df-11e5-927f-0fc9f35c1e50,5) memb { 0683c47e-a7df-11e5-927f-0fc9f35c1e50,0 9461a07f-a7cb-11e5-8c0f-c6065cd4d4b2,0 98b09d9a-a7cb-11e5-bcc4-eb4bc3071137,0 } joined { } left { } partitioned { }) 151221 7:33:23 [Note] WSREP: gcomm: connected 151221 7:33:23 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 151221 7:33:23 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0) 151221 7:33:23 [Note] WSREP: Opened channel 'galera_cluster' 151221 7:33:23 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3 151221 7:33:23 [Note] WSREP: Waiting for SST to complete. 151221 7:33:23 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 06d07e10-a7df-11e5-8b58-1aec02418f54 151221 7:33:23 [Note] WSREP: STATE EXCHANGE: sent state msg: 06d07e10-a7df-11e5-8b58-1aec02418f54 151221 7:33:23 [Note] WSREP: STATE EXCHANGE: got state msg: 06d07e10-a7df-11e5-8b58-1aec02418f54 from 0 (overcloud-controller-1.localdomain) 151221 7:33:23 [Note] WSREP: STATE EXCHANGE: got state msg: 06d07e10-a7df-11e5-8b58-1aec02418f54 from 1 (overcloud-controller-2.localdomain) 151221 7:33:23 [Note] WSREP: STATE EXCHANGE: got state msg: 06d07e10-a7df-11e5-8b58-1aec02418f54 from 2 (overcloud-controller-0.localdomain) 151221 7:33:23 [Note] WSREP: Quorum results: version = 3, component = PRIMARY, conf_id = 4, members = 2/3 (joined/total), act_id = 1264393, last_appl. = -1, protocols = 0/5/3 (gcs/repl/appl), group UUID = e68adf00-a3ee-11e5-9689-aa25c5e9f8f8 151221 7:33:23 [Note] WSREP: Flow-control interval: [28, 28] 151221 7:33:23 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 1264393) 151221 7:33:23 [Note] WSREP: State transfer required: Group state: e68adf00-a3ee-11e5-9689-aa25c5e9f8f8:1264393 Local state: e68adf00-a3ee-11e5-9689-aa25c5e9f8f8:1264209 151221 7:33:23 [Note] WSREP: New cluster view: global state: e68adf00-a3ee-11e5-9689-aa25c5e9f8f8:1264393, view# 5: Primary, number of nodes: 3, my index: 0, protocol version 3 151221 7:33:23 [Warning] WSREP: Gap in state sequence. Need state transfer. 151221 7:33:23 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '10.35.169.13' --auth '' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --parent '37395'' 151221 7:33:23 [Note] WSREP: Prepared SST request: rsync|10.35.169.13:4444/rsync_sst 151221 7:33:23 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 151221 7:33:23 [Note] WSREP: REPL Protocols: 5 (3, 1) 151221 7:33:23 [Note] WSREP: Service thread queue flushed. 151221 7:33:23 [Note] WSREP: Assign initial position for certification: 1264393, protocol version: 3 151221 7:33:23 [Note] WSREP: Service thread queue flushed. 151221 7:33:23 [Note] WSREP: IST receiver using ssl 151221 7:33:23 [Note] WSREP: Prepared IST receiver, listening at: ssl://10.35.169.13:4568 151221 7:33:23 [Note] WSREP: Member 0.0 (overcloud-controller-1.localdomain) requested state transfer from '*any*'. Selected 1.0 (overcloud-controller-2.localdomain)(SYNCED) as donor. 151221 7:33:23 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 1264393) 151221 7:33:23 [Note] WSREP: Requesting state transfer: success, donor: 1 151221 7:33:23 [Note] WSREP: declaring 98b09d9a-a7cb-11e5-bcc4-eb4bc3071137 stable 151221 7:33:23 [Note] WSREP: (0683c47e-a7df-11e5-927f-0fc9f35c1e50, 'tcp://0.0.0.0:4567') address 'tcp://10.35.169.13:4567' pointing to uuid 0683c47e-a7df-11e5-927f-0fc9f35c1e50 is blacklisted, skipping 151221 7:33:23 [Note] WSREP: (0683c47e-a7df-11e5-927f-0fc9f35c1e50, 'tcp://0.0.0.0:4567') address 'tcp://10.35.169.13:4567' pointing to uuid 0683c47e-a7df-11e5-927f-0fc9f35c1e50 is blacklisted, skipping 151221 7:33:23 [Note] WSREP: (0683c47e-a7df-11e5-927f-0fc9f35c1e50, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.35.169.14:4567 151221 7:33:23 [Note] WSREP: Node 0683c47e-a7df-11e5-927f-0fc9f35c1e50 state prim 151221 7:33:23 [Note] WSREP: view(view_id(PRIM,0683c47e-a7df-11e5-927f-0fc9f35c1e50,6) memb { 0683c47e-a7df-11e5-927f-0fc9f35c1e50,0 98b09d9a-a7cb-11e5-bcc4-eb4bc3071137,0 } joined { } left { } partitioned { 9461a07f-a7cb-11e5-8c0f-c6065cd4d4b2,0 }) 151221 7:33:23 [Note] WSREP: forgetting 9461a07f-a7cb-11e5-8c0f-c6065cd4d4b2 (tcp://10.35.169.14:4567) 151221 7:33:23 [Note] WSREP: (0683c47e-a7df-11e5-927f-0fc9f35c1e50, 'tcp://0.0.0.0:4567') address 'tcp://10.35.169.13:4567' pointing to uuid 0683c47e-a7df-11e5-927f-0fc9f35c1e50 is blacklisted, skipping 151221 7:33:23 [Note] WSREP: (0683c47e-a7df-11e5-927f-0fc9f35c1e50, 'tcp://0.0.0.0:4567') turning message relay requesting off 151221 7:33:23 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2 151221 7:33:23 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 06dc4e17-a7df-11e5-b446-c7eed2b57770 151221 7:33:23 [Note] WSREP: STATE EXCHANGE: sent state msg: 06dc4e17-a7df-11e5-b446-c7eed2b57770 151221 7:33:23 [Note] WSREP: STATE EXCHANGE: got state msg: 06dc4e17-a7df-11e5-b446-c7eed2b57770 from 0 (overcloud-controller-1.localdomain) 151221 7:33:23 [Note] WSREP: STATE EXCHANGE: got state msg: 06dc4e17-a7df-11e5-b446-c7eed2b57770 from 1 (overcloud-controller-0.localdomain) 151221 7:33:23 [Note] WSREP: Quorum results: version = 3, component = PRIMARY, conf_id = 5, members = 1/2 (joined/total), act_id = 1264393, last_appl. = 0, protocols = 0/5/3 (gcs/repl/appl), group UUID = e68adf00-a3ee-11e5-9689-aa25c5e9f8f8 151221 7:33:23 [Warning] WSREP: Donor 9461a07f-a7cb-11e5-8c0f-c6065cd4d4b2 is no longer in the group. State transfer cannot be completed, need to abort. Aborting... 151221 7:33:23 [Note] WSREP: /usr/libexec/mysqld: Terminated. 151221 07:33:23 mysqld_safe mysqld from pid file /var/run/mysql/mysqld.pid ended WSREP_SST: [ERROR] Parent mysqld process (PID:37395) terminated unexpectedly. (20151221 07:33:23.984) WSREP_SST: [INFO] Joiner cleanup. (20151221 07:33:23.986) WSREP_SST: [INFO] Joiner cleanup done. (20151221 07:33:24.492)