Bug 1194411
| Summary: | MariaDB Galera Cluster tries to use SSL during IST even when SSL is disabled. | |||
|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | David Gurtner <dgurtner> | |
| Component: | openstack-puppet-modules | Assignee: | Chris Jones <chjones> | |
| Status: | CLOSED EOL | QA Contact: | Ofer Blaut <oblaut> | |
| Severity: | high | Docs Contact: | ||
| Priority: | high | |||
| Version: | 5.0 (RHEL 7) | CC: | apevec, cwolfe, emacchi, fdinitto, ichavero, kbasil, mbayer, mburns, mmagr, morazi, rhos-maint, sclewis, srevivo | |
| Target Milestone: | z6 | Keywords: | Reopened, Triaged, ZStream | |
| Target Release: | 5.0 (RHEL 7) | Flags: | mbayer:
needinfo-
|
|
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | Bug Fix | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1277539 1283463 (view as bug list) | Environment: | ||
| Last Closed: | 2017-06-30 13:29:13 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1283463 | |||
First, you really should not disable SSL for the galera cluster communication. It is highly recommended that you keep it enabled. The problem here as far as I can tell is that if you specify a key and cert but have socket.ssl set to false, galera still seems to think that SSL is enabled. The quickest way to resolve this would be to fix the puppetg-galera module such that if the wsrep_ssl parameter is false, don't set socket.ssl_key or socket.ssl_cert. Adding Crag for his input. Ryan's comment above is true for OSP 5.0. The fix would have to be in puppet-galera in the manifest galera::server. For OSP 6.0, neither socket.ssl_key or socket.ssl_cert should get set if $wsrep_ssl is false. This is happening in quickstack in https://github.com/redhat-openstack/astapor/blob/master/puppet/modules/quickstack/manifests/pacemaker/galera.pp. Note, galera::server is not invoked here as it was in 5.0. I created a pull request with the necessary change to puppet-galera: https://github.com/redhat-openstack/puppet-galera/pull/7 Because I'm curious as to why this issue even occurs, I went poking through Galera's source to find it. It's local to galera/src/ist.cpp and I think is a bug we can demonstrate. It internally prepends the prefix "ssl://" to the host here, based only on if it can find CONF_SSL_KEY (which is COMMON_CONF_SSL_KEY, which is "socket.ssl_key": https://github.com/codership/galera/blob/release_25.3.5/common/common.h#L21): https://github.com/codership/galera/blob/release_25.3.5/galera/src/ist.cpp#L220 which then tells it to use ssl here: https://github.com/codership/galera/blob/release_25.3.5/galera/src/ist.cpp#L302 galera's fix would be that logic in ist.cpp should be checking the "socket.ssl" flag as well, or better yet they just use some common functionality somewhere. ryan can you let me know next steps on this? I think that PR should be merged and then I'm not sure where puppet-galera gets released to. (In reply to Michael Bayer from comment #10) > ryan can you let me know next steps on this? I think that PR should be > merged and then I'm not sure where puppet-galera gets released to. Right. We should merge the PR to the git repo on redhat-openstack. I am not sure how this gets pulled into openstack-puppet-modules. I'm going to set the component here to openstack-puppet-modules so that it gets pulled in. This setting is already on the OPM package Can i have acks for this bug please. This is now a high priority bug so we need the backport to OSP 5 Red Hat OpenStack Platform version 5 is now End-of-Life, and as such will not have further updates. See https://access.redhat.com/support/policy/updates/openstack/platform/ for full support lifecycle details. |
Description of problem: When running mariadb galera cluster with SSL disabled and a cluster member tries to get back in sync via IST during a rejoin, IST tries to happen via SSL and thus fails. Version-Release number of selected component (if applicable): mariadb-5.5.40-2.el7_0.x86_64 mariadb-galera-common-5.5.40-2.el7ost.x86_64 mariadb-galera-server-5.5.40-2.el7ost.x86_64 mariadb-libs-5.5.40-2.el7_0.x86_64 galera-25.3.5-7.el7.x86_64 How reproducible: always Steps to Reproduce: 1. configure mariadb galera cluster with the following options in /etc/my.cnf.d/galera.cnf [mysqld] wsrep_provider_options="socket.ssl=false; socket.ssl_cert=/etc/pki/galera/galera.crt; socket.ssl_key=/etc/pki/galera/galera.key" 2. /etc/pki/galera/galera.crt and /etc/pki/galera/galera.key exist (they are created during package installation) 3. stop mariadb on one of the cluster nodes 4. connect to mariadb on one of the other nodes and create some changes 5. restart mariadb on the cluster node where it was stopped Actual results: restarting fails with the following log output: on the donor: 150219 19:04:01 [Note] WSREP: Member 2.0 (rh7pt-5) requested state transfer from '*any*'. Selected 0.0 (rh7pt-4)(SYNCED) as donor. 150219 19:04:01 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 19) 150219 19:04:01 [Note] WSREP: IST request: 80a3127e-b777-11e4-9456-4b89f399845e:18-19|ssl://172.17.26.38:4568 150219 19:04:01 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 150219 19:04:01 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'donor' --address '172.17.26.38:4444/rsync_sst' --auth 'sst_user:SSTP@ss' --socket '/var/lib/mysql/mysql. sock' --datadir '/var/lib/mysql/data/' --defaults-file '/etc/my.cnf' --gtid '80a3127e-b777-11e4-9456-4b89f399845e:18' --bypass' 150219 19:04:01 [Note] WSREP: sst_donor_thread signaled with 0 150219 19:04:01 [Note] WSREP: IST sender using ssl WSREP_SST: [INFO] Bypassing state dump. (20150219 19:04:01.428) 150219 19:04:01 [Note] WSREP: (5dc0a92c-b861-11e4-9cc8-062009749d5d, 'tcp://0.0.0.0:4567') address 'tcp://172.17.26.37:4567' pointing to uuid 5dc0a92c-b861-11e4-9cc8-06 2009749d5d is blacklisted, skipping 150219 19:04:01 [Note] WSREP: (5dc0a92c-b861-11e4-9cc8-062009749d5d, 'tcp://0.0.0.0:4567') address 'tcp://172.17.26.37:4567' pointing to uuid 5dc0a92c-b861-11e4-9cc8-06 2009749d5d is blacklisted, skipping 150219 19:04:01 [Note] WSREP: (5dc0a92c-b861-11e4-9cc8-062009749d5d, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://172.17.26.38:4567 150219 19:04:01 [ERROR] WSREP: IST failed: IST sender, failed to connect 'ssl://172.17.26.38:4568': End of file.: 2 (No such file or directory) at galera/src/ist.cpp:Sender():654 150219 19:04:02 [Note] WSREP: (5dc0a92c-b861-11e4-9cc8-062009749d5d, 'tcp://0.0.0.0:4567') reconnecting to ae989120-b861-11e4-8454-4e8a7b003622 (tcp://172.17.26.38:4567 ), attempt 0 on the joiner/receiver: 150219 19:04:01 [Warning] WSREP: Gap in state sequence. Need state transfer. 150219 19:04:01 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '172.17.26.38' --auth 'sst_user:SSTP@ss' --datadir '/var/lib/mysql/data/' --defaults-f ile '/etc/my.cnf' --parent '18477'' 150219 19:04:01 [Note] WSREP: Prepared SST request: rsync|172.17.26.38:4444/rsync_sst 150219 19:04:01 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 150219 19:04:01 [Note] WSREP: REPL Protocols: 5 (3, 1) 150219 19:04:01 [Note] WSREP: Service thread queue flushed. 150219 19:04:01 [Note] WSREP: Assign initial position for certification: 19, protocol version: 3 150219 19:04:01 [Note] WSREP: Service thread queue flushed. 150219 19:04:01 [Note] WSREP: IST receiver using ssl 150219 19:04:01 [Note] WSREP: Prepared IST receiver, listening at: ssl://172.17.26.38:4568 150219 19:04:01 [Note] WSREP: Member 2.0 (rh7pt-5) requested state transfer from '*any*'. Selected 0.0 (rh7pt-4)(SYNCED) as donor. 150219 19:04:01 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 19) 150219 19:04:01 [Note] WSREP: Requesting state transfer: success, donor: 0 terminate called after throwing an instance of 'gu::Exception' what(): accept() failed', asio error 'SSL error.': 336105671 (Unknown error 336105671) at galera/src/ist.cpp:run():383 150219 19:04:01 [ERROR] mysqld got signal 6 ; Expected results: restarting succeeds, IST happens via TCP Additional info: without the cert and key option (see below) it works as intended. [mysqld] wsrep_provider_options="socket.ssl=false" on the donor: 150219 19:12:53 [Note] WSREP: Member 2.0 (rh7pt-6) requested state transfer from '*any*'. Selected 0.0 (rh7pt-4)(SYNCED) as donor. 150219 19:12:53 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 20) 150219 19:12:53 [Note] WSREP: IST request: 80a3127e-b777-11e4-9456-4b89f399845e:19-20|tcp://172.17.26.39:4568 150219 19:12:53 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 150219 19:12:53 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'donor' --address '172.17.26.39:4444/rsync_sst' --auth 'sst_user:SSTP@ss' --socket '/var/lib/mysql/mysql. sock' --datadir '/var/lib/mysql/data/' --defaults-file '/etc/my.cnf' --gtid '80a3127e-b777-11e4-9456-4b89f399845e:19' --bypass' 150219 19:12:53 [Note] WSREP: sst_donor_thread signaled with 0 150219 19:12:53 [Note] WSREP: async IST sender starting to serve tcp://172.17.26.39:4568 sending 20-20 WSREP_SST: [INFO] Bypassing state dump. (20150219 19:12:53.341) 150219 19:12:55 [Note] WSREP: async IST sender served 150219 19:12:55 [Note] WSREP: 0.0 (rh7pt-4): State transfer to 2.0 (rh7pt-6) complete. 150219 19:12:55 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 20) 150219 19:12:55 [Note] WSREP: 2.0 (rh7pt-6): State transfer from 0.0 (rh7pt-4) complete. 150219 19:12:55 [Note] WSREP: Member 0.0 (rh7pt-4) synced with group. 150219 19:12:55 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 20) 150219 19:12:55 [Note] WSREP: Member 2.0 (rh7pt-6) synced with group. on the joiner/receiver: 150219 19:08:21 InnoDB: Waiting for the background threads to start 150219 19:08:22 Percona XtraDB (http://www.percona.com) 5.5.40-MariaDB-36.1 started; log sequence number 1601469 150219 19:08:22 [Note] Plugin 'FEEDBACK' is disabled. 150219 19:08:22 [Warning] Failed to setup SSL 150219 19:08:22 [Warning] SSL error: SSL_CTX_set_default_verify_paths failed 150219 19:08:22 [Note] Server socket created on IP: '0.0.0.0'. 150219 19:08:22 [Note] Event Scheduler: Loaded 0 events 150219 19:08:22 [Note] WSREP: Signalling provider to continue. 150219 19:08:22 [Note] WSREP: SST received: 80a3127e-b777-11e4-9456-4b89f399845e:19 150219 19:08:22 [Note] /usr/libexec/mysqld: ready for connections. Version: '5.5.40-MariaDB-wsrep' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server, wsrep_25.11.r4026 150219 19:08:22 [Note] WSREP: 0.0 (rh7pt-5): State transfer from 1.0 (rh7pt-4) complete. 150219 19:08:22 [Note] WSREP: Shifting JOINER -> JOINED (TO: 19) 150219 19:08:22 [Note] WSREP: Member 0.0 (rh7pt-5) synced with group. 150219 19:08:22 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 19) 150219 19:08:22 [Note] WSREP: Synchronized with group, ready for connections