Hide Forgot
Description of problem: With the Internal TLS work upstream, it is now possible to configure Galera so that both regular MySQL traffic and WSREP replication traffic use encryption. However the SST traffic - used during bootstrap and for rejoining cluster - is still happening unencrypted. This is because the SST implementation is rsync (unencrypted), whereas the latest version of mariadb-galera [1] package ship a special SST implementation that enables encryption of SST traffic (rsync_tunnel) [1] https://bugzilla.redhat.com/show_bug.cgi?id=1497981 Version-Release number of selected component (if applicable): How reproducible:Always Steps to Reproduce: 1. Deploying an OSP overcloud with Internal TLS enabled. 2. Look into /var/log/pacemaker/bundles/galera-bundle*/mysqld.log for some logs like: WSREP: Prepared SST request: rsync|overcloud-controller-0.internalapi.tripleodomain.example.com:4444/rsync_sst Actual results: The traffic goes via rsync, thus enuncrypted Expected results: When Internal TLS is enabled, all internal endpoint of the controller service should encrypt their communication Additional info: An alternative SST implementation [1] allows encryption of traffic via a socat tunnel. Associated change in tripleo is being tracker upstream [2] [1] https://github.com/dciabrin/wsrep_sst_rsync_tunnel [2] https://bugs.launchpad.net/tripleo/+bug/1719885 Version-Release number of selected component (if applicable):
Feature committed upstream in Pike. Details of the two tracked reviews: tripleo-heat-templates: https://review.openstack.org/#/c/512203/ puppet-tripleo: https://review.openstack.org/#/c/518521/
Update on comment #1 I forgot to link a puppet-tripleo patch: https://review.openstack.org/#/c/517920/ This patch must be applied before https://review.openstack.org/#/c/518521/ so that the latter can apply cleanly. Tracker updated accordingly.
VERIFIED for IPv4 scenario (undercloud) [stack@undercloud-0 ~]$ sudo rpm -q puppet-tripleo puppet-tripleo-7.4.3-7.el7ost.noarch sudo cat /var/log/pacemaker/bundles/galera-bundle-0/mysqld.log |grep wsrep_sst_rsync_tunnel 171122 9:36:40 [Note] WSREP: Running: 'wsrep_sst_rsync_tunnel --role 'joiner' --address 'overcloud-controller-0.internalapi.redhat.local' --auth '' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --parent '2536'' [heat-admin@overcloud-controller-0 ~]$ sudo cat /var/log/pacemaker/bundles/galera-bundle-0/mysqld.log |grep "ssl" 171122 9:36:39 [Note] WSREP: initializing ssl context 171122 9:36:39 [Note] WSREP: (a3ef0354, 'ssl://172.17.1.24:4567') listening at ssl://172.17.1.24:4567 171122 9:36:39 [Note] WSREP: (a3ef0354, 'ssl://172.17.1.24:4567') multicast: , ttl: 1 171122 9:36:39 [Note] WSREP: SSL handshake successful, remote endpoint ssl://172.17.1.11:4567 local endpoint ssl://172.17.1.24:49972 cipher: AES128-SHA compression: 171122 9:36:39 [Note] WSREP: (a3ef0354, 'ssl://172.17.1.24:4567') turning message relay requesting on, nonlive peers: 171122 9:36:39 [Note] WSREP: declaring 9e9bec19 at ssl://172.17.1.11:4567 stable 171122 9:36:39 [Note] WSREP: discarding pending addr without UUID: ssl://172.17.1.21:4567 WSREP_SST: [INFO] Setting up tunnel for joiner: socat openssl-listen:4444,bind=overcloud-controller-0.internalapi.redhat.local,reuseaddr,fork,cert=/etc/pki/tls/certs/mysql.crt,key=/etc/pki/tls/private/mysql.key,cafile=/etc/ipa/ca.crt,cipher=!SSLv2:kEEH:kRSA:kEDH:kPSK:+3DES:!aNULL:!eNULL:!MD5:!EXP:!RC4:!SEED:!IDEA:!DES tcp:localhost:4444 (20171122 09:36:40.262) 171122 9:36:42 [Note] WSREP: (a3ef0354, 'ssl://172.17.1.24:4567') turning message relay requesting off 171122 9:36:51 [Note] WSREP: SSL handshake successful, remote endpoint ssl://172.17.1.21:44168 local endpoint ssl://172.17.1.24:4567 cipher: AES128-SHA compression: 171122 9:36:51 [Note] WSREP: (a3ef0354, 'ssl://172.17.1.24:4567') turning message relay requesting on, nonlive peers: 171122 9:36:51 [Note] WSREP: declaring 9e9bec19 at ssl://172.17.1.11:4567 stable 171122 9:36:51 [Note] WSREP: declaring ab0ac0ee at ssl://172.17.1.21:4567 stable 171122 9:36:54 [Note] WSREP: (a3ef0354, 'ssl://172.17.1.24:4567') turning message relay requesting off Re-assign to ushkalim for testing IPv6 scenario
(In reply to Artem Hrechanychenko from comment #5) > VERIFIED for IPv4 scenario > (undercloud) [stack@undercloud-0 ~]$ sudo rpm -q puppet-tripleo > puppet-tripleo-7.4.3-7.el7ost.noarch > > sudo cat /var/log/pacemaker/bundles/galera-bundle-0/mysqld.log |grep > wsrep_sst_rsync_tunnel > 171122 9:36:40 [Note] WSREP: Running: 'wsrep_sst_rsync_tunnel --role > 'joiner' --address 'overcloud-controller-0.internalapi.redhat.local' --auth > '' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --parent '2536'' > > > > [heat-admin@overcloud-controller-0 ~]$ sudo cat > /var/log/pacemaker/bundles/galera-bundle-0/mysqld.log |grep "ssl" > 171122 9:36:39 [Note] WSREP: initializing ssl context > 171122 9:36:39 [Note] WSREP: (a3ef0354, 'ssl://172.17.1.24:4567') listening > at ssl://172.17.1.24:4567 > 171122 9:36:39 [Note] WSREP: (a3ef0354, 'ssl://172.17.1.24:4567') > multicast: , ttl: 1 > 171122 9:36:39 [Note] WSREP: SSL handshake successful, remote endpoint > ssl://172.17.1.11:4567 local endpoint ssl://172.17.1.24:49972 cipher: > AES128-SHA compression: > 171122 9:36:39 [Note] WSREP: (a3ef0354, 'ssl://172.17.1.24:4567') turning > message relay requesting on, nonlive peers: > 171122 9:36:39 [Note] WSREP: declaring 9e9bec19 at ssl://172.17.1.11:4567 > stable > 171122 9:36:39 [Note] WSREP: discarding pending addr without UUID: > ssl://172.17.1.21:4567 > WSREP_SST: [INFO] Setting up tunnel for joiner: socat > openssl-listen:4444,bind=overcloud-controller-0.internalapi.redhat.local, > reuseaddr,fork,cert=/etc/pki/tls/certs/mysql.crt,key=/etc/pki/tls/private/ > mysql.key,cafile=/etc/ipa/ca.crt,cipher=!SSLv2:kEEH:kRSA:kEDH:kPSK:+3DES:! > aNULL:!eNULL:!MD5:!EXP:!RC4:!SEED:!IDEA:!DES tcp:localhost:4444 (20171122 > 09:36:40.262) > 171122 9:36:42 [Note] WSREP: (a3ef0354, 'ssl://172.17.1.24:4567') turning > message relay requesting off > 171122 9:36:51 [Note] WSREP: SSL handshake successful, remote endpoint > ssl://172.17.1.21:44168 local endpoint ssl://172.17.1.24:4567 cipher: > AES128-SHA compression: > 171122 9:36:51 [Note] WSREP: (a3ef0354, 'ssl://172.17.1.24:4567') turning > message relay requesting on, nonlive peers: > 171122 9:36:51 [Note] WSREP: declaring 9e9bec19 at ssl://172.17.1.11:4567 > stable > 171122 9:36:51 [Note] WSREP: declaring ab0ac0ee at ssl://172.17.1.21:4567 > stable > 171122 9:36:54 [Note] WSREP: (a3ef0354, 'ssl://172.17.1.24:4567') turning > message relay requesting off > > Re-assign to ushkalim for testing IPv6 scenario Hi Artem, Do you have an IPv6 setup we can use to speedup testing? our HA testing on the IPv4 setup still on-going.
(In reply to Udi Shkalim from comment #7) > (In reply to Artem Hrechanychenko from comment #5) > > VERIFIED for IPv4 scenario > > (undercloud) [stack@undercloud-0 ~]$ sudo rpm -q puppet-tripleo > > puppet-tripleo-7.4.3-7.el7ost.noarch > > > > sudo cat /var/log/pacemaker/bundles/galera-bundle-0/mysqld.log |grep > > wsrep_sst_rsync_tunnel > > 171122 9:36:40 [Note] WSREP: Running: 'wsrep_sst_rsync_tunnel --role > > 'joiner' --address 'overcloud-controller-0.internalapi.redhat.local' --auth > > '' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --parent '2536'' > > > > > > > > [heat-admin@overcloud-controller-0 ~]$ sudo cat > > /var/log/pacemaker/bundles/galera-bundle-0/mysqld.log |grep "ssl" > > 171122 9:36:39 [Note] WSREP: initializing ssl context > > 171122 9:36:39 [Note] WSREP: (a3ef0354, 'ssl://172.17.1.24:4567') listening > > at ssl://172.17.1.24:4567 > > 171122 9:36:39 [Note] WSREP: (a3ef0354, 'ssl://172.17.1.24:4567') > > multicast: , ttl: 1 > > 171122 9:36:39 [Note] WSREP: SSL handshake successful, remote endpoint > > ssl://172.17.1.11:4567 local endpoint ssl://172.17.1.24:49972 cipher: > > AES128-SHA compression: > > 171122 9:36:39 [Note] WSREP: (a3ef0354, 'ssl://172.17.1.24:4567') turning > > message relay requesting on, nonlive peers: > > 171122 9:36:39 [Note] WSREP: declaring 9e9bec19 at ssl://172.17.1.11:4567 > > stable > > 171122 9:36:39 [Note] WSREP: discarding pending addr without UUID: > > ssl://172.17.1.21:4567 > > WSREP_SST: [INFO] Setting up tunnel for joiner: socat > > openssl-listen:4444,bind=overcloud-controller-0.internalapi.redhat.local, > > reuseaddr,fork,cert=/etc/pki/tls/certs/mysql.crt,key=/etc/pki/tls/private/ > > mysql.key,cafile=/etc/ipa/ca.crt,cipher=!SSLv2:kEEH:kRSA:kEDH:kPSK:+3DES:! > > aNULL:!eNULL:!MD5:!EXP:!RC4:!SEED:!IDEA:!DES tcp:localhost:4444 (20171122 > > 09:36:40.262) > > 171122 9:36:42 [Note] WSREP: (a3ef0354, 'ssl://172.17.1.24:4567') turning > > message relay requesting off > > 171122 9:36:51 [Note] WSREP: SSL handshake successful, remote endpoint > > ssl://172.17.1.21:44168 local endpoint ssl://172.17.1.24:4567 cipher: > > AES128-SHA compression: > > 171122 9:36:51 [Note] WSREP: (a3ef0354, 'ssl://172.17.1.24:4567') turning > > message relay requesting on, nonlive peers: > > 171122 9:36:51 [Note] WSREP: declaring 9e9bec19 at ssl://172.17.1.11:4567 > > stable > > 171122 9:36:51 [Note] WSREP: declaring ab0ac0ee at ssl://172.17.1.21:4567 > > stable > > 171122 9:36:54 [Note] WSREP: (a3ef0354, 'ssl://172.17.1.24:4567') turning > > message relay requesting off > > > > Re-assign to ushkalim for testing IPv6 scenario > > Hi Artem, > > Do you have an IPv6 setup we can use to speedup testing? > our HA testing on the IPv4 setup still on-going. Hi, nope, any Ipv6 setup
Verified ON puppet-tripleo-7.4.3-11.el7ost.noarch IPv4 setup passed HA sanity tests
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:3462