Created attachment 1114889 [details]
Description of problem:
Galera fails to start in ipv6 environment with error:
160114 17:25:21 [ERROR] WSREP: Can't parse port number from '4af4': 22 (Invalid argument)
160114 17:25:21 [ERROR] WSREP: wsrep::init() failed: 7, must shutdown
160114 17:25:21 [ERROR] Aborting
Version-Release number of selected component (if applicable):
I'm doing the test following the instructions in:
and enabling pacemaker by passing an additional $THT/environments/puppet-pacemaker.yaml environment file
Steps to Reproduce:
1. Apply workarounds for BZ#1295986, BZ#1297850 and BZ#1298506
2. Deploy ipv6 enabled overcloud
Galera resource fails to start:
Master/Slave Set: galera-master [galera]
galera (ocf::heartbeat:galera): FAILED Master overcloud-controller-0 (unmanaged)
* galera_promote_0 on overcloud-controller-0 'unknown error' (1): call=45, status=complete, exitreason='MySQL server failed to start (pid=8247) (rc=0), please check your installation',
last-rc-change='Thu Jan 14 17:25:18 2016', queued=0ms, exec=4365ms
Galera resource starts
Attaching galera.cnf, mysqld.log and /var/log/messages.
Created attachment 1114890 [details]
Created attachment 1114891 [details]
This could be related to https://mariadb.atlassian.net/browse/MDEV-8034
I'm not sure what the cause is but I was able to get it running by setting the bind address to the hostname in galera.cnf:
bind-address = overcloud-controller-0
Using a hostname seems the best work around attended there is a DNS resolution (or correct /etc/hosts file) in regards of IPv6.
It seems there is an issue with IPv6 address.
IPv6 addresses should always be using brackets in cases where a port number could be used. Whether it's not provided and assumes a default or not.
So would it be possible to test your environment with an [::ipv6address] value?
Also, could you please provide all the parameters used for the installation?
So we can identify what value TrippleO Heat Templates has been passed to Galera Puppet module.
(In reply to Gilles Dubreuil from comment #4)
> Using a hostname seems the best work around attended there is a DNS
> resolution (or correct /etc/hosts file) in regards of IPv6.
> It seems there is an issue with IPv6 address.
> IPv6 addresses should always be using brackets in cases where a port number
> could be used. Whether it's not provided and assumes a default or not.
> So would it be possible to test your environment with an [::ipv6address]
Setting the following in galera.cnf
bind-address = [fd00:fd00:fd00:2000:f816:3eff:fe94:c469]
[ERROR] Can't create IP socket: Connection timed out
> Also, could you please provide all the parameters used for the installation?
> So we can identify what value TrippleO Heat Templates has been passed to
> Galera Puppet module.
Here is the mysql_bind_host hiera value:
This is the deploy command:
export THT=/home/stack/templates/tripleo-heat-templates #checkout from https://review.openstack.org/#/c/235423/
openstack overcloud deploy --templates $THT \
-e $THT/environments/network-isolation-v6.yaml \
-e $THT/environments/net-single-nic-with-vlans.yaml \
-e $THT/environments/puppet-pacemaker.yaml \
-e /home/stack/templates/network-environment-v6.yaml \
--control-scale 1 \
--compute-scale 1 \
--neutron-network-type vxlan \
--neutron-tunnel-types vxlan \
Contens of network-environment-v6.yaml:
MongoDbNetwork: ctlplane # changed from storage_mgmt for ipv6 testing
KeystoneAdminApiNetwork: ctlplane # allows undercloud to config endpoints
SwiftMgmtNetwork: ctlplane # changed from storage_mgmt for ipv6 testing
#DnsServers: ["2001:db8:fd00:1000:dead:beef:cafe:f00", "2001:db8:fd00:1000:dead:beef:cafe:f01"]
Giulio, can you check on this? Dan thinks it may already be fixed.
FWIW the issue was reported here for Galera:
Under investigation; by commenting out:
wsrep_provider_options = "gmcast.listen_addr=tcp://[::]:4567;"
where [::] is replaced by the desired binding address, we get galera to bind on the ipv6 address.
If bind-address is also set to an ipv6 address though the problem remains as described in the report:
160115 15:48:45 [ERROR] WSREP: Can't parse port number from '8d64': 22 (Invalid argument) at galerautils/src/gu_uri.cpp:parse_authority():69
When mysql is trying to bind to an ipv6 address with anything else than ::, it seems to fail here:
I've managed to get Galera to work on an IPv6 environment by manually setting up this way:
- In /etc/my.cnf.d/galera.cnf, set:
bind_address = <hostname>
Where <hostname> is an entry in /etc/hosts pointing to the server IPv6 address
- Then, in the pacemaker resource, set:
Where hacontroller1, hacontroller2 and hacontroller3 are the /etc/hosts entries (or DNS) pointing to the IPv6 addresses.
With those, mysqld was listening on an IPv6 address, and wsrep communication succeeded.
[root@overcloud-controller-0 ~]# pcs status | grep -A 1 galera
Master/Slave Set: galera-master [galera]
Masters: [ overcloud-controller-0 ]
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.