Created attachment 1114889 [details] mysqld.log Description of problem: Galera fails to start in ipv6 environment with error: 160114 17:25:21 [ERROR] WSREP: Can't parse port number from '4af4': 22 (Invalid argument) at galerautils/src/gu_uri.cpp:parse_authority():69 160114 17:25:21 [ERROR] WSREP: wsrep::init() failed: 7, must shutdown 160114 17:25:21 [ERROR] Aborting Version-Release number of selected component (if applicable): I'm doing the test following the instructions in: https://etherpad.openstack.org/p/tripleo-ipv6-support and enabling pacemaker by passing an additional $THT/environments/puppet-pacemaker.yaml environment file How reproducible: 100% Steps to Reproduce: 1. Apply workarounds for BZ#1295986, BZ#1297850 and BZ#1298506 2. Deploy ipv6 enabled overcloud Actual results: Galera resource fails to start: Master/Slave Set: galera-master [galera] galera (ocf::heartbeat:galera): FAILED Master overcloud-controller-0 (unmanaged) Failed Actions: * galera_promote_0 on overcloud-controller-0 'unknown error' (1): call=45, status=complete, exitreason='MySQL server failed to start (pid=8247) (rc=0), please check your installation', last-rc-change='Thu Jan 14 17:25:18 2016', queued=0ms, exec=4365ms Expected results: Galera resource starts Additional info: Attaching galera.cnf, mysqld.log and /var/log/messages.
Created attachment 1114890 [details] galera.cnf
Created attachment 1114891 [details] /var/log/messages
This could be related to https://mariadb.atlassian.net/browse/MDEV-8034 I'm not sure what the cause is but I was able to get it running by setting the bind address to the hostname in galera.cnf: bind-address = overcloud-controller-0
Using a hostname seems the best work around attended there is a DNS resolution (or correct /etc/hosts file) in regards of IPv6. It seems there is an issue with IPv6 address. IPv6 addresses should always be using brackets in cases where a port number could be used. Whether it's not provided and assumes a default or not. So would it be possible to test your environment with an [::ipv6address] value? Also, could you please provide all the parameters used for the installation? So we can identify what value TrippleO Heat Templates has been passed to Galera Puppet module.
(In reply to Gilles Dubreuil from comment #4) > Using a hostname seems the best work around attended there is a DNS > resolution (or correct /etc/hosts file) in regards of IPv6. > > It seems there is an issue with IPv6 address. > IPv6 addresses should always be using brackets in cases where a port number > could be used. Whether it's not provided and assumes a default or not. > So would it be possible to test your environment with an [::ipv6address] > value? Setting the following in galera.cnf bind-address = [fd00:fd00:fd00:2000:f816:3eff:fe94:c469] fails with: [ERROR] Can't create IP socket: Connection timed out [ERROR] Aborting > Also, could you please provide all the parameters used for the installation? > So we can identify what value TrippleO Heat Templates has been passed to > Galera Puppet module. Here is the mysql_bind_host hiera value: mysql_bind_host: fd00:fd00:fd00:2000:f816:3eff:fe94:c469 This is the deploy command: export THT=/home/stack/templates/tripleo-heat-templates #checkout from https://review.openstack.org/#/c/235423/ openstack overcloud deploy --templates $THT \ -e $THT/environments/network-isolation-v6.yaml \ -e $THT/environments/net-single-nic-with-vlans.yaml \ -e $THT/environments/puppet-pacemaker.yaml \ -e /home/stack/templates/network-environment-v6.yaml \ --control-scale 1 \ --compute-scale 1 \ --neutron-network-type vxlan \ --neutron-tunnel-types vxlan \ --libvirt-type qemu Contens of network-environment-v6.yaml: parameters: ServiceNetMap: NeutronTenantNetwork: tenant CeilometerApiNetwork: internal_api MongoDbNetwork: ctlplane # changed from storage_mgmt for ipv6 testing CinderApiNetwork: internal_api CinderIscsiNetwork: storage GlanceApiNetwork: storage GlanceRegistryNetwork: internal_api KeystoneAdminApiNetwork: ctlplane # allows undercloud to config endpoints KeystonePublicApiNetwork: internal_api NeutronApiNetwork: internal_api HeatApiNetwork: internal_api NovaApiNetwork: internal_api NovaMetadataNetwork: internal_api NovaVncProxyNetwork: internal_api SwiftMgmtNetwork: ctlplane # changed from storage_mgmt for ipv6 testing SwiftProxyNetwork: storage HorizonNetwork: internal_api MemcachedNetwork: internal_api RabbitMqNetwork: internal_api RedisNetwork: internal_api MysqlNetwork: internal_api CephClusterNetwork: storage_mgmt CephPublicNetwork: storage ControllerHostnameResolveNetwork: internal_api ComputeHostnameResolveNetwork: internal_api BlockStorageHostnameResolveNetwork: internal_api ObjectStorageHostnameResolveNetwork: internal_api CephStorageHostnameResolveNetwork: storage NeutronMetadataProxySharedSecret: marius parameter_defaults: ControlPlaneSubnetCidr: "24" EC2MetadataIp: 192.0.2.1 ControlPlaneDefaultRoute: 192.0.2.1 ExternalInterfaceDefaultRoute: 2001:db8:fd00:1000:dead:beef:cafe:f00 #DnsServers: ["2001:db8:fd00:1000:dead:beef:cafe:f00", "2001:db8:fd00:1000:dead:beef:cafe:f01"]
Giulio, can you check on this? Dan thinks it may already be fixed.
FWIW the issue was reported here for Galera: https://bugs.launchpad.net/galera/+bug/1130595
Under investigation; by commenting out: bind-address and adding: wsrep_provider_options = "gmcast.listen_addr=tcp://[::]:4567;" where [::] is replaced by the desired binding address, we get galera to bind on the ipv6 address. If bind-address is also set to an ipv6 address though the problem remains as described in the report: 160115 15:48:45 [ERROR] WSREP: Can't parse port number from '8d64': 22 (Invalid argument) at galerautils/src/gu_uri.cpp:parse_authority():69
When mysql is trying to bind to an ipv6 address with anything else than ::, it seems to fail here: https://github.com/atcurtis/mariadb/blob/5.5/sql/mysqld.cc#L2198 or https://github.com/atcurtis/mariadb/blob/5.5/sql/mysqld.cc#L2164 Investigating...
I've managed to get Galera to work on an IPv6 environment by manually setting up this way: - In /etc/my.cnf.d/galera.cnf, set: bind_address = <hostname> wsrep_provider_options="gmcast.listen_addr=tcp://[::]:4567;" Where <hostname> is an entry in /etc/hosts pointing to the server IPv6 address - Then, in the pacemaker resource, set: wsrep_cluster_address=gcomm://hacontroller1,hacontroller2,hacontroller3 Where hacontroller1, hacontroller2 and hacontroller3 are the /etc/hosts entries (or DNS) pointing to the IPv6 addresses. With those, mysqld was listening on an IPv6 address, and wsrep communication succeeded.
openstack-tripleo-heat-templates-0.8.6-106.el7ost.noarch [root@overcloud-controller-0 ~]# pcs status | grep -A 1 galera Master/Slave Set: galera-master [galera] Masters: [ overcloud-controller-0 ]
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-0264.html