Bug 1298671

Created attachment 1114890 [details]
galera.cnf

Created attachment 1114891 [details]
/var/log/messages

This could be related to https://mariadb.atlassian.net/browse/MDEV-8034

I'm not sure what the  cause is but I was able to get it running by setting the bind address to the hostname in galera.cnf:

bind-address = overcloud-controller-0

Using a hostname seems the best work around attended there is a DNS resolution (or correct /etc/hosts file) in regards of IPv6.

It seems there is an issue with IPv6 address. 
IPv6 addresses should always be using brackets in cases where a port number could be used. Whether it's not provided and assumes a default or not.
So would it be possible to test your environment with an [::ipv6address] value?

Also, could you please provide all the parameters used for the installation?
So we can identify what value TrippleO Heat Templates has been passed to Galera Puppet module.

(In reply to Gilles Dubreuil from comment #4)
> Using a hostname seems the best work around attended there is a DNS
> resolution (or correct /etc/hosts file) in regards of IPv6.
> 
> It seems there is an issue with IPv6 address. 
> IPv6 addresses should always be using brackets in cases where a port number
> could be used. Whether it's not provided and assumes a default or not.
> So would it be possible to test your environment with an [::ipv6address]
> value?

Setting the following in galera.cnf
bind-address = [fd00:fd00:fd00:2000:f816:3eff:fe94:c469]

fails with:
[ERROR] Can't create IP socket: Connection timed out
[ERROR] Aborting

> Also, could you please provide all the parameters used for the installation?
> So we can identify what value TrippleO Heat Templates has been passed to
> Galera Puppet module.

Here is the mysql_bind_host hiera value:
mysql_bind_host: fd00:fd00:fd00:2000:f816:3eff:fe94:c469

This is the deploy command:

export THT=/home/stack/templates/tripleo-heat-templates #checkout from https://review.openstack.org/#/c/235423/ 
openstack overcloud deploy --templates $THT \
-e $THT/environments/network-isolation-v6.yaml \
-e $THT/environments/net-single-nic-with-vlans.yaml \
-e $THT/environments/puppet-pacemaker.yaml \
-e /home/stack/templates/network-environment-v6.yaml \
--control-scale 1 \
--compute-scale 1 \
--neutron-network-type vxlan \
--neutron-tunnel-types vxlan \
--libvirt-type qemu 

Contens of network-environment-v6.yaml:
parameters:
  ServiceNetMap:
    NeutronTenantNetwork: tenant
    CeilometerApiNetwork: internal_api
    MongoDbNetwork: ctlplane # changed from storage_mgmt for ipv6 testing
    CinderApiNetwork: internal_api
    CinderIscsiNetwork: storage
    GlanceApiNetwork: storage
    GlanceRegistryNetwork: internal_api
    KeystoneAdminApiNetwork: ctlplane # allows undercloud to config endpoints
    KeystonePublicApiNetwork: internal_api
    NeutronApiNetwork: internal_api
    HeatApiNetwork: internal_api
    NovaApiNetwork: internal_api
    NovaMetadataNetwork: internal_api
    NovaVncProxyNetwork: internal_api
    SwiftMgmtNetwork: ctlplane # changed from storage_mgmt for ipv6 testing
    SwiftProxyNetwork: storage
    HorizonNetwork: internal_api
    MemcachedNetwork: internal_api
    RabbitMqNetwork: internal_api
    RedisNetwork: internal_api
    MysqlNetwork: internal_api
    CephClusterNetwork: storage_mgmt
    CephPublicNetwork: storage
    ControllerHostnameResolveNetwork: internal_api
    ComputeHostnameResolveNetwork: internal_api
    BlockStorageHostnameResolveNetwork: internal_api
    ObjectStorageHostnameResolveNetwork: internal_api
    CephStorageHostnameResolveNetwork: storage
  NeutronMetadataProxySharedSecret: marius

parameter_defaults:
  ControlPlaneSubnetCidr: "24"
  EC2MetadataIp: 192.0.2.1
  ControlPlaneDefaultRoute: 192.0.2.1
  ExternalInterfaceDefaultRoute: 2001:db8:fd00:1000:dead:beef:cafe:f00
  #DnsServers: ["2001:db8:fd00:1000:dead:beef:cafe:f00", "2001:db8:fd00:1000:dead:beef:cafe:f01"]

Giulio, can you check on this? Dan thinks it may already be fixed.

FWIW the issue was reported here for Galera:
https://bugs.launchpad.net/galera/+bug/1130595

Under investigation; by commenting out:

  bind-address

and adding:

  wsrep_provider_options = "gmcast.listen_addr=tcp://[::]:4567;"

where [::] is replaced by the desired binding address, we get galera to bind on the ipv6 address.

If bind-address is also set to an ipv6 address though the problem remains as described in the report:

  160115 15:48:45 [ERROR] WSREP: Can't parse port number from '8d64': 22 (Invalid argument) at galerautils/src/gu_uri.cpp:parse_authority():69

When mysql is trying to bind to an ipv6 address with anything else than ::, it seems to fail here:
 https://github.com/atcurtis/mariadb/blob/5.5/sql/mysqld.cc#L2198 or
 https://github.com/atcurtis/mariadb/blob/5.5/sql/mysqld.cc#L2164

Investigating...

I've managed to get Galera to work on an IPv6 environment by manually setting up this way:

- In /etc/my.cnf.d/galera.cnf, set:

bind_address = <hostname>
wsrep_provider_options="gmcast.listen_addr=tcp://[::]:4567;"

Where <hostname> is an entry in /etc/hosts pointing to the server IPv6 address

- Then, in the pacemaker resource, set:

wsrep_cluster_address=gcomm://hacontroller1,hacontroller2,hacontroller3

Where hacontroller1, hacontroller2 and hacontroller3 are the /etc/hosts entries (or DNS) pointing to the IPv6 addresses.

With those, mysqld was listening on an IPv6 address, and wsrep communication succeeded.

openstack-tripleo-heat-templates-0.8.6-106.el7ost.noarch

[root@overcloud-controller-0 ~]# pcs status | grep -A 1 galera
 Master/Slave Set: galera-master [galera]
     Masters: [ overcloud-controller-0 ]

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0264.html