Description of problem: HAProxy is able to bind to nonlocal virtual IP addresses using IPv4 addresses, because net.ipv4.ip_nonlocal_bind=1 in the sysctl settings. No such feature exists for IPv6 addresses, so when an IPv6 address is configured, HAProxy reports an error. Version-Release number of selected component (if applicable): OSP 8 beta How reproducible: 100% Steps to Reproduce: 1. Deploy with IPv6 TripleO Heat templates 2. 3. Actual results: Although the service VIPs are configured properly in HAProxy, the IP addresses are virtual IPs that don't actually exist on an interface. When HAProxy tries to bind to the port, the following errors are reported: Jan 06 01:14:38 overcloud-controller-0 haproxy-systemd-wrapper[8783]: [ALERT] 005/011438 (8785) : Starting proxy ceilometer: cannot bind socket [2001:db8:fd00:1000:f816:3eff:fe62:31b2:8777] Jan 06 01:14:38 overcloud-controller-0 haproxy-systemd-wrapper[8783]: [ALERT] 005/011438 (8785) : Starting proxy cinder: cannot bind socket [2001:db8:fd00:1000:f816:3eff:fe62:31b2:8776] Jan 06 01:14:38 overcloud-controller-0 haproxy-systemd-wrapper[8783]: [ALERT] 005/011438 (8785) : Starting proxy glance_api: cannot bind socket [2001:db8:fd00:1000:f816:3eff:fe62:31b2:9292] Jan 06 01:14:38 overcloud-controller-0 haproxy-systemd-wrapper[8783]: [ALERT] 005/011438 (8785) : Starting proxy glance_api: cannot bind socket [fd00:fd00:fd00:3000:f816:3eff:fea3:64ca:9292] etc. Expected results: The VIPs should be bound and the HAProxy should be listening for the services. Additional info: When I manually assign the IPv6 address to the loopback adapter, the HAProxy bind succeeds and the service is live.
It seems patches to add this feature to the kernel have been rejected in the past: http://comments.gmane.org/gmane.linux.network/215163 Seems that they think that ip_nonlocal_bind for ipv4 was a mistake and don't want to add it for ipv6. Instead they suggest apps to use IP_FREEBIND. I do see some IP_FREEBIND usage in haproxy git, but I have not investigated much more. Equivalent Debian bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=726569
I guess opinions change over time ;) git log --oneline -i --grep ip_nonlocal net |head -1 35a256fee52c ipv6: Nonlocal bind So best to clone/move this BZ to the kernel and ask for this commit to be backported
I found a potential fix for this issue, but I'm not sure how to implement it. If you add "transparent" to the end of the bind statement, it magically works (suggestion from Ryan O'Hara): bind fd00:fd00:fd00:2000:3f93:ba43:34ad:3201:80 transparent But the problem is that I can't figure out how to make the HAProxy puppet module output the configuration with "transparent" at the end. Is that possible, or will it require updates to the haproxy puppet module?
you can patch openstack/puppet-tripleo/loadbalancer.pp to make it. HTH
Adding the port number to the IPv6 address fixed the issue for me. Can you please try/confirm?
Bound fc00::abcd on br-ex, tried a few configs (showing just snippets for brevity). Gilles, i'm not sure what you mean by adding port number? I tried bracket syntax to separate IP and port: listen ceilometer bind [fc00::abcd]:8777 but i'm getting this error, seems like HAproxy doesn't like that syntax: parsing [/etc/haproxy/haproxy.cfg:23] : 'bind' : invalid address: '[fc00::abcd]' in '[fc00::abcd]:8777' What Dan suggested worked for me though: # worked (the IP is bound, as i wrote above, so that's expected) listen ceilometer bind fc00::abcd:8777 # didn't work (IP not bound), showing this error: Starting proxy ceilometer: cannot bind socket [fc00::eeee:8777] listen ceilometer bind fc00::eeee:8777 # worked listen ceilometer bind fc00::eeee:8777 transparent
[root@overcloud-controller-0 ~]# rpm -qa | grep puppet-modules openstack-puppet-modules-2015.1.8-41.el7ost.noarch [root@overcloud-controller-0 ~]# systemctl status haproxy ● haproxy.service - Cluster Controlled haproxy Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled; vendor preset: disabled) Drop-In: /run/systemd/system/haproxy.service.d └─50-pacemaker.conf Active: active (running) since Tue 2016-01-19 05:41:18 EST; 7min ago Main PID: 3052 (haproxy-systemd) CGroup: /system.slice/haproxy.service ├─3052 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid ├─3053 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds └─3082 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds [root@overcloud-controller-0 ~]# netstat -tupan | grep haproxy | grep LISTEN tcp 0 0 192.0.2.6:1993 0.0.0.0:* LISTEN 3082/haproxy tcp 0 0 192.0.2.6:35357 0.0.0.0:* LISTEN 3082/haproxy tcp6 0 0 fd00:fd00:fd00:200:5000 :::* LISTEN 3082/haproxy tcp6 0 0 2001:db8:fd00:1000:5000 :::* LISTEN 3082/haproxy tcp6 0 0 fd00:fd00:fd00:200:8776 :::* LISTEN 3082/haproxy tcp6 0 0 2001:db8:fd00:1000:8776 :::* LISTEN 3082/haproxy tcp6 0 0 fd00:fd00:fd00:200:8777 :::* LISTEN 3082/haproxy tcp6 0 0 2001:db8:fd00:1000:8777 :::* LISTEN 3082/haproxy tcp6 0 0 fd00:fd00:fd00:200:3306 :::* LISTEN 3082/haproxy tcp6 0 0 fd00:fd00:fd00:200:6379 :::* LISTEN 3082/haproxy tcp6 0 0 fd00:fd00:fd00:300:9292 :::* LISTEN 3082/haproxy tcp6 0 0 2001:db8:fd00:1000:9292 :::* LISTEN 3082/haproxy tcp6 0 0 fd00:fd00:fd00:300:8080 :::* LISTEN 3082/haproxy tcp6 0 0 2001:db8:fd00:1000:8080 :::* LISTEN 3082/haproxy tcp6 0 0 fd00:fd00:fd00:2000::80 :::* LISTEN 3082/haproxy tcp6 0 0 2001:db8:fd00:1000:f:80 :::* LISTEN 3082/haproxy tcp6 0 0 2001:db8:fd00:100:35357 :::* LISTEN 3082/haproxy tcp6 0 0 fd00:fd00:fd00:200:6080 :::* LISTEN 3082/haproxy tcp6 0 0 2001:db8:fd00:1000:6080 :::* LISTEN 3082/haproxy tcp6 0 0 fd00:fd00:fd00:200:9696 :::* LISTEN 3082/haproxy tcp6 0 0 2001:db8:fd00:1000:9696 :::* LISTEN 3082/haproxy tcp6 0 0 fd00:fd00:fd00:200:8000 :::* LISTEN 3082/haproxy tcp6 0 0 2001:db8:fd00:1000:8000 :::* LISTEN 3082/haproxy tcp6 0 0 fd00:fd00:fd00:200:8003 :::* LISTEN 3082/haproxy tcp6 0 0 2001:db8:fd00:1000:8003 :::* LISTEN 3082/haproxy tcp6 0 0 fd00:fd00:fd00:200:8004 :::* LISTEN 3082/haproxy tcp6 0 0 2001:db8:fd00:1000:8004 :::* LISTEN 3082/haproxy tcp6 0 0 fd00:fd00:fd00:200:8773 :::* LISTEN 3082/haproxy tcp6 0 0 2001:db8:fd00:1000:8773 :::* LISTEN 3082/haproxy tcp6 0 0 fd00:fd00:fd00:200:8774 :::* LISTEN 3082/haproxy tcp6 0 0 2001:db8:fd00:1000:8774 :::* LISTEN 3082/haproxy tcp6 0 0 fd00:fd00:fd00:200:8775 :::* LISTEN 3082/haproxy
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-0265.html