Bug 1295986 - HAProxy Cannot Bind IPv6 Addresses for OpenStack VIPs
HAProxy Cannot Bind IPv6 Addresses for OpenStack VIPs
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-puppet-modules (Show other bugs)
7.0 (Kilo)
Unspecified Unspecified
high Severity high
: z4
: 7.0 (Kilo)
Assigned To: Giulio Fidente
yeylon@redhat.com
: ZStream
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-01-05 20:44 EST by Dan Sneddon
Modified: 2016-04-18 03:13 EDT (History)
17 users (show)

See Also:
Fixed In Version: openstack-puppet-modules-2015.1.8-36.el7ost
Doc Type: Bug Fix
Doc Text:
Previously, by default, the HAProxy did not listen on IPv6 virtual IP addresses. As a result, OpenStack services that used the 'bind' statement were not available. As a workaround, HAProxy now requires you to set up 'transparent' on the 'bind' statements that use the IPv6 virtual IP addresses. For example, listen ceilometer bind fc00::eeee:8777 transparent Note: The 'transparent' option must never be used with "wildcard" 'bind' statements, as follows: listen wildcardservice bind 0.0.0.0:8000 As a result, HAProxy now listens to IPv6 virtual IP addresses.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-02-18 11:46:29 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
OpenStack gerrit 267211 None None None 2016-01-13 18:27 EST
OpenStack gerrit 267212 None None None 2016-01-13 18:27 EST

  None (edit)
Description Dan Sneddon 2016-01-05 20:44:11 EST
Description of problem:
HAProxy is able to bind to nonlocal virtual IP addresses using IPv4 addresses, because net.ipv4.ip_nonlocal_bind=1 in the sysctl settings. No such feature exists for IPv6 addresses, so when an IPv6 address is configured, HAProxy reports an error.

Version-Release number of selected component (if applicable):
OSP 8 beta

How reproducible:
100%

Steps to Reproduce:
1. Deploy with IPv6 TripleO Heat templates
2.
3.

Actual results:
Although the service VIPs are configured properly in HAProxy, the IP addresses are virtual IPs that don't actually exist on an interface. When HAProxy tries to bind to the port, the following errors are reported:

Jan 06 01:14:38 overcloud-controller-0 haproxy-systemd-wrapper[8783]: [ALERT] 005/011438 (8785) : Starting proxy ceilometer: cannot bind socket [2001:db8:fd00:1000:f816:3eff:fe62:31b2:8777]
Jan 06 01:14:38 overcloud-controller-0 haproxy-systemd-wrapper[8783]: [ALERT] 005/011438 (8785) : Starting proxy cinder: cannot bind socket [2001:db8:fd00:1000:f816:3eff:fe62:31b2:8776]
Jan 06 01:14:38 overcloud-controller-0 haproxy-systemd-wrapper[8783]: [ALERT] 005/011438 (8785) : Starting proxy glance_api: cannot bind socket [2001:db8:fd00:1000:f816:3eff:fe62:31b2:9292]
Jan 06 01:14:38 overcloud-controller-0 haproxy-systemd-wrapper[8783]: [ALERT] 005/011438 (8785) : Starting proxy glance_api: cannot bind socket [fd00:fd00:fd00:3000:f816:3eff:fea3:64ca:9292]

etc.

Expected results:
The VIPs should be bound and the HAProxy should be listening for the services.

Additional info:
When I manually assign the IPv6 address to the loopback adapter, the HAProxy bind succeeds and the service is live.
Comment 2 Michele Baldessari 2016-01-06 04:34:00 EST
It seems patches to add this feature to the kernel have been rejected in the
past:
http://comments.gmane.org/gmane.linux.network/215163

Seems that they think that ip_nonlocal_bind for ipv4 was a mistake and don't
want to add it for ipv6. Instead they suggest apps to use IP_FREEBIND. I do
see some IP_FREEBIND usage in haproxy git, but I have not investigated much more.

Equivalent Debian bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=726569
Comment 3 Michele Baldessari 2016-01-06 05:47:06 EST
I guess opinions change over time ;)

git log --oneline -i --grep ip_nonlocal net |head -1
35a256fee52c ipv6: Nonlocal bind

So best to clone/move this BZ to the kernel and ask for this commit to be backported
Comment 4 Dan Sneddon 2016-01-07 15:19:06 EST
I found a potential fix for this issue, but I'm not sure how to implement it.

If you add "transparent" to the end of the bind statement, it magically works (suggestion from Ryan O'Hara):

bind fd00:fd00:fd00:2000:3f93:ba43:34ad:3201:80 transparent

But the problem is that I can't figure out how to make the HAProxy puppet module output the configuration with "transparent" at the end. Is that possible, or will it require updates to the haproxy puppet module?
Comment 5 Emilien Macchi 2016-01-07 15:48:00 EST
you can patch openstack/puppet-tripleo/loadbalancer.pp to make it.

HTH
Comment 7 Gilles Dubreuil 2016-01-14 00:27:04 EST
Adding the port number to the IPv6 address fixed the issue for me.
Can you please try/confirm?
Comment 8 Jiri Stransky 2016-01-15 06:07:51 EST
Bound fc00::abcd on br-ex, tried a few configs (showing just snippets for brevity). Gilles, i'm not sure what you mean by adding port number? I tried bracket syntax to separate IP and port:

listen ceilometer
  bind [fc00::abcd]:8777

but i'm getting this error, seems like HAproxy doesn't like that syntax:

parsing [/etc/haproxy/haproxy.cfg:23] : 'bind' : invalid address: '[fc00::abcd]' in '[fc00::abcd]:8777'


What Dan suggested worked for me though:

# worked (the IP is bound, as i wrote above, so that's expected)
listen ceilometer
  bind fc00::abcd:8777

# didn't work (IP not bound), showing this error: Starting proxy ceilometer: cannot bind socket [fc00::eeee:8777]
listen ceilometer
  bind fc00::eeee:8777

# worked
listen ceilometer
  bind fc00::eeee:8777 transparent
Comment 10 Marius Cornea 2016-01-19 05:49:21 EST
[root@overcloud-controller-0 ~]# rpm -qa | grep puppet-modules
openstack-puppet-modules-2015.1.8-41.el7ost.noarch

[root@overcloud-controller-0 ~]# systemctl status haproxy
● haproxy.service - Cluster Controlled haproxy
   Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled; vendor preset: disabled)
  Drop-In: /run/systemd/system/haproxy.service.d
           └─50-pacemaker.conf
   Active: active (running) since Tue 2016-01-19 05:41:18 EST; 7min ago
 Main PID: 3052 (haproxy-systemd)
   CGroup: /system.slice/haproxy.service
           ├─3052 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
           ├─3053 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
           └─3082 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds

[root@overcloud-controller-0 ~]# netstat -tupan | grep haproxy | grep LISTEN
tcp        0      0 192.0.2.6:1993          0.0.0.0:*               LISTEN      3082/haproxy        
tcp        0      0 192.0.2.6:35357         0.0.0.0:*               LISTEN      3082/haproxy        
tcp6       0      0 fd00:fd00:fd00:200:5000 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 2001:db8:fd00:1000:5000 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 fd00:fd00:fd00:200:8776 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 2001:db8:fd00:1000:8776 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 fd00:fd00:fd00:200:8777 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 2001:db8:fd00:1000:8777 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 fd00:fd00:fd00:200:3306 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 fd00:fd00:fd00:200:6379 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 fd00:fd00:fd00:300:9292 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 2001:db8:fd00:1000:9292 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 fd00:fd00:fd00:300:8080 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 2001:db8:fd00:1000:8080 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 fd00:fd00:fd00:2000::80 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 2001:db8:fd00:1000:f:80 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 2001:db8:fd00:100:35357 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 fd00:fd00:fd00:200:6080 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 2001:db8:fd00:1000:6080 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 fd00:fd00:fd00:200:9696 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 2001:db8:fd00:1000:9696 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 fd00:fd00:fd00:200:8000 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 2001:db8:fd00:1000:8000 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 fd00:fd00:fd00:200:8003 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 2001:db8:fd00:1000:8003 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 fd00:fd00:fd00:200:8004 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 2001:db8:fd00:1000:8004 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 fd00:fd00:fd00:200:8773 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 2001:db8:fd00:1000:8773 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 fd00:fd00:fd00:200:8774 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 2001:db8:fd00:1000:8774 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 fd00:fd00:fd00:200:8775 :::*                    LISTEN      3082/haproxy
Comment 14 errata-xmlrpc 2016-02-18 11:46:29 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0265.html

Note You need to log in before you can comment on or make changes to this bug.