Bug 1295986

Summary:	HAProxy Cannot Bind IPv6 Addresses for OpenStack VIPs
Product:	Red Hat OpenStack	Reporter:	Dan Sneddon <dsneddon>
Component:	openstack-puppet-modules	Assignee:	Giulio Fidente <gfidente>
Status:	CLOSED ERRATA	QA Contact:	yeylon <yeylon>
Severity:	high	Docs Contact:
Priority:	high
Version:	7.0 (Kilo)	CC:	bperkins, dnavale, dsneddon, emacchi, gchamoul, gdubreui, gfidente, jstransk, mburns, mcornea, michele, oblaut, rhel-osp-director-maint, sputhenp, srevivo, ushkalim, yeylon
Target Milestone:	z4	Keywords:	ZStream
Target Release:	7.0 (Kilo)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-puppet-modules-2015.1.8-36.el7ost	Doc Type:	Bug Fix
Doc Text:	Previously, by default, the HAProxy did not listen on IPv6 virtual IP addresses. As a result, OpenStack services that used the 'bind' statement were not available. As a workaround, HAProxy now requires you to set up 'transparent' on the 'bind' statements that use the IPv6 virtual IP addresses. For example, listen ceilometer bind fc00::eeee:8777 transparent Note: The 'transparent' option must never be used with "wildcard" 'bind' statements, as follows: listen wildcardservice bind 0.0.0.0:8000 As a result, HAProxy now listens to IPv6 virtual IP addresses.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-02-18 16:46:29 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Dan Sneddon 2016-01-06 01:44:11 UTC

Description of problem:
HAProxy is able to bind to nonlocal virtual IP addresses using IPv4 addresses, because net.ipv4.ip_nonlocal_bind=1 in the sysctl settings. No such feature exists for IPv6 addresses, so when an IPv6 address is configured, HAProxy reports an error.

Version-Release number of selected component (if applicable):
OSP 8 beta

How reproducible:
100%

Steps to Reproduce:
1. Deploy with IPv6 TripleO Heat templates
2.
3.

Actual results:
Although the service VIPs are configured properly in HAProxy, the IP addresses are virtual IPs that don't actually exist on an interface. When HAProxy tries to bind to the port, the following errors are reported:

Jan 06 01:14:38 overcloud-controller-0 haproxy-systemd-wrapper[8783]: [ALERT] 005/011438 (8785) : Starting proxy ceilometer: cannot bind socket [2001:db8:fd00:1000:f816:3eff:fe62:31b2:8777]
Jan 06 01:14:38 overcloud-controller-0 haproxy-systemd-wrapper[8783]: [ALERT] 005/011438 (8785) : Starting proxy cinder: cannot bind socket [2001:db8:fd00:1000:f816:3eff:fe62:31b2:8776]
Jan 06 01:14:38 overcloud-controller-0 haproxy-systemd-wrapper[8783]: [ALERT] 005/011438 (8785) : Starting proxy glance_api: cannot bind socket [2001:db8:fd00:1000:f816:3eff:fe62:31b2:9292]
Jan 06 01:14:38 overcloud-controller-0 haproxy-systemd-wrapper[8783]: [ALERT] 005/011438 (8785) : Starting proxy glance_api: cannot bind socket [fd00:fd00:fd00:3000:f816:3eff:fea3:64ca:9292]

etc.

Expected results:
The VIPs should be bound and the HAProxy should be listening for the services.

Additional info:
When I manually assign the IPv6 address to the loopback adapter, the HAProxy bind succeeds and the service is live.

Comment 2 Michele Baldessari 2016-01-06 09:34:00 UTC

It seems patches to add this feature to the kernel have been rejected in the
past:
http://comments.gmane.org/gmane.linux.network/215163

Seems that they think that ip_nonlocal_bind for ipv4 was a mistake and don't
want to add it for ipv6. Instead they suggest apps to use IP_FREEBIND. I do
see some IP_FREEBIND usage in haproxy git, but I have not investigated much more.

Equivalent Debian bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=726569

Comment 3 Michele Baldessari 2016-01-06 10:47:06 UTC

I guess opinions change over time ;)

git log --oneline -i --grep ip_nonlocal net |head -1
35a256fee52c ipv6: Nonlocal bind

So best to clone/move this BZ to the kernel and ask for this commit to be backported

Comment 4 Dan Sneddon 2016-01-07 20:19:06 UTC

I found a potential fix for this issue, but I'm not sure how to implement it.

If you add "transparent" to the end of the bind statement, it magically works (suggestion from Ryan O'Hara):

bind fd00:fd00:fd00:2000:3f93:ba43:34ad:3201:80 transparent

But the problem is that I can't figure out how to make the HAProxy puppet module output the configuration with "transparent" at the end. Is that possible, or will it require updates to the haproxy puppet module?

Comment 5 Emilien Macchi 2016-01-07 20:48:00 UTC

you can patch openstack/puppet-tripleo/loadbalancer.pp to make it.

HTH

Comment 7 Gilles Dubreuil 2016-01-14 05:27:04 UTC

Adding the port number to the IPv6 address fixed the issue for me.
Can you please try/confirm?

Comment 8 Jiri Stransky 2016-01-15 11:07:51 UTC

Bound fc00::abcd on br-ex, tried a few configs (showing just snippets for brevity). Gilles, i'm not sure what you mean by adding port number? I tried bracket syntax to separate IP and port:

listen ceilometer
  bind [fc00::abcd]:8777

but i'm getting this error, seems like HAproxy doesn't like that syntax:

parsing [/etc/haproxy/haproxy.cfg:23] : 'bind' : invalid address: '[fc00::abcd]' in '[fc00::abcd]:8777'


What Dan suggested worked for me though:

# worked (the IP is bound, as i wrote above, so that's expected)
listen ceilometer
  bind fc00::abcd:8777

# didn't work (IP not bound), showing this error: Starting proxy ceilometer: cannot bind socket [fc00::eeee:8777]
listen ceilometer
  bind fc00::eeee:8777

# worked
listen ceilometer
  bind fc00::eeee:8777 transparent

Comment 10 Marius Cornea 2016-01-19 10:49:21 UTC

[root@overcloud-controller-0 ~]# rpm -qa | grep puppet-modules
openstack-puppet-modules-2015.1.8-41.el7ost.noarch

[root@overcloud-controller-0 ~]# systemctl status haproxy
● haproxy.service - Cluster Controlled haproxy
   Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled; vendor preset: disabled)
  Drop-In: /run/systemd/system/haproxy.service.d
           └─50-pacemaker.conf
   Active: active (running) since Tue 2016-01-19 05:41:18 EST; 7min ago
 Main PID: 3052 (haproxy-systemd)
   CGroup: /system.slice/haproxy.service
           ├─3052 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
           ├─3053 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
           └─3082 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds

[root@overcloud-controller-0 ~]# netstat -tupan | grep haproxy | grep LISTEN
tcp        0      0 192.0.2.6:1993          0.0.0.0:*               LISTEN      3082/haproxy        
tcp        0      0 192.0.2.6:35357         0.0.0.0:*               LISTEN      3082/haproxy        
tcp6       0      0 fd00:fd00:fd00:200:5000 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 2001:db8:fd00:1000:5000 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 fd00:fd00:fd00:200:8776 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 2001:db8:fd00:1000:8776 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 fd00:fd00:fd00:200:8777 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 2001:db8:fd00:1000:8777 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 fd00:fd00:fd00:200:3306 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 fd00:fd00:fd00:200:6379 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 fd00:fd00:fd00:300:9292 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 2001:db8:fd00:1000:9292 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 fd00:fd00:fd00:300:8080 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 2001:db8:fd00:1000:8080 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 fd00:fd00:fd00:2000::80 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 2001:db8:fd00:1000:f:80 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 2001:db8:fd00:100:35357 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 fd00:fd00:fd00:200:6080 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 2001:db8:fd00:1000:6080 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 fd00:fd00:fd00:200:9696 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 2001:db8:fd00:1000:9696 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 fd00:fd00:fd00:200:8000 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 2001:db8:fd00:1000:8000 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 fd00:fd00:fd00:200:8003 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 2001:db8:fd00:1000:8003 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 fd00:fd00:fd00:200:8004 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 2001:db8:fd00:1000:8004 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 fd00:fd00:fd00:200:8773 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 2001:db8:fd00:1000:8773 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 fd00:fd00:fd00:200:8774 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 2001:db8:fd00:1000:8774 :::*                    LISTEN      3082/haproxy        
tcp6       0      0 fd00:fd00:fd00:200:8775 :::*                    LISTEN      3082/haproxy

Comment 14 errata-xmlrpc 2016-02-18 16:46:29 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0265.html