Bug 1181796 - Any change to floating ips causes connectivity issues using HA routers
Summary: Any change to floating ips causes connectivity issues using HA routers
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-foreman-installer
Version: 6.0 (Juno)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ga
: Installer
Assignee: Crag Wolfe
QA Contact: Alexander Chuzhoy
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-01-13 19:02 UTC by Miguel Angel Ajo
Modified: 2016-04-26 18:03 UTC (History)
12 users (show)

Fixed In Version: openstack-foreman-installer-3.0.11-1.el7ost
Doc Type: Bug Fix
Doc Text:
If the external network could not access the DNS server on the network (controller) node, keepalived would stagnate for approximately 60 seconds on floating IP address changes. This caused the MASTER server to flap between HA routers and create connectivity issues with the external network. This fix adds an entry into /etc/hosts for 127.0.0.1 to include the network (controller) hostname. The external connectivity now works without issue.
Clone Of: 1181592
Environment:
Last Closed: 2015-02-09 15:19:20 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2015:0156 normal SHIPPED_LIVE Red Hat Enterprise Linux OpenStack Platform Installer Bug Fix Advisory 2015-02-09 20:13:39 UTC

Comment 3 Jason Guiditta 2015-01-14 15:31:57 UTC
Crag posted a patch here:
https://github.com/redhat-openstack/astapor/pull/454

Comment 4 Miguel Angel Ajo 2015-01-14 15:32:51 UTC
Important, after talking with Fabio di Nitto, we found the provided workaround into the first comment is risky:

Use this one, otherwise it could cause problems to pacemaker:

dig A $(hostname) | grep -A1 "ANSWER SEC" | tail -n 1 | awk '{print $NF " " $1}' | sed -e 's/.$//g'  >>/etc/hosts ;   grep $(hostname) /etc/hosts || echo "Failure setting up the hostname entry"

Setting the needinfo to get an ack on this comment

Comment 5 Miguel Angel Ajo 2015-01-14 15:42:32 UTC
After seeing the /etc/hosts posted by Crag on github:

https://github.com/redhat-openstack/astapor/pull/454

I suspect we may not need this fix when deployed from foreman,
is the /etc/hosts always populated like that?

we just need a resolving $(hostname) entry in /etc/hosts

Comment 6 Crag Wolfe 2015-01-14 17:30:33 UTC
Actually, that paste from /etc/hosts was not a staypuft install but from one of my development VM's using foreman (just wanted to illustrate the last line being added).  This is /etc/hosts from a staypuft install:

[root@maca25400702876 ~]# cat /etc/hosts
# HEADER: This file was autogenerated at 2015-01-14 10:30:46 -0500
# HEADER: by puppet.  While it can still be managed manually, it
# HEADER: is definitely not recommended.
127.0.0.1	localhost	localhost.localdomain localhost4 localhost4.localdomain4
::1	localhost	localhost.localdomain localhost6 localhost6.localdomain6
192.168.0.7	lb-backend-maca25400702875
192.168.0.9	lb-backend-maca25400702877
192.168.0.8	pcmk-maca25400702876
192.168.0.9	pcmk-maca25400702877
192.168.0.7	pcmk-maca25400702875
192.168.0.8	lb-backend-maca25400702876
[root@maca25400702876 ~]# hostname
maca25400702876.example.com

So, I think we still need a patch.

Comment 7 Miguel Angel Ajo 2015-01-15 14:07:37 UTC
(In reply to Crag Wolfe from comment #6)
> Actually, that paste from /etc/hosts was not a staypuft install but from one
> of my development VM's using foreman (just wanted to illustrate the last
> line being added).  This is /etc/hosts from a staypuft install:
> 
> [root@maca25400702876 ~]# cat /etc/hosts
> # HEADER: This file was autogenerated at 2015-01-14 10:30:46 -0500
> # HEADER: by puppet.  While it can still be managed manually, it
> # HEADER: is definitely not recommended.
> 127.0.0.1	localhost	localhost.localdomain localhost4 localhost4.localdomain4
> ::1	localhost	localhost.localdomain localhost6 localhost6.localdomain6
> 192.168.0.7	lb-backend-maca25400702875
> 192.168.0.9	lb-backend-maca25400702877
> 192.168.0.8	pcmk-maca25400702876
> 192.168.0.9	pcmk-maca25400702877
> 192.168.0.7	pcmk-maca25400702875
> 192.168.0.8	lb-backend-maca25400702876
> [root@maca25400702876 ~]# hostname
> maca25400702876.example.com
> 
> So, I think we still need a patch.

Thank you for checking that Crag, please use the "dig" version or something equivalent, we need to put the real IP instead of 127.0.0.1

dig A $(hostname) | grep -A1 "ANSWER SEC" | tail -n 1 | awk '{print $NF " " $1}' | sed -e 's/.$//g'  >>/etc/hosts ;   grep $(hostname) /etc/hosts || echo "Failure setting up the hostname entry"

Comment 9 Jason Guiditta 2015-01-23 19:08:19 UTC
Tested and merged updated patch.

Comment 11 Alexander Chuzhoy 2015-01-27 18:43:04 UTC
Verified:
Environment:
ruby193-rubygem-foreman_openstack_simplify-0.0.6-8.el7ost.noarch
openstack-foreman-installer-3.0.11-1.el7ost.noarch
ruby193-rubygem-staypuft-0.5.15-1.el7ost.noarch
rhel-osp-installer-client-0.5.5-2.el7ost.noarch
openstack-puppet-modules-2014.2.8-2.el7ost.noarch
rhel-osp-installer-0.5.5-2.el7ost.noarch


Verified the output from `hostname` actually exists in the /etc/hosts file next to its IP on all controllers.

Comment 13 errata-xmlrpc 2015-02-09 15:19:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-0156.html


Note You need to log in before you can comment on or make changes to this bug.