Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1215177

Summary: VRRP failback after failover does not work
Product: Red Hat OpenStack Reporter: wdaniel
Component: openstack-neutronAssignee: Nir Magnezi <nmagnezi>
Status: CLOSED ERRATA QA Contact: Ofer Blaut <oblaut>
Severity: high Docs Contact:
Priority: high    
Version: 6.0 (Juno)CC: benglish, chrisw, ihrachys, lpeer, mlopes, nyechiel, oblaut, tfreger, yeylon
Target Milestone: z1Keywords: ZStream
Target Release: 7.0 (Kilo)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-neutron-2015.1.0-13.el7ost Doc Type: Bug Fix
Doc Text:
Under certain circumstances, the L3 agent might have incorrectly started before the OVS agent during the node boot process. Consequently, when the L3 agent finished configuring a HA router and started up keepalived, it would transition to master and send out gratuitous ARPs before the OVS agent finished plugging its ports. This meant that the gratuitous ARP was lost, but with the router still acting as master. This update addresses this issue by adding `garp_master_repeat=5` and `garp_master_refresh=10` to keepalived.conf: garp_master_repeat - Defines how often to broadcast the gratuitous ARP after MASTER. garp_master_refresh - Periodic delay in seconds between sending gratuitous ARP while in MASTER. As a result, when transitioning to MASTER state, the keepalived will send several gratuitous ARPs to ensure that they reach the OVS agent after it has plugged its ports.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2015-09-03 17:57:00 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1215179    

Description wdaniel 2015-04-24 13:49:52 UTC 
Description of problem:

We are using VRRP for L3 high availability across 3 management nodes.  We are testing VRRP failover by rebooting management node 1 and continually pinging a Floating IP that is routing through that node that will be rebooted.  After rebooting node 1, the floating IP failover happens successfully after a few seconds and is reachable through node management node 2. However, when node 1 (the node that was rebooted) comes back online the floating IP will stop being reachable after ~1 minute.  Doing some more investigation it appears that the virtual router on node 2 was still receiving ping requests for the floating IP but was not sending any responses.  In looking at the router namespace, the router on node 1 had the floating IPs in its namespace but was not receiving any of the ping requests.  The router on node 2 did not have any floating IPs in its namespace.  I will attach terminal output which illustrates this more clearly.   After 10-20 minutes though, the floating IP becomes reachable again and is routing through node 1 successfully.

Version-Release number of selected component (if applicable):

openstack-neutron-2014.2.1-6.el7ost.noarch

How reproducible:

Very

Steps to Reproduce:
1. Set up VRRP for L3 HA
2. Reboot one of the controllers
3. Ping floating IP that was originally routed through the rebooted node

Actual results:

Once the node finishes rebooting the floating IP becomes unavailable for some time

Expected results:

Floating IP failback transitions smoothly without any interruption in service
Comment 10 Ofer Blaut 2015-08-31 15:18:49 UTC 
Tested on setup with 1 controller and 2 network nodes


ping fail when network node 1 failed for few sec, but not when it came up 

[root@controller ~(keystone_admin)]# neutron l3-agent-list-hosting-router router1
+--------------------------------------+----------------------------------------------------------+----------------+-------+----------+
| id                                   | host                                                     | admin_state_up | alive | ha_state |
+--------------------------------------+----------------------------------------------------------+----------------+-------+----------+
| dd115bf3-1613-4b52-8e71-1622b0bcd7d8 | networker-4348a4bf-4d80-4d16-9b43-cf0f32923b2c.novalocal | True           | :-)   | active   |
| 9707129c-fb57-4a30-b321-99e414e0be07 | networker-a29ff2fb-13d1-4f8e-b24c-a7141ffd0a44.novalocal | True           | :-)   | active   |
+--------------------------------------+----------------------------------------------------------+----------------+-------+----------+
[root@controller ~(keystone_admin)]# neutron l3-agent-list-hosting-router router1
+--------------------------------------+----------------------------------------------------------+----------------+-------+----------+
| id                                   | host                                                     | admin_state_up | alive | ha_state |
+--------------------------------------+----------------------------------------------------------+----------------+-------+----------+
| dd115bf3-1613-4b52-8e71-1622b0bcd7d8 | networker-4348a4bf-4d80-4d16-9b43-cf0f32923b2c.novalocal | True           | :-)   | active   |
| 9707129c-fb57-4a30-b321-99e414e0be07 | networker-a29ff2fb-13d1-4f8e-b24c-a7141ffd0a44.novalocal | True           | :-)   | standby  |
+--------------------------------------+----------------------------------------------------------+----------------+-------+----------+
[root@controller ~(keystone_admin)]# neutron l3-agent-list-hosting-router router1
+--------------------------------------+----------------------------------------------------------+----------------+-------+----------+
| id                                   | host                                                     | admin_state_up | alive | ha_state |
+--------------------------------------+----------------------------------------------------------+----------------+-------+----------+
| dd115bf3-1613-4b52-8e71-1622b0bcd7d8 | networker-4348a4bf-4d80-4d16-9b43-cf0f32923b2c.novalocal | True           | :-)   | active   |
| 9707129c-fb57-4a30-b321-99e414e0be07 | networker-a29ff2fb-13d1-4f8e-b24c-a7141ffd0a44.novalocal | True           | :-)   | standby  |
+--------------------------------------+----------------------------------------------------------+----------------+-------+----------+
[root@controller ~(keystone_admin)]# neutron l3-agent-list-hosting-router router1
+--------------------------------------+----------------------------------------------------------+----------------+-------+----------+
| id                                   | host                                                     | admin_state_up | alive | ha_state |
+--------------------------------------+----------------------------------------------------------+----------------+-------+----------+
| dd115bf3-1613-4b52-8e71-1622b0bcd7d8 | networker-4348a4bf-4d80-4d16-9b43-cf0f32923b2c.novalocal | True           | :-)   | active   |
| 9707129c-fb57-4a30-b321-99e414e0be07 | networker-a29ff2fb-13d1-4f8e-b24c-a7141ffd0a44.novalocal | True           | :-)   | standby  |
+--------------------------------------+----------------------------------------------------------+----------------+-------+----------+
[root@controller ~(keystone_admin)]# 
[root@controller ~(keystone_admin)]# 
[root@controller ~(keystone_admin)]# 
[root@controller ~(keystone_admin)]# 
[root@controller ~(keystone_admin)]# rpm -qa | grep neutron
python-neutron-2015.1.0-16.el7ost.noarch
openstack-neutron-fwaas-2015.1.0-3.el7ost.noarch
openstack-neutron-2015.1.0-16.el7ost.noarch
openstack-neutron-lbaas-2015.1.0-5.el7ost.noarch
python-neutron-fwaas-2015.1.0-3.el7ost.noarch
python-neutronclient-2.4.0-2.el7ost.noarch
openstack-neutron-common-2015.1.0-16.el7ost.noarch
python-neutron-lbaas-2015.1.0-5.el7ost.noarch
openstack-neutron-ml2-2015.1.0-16.el7ost.noarch
[root@controller ~(keystone_admin)]#
Comment 12 errata-xmlrpc 2015-09-03 17:57:00 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2015:1722