Bug 1443024

Assigned to Brian for triaging.

This looks similar to https://bugzilla.redhat.com/show_bug.cgi?id=1441651 where there is a race condition setting up the FIP namespace after a restart.

Can you check the last few comments there?  I believe a fix for this would be in the next OSP10 release.

Hi Brian,

I skim through the mentioned Bug, I remember that rhos-tech thread but I was not able to relate that issue to this one because I thought scenario which Ben is talking about happened when Cu. is taking manual actions to remove the interface. In my case it happened after reboot of compute node, cu. is not altering anything manually. 

Surprisingly, it's consistently reproduced on two compute nodes but not on other two. 

anyway, if you want I can ask Cu. to apply both the patches manually or you may share the test package to verify that it's fixing the issue for customer. 


[1] https://review.openstack.org/#/c/413240/
[2] https://review.openstack.org/#/c/385617/

Vikrant - I would like to ask the customer to try both those patches since I believe it should fix the problem.

Looking at the other bug I linked the fixes in question are on the next OSP 10 release, z-3, so there would be no other backports necessary.

Brian,

Thanks for the update. I downloaded both the patch set and checked the different between them I can see only this difference: 

~~~
-                                           self.agent_conf)
+                                           self.agent_conf.send_arp_for_ha)
~~~

Both patches look same, can you please let us know which patch we really need to apply? or you want us to try both one by one. 

~~~
# diff -Nurp  95d93495.diff d40322c7.diff 
--- 95d93495.diff	2016-12-20 21:42:50.000000000 +0530
+++ d40322c7.diff	2016-12-13 15:58:08.000000000 +0530
@@ -1,4 +1,4 @@
-From 95d93495d743115808b511740899d649cab6b5ae Mon Sep 17 00:00:00 2001
+From d40322c7d4aa1dd6d595dfe415278c9f252f4da2 Mon Sep 17 00:00:00 2001
 From: Swaminathan Vasudevan <swaminathan.vasudevan>
 Date: Fri, 07 Oct 2016 10:30:40 -0700
 Subject: [PATCH] DVR: Fix race condition in creation of fip gateway
@@ -22,11 +22,10 @@ re-scheduled for the next iteration.
 
 Closes-Bug: #1631513
 Change-Id: Ia8c92cea2f8798582c39ad3450ab3b3c45a356f7
-(cherry picked from commit d40322c7d4aa1dd6d595dfe415278c9f252f4da2)
 ---
 
 diff --git a/neutron/agent/l3/dvr_fip_ns.py b/neutron/agent/l3/dvr_fip_ns.py
-index 6e6fcfc..99b5908 100644
+index 350ad94..5d4dccb 100644
 --- a/neutron/agent/l3/dvr_fip_ns.py
 +++ b/neutron/agent/l3/dvr_fip_ns.py
 @@ -12,16 +12,21 @@
@@ -123,7 +122,7 @@ index 6e6fcfc..99b5908 100644
  
          cmd = ['sysctl', '-w', 'net.ipv4.conf.%s.proxy_arp=1' % interface_name]
          ip_wrapper.netns.execute(cmd, check_exit_code=False)
-@@ -178,17 +226,6 @@
+@@ -177,17 +225,6 @@
          LOG.debug('DVR: destroy fip namespace: %s', self.name)
          super(FipNamespace, self).delete()
  
@@ -141,7 +140,7 @@ index 6e6fcfc..99b5908 100644
      def _check_for_gateway_ip_change(self, new_agent_gateway_port):
  
          def get_gateway_ips(gateway_port):
-@@ -206,22 +243,33 @@
+@@ -205,22 +242,33 @@
  
          return new_gw_ips != old_gw_ips
  
@@ -177,13 +176,13 @@ index 6e6fcfc..99b5908 100644
              ip_lib.send_ip_addr_adv_notif(ns_name,
                                            interface_name,
                                            fixed_ip['ip_address'],
-                                           self.agent_conf)
+                                           self.agent_conf.send_arp_for_ha)
  
 -        ipd = ip_lib.IPDevice(interface_name, namespace=ns_name)
          for subnet in agent_gateway_port['subnets']:
              gw_ip = subnet.get('gateway_ip')
              if gw_ip:
-@@ -234,6 +282,10 @@
+@@ -233,6 +281,10 @@
                  current_gateway = ipd.route.get_gateway()
                  if current_gateway and current_gateway.get('gateway'):
                      ipd.route.delete_gateway(current_gateway.get('gateway'))
@@ -195,10 +194,10 @@ index 6e6fcfc..99b5908 100644
      def _add_cidr_to_device(self, device, ip_cidr):
          if not device.addr.list(to=ip_cidr):
 diff --git a/neutron/agent/l3/dvr_local_router.py b/neutron/agent/l3/dvr_local_router.py
-index 08afb8b..1fa99f5 100644
+index b9f26f5..deba6ab 100644
 --- a/neutron/agent/l3/dvr_local_router.py
 +++ b/neutron/agent/l3/dvr_local_router.py
-@@ -522,10 +522,8 @@
+@@ -518,10 +518,8 @@
          if fip_agent_port:
              LOG.debug("FloatingIP agent gateway port received from the "
                  "plugin: %s", fip_agent_port)
@@ -210,7 +209,7 @@ index 08afb8b..1fa99f5 100644
                  LOG.debug("No FloatingIP agent gateway port possibly due to "
                            "late binding of the private port to the host, "
                            "requesting agent gateway port for 'network-id' :"
-@@ -540,10 +538,7 @@
+@@ -536,10 +534,7 @@
                  if 'subnets' not in fip_agent_port:
                      LOG.error(_LE('Missing subnet/agent_gateway_port'))
                  else:
@@ -443,10 +442,10 @@ index 0a1bfa0..b027fae 100644
          IPDevice().route.add_route.assert_called_once_with('20.0.1.1',
                                                             scope='link')
 diff --git a/neutron/tests/unit/agent/l3/test_dvr_local_router.py b/neutron/tests/unit/agent/l3/test_dvr_local_router.py
-index a34891a..e09fcb8 100644
+index df649c4..6b6c58e 100644
 --- a/neutron/tests/unit/agent/l3/test_dvr_local_router.py
 +++ b/neutron/tests/unit/agent/l3/test_dvr_local_router.py
-@@ -169,7 +169,7 @@
+@@ -170,7 +170,7 @@
          ri.fip_ns.subscribe.return_value = False
          ex_gw_port = {'network_id': 'fake_net_id'}
          ri.create_dvr_fip_interfaces(ex_gw_port)
~~~

You would want to try the first one, https://review.openstack.org/#/c/413240/ as that is the newton-based version.

Let me know if there is a problem.

Created DVR environment with several computes and put Vms on them.
Rebooted the compute node with VM, then when compute was up started the VM and tested connectivity to that VM.
Tested on openstack-neutron-9.4.0-2.el7ost.noarch

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2663