Bug 1663265

Description of problem:
No ping between 2 instances of 2 different tenants via floating IPs after rebooting controllers.

Version-Release number of selected component (if applicable):
Red Hat OpenStack Platform release 14.0.0 RC (Rocky)
core_puddle=2018-12-20.4

openvswitch2.10-ovn-common-2.10.0-28.el7fdp.1.x86_64
openvswitch2.10-ovn-central-2.10.0-28.el7fdp.1.x86_64
rhosp-openvswitch-ovn-common-2.10-0.1.el7ost.noarch
rhosp-openvswitch-ovn-central-2.10-0.1.el7ost.noarch


How reproducible:
Ocassionally. I tried to reproduce once again without success.

Steps to Reproduce:

1. Prepare OSP14 HA setup with 3 controller and 2 compute nodes. Enable OVN and DVR. 

2.Set 'allow_overlapping_ips=True' in /etc/neutron/neutron.conf on all controller nodes (inside neutron_api containers) and restart the containers.

3.Create 2 tenants. For each tenant create a network, a subnet (in my case it was overlapping subnet, i.e. 192.168.3.0/24 for both), router between the tenant internal network and public network. Run an instance in each tenant and create floating IPs for them. Allow ssh and icmp for used security group.

4. From both of the instances try to ping floating IP address of the other instance.
-->Ping echo replies received - OK

4. Restart all controller nodes simultaneously. Wait until all controllers are up and running.

5. Shutdown master controller.[Note: I used 'virsh shutdown' command]. Then start it back.

6. From both of the instances try to ping floating IP address of the other instance.


Actual results:
No ping replies from other instance floating IP - BUG 

Expected results:
There are ping replies from floating IP of an instance on other compute node. 

Additional info:
Numan already investigated the issue. After destroying some mac_bindings in Southbound DB ping between floating IPs works again.

The issue was found during executing scenario from test: RHELOSP-26294 - controller - recovery
from OVN Neutron HA test plan:
https://polarion.engineering.redhat.com/polarion/#/project/RHELOpenStackPlatform/wiki/OVN/OVN%20Neutron%20-%20HA