Bug 2188326

Summary:	Route to FIP exposed/withdrawn by the sync process
Product:	Red Hat OpenStack	Reporter:	Eduardo Olivares <eolivare>
Component:	ovn-bgp-agent	Assignee:	Luis Tomas Bolivar <ltomasbo>
Status:	CLOSED ERRATA	QA Contact:	Eduardo Olivares <eolivare>
Severity:	high	Docs Contact:
Priority:	urgent
Version:	17.1 (Wallaby)	CC:	dalvarez, lmartins, ltomasbo
Target Milestone:	ga	Keywords:	Triaged
Target Release:	17.1
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	ovn-bgp-agent-0.3.1-1.20230324161202.ddb740e.el9ost	Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2023-08-16 01:14:48 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Eduardo Olivares 2023-04-20 13:35:54 UTC

Description of problem:
When a FIP is attached to a VM's port, the ovn bgp agent should expose the route to its IP immediately. However, this was not working like that, but the sync process was exposing the FIP some time later (depending on the sync interval configured, 2 minutes by default).

The same happens when the FIP is deleted (or dettached from the VM's port).

Apparently, this issue was introduced with the following patch:
https://opendev.org/openstack/ovn-bgp-agent/commit/d9c7de64650a8a0c31172a1fd50200ae41b6281f


Version-Release number of selected component (if applicable):
RHOS-17.1-RHEL-9-20230419.n.1

How reproducible:
100%

Steps to Reproduce:
1. create a VM connected to a tenant network and add a FIP to it
2. check the routes on the extvm or undercloud or simply check the IPs added to the dummy interface bgp-nic on the compute where the VM is running


Actual results:
The IP is exposed/withdrawn some time later, it may take up to 2 mins (default sync interval).


Expected results:
The IP should be exposed/withdrawn immediately when a FIP is attached/detached to/from a VM's port.

Comment 7 Eduardo Olivares 2023-04-28 10:56:07 UTC

Verified on RHOS-17.1-RHEL-9-20230426.n.1.

1) Add a FIP to a VM:
$ openstack server add floating ip kyrhrvei 172.24.100.211
As soon as the FIP is added, it is exposed.

The ovn-bgp-agent logs show that as soon as the FIPSetEvent is detected, the FIP is exposed:
2023-04-28T10:45:50.616174461+00:00 stdout F 2023-04-28 10:45:50.615 3139 DEBUG ovsdbapp.backend.ovs_idl.event [-] Matched UPDATE: FIPSetEvent(events=('update',), table='Port_Binding', conditions=None, old_conditions=None) to row=Port_Bin
ding(mac=['router'], port_security=[], type=patch, nat_addresses=['fa:16:3e:0b:a4:8e 172.24.100.75 is_chassis_resident("5d4a9317-0e15-4993-b1d0-0cb854112390")', 'fa:16:3e:3e:33:22 172.24.100.56 is_chassis_resident("4773e234-993c-46f8-a5f7
-2ea4ced5b4f8")', 'fa:16:3e:57:90:35 172.24.100.171 172.24.100.10 172.24.100.184 172.24.100.173 is_chassis_resident("cr-lrp-2a780f07-be18-45c5-a939-813d52fe55a6")', 'fa:16:3e:57:90:35 172.24.100.209 is_chassis_resident("cr-lrp-2a780f07-be
18-45c5-a939-813d52fe55a6")', 'fa:16:3e:7a:1d:90 172.24.100.102 is_chassis_resident("750a5f7c-9225-4984-b0ab-5ddbd1b71782")', 'fa:16:3e:9f:4b:bb 172.24.100.211 is_chassis_resident("4ec7cc4e-1cc3-4bf3-88a6-6664f42ffebd")', 'fa:16:3e:ad:1a:
2c 172.24.100.155 is_chassis_resident("49c7f905-d631-4b1e-88c2-edcd1a6aa60c")', 'fa:16:3e:ad:dd:ce 172.24.100.160 is_chassis_resident("937b3e8b-683c-467e-950b-a955d3c69043")', 'fa:16:3e:b7:a6:db 172.24.100.148 is_chassis_resident("2566c45
4-6e5b-443e-b0ac-7f7d4338a9a4")', 'fa:16:3e:bf:ec:5e 172.24.100.70 is_chassis_resident("f2ddd0d6-ecfc-43b1-8e62-082e7ecac695")', 'fa:16:3e:c7:de:38 172.24.100.85 is_chassis_resident("f09d3bef-540a-4ee4-8c7d-73ba5987e89d")', 'fa:16:3e:d4:7
9:55 172.24.100.35 is_chassis_resident("3c3a9516-7ffa-4def-9ca6-e1900e31ee81")', 'fa:16:3e:e5:95:81 172.24.100.241 is_chassis_resident("6f674c65-0dff-4521-bd0c-42c4d6b0037b")', 'fa:16:3e:f1:31:95 172.24.100.212 is_chassis_resident("2b9a67
8f-0c63-426e-9a8a-d19231ab2293")'], virtual_parent=[], up=[False], options={'peer': 'lrp-2a780f07-be18-45c5-a939-813d52fe55a6'}, parent_port=[], requested_additional_chassis=[], ha_chassis_group=[], external_ids={'neutron:cidrs': '172.24.
100.209/24 2001:db8::f816:3eff:fe57:9035/64', 'neutron:device_id': '18858d81-d07a-4115-b743-199b6b09198d', 'neutron:device_owner': 'network:router_gateway', 'neutron:network_name': 'neutron-b5b268c3-986b-4c80-999b-b0b6d68c654f', 'neutron:
port_capabilities': '', 'neutron:port_name': '', 'neutron:project_id': '', 'neutron:revision_number': '20', 'neutron:security_group_ids': '', 'neutron:subnet_pool_addr_scope4': '', 'neutron:subnet_pool_addr_scope6': '', 'neutron:vnic_type
': 'normal'}, additional_chassis=[], tag=[], additional_encap=[], encap=[], mirror_rules=[], datapath=2f8ccd39-7279-442a-9189-f8268c4d3e9f, chassis=[], tunnel_key=4, gateway_chassis=[], requested_chassis=[], logical_port=2a780f07-be18-45c
5-a939-813d52fe55a6) old=Port_Binding(nat_addresses=['fa:16:3e:0b:a4:8e 172.24.100.75 is_chassis_resident("5d4a9317-0e15-4993-b1d0-0cb854112390")', 'fa:16:3e:3e:33:22 172.24.100.56 is_chassis_resident("4773e234-993c-46f8-a5f7-2ea4ced5b4f8
")', 'fa:16:3e:57:90:35 172.24.100.171 172.24.100.10 172.24.100.184 172.24.100.173 is_chassis_resident("cr-lrp-2a780f07-be18-45c5-a939-813d52fe55a6")', 'fa:16:3e:57:90:35 172.24.100.209 is_chassis_resident("cr-lrp-2a780f07-be18-45c5-a939-
813d52fe55a6")', 'fa:16:3e:7a:1d:90 172.24.100.102 is_chassis_resident("750a5f7c-9225-4984-b0ab-5ddbd1b71782")', 'fa:16:3e:ad:1a:2c 172.24.100.155 is_chassis_resident("49c7f905-d631-4b1e-88c2-edcd1a6aa60c")', 'fa:16:3e:ad:dd:ce 172.24.100
.160 is_chassis_resident("937b3e8b-683c-467e-950b-a955d3c69043")', 'fa:16:3e:b7:a6:db 172.24.100.148 is_chassis_resident("2566c454-6e5b-443e-b0ac-7f7d4338a9a4")', 'fa:16:3e:bf:ec:5e 172.24.100.70 is_chassis_resident("f2ddd0d6-ecfc-43b1-8$
62-082e7ecac695")', 'fa:16:3e:c7:de:38 172.24.100.85 is_chassis_resident("f09d3bef-540a-4ee4-8c7d-73ba5987e89d")', 'fa:16:3e:d4:79:55 172.24.100.35 is_chassis_resident("3c3a9516-7ffa-4def-9ca6-e1900e31ee81")', 'fa:16:3e:e5:95:81 172.24.1$
0.241 is_chassis_resident("6f674c65-0dff-4521-bd0c-42c4d6b0037b")', 'fa:16:3e:f1:31:95 172.24.100.212 is_chassis_resident("2b9a678f-0c63-426e-9a8a-d19231ab2293")']) matches /usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/event.$
y:43                                                                                                                                                                                                                                         
2023-04-28T10:45:50.617666681+00:00 stdout F 2023-04-28 10:45:50.617 3139 DEBUG oslo_concurrency.lockutils [-] Lock "bgp" acquired by "ovn_bgp_agent.drivers.openstack.ovn_bgp_driver.OVNBGPDriver.expose_ip" :: waited 0.000s inner /usr/lib$
python3.9/site-packages/oslo_concurrency/lockutils.py:355                                                                                                                                                                                    
2023-04-28T10:45:50.621297855+00:00 stdout F 2023-04-28 10:45:50.621 3139 DEBUG ovn_bgp_agent.drivers.openstack.ovn_bgp_driver [-] Adding BGP route for FIP with ip ['172.24.100.211'] _expose_ip /usr/lib/python3.9/site-packages/ovn_bgp_ag$
nt/drivers/openstack/ovn_bgp_driver.py:587                                                                                                                                                                                                   


2) Remove a FIP from a VM:
$ openstack server remove floating ip kyrhrvei 172.24.100.211
As soon as the FIP is removed, its route is withdrawn.


The ovn-bgp-agent logs show that as soon as the FIPSetEvent is detected, the FIP is exposed:
2023-04-28T10:45:07.877619466+00:00 stdout F 2023-04-28 10:45:07.877 3139 DEBUG ovsdbapp.backend.ovs_idl.event [-] Matched UPDATE: FIPUnsetEvent(events=('update',), table='Port_Binding', conditions=None, old_conditions=None) to row=Port_B
inding(mac=['router'], port_security=[], type=patch, nat_addresses=['fa:16:3e:0b:a4:8e 172.24.100.75 is_chassis_resident("5d4a9317-0e15-4993-b1d0-0cb854112390")', 'fa:16:3e:3e:33:22 172.24.100.56 is_chassis_resident("4773e234-993c-46f8-a5
f7-2ea4ced5b4f8")', 'fa:16:3e:57:90:35 172.24.100.171 172.24.100.10 172.24.100.184 172.24.100.173 is_chassis_resident("cr-lrp-2a780f07-be18-45c5-a939-813d52fe55a6")', 'fa:16:3e:57:90:35 172.24.100.209 is_chassis_resident("cr-lrp-2a780f07-
be18-45c5-a939-813d52fe55a6")', 'fa:16:3e:7a:1d:90 172.24.100.102 is_chassis_resident("750a5f7c-9225-4984-b0ab-5ddbd1b71782")', 'fa:16:3e:ad:1a:2c 172.24.100.155 is_chassis_resident("49c7f905-d631-4b1e-88c2-edcd1a6aa60c")', 'fa:16:3e:ad:d
d:ce 172.24.100.160 is_chassis_resident("937b3e8b-683c-467e-950b-a955d3c69043")', 'fa:16:3e:b7:a6:db 172.24.100.148 is_chassis_resident("2566c454-6e5b-443e-b0ac-7f7d4338a9a4")', 'fa:16:3e:bf:ec:5e 172.24.100.70 is_chassis_resident("f2ddd0
d6-ecfc-43b1-8e62-082e7ecac695")', 'fa:16:3e:c7:de:38 172.24.100.85 is_chassis_resident("f09d3bef-540a-4ee4-8c7d-73ba5987e89d")', 'fa:16:3e:d4:79:55 172.24.100.35 is_chassis_resident("3c3a9516-7ffa-4def-9ca6-e1900e31ee81")', 'fa:16:3e:e5:
95:81 172.24.100.241 is_chassis_resident("6f674c65-0dff-4521-bd0c-42c4d6b0037b")', 'fa:16:3e:f1:31:95 172.24.100.212 is_chassis_resident("2b9a678f-0c63-426e-9a8a-d19231ab2293")'], virtual_parent=[], up=[False], options={'peer': 'lrp-2a780
f07-be18-45c5-a939-813d52fe55a6'}, parent_port=[], requested_additional_chassis=[], ha_chassis_group=[], external_ids={'neutron:cidrs': '172.24.100.209/24 2001:db8::f816:3eff:fe57:9035/64', 'neutron:device_id': '18858d81-d07a-4115-b743-19
9b6b09198d', 'neutron:device_owner': 'network:router_gateway', 'neutron:network_name': 'neutron-b5b268c3-986b-4c80-999b-b0b6d68c654f', 'neutron:port_capabilities': '', 'neutron:port_name': '', 'neutron:project_id': '', 'neutron:revision_n
umber': '20', 'neutron:security_group_ids': '', 'neutron:subnet_pool_addr_scope4': '', 'neutron:subnet_pool_addr_scope6': '', 'neutron:vnic_type': 'normal'}, additional_chassis=[], tag=[], additional_encap=[], encap=[], mirror_rules=[], d
atapath=2f8ccd39-7279-442a-9189-f8268c4d3e9f, chassis=[], tunnel_key=4, gateway_chassis=[], requested_chassis=[], logical_port=2a780f07-be18-45c5-a939-813d52fe55a6) old=Port_Binding(nat_addresses=['fa:16:3e:0b:a4:8e 172.24.100.75 is_chass
is_resident("5d4a9317-0e15-4993-b1d0-0cb854112390")', 'fa:16:3e:3e:33:22 172.24.100.56 is_chassis_resident("4773e234-993c-46f8-a5f7-2ea4ced5b4f8")', 'fa:16:3e:57:90:35 172.24.100.171 172.24.100.10 172.24.100.184 172.24.100.173 is_chassis_
resident("cr-lrp-2a780f07-be18-45c5-a939-813d52fe55a6")', 'fa:16:3e:57:90:35 172.24.100.209 is_chassis_resident("cr-lrp-2a780f07-be18-45c5-a939-813d52fe55a6")', 'fa:16:3e:7a:1d:90 172.24.100.102 is_chassis_resident("750a5f7c-9225-4984-b0a
b-5ddbd1b71782")', 'fa:16:3e:9f:4b:bb 172.24.100.211 is_chassis_resident("4ec7cc4e-1cc3-4bf3-88a6-6664f42ffebd")', 'fa:16:3e:ad:1a:2c 172.24.100.155 is_chassis_resident("49c7f905-d631-4b1e-88c2-edcd1a6aa60c")', 'fa:16:3e:ad:dd:ce 172.24.1
00.160 is_chassis_resident("937b3e8b-683c-467e-950b-a955d3c69043")', 'fa:16:3e:b7:a6:db 172.24.100.148 is_chassis_resident("2566c454-6e5b-443e-b0ac-7f7d4338a9a4")', 'fa:16:3e:bf:ec:5e 172.24.100.70 is_chassis_resident("f2ddd0d6-ecfc-43b1-
8e62-082e7ecac695")', 'fa:16:3e:c7:de:38 172.24.100.85 is_chassis_resident("f09d3bef-540a-4ee4-8c7d-73ba5987e89d")', 'fa:16:3e:d4:79:55 172.24.100.35 is_chassis_resident("3c3a9516-7ffa-4def-9ca6-e1900e31ee81")', 'fa:16:3e:e5:95:81 172.24.
100.241 is_chassis_resident("6f674c65-0dff-4521-bd0c-42c4d6b0037b")', 'fa:16:3e:f1:31:95 172.24.100.212 is_chassis_resident("2b9a678f-0c63-426e-9a8a-d19231ab2293")']) matches /usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/event
.py:43
2023-04-28T10:45:07.879591679+00:00 stdout F 2023-04-28 10:45:07.879 3139 DEBUG oslo_concurrency.lockutils [-] Lock "bgp" acquired by "ovn_bgp_agent.drivers.openstack.ovn_bgp_driver.OVNBGPDriver.withdraw_ip" :: waited 0.000s inner /usr/li
b/python3.9/site-packages/oslo_concurrency/lockutils.py:355
2023-04-28T10:45:07.881203016+00:00 stdout F 2023-04-28 10:45:07.881 3139 DEBUG ovn_bgp_agent.drivers.openstack.ovn_bgp_driver [-] Deleting BGP route for FIP with ip ['172.24.100.211'] withdraw_ip /usr/lib/python3.9/site-packages/ovn_bgp_
agent/drivers/openstack/ovn_bgp_driver.py:729

Comment 16 errata-xmlrpc 2023-08-16 01:14:48 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 17.1 (Wallaby)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2023:4577