Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2186511

Summary:	ip nei entries not cleaned up when a router is deleted
Product:	Red Hat OpenStack	Reporter:	Eduardo Olivares <eolivare>
Component:	ovn-bgp-agent	Assignee:	Luis Tomas Bolivar <ltomasbo>
Status:	CLOSED ERRATA	QA Contact:	Eduardo Olivares <eolivare>
Severity:	high	Docs Contact:
Priority:	medium
Version:	17.1 (Wallaby)	CC:	dalvarez, lmartins, ltomasbo
Target Milestone:	ga	Keywords:	Triaged
Target Release:	17.1
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	ovn-bgp-agent-0.3.1-1.20230422171003.2553998.el9ost	Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2023-08-16 01:14:35 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Eduardo Olivares 2023-04-13 15:04:03 UTC

Description of problem:
When a router is connected to an external network, the IP from its external GW IP is exposed. The ovn-bgp-agent performs the following actions:
- adds that IP to the bgp-nic dummy interface on the controller/networker node
- adds an entry to the ip rule table to redirect traffic to that IP via the bridge corresponding to the external network
- adds an entry to the ip nei table (ARP) with information from the bridge and the MAC address

When the router is deleted or disconnected from the external network, the opposite operations need to be performed.
That works fine for flat external networks, but not for vlan external networks. The created ip nei entry is never removed. The following exception is shown:
2023-04-13T14:46:13.815468162+00:00 stdout F 2023-04-13 14:46:13.809 2778 ERROR ovsdbapp.event [-] Unexpected exception in notify_loop: pr2modules.netlink.exceptions.NetlinkError: (2, 'No such file or directory')                                                                                                         
2023-04-13T14:46:13.815468162+00:00 stdout F 2023-04-13 14:46:13.809 2778 ERROR ovsdbapp.event Traceback (most recent call last):
2023-04-13T14:46:13.815468162+00:00 stdout F 2023-04-13 14:46:13.809 2778 ERROR ovsdbapp.event   File "/usr/lib/python3.9/site-packages/ovsdbapp/event.py", line 159, in notify_loop                                                                                                                                         
2023-04-13T14:46:13.815468162+00:00 stdout F 2023-04-13 14:46:13.809 2778 ERROR ovsdbapp.event     match.run(event, row, updates)
2023-04-13T14:46:13.815468162+00:00 stdout F 2023-04-13 14:46:13.809 2778 ERROR ovsdbapp.event   File "/usr/lib/python3.9/site-packages/ovn_bgp_agent/drivers/openstack/watchers/bgp_watcher.py", line 77, in run                                                                                                            
2023-04-13T14:46:13.815468162+00:00 stdout F 2023-04-13 14:46:13.809 2778 ERROR ovsdbapp.event     self.agent.withdraw_ip(ips, row)
2023-04-13T14:46:13.815468162+00:00 stdout F 2023-04-13 14:46:13.809 2778 ERROR ovsdbapp.event   File "/usr/lib/python3.9/site-packages/oslo_concurrency/lockutils.py", line 360, in inner                                                                                                                                   
2023-04-13T14:46:13.815468162+00:00 stdout F 2023-04-13 14:46:13.809 2778 ERROR ovsdbapp.event     return f(*args, **kwargs)
2023-04-13T14:46:13.815468162+00:00 stdout F 2023-04-13 14:46:13.809 2778 ERROR ovsdbapp.event   File "/usr/lib/python3.9/site-packages/ovn_bgp_agent/drivers/openstack/ovn_bgp_driver.py", line 751, in withdraw_ip                                                                                                         
2023-04-13T14:46:13.815468162+00:00 stdout F 2023-04-13 14:46:13.809 2778 ERROR ovsdbapp.event     self._withdraw_cr_lrp_port(ips, mac, bridge_device, bridge_vlan,                                                                                                                                                          
2023-04-13T14:46:13.815468162+00:00 stdout F 2023-04-13 14:46:13.809 2778 ERROR ovsdbapp.event   File "/usr/lib/python3.9/site-packages/ovn_bgp_agent/drivers/openstack/ovn_bgp_driver.py", line 916, in _withdraw_cr_lrp_port                                                                                               
2023-04-13T14:46:13.815468162+00:00 stdout F 2023-04-13 14:46:13.809 2778 ERROR ovsdbapp.event     if not self._withdraw_provider_port(
2023-04-13T14:46:13.815468162+00:00 stdout F 2023-04-13 14:46:13.809 2778 ERROR ovsdbapp.event   File "/usr/lib/python3.9/site-packages/ovn_bgp_agent/drivers/openstack/ovn_bgp_driver.py", line 399, in _withdraw_provider_port                                                                                             
2023-04-13T14:46:13.815468162+00:00 stdout F 2023-04-13 14:46:13.809 2778 ERROR ovsdbapp.event     return wire_utils.unwire_provider_port(
2023-04-13T14:46:13.815468162+00:00 stdout F 2023-04-13 14:46:13.809 2778 ERROR ovsdbapp.event   File "/usr/lib/python3.9/site-packages/ovn_bgp_agent/drivers/openstack/utils/wire.py", line 66, in unwire_provider_port                                                                                                     
2023-04-13T14:46:13.815468162+00:00 stdout F 2023-04-13 14:46:13.809 2778 ERROR ovsdbapp.event     linux_net.del_ip_rule(cr_lrp_ip, routing_table, bridge_device,                                                                                                                                                            
2023-04-13T14:46:13.815468162+00:00 stdout F 2023-04-13 14:46:13.809 2778 ERROR ovsdbapp.event   File "/usr/lib/python3.9/site-packages/ovn_bgp_agent/utils/linux_net.py", line 545, in del_ip_rule                                                                                                                          
2023-04-13T14:46:13.815468162+00:00 stdout F 2023-04-13 14:46:13.809 2778 ERROR ovsdbapp.event     del_ip_nei(ip, lladdr, dev)
2023-04-13T14:46:13.815468162+00:00 stdout F 2023-04-13 14:46:13.809 2778 ERROR ovsdbapp.event   File "/usr/lib/python3.9/site-packages/ovn_bgp_agent/utils/linux_net.py", line 557, in del_ip_nei                                                                                                                           
2023-04-13T14:46:13.815468162+00:00 stdout F 2023-04-13 14:46:13.809 2778 ERROR ovsdbapp.event     ovn_bgp_agent.privileged.linux_net.del_ip_nei(ip, lladdr, dev)                                                                                                                                                            
2023-04-13T14:46:13.815468162+00:00 stdout F 2023-04-13 14:46:13.809 2778 ERROR ovsdbapp.event   File "/usr/lib/python3.9/site-packages/oslo_privsep/priv_context.py", line 247, in _wrap                                                                                                                                    
2023-04-13T14:46:13.815468162+00:00 stdout F 2023-04-13 14:46:13.809 2778 ERROR ovsdbapp.event     return self.channel.remote_call(name, args, kwargs)
2023-04-13T14:46:13.815468162+00:00 stdout F 2023-04-13 14:46:13.809 2778 ERROR ovsdbapp.event   File "/usr/lib/python3.9/site-packages/oslo_privsep/daemon.py", line 224, in remote_call                                                                                                                                    
2023-04-13T14:46:13.815468162+00:00 stdout F 2023-04-13 14:46:13.809 2778 ERROR ovsdbapp.event     raise exc_type(*result[2])
2023-04-13T14:46:13.815468162+00:00 stdout F 2023-04-13 14:46:13.809 2778 ERROR ovsdbapp.event pr2modules.netlink.exceptions.NetlinkError: (2, 'No such file or directory')                                                                                                                                                  


And this entry is not removed:
$ ip nei show 172.24.200.82
172.24.200.82 dev br-vlan.419 lladdr fa:16:3e:b6:1c:c8 PERMANENT


Version-Release number of selected component (if applicable):
RHOS-17.1-RHEL-9-20230404.n.1

How reproducible:
100%

Steps to Reproduce:
1. create a router
2. connect it to a vlan external network
3. disconnect it
4. check the ip nei entry has not been removed from the controller/networker node (or check the error message previously mentioned)

Actual results:
the entry is not deleted from the ip nei table

Expected results:
the entry should be deleted from the ip nei table

Comment 6 Eduardo Olivares 2023-04-28 10:41:26 UTC

Verified on RHOS-17.1-RHEL-9-20230426.n.1

External vlan network and subnet created
$ openstack network create public-vlan --external --provider-physical-network provider2 --provider-network-type vlan
$ openstack subnet create --network public-vlan --subnet-range 172.24.200.0/24 subnet-provider-vlan

Router created and connected to the external vlan network
$ openstack router create r
$ openstack router set --external-gateway public-vlan r

Obtain the router external GW IP:
$ openstack router show r | grep external_gateway_info
| external_gateway_info   | {"network_id": "d5a83758-7e1e-4da7-b4d7-41582c56b3b4", "external_fixed_ips": [{"subnet_id": "7a78acfe-8f64-4711-8e7f-b2599ec06f01", "ip_address": "172.24.200.140"}], "enable_snat": true} |                     


The router external GW IP is exposed from a controller node:
[root@ctrl-1-0 ~]# ip a s bgp-nic | grep 172.24.200.140                                                                                                                                                                                      
    inet 172.24.200.140/32 scope global bgp-nic                                                                                                                                                                                              
[root@ctrl-1-0 ~]# ip nei show 172.24.200.140                                                                                                                                                                                                
172.24.200.140 dev br-vlan.870 lladdr fa:16:3e:6f:58:1c PERMANENT                                                                                                                                                                            

Router deleted:
$ openstack router delete r

Checked the external IP 172.24.200.140 is not exposed from the controller anymore.
There are no exceptions raised by the ovn-bgp-agent either.

Repeated the test disconnecting the external network from router instead of deleting the router:
$ openstack router unset --external-gateway r

Comment 15 errata-xmlrpc 2023-08-16 01:14:35 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 17.1 (Wallaby)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2023:4577