Bug 2054652

Summary:	[OSP16.1][OVN][Migration] Interfaces like fg-, sg- are not being erased after migration
Product:	Red Hat OpenStack	Reporter:	Roman Safronov <rsafrono>
Component:	python-networking-ovn	Assignee:	Arnau Verdaguer <averdagu>
Status:	CLOSED ERRATA	QA Contact:	Roman Safronov <rsafrono>
Severity:	high	Docs Contact:
Priority:	high
Version:	16.1 (Train)	CC:	apevec, averdagu, egarciar, lhh, majopela, scohen
Target Milestone:	z8	Keywords:	Triaged
Target Release:	16.1 (Train on RHEL 8.2)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	python-networking-ovn-7.3.1-1.20220422083617.4e24f4c.el8ost	Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:
Clones:	2054655 (view as bug list)		Environment:
Last Closed:	2022-12-07 20:25:48 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	2054655

Description Roman Safronov 2022-02-15 12:33:49 UTC

Description of problem:
After OVS to OVN migration is completed there are still some ML2/OVS specific interfaces on overcloud nodes, e.g. compute nodes have fg- and interfaces, controller nodes have sg- interfaces. All the interfaces are DOWN.

Note: the environment before the OVN migration was using openvswitch firewall and had VMs with trunk ports.

Version-Release number of selected component (if applicable):
RHOS-16.1-RHEL-8-20220210.n.1
python3-networking-ovn-migration-tool-7.3.1-1.20220113183502.el8ost.noarch

How reproducible:
100% 

Steps to Reproduce:
1. Deploy an ML2/OVS environment with openvswitch firewall (Note: OVN migration with iptables_hybrid firewall is not supported, see BZ2022040)
2. Create router, private network, security group, Create VMs connected to the privare network. 
3. Create workload VMs with trunk ports (Note, I connected the VMs directly to the external network)
4. Migrate to ML2/OVN according to the official procedure from the OSP documentation.
5. Check that there are no ML2/OVS-specific leftovers on the overcloud nodes.

Actual results:
Interfaces like fg- and sg- exist on the overcloud nodes

Expected results:
Interfaces like fg- and sg- do not exist on the overcloud nodes

Additional info:


[heat-admin@controller-0 ~]$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:f1:3d:20 brd ff:ff:ff:ff:ff:ff
    inet 192.168.24.51/24 brd 192.168.24.255 scope global ens3
       valid_lft forever preferred_lft forever
    inet 192.168.24.8/32 brd 192.168.24.255 scope global ens3
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fef1:3d20/64 scope link 
       valid_lft forever preferred_lft forever
3: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master ovs-system state UP group default qlen 1000
    link/ether 52:54:00:02:01:1a brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe02:11a/64 scope link 
       valid_lft forever preferred_lft forever
4: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master ovs-system state UP group default qlen 1000
    link/ether 52:54:00:87:8a:1b brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe87:8a1b/64 scope link 
       valid_lft forever preferred_lft forever
5: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether b2:5d:42:99:1b:4b brd ff:ff:ff:ff:ff:ff
6: br-isolated: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 52:54:00:02:01:1a brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe02:11a/64 scope link 
       valid_lft forever preferred_lft forever
7: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 52:54:00:87:8a:1b brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.148/24 brd 10.0.0.255 scope global br-ex
       valid_lft forever preferred_lft forever
    inet 10.0.0.146/32 brd 10.0.0.255 scope global br-ex
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe87:8a1b/64 scope link 
       valid_lft forever preferred_lft forever
8: vlan30: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether d2:92:cc:12:d1:0b brd ff:ff:ff:ff:ff:ff
    inet 172.17.3.64/24 brd 172.17.3.255 scope global vlan30
       valid_lft forever preferred_lft forever
    inet6 fe80::d092:ccff:fe12:d10b/64 scope link 
       valid_lft forever preferred_lft forever
9: vlan50: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 96:32:56:d7:f5:cd brd ff:ff:ff:ff:ff:ff
    inet 172.17.2.36/24 brd 172.17.2.255 scope global vlan50
       valid_lft forever preferred_lft forever
    inet6 fe80::9432:56ff:fed7:f5cd/64 scope link 
       valid_lft forever preferred_lft forever
10: vlan40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether e6:a4:d2:fe:32:cc brd ff:ff:ff:ff:ff:ff
    inet 172.17.4.19/24 brd 172.17.4.255 scope global vlan40
       valid_lft forever preferred_lft forever
    inet6 fe80::e4a4:d2ff:fefe:32cc/64 scope link 
       valid_lft forever preferred_lft forever
11: vlan20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 16:76:a5:aa:24:ae brd ff:ff:ff:ff:ff:ff
    inet 172.17.1.85/24 brd 172.17.1.255 scope global vlan20
       valid_lft forever preferred_lft forever
    inet 172.17.1.143/32 brd 172.17.1.255 scope global vlan20
       valid_lft forever preferred_lft forever
    inet6 fe80::1476:a5ff:feaa:24ae/64 scope link 
       valid_lft forever preferred_lft forever
12: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether fe:d0:01:3d:65:4c brd ff:ff:ff:ff:ff:ff
81: genev_sys_6081: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue master ovs-system state UNKNOWN group default qlen 1000
    link/ether e2:b3:09:ee:ce:7d brd ff:ff:ff:ff:ff:ff
    inet6 fe80::e0b3:9ff:feee:ce7d/64 scope link 
       valid_lft forever preferred_lft forever
90: sg-41e97ad4-49: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 6a:a5:27:29:ed:d2 brd ff:ff:ff:ff:ff:ff
91: sg-116af0e0-df: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 7a:2b:52:8a:1e:ec brd ff:ff:ff:ff:ff:ff
92: sg-74c1540c-f7: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether f2:45:50:42:cd:43 brd ff:ff:ff:ff:ff:ff
93: fg-6352a82b-56: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 9a:8e:82:a2:a1:af brd ff:ff:ff:ff:ff:ff
94: sg-365c534c-b7: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 66:4b:c7:4f:b4:c8 brd ff:ff:ff:ff:ff:ff
95: sg-b4b16007-f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 8a:28:19:b0:9b:ea brd ff:ff:ff:ff:ff:ff
96: sg-c0eed5dc-b2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 1e:da:fe:cb:69:92 brd ff:ff:ff:ff:ff:ff
97: sg-8c093165-fe: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether ee:6f:bf:c4:69:24 brd ff:ff:ff:ff:ff:ff




[heat-admin@compute-0 ~]$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:53:11:56 brd ff:ff:ff:ff:ff:ff
    inet 192.168.24.18/24 brd 192.168.24.255 scope global ens3
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe53:1156/64 scope link 
       valid_lft forever preferred_lft forever
3: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master ovs-system state UP group default qlen 1000
    link/ether 52:54:00:06:b9:3d brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe06:b93d/64 scope link 
       valid_lft forever preferred_lft forever
4: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master ovs-system state UP group default qlen 1000
    link/ether 52:54:00:20:d7:41 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe20:d741/64 scope link 
       valid_lft forever preferred_lft forever
5: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 96:3e:97:ef:39:79 brd ff:ff:ff:ff:ff:ff
6: br-isolated: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 52:54:00:06:b9:3d brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe06:b93d/64 scope link 
       valid_lft forever preferred_lft forever
7: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 52:54:00:20:d7:41 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe20:d741/64 scope link 
       valid_lft forever preferred_lft forever
8: vlan30: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 46:79:bb:6b:77:7a brd ff:ff:ff:ff:ff:ff
    inet 172.17.3.17/24 brd 172.17.3.255 scope global vlan30
       valid_lft forever preferred_lft forever
    inet6 fe80::4479:bbff:fe6b:777a/64 scope link 
       valid_lft forever preferred_lft forever
9: vlan50: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:21:01:7d:10:2c brd ff:ff:ff:ff:ff:ff
    inet 172.17.2.17/24 brd 172.17.2.255 scope global vlan50
       valid_lft forever preferred_lft forever
    inet6 fe80::f821:1ff:fe7d:102c/64 scope link 
       valid_lft forever preferred_lft forever
10: vlan20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 5a:29:6f:89:c5:d1 brd ff:ff:ff:ff:ff:ff
    inet 172.17.1.39/24 brd 172.17.1.255 scope global vlan20
       valid_lft forever preferred_lft forever
    inet6 fe80::5829:6fff:fe89:c5d1/64 scope link 
       valid_lft forever preferred_lft forever
11: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 2a:d0:2d:f3:ae:47 brd ff:ff:ff:ff:ff:ff
30: tbr-aecb9cc6-9: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether ee:42:4b:7f:8a:4e brd ff:ff:ff:ff:ff:ff
34: tbr-2d0b07bd-e: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 16:67:f2:eb:b6:4e brd ff:ff:ff:ff:ff:ff
35: tap6abd4bdd-88: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master ovs-system state UNKNOWN group default qlen 1000
    link/ether fe:16:3e:a0:73:e9 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc16:3eff:fea0:73e9/64 scope link 
       valid_lft forever preferred_lft forever
303: genev_sys_6081: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue master ovs-system state UNKNOWN group default qlen 1000
    link/ether 0e:3d:1a:8e:af:a1 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::c3d:1aff:fe8e:afa1/64 scope link 
       valid_lft forever preferred_lft forever
51: tap39f52589-60@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master ovs-system state UP group default qlen 1000
    link/ether e2:d5:84:ae:57:ba brd ff:ff:ff:ff:ff:ff link-netns ovnmeta-39f52589-63fa-4fd0-a57f-8dc8b23864b8
    inet6 fe80::e0d5:84ff:feae:57ba/64 scope link 
       valid_lft forever preferred_lft forever
55: tapf16dff19-90@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master ovs-system state UP group default qlen 1000
    link/ether a6:08:cd:e1:22:c9 brd ff:ff:ff:ff:ff:ff link-netns ovnmeta-f16dff19-98f6-4e08-9961-87fc457c7c89
    inet6 fe80::a408:cdff:fee1:22c9/64 scope link 
       valid_lft forever preferred_lft forever
56: fg-98d5b897-a4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether e2:07:39:ef:fa:84 brd ff:ff:ff:ff:ff:ff

Comment 1 Roman Safronov 2022-02-15 12:36:25 UTC

Note: in my case the OVS environment had DVR enabled and OVN migration scenario was "DVR to DVR"

Comment 15 errata-xmlrpc 2022-12-07 20:25:48 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 16.1.9 bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:8795