Bug 2054655

Summary: [OSP16.2][OVN][Migration] Interfaces like fg-, sg- are not being erased after migration
Product: Red Hat OpenStack Reporter: Roman Safronov <rsafrono>
Component: python-networking-ovnAssignee: Arnau Verdaguer <averdagu>
Status: CLOSED ERRATA QA Contact: Roman Safronov <rsafrono>
Severity: high Docs Contact:
Priority: high    
Version: 16.2 (Train)CC: apevec, egarciar, ekuris, lhh, majopela, rhos-maint, scohen
Target Milestone: z3Keywords: Triaged
Target Release: 16.2 (Train on RHEL 8.4)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: python-networking-ovn-7.4.2-2.20220331191610.788d8ef.el8ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 2054652 Environment:
Last Closed: 2022-06-22 16:04:16 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2054652    
Bug Blocks:    

Description Roman Safronov 2022-02-15 12:45:00 UTC 
+++ This bug was initially created as a clone of Bug #2054652 +++

Description of problem:
After OVS to OVN migration is completed there are still some ML2/OVS specific interfaces on overcloud nodes, e.g. compute nodes have fg- interfaces, controller nodes have sg- interfaces. All the interfaces are DOWN.

Note: the environment before the OVN migration was using openvswitch firewall, had VMs with trunk ports and DVR enabled.

Version-Release number of selected component (if applicable):
RHOS-16.2-RHEL-8-20220210.n.1
python3-networking-ovn-migration-tool-7.4.2-2.20220113214852.a2eba10.el8ost.noarch

How reproducible:
100% 

Steps to Reproduce:
1. Deploy an ML2/OVS environment with openvswitch firewall (Note: OVN migration with iptables_hybrid firewall is not supported, see BZ2021987)
2. Create router, private network, security group, Create VMs connected to the privare network. 
3. Create workload VMs with trunk ports (Note, I connected the VMs directly to the external network)
4. Migrate to ML2/OVN according to the official procedure from the OSP documentation.
5. Check that there are no ML2/OVS-specific leftovers on the overcloud nodes.

Actual results:
Interfaces like fg-, sg- exist on the overcloud nodes

Expected results:
Interfaces like fg-, sg- do not exist on the overcloud nodes

Additional info:


[heat-admin@controller-0 ~]$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:f1:3d:20 brd ff:ff:ff:ff:ff:ff
    inet 192.168.24.51/24 brd 192.168.24.255 scope global ens3
       valid_lft forever preferred_lft forever
    inet 192.168.24.8/32 brd 192.168.24.255 scope global ens3
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fef1:3d20/64 scope link 
       valid_lft forever preferred_lft forever
3: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master ovs-system state UP group default qlen 1000
    link/ether 52:54:00:02:01:1a brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe02:11a/64 scope link 
       valid_lft forever preferred_lft forever
4: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master ovs-system state UP group default qlen 1000
    link/ether 52:54:00:87:8a:1b brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe87:8a1b/64 scope link 
       valid_lft forever preferred_lft forever
5: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether b2:5d:42:99:1b:4b brd ff:ff:ff:ff:ff:ff
6: br-isolated: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 52:54:00:02:01:1a brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe02:11a/64 scope link 
       valid_lft forever preferred_lft forever
7: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 52:54:00:87:8a:1b brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.148/24 brd 10.0.0.255 scope global br-ex
       valid_lft forever preferred_lft forever
    inet 10.0.0.146/32 brd 10.0.0.255 scope global br-ex
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe87:8a1b/64 scope link 
       valid_lft forever preferred_lft forever
8: vlan30: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether d2:92:cc:12:d1:0b brd ff:ff:ff:ff:ff:ff
    inet 172.17.3.64/24 brd 172.17.3.255 scope global vlan30
       valid_lft forever preferred_lft forever
    inet6 fe80::d092:ccff:fe12:d10b/64 scope link 
       valid_lft forever preferred_lft forever
9: vlan50: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 96:32:56:d7:f5:cd brd ff:ff:ff:ff:ff:ff
    inet 172.17.2.36/24 brd 172.17.2.255 scope global vlan50
       valid_lft forever preferred_lft forever
    inet6 fe80::9432:56ff:fed7:f5cd/64 scope link 
       valid_lft forever preferred_lft forever
10: vlan40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether e6:a4:d2:fe:32:cc brd ff:ff:ff:ff:ff:ff
    inet 172.17.4.19/24 brd 172.17.4.255 scope global vlan40
       valid_lft forever preferred_lft forever
    inet6 fe80::e4a4:d2ff:fefe:32cc/64 scope link 
       valid_lft forever preferred_lft forever
11: vlan20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 16:76:a5:aa:24:ae brd ff:ff:ff:ff:ff:ff
    inet 172.17.1.85/24 brd 172.17.1.255 scope global vlan20
       valid_lft forever preferred_lft forever
    inet 172.17.1.143/32 brd 172.17.1.255 scope global vlan20
       valid_lft forever preferred_lft forever
    inet6 fe80::1476:a5ff:feaa:24ae/64 scope link 
       valid_lft forever preferred_lft forever
12: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether fe:d0:01:3d:65:4c brd ff:ff:ff:ff:ff:ff
81: genev_sys_6081: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue master ovs-system state UNKNOWN group default qlen 1000
    link/ether e2:b3:09:ee:ce:7d brd ff:ff:ff:ff:ff:ff
    inet6 fe80::e0b3:9ff:feee:ce7d/64 scope link 
       valid_lft forever preferred_lft forever
90: sg-41e97ad4-49: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 6a:a5:27:29:ed:d2 brd ff:ff:ff:ff:ff:ff
91: sg-116af0e0-df: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 7a:2b:52:8a:1e:ec brd ff:ff:ff:ff:ff:ff
92: sg-74c1540c-f7: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether f2:45:50:42:cd:43 brd ff:ff:ff:ff:ff:ff
93: fg-6352a82b-56: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 9a:8e:82:a2:a1:af brd ff:ff:ff:ff:ff:ff
94: sg-365c534c-b7: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 66:4b:c7:4f:b4:c8 brd ff:ff:ff:ff:ff:ff
95: sg-b4b16007-f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 8a:28:19:b0:9b:ea brd ff:ff:ff:ff:ff:ff
96: sg-c0eed5dc-b2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 1e:da:fe:cb:69:92 brd ff:ff:ff:ff:ff:ff
97: sg-8c093165-fe: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether ee:6f:bf:c4:69:24 brd ff:ff:ff:ff:ff:ff




[heat-admin@compute-0 ~]$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:53:11:56 brd ff:ff:ff:ff:ff:ff
    inet 192.168.24.18/24 brd 192.168.24.255 scope global ens3
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe53:1156/64 scope link 
       valid_lft forever preferred_lft forever
3: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master ovs-system state UP group default qlen 1000
    link/ether 52:54:00:06:b9:3d brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe06:b93d/64 scope link 
       valid_lft forever preferred_lft forever
4: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master ovs-system state UP group default qlen 1000
    link/ether 52:54:00:20:d7:41 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe20:d741/64 scope link 
       valid_lft forever preferred_lft forever
5: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 96:3e:97:ef:39:79 brd ff:ff:ff:ff:ff:ff
6: br-isolated: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 52:54:00:06:b9:3d brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe06:b93d/64 scope link 
       valid_lft forever preferred_lft forever
7: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 52:54:00:20:d7:41 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe20:d741/64 scope link 
       valid_lft forever preferred_lft forever
8: vlan30: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 46:79:bb:6b:77:7a brd ff:ff:ff:ff:ff:ff
    inet 172.17.3.17/24 brd 172.17.3.255 scope global vlan30
       valid_lft forever preferred_lft forever
    inet6 fe80::4479:bbff:fe6b:777a/64 scope link 
       valid_lft forever preferred_lft forever
9: vlan50: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:21:01:7d:10:2c brd ff:ff:ff:ff:ff:ff
    inet 172.17.2.17/24 brd 172.17.2.255 scope global vlan50
       valid_lft forever preferred_lft forever
    inet6 fe80::f821:1ff:fe7d:102c/64 scope link 
       valid_lft forever preferred_lft forever
10: vlan20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 5a:29:6f:89:c5:d1 brd ff:ff:ff:ff:ff:ff
    inet 172.17.1.39/24 brd 172.17.1.255 scope global vlan20
       valid_lft forever preferred_lft forever
    inet6 fe80::5829:6fff:fe89:c5d1/64 scope link 
       valid_lft forever preferred_lft forever
11: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 2a:d0:2d:f3:ae:47 brd ff:ff:ff:ff:ff:ff
30: tbr-aecb9cc6-9: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether ee:42:4b:7f:8a:4e brd ff:ff:ff:ff:ff:ff
34: tbr-2d0b07bd-e: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 16:67:f2:eb:b6:4e brd ff:ff:ff:ff:ff:ff
35: tap6abd4bdd-88: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master ovs-system state UNKNOWN group default qlen 1000
    link/ether fe:16:3e:a0:73:e9 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc16:3eff:fea0:73e9/64 scope link 
       valid_lft forever preferred_lft forever
303: genev_sys_6081: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue master ovs-system state UNKNOWN group default qlen 1000
    link/ether 0e:3d:1a:8e:af:a1 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::c3d:1aff:fe8e:afa1/64 scope link 
       valid_lft forever preferred_lft forever
51: tap39f52589-60@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master ovs-system state UP group default qlen 1000
    link/ether e2:d5:84:ae:57:ba brd ff:ff:ff:ff:ff:ff link-netns ovnmeta-39f52589-63fa-4fd0-a57f-8dc8b23864b8
    inet6 fe80::e0d5:84ff:feae:57ba/64 scope link 
       valid_lft forever preferred_lft forever
55: tapf16dff19-90@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master ovs-system state UP group default qlen 1000
    link/ether a6:08:cd:e1:22:c9 brd ff:ff:ff:ff:ff:ff link-netns ovnmeta-f16dff19-98f6-4e08-9961-87fc457c7c89
    inet6 fe80::a408:cdff:fee1:22c9/64 scope link 
       valid_lft forever preferred_lft forever
56: fg-98d5b897-a4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether e2:07:39:ef:fa:84 brd ff:ff:ff:ff:ff:ff

--- Additional comment from Roman Safronov on 2022-02-15 12:36:25 UTC ---

Note: the output above if from OVS environment with DVR enabled and OVN migration scenario was "DVR to DVR".  On non-DVR environment (with VMs with trunk ports) I see only tbr- interfaces and no fg- and sg-.
Comment 6 Roman Safronov 2022-05-04 12:07:22 UTC 
Verified on RHOS-16.2-RHEL-8-20220427.n.3 with python3-networking-ovn-migration-tool-7.4.2-2.20220409154847.el8ost.noarch
Performed migration to OVN from an OVS environment with DVR enabled (DVR to DVR scenario) and workload VMs running. The migration task "Cleanup neutron router and dhcp interfaces" completed successfully.   After the migration there are no fg- and sg- interfaces on overcloud nodes.
Comment 12 errata-xmlrpc 2022-06-22 16:04:16 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 16.2.3 (Train)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:4793