Bug 2224999

Summary: Trunk subports connectivity is broken after reverting ovn migration
Product: Red Hat OpenStack Reporter: Roman Safronov <rsafrono>
Component: openstack-neutronAssignee: Arnau Verdaguer <averdagu>
Status: CLOSED ERRATA QA Contact: Roman Safronov <rsafrono>
Severity: high Docs Contact:
Priority: high    
Version: 17.1 (Wallaby)CC: averdagu, bcafarel, chrisw, gbrinn, jamsmith, mariel, mburns, mlavalle, njohnston, pgrist, prgutier, scohen
Target Milestone: z3Keywords: Triaged
Target Release: 17.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-neutron-18.6.1-17.1.20231025110807.el9ost Doc Type: Known Issue
Doc Text:
If your migration to the OVN mechanism driver fails, the optional OVN migration revert feature helps you restore your environment to the OVS mechanism driver. + If your original ML2/OVS deployment has instances that use trunk ports, and you plan to provide for a potential migration revert, communicate that information in the proactive support ticket that you file before beginning the migration. + After the migration revert, trunk port connectivity will not be restored as expected. Your Red Hat support representative can take steps to restore trunk port connectivity.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2024-05-22 20:39:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Roman Safronov 2023-07-24 08:29:45 UTC 
Description of problem:
Trunk subports connectivity is broken after reverting ovn migration

Version-Release number of selected component (if applicable):
RHOS-17.1-RHEL-9-20230719.n.1
openstack-neutron-ovn-migration-tool-18.6.1-1.20230518200971.el9ost.noarch
python3-neutron-18.6.1-1.20230518200971.el9ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy HA environment with neutron using ovs backend
2. Create 2 VMs with trunk ports connected to external network
3. Create an internal network, add trunk subports connected to this network to both VMs. Make sure VMs are able to communicate via the subports and the network.
4. Create a backup of control plane nodes using ReaR tool in order to be able to revert.
5. Migrate neutron to OVN and make sure VMs are able to communicate via the subports and the network.
5. Revert ovn migration, i.e. restore controller nodes from the backup, run revert.yml playbook and then overcloud update using the original overcloud_deploy.sh script (used for deploying initial ovs environment)

Actual results:
VMs with trunk ports are not able to communicate via trunk subports

Expected results:
VMs with trunk ports are able to communicate via trunk subports

Additional info:
(overcloud) [stack@undercloud-0 ~]$ openstack server list
+--------------------------------------+------------------------------------------+--------+-------------------------------------------------------------------------------------------+-----------------------------+----------------------+
| ID                                   | Name                                     | Status | Networks                                                                                  | Image                       | Flavor               |
+--------------------------------------+------------------------------------------+--------+-------------------------------------------------------------------------------------------+-----------------------------+----------------------+
| a053618f-6499-41bd-b543-e71c0ded13ed | ovn-migration-server-trunk-ext-pinger-2  | ACTIVE | public=10.0.0.158, 2620:52:0:13b8::1000:81                                                | rhel-8-ovn-migration-pinger | ovn-migration-pinger |
| 094ed699-6e36-4fd6-ac26-41edd525da7c | ovn-migration-server-trunk-ext-pinger-1  | ACTIVE | public=10.0.0.233, 2620:52:0:13b8::1000:1c                                                | rhel-8-ovn-migration-pinger | ovn-migration-pinger |
| d9a4bb3c-c239-4188-b968-4488b26a6bb7 | ovn-migration-server-normal-int-pinger-2 | ACTIVE | ovn-migration-net-pinger=10.0.0.213, 192.168.168.151, 2001:db8:cafe:1:f816:3eff:fe86:d167 | rhel-8-ovn-migration-pinger | ovn-migration-pinger |
| c7e58823-100c-43d4-bff8-68d9275c07ea | ovn-migration-server-normal-int-pinger-1 | ACTIVE | ovn-migration-net-pinger=10.0.0.231, 192.168.168.193, 2001:db8:cafe:1:f816:3eff:fe68:4fc3 | rhel-8-ovn-migration-pinger | ovn-migration-pinger |
| cd7cc79e-2be6-479a-93a1-2f69f0ef22b6 | workload_instance_0                      | ACTIVE | workload_internal_net_0=10.0.0.205, 192.168.0.98                                          | workload_image_0            | workload_flavor_0    |
+--------------------------------------+------------------------------------------+--------+-------------------------------------------------------------------------------------------+-----------------------------+----------------------+
(overcloud) [stack@undercloud-0 ~]$ 
(overcloud) [stack@undercloud-0 ~]$ 
(overcloud) [stack@undercloud-0 ~]$ ssh -i ~/.ssh/id_extra_keys cloud-user.0.158
Warning: Permanently added '10.0.0.158' (ED25519) to the list of known hosts.
Activate the web console with: systemctl enable --now cockpit.socket

This system is not registered to Red Hat Insights. See https://cloud.redhat.com/
To register this system, run: insights-client --register

Last login: Mon Jul 24 04:25:32 2023 from 10.0.0.19
[cloud-user@ovn-migration-server-trunk-ext-pinger-2 ~]$ ll
total 340
-rw-rw-r--. 1 cloud-user cloud-user  24972 Jul 21 11:43 ping_192.168.200.196_output
-rw-rw-r--. 1 cloud-user cloud-user 315564 Jul 21 10:35 ping_192.168.200.196_output.bak
[cloud-user@ovn-migration-server-trunk-ext-pinger-2 ~]$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether fa:16:3e:b8:40:8f brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.158/24 brd 10.0.0.255 scope global dynamic noprefixroute eth0
       valid_lft 51684sec preferred_lft 51684sec
    inet6 2620:52:0:13b8::1000:81/128 scope global dynamic noprefixroute 
       valid_lft 61027sec preferred_lft 61027sec
    inet6 fe80::f816:3eff:feb8:408f/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: eth0.1001@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1300 qdisc noqueue state UP group default qlen 1000
    link/ether fa:16:3e:b8:40:8f brd ff:ff:ff:ff:ff:ff
    inet 192.168.200.84/24 brd 192.168.200.255 scope global noprefixroute eth0.1001
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:feb8:408f/64 scope link 
       valid_lft forever preferred_lft forever
[cloud-user@ovn-migration-server-trunk-ext-pinger-2 ~]$ ping 192.168.200.196
PING 192.168.200.196 (192.168.200.196) 56(84) bytes of data.
From 192.168.200.84 icmp_seq=1 Destination Host Unreachable
From 192.168.200.84 icmp_seq=2 Destination Host Unreachable
From 192.168.200.84 icmp_seq=3 Destination Host Unreachable
^C
--- 192.168.200.196 ping statistics ---
5 packets transmitted, 0 received, +3 errors, 100% packet loss, time 110ms
pipe 4
[cloud-user@ovn-migration-server-trunk-ext-pinger-2 ~]$ exit
logout
Connection to 10.0.0.158 closed.
(overcloud) [stack@undercloud-0 ~]$ ssh -i ~/.ssh/id_extra_keys cloud-user.0.233
Warning: Permanently added '10.0.0.233' (ED25519) to the list of known hosts.
Activate the web console with: systemctl enable --now cockpit.socket

This system is not registered to Red Hat Insights. See https://cloud.redhat.com/
To register this system, run: insights-client --register

Last login: Mon Jul 24 04:25:14 2023 from 10.0.0.19
[cloud-user@ovn-migration-server-trunk-ext-pinger-1 ~]$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether fa:16:3e:df:ee:3b brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.233/24 brd 10.0.0.255 scope global dynamic noprefixroute eth0
       valid_lft 84972sec preferred_lft 84972sec
    inet6 2620:52:0:13b8::1000:1c/128 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fedf:ee3b/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: eth0.1001@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1300 qdisc noqueue state UP group default qlen 1000
    link/ether fa:16:3e:df:ee:3b brd ff:ff:ff:ff:ff:ff
    inet 192.168.200.196/24 brd 192.168.200.255 scope global noprefixroute eth0.1001
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fedf:ee3b/64 scope link 
       valid_lft forever preferred_lft forever
[cloud-user@ovn-migration-server-trunk-ext-pinger-1 ~]$ ll
total 336
-rw-rw-r--. 1 cloud-user cloud-user  25498 Jul 21 11:43 ping_192.168.200.84_output
-rw-rw-r--. 1 cloud-user cloud-user 311995 Jul 21 10:35 ping_192.168.200.84_output.bak
[cloud-user@ovn-migration-server-trunk-ext-pinger-1 ~]$ ping 192.168.200.84
PING 192.168.200.84 (192.168.200.84) 56(84) bytes of data.
From 192.168.200.196 icmp_seq=1 Destination Host Unreachable
From 192.168.200.196 icmp_seq=2 Destination Host Unreachable
From 192.168.200.196 icmp_seq=3 Destination Host Unreachable
^C
--- 192.168.200.84 ping statistics ---
4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 99ms
Comment 26 Roman Safronov 2024-03-28 18:53:25 UTC 
Verified on RHOS-17.1-RHEL-9-20240320.n.1 with openstack-neutron-ovn-migration-tool-18.6.1-17.1.20231025110808.el9ost.noarch
Verified that after migration to ml2/ovn and revert back to ml2/ovs the exisitng VMs are able to communicate via trunk port subports.
Comment 33 errata-xmlrpc 2024-05-22 20:39:25 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 17.1.3 bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2024:2741