Bug 2251023
| Summary: | Trunk subports do not respond to incoming ping after ovs->ovn migration | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Roman Safronov <rsafrono> |
| Component: | openstack-neutron | Assignee: | OSP Team <rhos-maint> |
| Status: | CLOSED NOTABUG | QA Contact: | Eran Kuris <ekuris> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 17.1 (Wallaby) | CC: | chrisw, jlibosva, scohen |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-11-27 15:22:35 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 2222082 | ||
I looked at the provided env and the subports have only the default security group while the other VMs have custom security group. There is no relation between those groups and the default does not allow ICMP. After creating a security group rule that allows ICMP the ping started to work. |
Description of problem: Trunk subports do not respond to incoming ping after ovs->ovn migration. Opposite direction works. connectivity tests summary looks like this after ovn migration INFO: ping 192.168.211.188 via 10.0.0.240 = passed INFO: ping 192.168.212.24 via 10.0.0.240 = passed INFO: ping 192.168.211.84 via 10.0.0.240 = passed INFO: ping 192.168.211.76 via 10.0.0.248 = failed INFO: ping 192.168.211.84 via 10.0.0.248 = passed INFO: ping 192.168.212.170 via 10.0.0.188 = failed INFO: ping 192.168.211.76 via 10.0.0.208 = failed INFO: ping 192.168.211.188 via 10.0.0.208 = passed INFO: ping 192.168.213.177 via 10.0.0.189 = passed INFO: ping 192.168.214.66 via 10.0.0.189 = passed INFO: ping 192.168.213.110 via 10.0.0.189 = passed INFO: ping 192.168.213.135 via 10.0.0.250 = failed INFO: ping 192.168.213.110 via 10.0.0.250 = passed INFO: ping 192.168.214.165 via 10.0.0.224 = failed INFO: ping 192.168.213.135 via 10.0.0.173 = failed INFO: ping 192.168.213.177 via 10.0.0.173 = passed No vm migration/reboot were performed, only all-at-once migration from ovs-to-ovn was performed See VMs and ports ip addresses below (overcloud) (myvenv) [stack@undercloud-0 roman]$ openstack server list --long -c ID -c Name -c Host -c Networks +--------------------------------------+------------------------------------------------------+------------------------------------------------------------------------------------------------------------+------------------------+ | ID | Name | Networks | Host | +--------------------------------------+------------------------------------------------------+------------------------------------------------------------------------------------------------------------+------------------------+ | 79aca57b-4246-40aa-be1d-fd5163c26cd9 | ovn-migration-trunk-server-granular-ext-pinger-zone1 | public=10.0.0.189, 2620:52:0:13b8:f816:3eff:fe82:820b | compute-2.redhat.local | | 34f25587-162e-47b1-b903-f68c9439c99a | ovn-migration-server-granular-ext-pinger-3-zone1 | ovn-migration-net-vnf1-pinger-zone1=192.168.213.177; public=10.0.0.250, 2620:52:0:13b8:f816:3eff:fe99:b16e | compute-3.redhat.local | | ebc2feae-ee4d-4f38-ae58-5cae8dae75b7 | ovn-migration-server-granular-ext-pinger-2-zone1 | ovn-migration-net-vnf2-pinger-zone1=192.168.214.66; public=10.0.0.224, 2620:52:0:13b8:f816:3eff:fe98:488c | compute-3.redhat.local | | 5c813d16-02d8-4459-b950-f1c34aaca6a9 | ovn-migration-server-granular-ext-pinger-1-zone1 | ovn-migration-net-vnf1-pinger-zone1=192.168.213.110; public=10.0.0.173, 2620:52:0:13b8:f816:3eff:fe09:1c19 | compute-2.redhat.local | | 0b28af63-3170-4d7e-9232-0ec7593ee0c6 | ovn-migration-trunk-server-granular-ext-pinger-zone0 | public=10.0.0.240, 2620:52:0:13b8:f816:3eff:feac:25d0 | compute-0.redhat.local | | 29a7e09b-32f4-4339-9598-d0770585dff9 | ovn-migration-server-granular-ext-pinger-3-zone0 | ovn-migration-net-vnf1-pinger-zone0=192.168.211.188; public=10.0.0.248, 2620:52:0:13b8:f816:3eff:fe4c:120 | compute-1.redhat.local | | 2fcf7e66-81a4-4854-8b75-4dab5fdece7d | ovn-migration-server-granular-ext-pinger-2-zone0 | ovn-migration-net-vnf2-pinger-zone0=192.168.212.24; public=10.0.0.188, 2620:52:0:13b8:f816:3eff:fe14:8477 | compute-1.redhat.local | | e84793ea-d62b-4404-a668-9af1e34991bf | ovn-migration-server-granular-ext-pinger-1-zone0 | ovn-migration-net-vnf1-pinger-zone0=192.168.211.84; public=10.0.0.208, 2620:52:0:13b8:f816:3eff:febc:9ebe | compute-0.redhat.local | | 6d38f797-0fd9-4133-9e8f-f7bec573b1ae | workload_instance_0 | workload_internal_net_0=10.0.0.176, 192.168.0.228 | compute-0.redhat.local | +--------------------------------------+------------------------------------------------------+------------------------------------------------------------------------------------------------------------+------------------------+ [stack@undercloud-0 ~]$ openstack network list +--------------------------------------+----------------------------------------------------+----------------------------------------------------------------------------+ | ID | Name | Subnets | +--------------------------------------+----------------------------------------------------+----------------------------------------------------------------------------+ | 15b2989d-a567-476e-9076-75fe68245894 | ovn-migration-net-vnf2-pinger-zone0 | 8d021e2a-e030-4a57-8105-78f94a127590 | | 20bf44d0-c075-4b12-9478-6e0b9209a0ac | ovn-migration-net-vnf1-pinger-zone1 | 51098edc-2703-4e2a-9a58-0c3a60a19aef | | 39b42d3f-ce69-4339-b357-310dfcc234fb | HA network tenant 861db2e83f854e0d852dfab20c37ab7d | 8246d451-57c7-4d21-b2f1-16aa9402a166 | | a8727d86-6593-4ed4-85ec-135d0d33e376 | public | 47d180c2-dce9-4645-85d7-3626b5ca26fc, e6206930-03ff-41b5-b044-6e82f389916b | | cb7fbbf2-2189-40a8-b030-c6535cedf872 | ovn-migration-net-vnf2-pinger-zone1 | 4296af40-ca60-4c6d-ab96-29357d1d559a | | e65acc30-135f-486c-9e21-d9acd013b1fe | workload_internal_net_0 | d3f9dde0-f4cc-4efc-94c8-1306986aa109 | | edcc9611-58e2-4fd2-ab8b-e6f8bba80dba | ovn-migration-net-vnf1-pinger-zone0 | f6d621b2-43f1-48db-9bc3-8a71be25ef8f | +--------------------------------------+----------------------------------------------------+----------------------------------------------------------------------------+ [stack@undercloud-0 ~]$ [stack@undercloud-0 ~]$ openstack subnet list +--------------------------------------+---------------------------------------------------+--------------------------------------+---------------------+ | ID | Name | Network | Subnet | +--------------------------------------+---------------------------------------------------+--------------------------------------+---------------------+ | 4296af40-ca60-4c6d-ab96-29357d1d559a | ovn-migration-subnet-vnf2-pinger-zone1 | cb7fbbf2-2189-40a8-b030-c6535cedf872 | 192.168.214.0/24 | | 47d180c2-dce9-4645-85d7-3626b5ca26fc | external_ipv6_subnet | a8727d86-6593-4ed4-85ec-135d0d33e376 | 2620:52:0:13b8::/64 | | 51098edc-2703-4e2a-9a58-0c3a60a19aef | ovn-migration-subnet-vnf1-pinger-zone1 | 20bf44d0-c075-4b12-9478-6e0b9209a0ac | 192.168.213.0/24 | | 8246d451-57c7-4d21-b2f1-16aa9402a166 | HA subnet tenant 861db2e83f854e0d852dfab20c37ab7d | 39b42d3f-ce69-4339-b357-310dfcc234fb | 169.254.192.0/18 | | 8d021e2a-e030-4a57-8105-78f94a127590 | ovn-migration-subnet-vnf2-pinger-zone0 | 15b2989d-a567-476e-9076-75fe68245894 | 192.168.212.0/24 | | d3f9dde0-f4cc-4efc-94c8-1306986aa109 | workload_internal_net_subnet_0 | e65acc30-135f-486c-9e21-d9acd013b1fe | 192.168.0.0/24 | | e6206930-03ff-41b5-b044-6e82f389916b | external_subnet | a8727d86-6593-4ed4-85ec-135d0d33e376 | 10.0.0.0/24 | | f6d621b2-43f1-48db-9bc3-8a71be25ef8f | ovn-migration-subnet-vnf1-pinger-zone0 | edcc9611-58e2-4fd2-ab8b-e6f8bba80dba | 192.168.211.0/24 | +--------------------------------------+---------------------------------------------------+--------------------------------------+---------------------+ Trunk VMs interfaces [cloud-user@ovn-migration-trunk-server-granular-ext-pinger-zone1 ~]$ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether fa:16:3e:82:82:0b brd ff:ff:ff:ff:ff:ff inet 10.0.0.189/24 brd 10.0.0.255 scope global dynamic noprefixroute eth0 valid_lft 55147sec preferred_lft 55147sec inet6 2620:52:0:13b8:f816:3eff:fe82:820b/64 scope global deprecated dynamic noprefixroute valid_lft 55145sec preferred_lft 0sec inet6 fe80::f816:3eff:fe82:820b/64 scope link noprefixroute valid_lft forever preferred_lft forever 3: eth0.103@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000 link/ether fa:16:3e:82:82:0b brd ff:ff:ff:ff:ff:ff inet 192.168.213.135/24 brd 192.168.213.255 scope global dynamic noprefixroute eth0.103 valid_lft 55148sec preferred_lft 55148sec inet6 fe80::f816:3eff:fe82:820b/64 scope link valid_lft forever preferred_lft forever 4: eth0.104@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000 link/ether fa:16:3e:82:82:0b brd ff:ff:ff:ff:ff:ff inet 192.168.214.165/24 brd 192.168.214.255 scope global dynamic noprefixroute eth0.104 valid_lft 55148sec preferred_lft 55148sec inet6 fe80::f816:3eff:fe82:820b/64 scope link valid_lft forever preferred_lft forever [cloud-user@ovn-migration-trunk-server-granular-ext-pinger-zone0 ~]$ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether fa:16:3e:ac:25:d0 brd ff:ff:ff:ff:ff:ff inet 10.0.0.240/24 brd 10.0.0.255 scope global dynamic noprefixroute eth0 valid_lft 54948sec preferred_lft 54948sec inet6 2620:52:0:13b8:f816:3eff:feac:25d0/64 scope global deprecated dynamic noprefixroute valid_lft 54945sec preferred_lft 0sec inet6 fe80::f816:3eff:feac:25d0/64 scope link noprefixroute valid_lft forever preferred_lft forever 3: eth0.101@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000 link/ether fa:16:3e:ac:25:d0 brd ff:ff:ff:ff:ff:ff inet 192.168.211.76/24 brd 192.168.211.255 scope global dynamic noprefixroute eth0.101 valid_lft 54948sec preferred_lft 54948sec inet6 fe80::f816:3eff:feac:25d0/64 scope link valid_lft forever preferred_lft forever 4: eth0.102@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000 link/ether fa:16:3e:ac:25:d0 brd ff:ff:ff:ff:ff:ff inet 192.168.212.170/24 brd 192.168.212.255 scope global dynamic noprefixroute eth0.102 valid_lft 54948sec preferred_lft 54948sec inet6 fe80::f816:3eff:feac:25d0/64 scope link valid_lft forever preferred_lft forever Version-Release number of selected component (if applicable): RHOS-17.1-RHEL-9-20231117.n.1 openstack-neutron-ovn-migration-tool-18.6.1-17.1.20231025110805.el9ost.noarch python3-neutron-18.6.1-17.1.20231025110805.el9ost.noarch ovn22.12-22.12.1-11.el9fdp.x86_64 openvswitch3.1-3.1.0-54.el9fdp.x86_64 How reproducible: always Steps to Reproduce: 1. Deploy HA environment (3 controllers + 4 compute nodes) with OVS neutron backend. In my case it was an environment with centralized (no-DVR) routing. 2. Create a workload, in my case it was the following (see also the attached server and network list) - 2 availability zones, first zone: compute nodes 0 and 1, second: compute nodes 2 and 3 - 8 VMs divided to 2 groups of 4 VMs, each group in a separate availability zone - All VMs are connected to the external network directly - 2 separate internal networks in each zone (4 networks total) - There is a single VM with a trunk port and 2 subports, in each zone - 3 other VMs in each zone have an intenral port that is connected to one of internal networks - No neutron routers involved and there are no VMs in the workload that are using floating ip addresses in order to connect to the external network 3. Migrate network backend from OVS to OVN using an official procedure https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/17.1/html/migrating_to_the_ovn_mechanism_driver/migrating-ovs-to-ovn I used no-DVR to DVR scenario. Actual results: After OVN migration subports in both trunk VMs are not accessible from other VMs. Ping from trunk VMs is working. i.e. ping from 192.168.212.170 to 192.168.212.24 works, from 192.168.212.24 to 192.168.212.170 does not. Both ports are connected to the same network. Expected results: Ping is working between ports connected to the same network, in both directions. Additional info: