Description of problem: Hi, a customer is hitting this upstream bug: https://bugs.launchpad.net/neutron/+bug/1973276 so the ports are missing the type "virtual" $ sudo podman exec -ti ovn-dbs-bundle-podman-2 ovn-nbctl --columns addresses,type,name,up find logical_switch_port 'external_ids:"neut ron:device_owner"=Octavia' addresses : ["fa:16:3e:5d:18:64 10.226.131.79"] type : virtual name : "92cf781d-6672-458b-82e6-6c3f21452e23" up : true addresses : ["fa:16:3e:bc:8e:6c 10.226.128.77"] type : "" name : "926ae169-72ce-4d75-926b-7ed6bcb44ec8" up : false addresses : ["fa:16:3e:f9:8f:4f 10.226.129.215"] type : virtual name : "9b2a44a6-8a25-4999-ac15-6fba843430a8" up : true addresses : ["fa:16:3e:8e:b5:f0 10.226.131.25"] type : "" name : "cd63d677-a8ca-4d10-9188-478883064217" up : false addresses : ["fa:16:3e:e0:68:31 10.226.129.4"] type : virtual name : "9aad38b3-5207-401c-9ace-d0f2f16aa405" up : true addresses : ["fa:16:3e:cd:5d:9c 10.226.129.111"] type : virtual name : "5c966262-123f-4485-b142-1444048ba972" up : true addresses : ["fa:16:3e:06:f1:ed 10.226.129.195"] type : "" name : "85f8f966-1e1b-4453-b7e9-79c824b09c71" up : false and it breaks ARP resolution. Version-Release number of selected component (if applicable): How reproducible: on customer environment Steps to Reproduce: 1. 2. 3. Actual results: logical_switch_port loses the type "virtual" Expected results: logical_switch_port keep the type "virtual" Additional info:
This is a test the customer ran on his environment: This is the port at the moment: $ sudo podman exec -ti ovn-dbs-bundle-podman-2 ovn-nbctl list logical_switch_port octavia-lb-5279d85e-3c98-46ca-aea5-58b898db0386 _uuid : 72dc280c-dbfa-4e3c-9f21-28b8631a7099 addresses : ["fa:16:3e:bc:8e:6c 10.226.128.77"] dhcpv4_options : 45a1446b-7bb3-4661-a7ec-5e340f992d34 dhcpv6_options : [] dynamic_addresses : [] enabled : false external_ids : {"neutron:cidrs"="10.226.128.77/17", "neutron:device_id"=lb-5279d85e-3c98-46ca-aea5-58b898db0386, "neutron:device_owner"=Octavia, "neutron:network_name"=neutron-96f906d7-fa5a-4c71-a f8c-d7a878b982d6, "neutron:port_name"=octavia-lb-5279d85e-3c98-46ca-aea5-58b898db0386, "neutron:project_id"="45defcc0fd08466e866a4ca3195507cb", "neutron:revision_number"="10", "neutron:security_group_ids "="6040034c-f072-4448-afd0-55e4dde12eff"} ha_chassis_group : [] name : "926ae169-72ce-4d75-926b-7ed6bcb44ec8" options : {mcast_flood_reports="true", requested-chassis=""} parent_name : [] port_security : ["fa:16:3e:bc:8e:6c 10.226.128.77"] tag : [] tag_request : [] type : "" up : false when running openstack port set --device-owner "" octavia-lb-5279d85e-3c98-46ca-aea5-58b898db0386 I can see it recover: ovn-nbctl list logical_switch_port octavia-lb-5279d85e-3c98-46ca-aea5-58b898db0386 _uuid : 72dc280c-dbfa-4e3c-9f21-28b8631a7099 addresses : ["fa:16:3e:bc:8e:6c 10.226.128.77"] dhcpv4_options : 45a1446b-7bb3-4661-a7ec-5e340f992d34 dhcpv6_options : [] dynamic_addresses : [] enabled : false external_ids : {"neutron:cidrs"="10.226.128.77/17", "neutron:device_id"=lb-5279d85e-3c98-46ca-aea5-58b898db0386, "neutron:device_owner"="", "neutron:network_name"=neutron-96f906d7-fa5a-4c71-af8c-d7a878b982d6, "neutron:port_name"=octavia-lb-5279d85e-3c98-46ca-aea5-58b898db0386, "neutron:project_id"="45defcc0fd08466e866a4ca3195507cb", "neutron:revision_number"="11", "neutron:security_group_ids"="6040034c-f072-4448-afd0-55e4dde12eff"} ha_chassis_group : [] name : "926ae169-72ce-4d75-926b-7ed6bcb44ec8" options : {mcast_flood_reports="true", requested-chassis="", virtual-ip="10.226.128.77", virtual-parents="16a746d7-b285-4046-ad43-8974346feaa3,c6b1aef9-f6e9-4893-b024-57b7b2bf78fe"} parent_name : [] port_security : ["fa:16:3e:bc:8e:6c 10.226.128.77"] tag : [] tag_request : [] type : virtual up : true also the nat is correct now: ovn-nbctl lr-nat-list neutron-0e650979-0c96-4fe7-88b5-929451c29931 | grep 101 dnat_and_snat 172.18.144.101 10.226.128.77 fa:16:3e:44:8a:2a 926ae169-72ce-4d75-926b-7ed6bcb44ec8 if I move back to openstack port set --device-owner "Octavia" octavia-lb-5279d85e-3c98-46ca-aea5-58b898db0386 I see again the type being empty: options : {mcast_flood_reports="true", requested-chassis=""} parent_name : [] port_security : ["fa:16:3e:bc:8e:6c 10.226.128.77"] tag : [] tag_request : [] type : "" up : false now trying ovn-nbctl lsp-set-type 926ae169-72ce-4d75-926b-7ed6bcb44ec8 virtual, logical port and nat become: options : {mcast_flood_reports="true", requested-chassis=""} parent_name : [] port_security : ["fa:16:3e:bc:8e:6c 10.226.128.77"] tag : [] tag_request : [] type : virtual up : false $ sudo podman exec -ti ovn-dbs-bundle-podman-0 ovn-nbctl lr-nat-list neutron-0e650979-0c96-4fe7-88b5-929451c29931 | grep 101 dnat_and_snat 172.18.144.101 10.226.128.77 so ovn-nbctl lsp-set-type <vip-port-id> virtual does not seem like a valid workaround: the lsp still has up=false options:virtual-ip and virtual-parents are missing from the lsp external_mac is still missing from the nat also in the case I mention At the moment the traffic is still sent to the chassis hosting the Amphora VM though. I think this a mistake. I double checked and the traffic is sent to the L3 gw router. I am more confident with setting openstack port set --device-owner "", on the loadbalancer having issues. Also this behavior does not affect all the load balancers, setting device-owner to "" then back to Octavia on other load balancer ports does not break the virtual type, it's weird.
*** Bug 2129253 has been marked as a duplicate of this bug. ***
Verified on RHOS-16.2-RHEL-8-20221104.n.0 with python-networking-ovn-7.4.2-2.20220409154863.el8ost All the OVN Octavia ports in all the controllers had "virtual" type: # Finding ovn-dbs-bundle-podman-X on Controller-0 [root@controller-0 ~]# podman ps | grep -i ovn 4225de3354d2 cluster.common.tag/rhosp16-openstack-ovn-northd:pcmklatest /bin/bash /usr/lo... 11 days ago Up 11 days ago ovn-dbs-bundle-podman-0 b0f038826b5d undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-neutron-server-ovn:16.2_20221013.1 kolla_start 10 days ago Up 10 days ago neutron_api be028e7da67f undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-nova-novncproxy:16.2_20221013.1 kolla_start 10 days ago Up 10 days ago nova_vnc_proxy f4ae7304d771 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-ovn-controller:16.2_20221013.1 kolla_start 10 days ago Up 10 days ago ovn_controller # Making sure the logical_switch_port(s) do have "virtual" port type [root@controller-0 ~]# podman exec -ti ovn-dbs-bundle-podman-0 ovn-nbctl --columns addresses,type,name,up find logical_switch_port 'external_ids:"neutron:device_owner"=Octavia' addresses : ["fa:16:3e:6f:61:9a 10.100.227.82 2001:db8:0:e333:f816:3eff:fe6f:619a"] type : virtual <------------------------------ name : "f7aeaf84-b01f-4429-a4d0-fecc39d2416f" up : true addresses : ["fa:16:3e:19:f9:06 10.0.0.157"] type : virtual <------------------------------ name : "27dddbae-c7e6-459d-8da8-3634ecac9978" up : true WARN[0000] Error resizing exec session 8ceb5e56c00f48e35063a3e41ae1b82c256f00578cf600fbafb8955c5e37add8: could not open ctl file for terminal resize for container 4225de3354d2e931af5cd57a889d8277d35601b6aed5bf9996a8860cdc1739ab: open /var/lib/containers/storage/overlay-containers/4225de3354d2e931af5cd57a889d8277d35601b6aed5bf9996a8860cdc1739ab/userdata/8ceb5e56c00f48e35063a3e41ae1b82c256f00578cf600fbafb8955c5e37add8/ctl: no such device or address [root@controller-0 ~]# logout [heat-admin@controller-0 ~]$ logout Connection to controller-0.ctlplane closed. # Finding ovn-dbs-bundle-podman-X on Controller-1 > ssh controller-1.ctlplane Warning: Permanently added 'controller-1.ctlplane,192.168.24.32' (ECDSA) to the list of known hosts. Last login: Thu Nov 10 15:19:06 2022 from 192.168.24.254 [heat-admin@controller-1 ~]$ sudo -i [root@controller-1 ~]# podman ps | grep -i ovn 2a09465571fb cluster.common.tag/rhosp16-openstack-ovn-northd:pcmklatest /bin/bash /usr/lo... 11 days ago Up 11 days ago ovn-dbs-bundle-podman-1 48f3d363d72d undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-neutron-server-ovn:16.2_20221013.1 kolla_start 10 days ago Up 10 days ago neutron_api 6ebbbe66b62b undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-nova-novncproxy:16.2_20221013.1 kolla_start 10 days ago Up 10 days ago nova_vnc_proxy 302346670ef8 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-ovn-controller:16.2_20221013.1 kolla_start 10 days ago Up 10 days ago ovn_controller # Making sure the logical_switch_port(s) do have "virtual" port type [root@controller-1 ~]# podman exec -ti ovn-dbs-bundle-podman-1 ovn-nbctl --columns addresses,type,name,up find logical_switch_port 'external_ids:"neutron:device_owner"=Octavia' addresses : ["fa:16:3e:6f:61:9a 10.100.227.82 2001:db8:0:e333:f816:3eff:fe6f:619a"] type : virtual <------------------------------ name : "f7aeaf84-b01f-4429-a4d0-fecc39d2416f" up : true addresses : ["fa:16:3e:19:f9:06 10.0.0.157"] type : virtual <------------------------------ name : "27dddbae-c7e6-459d-8da8-3634ecac9978" up : true [root@controller-1 ~]# logout [heat-admin@controller-1 ~]$ logout Connection to controller-1.ctlplane closed. # Finding ovn-dbs-bundle-podman-X on Controller-2 > ssh controller-2.ctlplane Warning: Permanently added 'controller-2.ctlplane,192.168.24.35' (ECDSA) to the list of known hosts. Last login: Thu Nov 10 15:19:06 2022 from 192.168.24.254 [heat-admin@controller-2 ~]$ sudo -i [root@controller-2 ~]# podman ps | grep -i ovn d31ed6e91be5 cluster.common.tag/rhosp16-openstack-ovn-northd:pcmklatest /bin/bash /usr/lo... 11 days ago Up 11 days ago ovn-dbs-bundle-podman-2 3f3c3e4a1ce7 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-neutron-server-ovn:16.2_20221013.1 kolla_start 10 days ago Up 10 days ago neutron_api 9cdaa7157b53 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-nova-novncproxy:16.2_20221013.1 kolla_start 10 days ago Up 10 days ago nova_vnc_proxy 22a72e9715c9 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-ovn-controller:16.2_20221013.1 kolla_start 10 days ago Up 10 days ago ovn_controller # Making sure the logical_switch_port(s) do have "virtual" port type [root@controller-2 ~]# podman exec -ti ovn-dbs-bundle-podman-2 ovn-nbctl --columns addresses,type,name,up find logical_switch_port 'external_ids:"neutron:device_owner"=Octavia' addresses : ["fa:16:3e:6f:61:9a 10.100.227.82 2001:db8:0:e333:f816:3eff:fe6f:619a"] type : virtual <------------------------------ name : "f7aeaf84-b01f-4429-a4d0-fecc39d2416f" up : true addresses : ["fa:16:3e:19:f9:06 10.0.0.157"] type : virtual <------------------------------ name : "27dddbae-c7e6-459d-8da8-3634ecac9978" up : true WARN[0000] Error resizing exec session 029e85aa8e54372f350185dc0b8a7f06263017395feaa40533f4ca4a06edc569: could not open ctl file for terminal resize for container d31ed6e91be534a41e4f0bebf7ed976e3c4a2e259bf757874e50db9e464a8329: open /var/lib/containers/storage/overlay-containers/d31ed6e91be534a41e4f0bebf7ed976e3c4a2e259bf757874e50db9e464a8329/userdata/029e85aa8e54372f350185dc0b8a7f06263017395feaa40533f4ca4a06edc569/ctl: no such device or address Looks good to me. I am moving the BZ status to VERIFIED.
*** Bug 2144883 has been marked as a duplicate of this bug. ***
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Release of components for Red Hat OpenStack Platform 16.2.4), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:8794