Description of problem: The test neutron_plugin.tests.scenario.test_dvr_ovn.OvnDvrTest.test_dvr_vip_failover [1] is failing in low frequency (4/600). When de VIP_FIP is assigned for the first time to a VM, the mac address is the router-gw MAC instead of the VIP-FIP MAC itself. The drop flow rule with priority 80 is not added in "ovn-sbctl lflow-list " [root@controller-1 ~]# ovn-sbctl lflow-list tempest-router-571443429 | grep lr_in_gw_redirect table=18(lr_in_gw_redirect ), priority=100 , match=(ip4.src == 192.168.0.192 && outport == "lrp-68d50fa4-990f-4e73-9bc5-89343a958912" && is_chassis_resident("eb6a1352-a7c2-4a75-b78a-db009640f15e")), action=(eth.src = fa:16:3e:d1:b9:45; reg1 = 10.0.0.185; next;) table=18(lr_in_gw_redirect ), priority=100 , match=(ip4.src == 192.168.0.226 && outport == "lrp-68d50fa4-990f-4e73-9bc5-89343a958912" && is_chassis_resident("4ee1cb83-56af-4726-8408-7db201324ebe")), action=(eth.src = fa:16:3e:ec:1c:bd; reg1 = 10.0.0.182; next;) table=18(lr_in_gw_redirect ), priority=100 , match=(ip4.src == 192.168.0.52 && outport == "lrp-68d50fa4-990f-4e73-9bc5-89343a958912" && is_chassis_resident("72767d62-1438-42b5-8c09-e1a4fb50ed03")), action=(eth.src = fa:16:3e:75:7c:9f; reg1 = 10.0.0.216; next;) table=18(lr_in_gw_redirect ), priority=80 , match=(ip4.src == 192.168.0.226 && outport == "lrp-68d50fa4-990f-4e73-9bc5-89343a958912"), action=(drop;) <-----------------------------------------------This rule is not shows when the test is fail table=18(lr_in_gw_redirect ), priority=50 , match=(outport == "lrp-68d50fa4-990f-4e73-9bc5-89343a958912"), action=(outport = "cr-lrp-68d50fa4-990f-4e73-9bc5-89343a958912"; next;) table=18(lr_in_gw_redirect ), priority=0 , match=(1), action=(next;) The traffic is not distributed, is centralized despite enabling DVR. However, after doing the VIP failover the problem is solved (also the drop flow rule is added) The problem is similar as this bug: https://bugzilla.redhat.com/show_bug.cgi?id=2083527. This create as a new bug because the fail is not always reproduce. Version: RHOS-16.2-RHEL-8-20220603.n.1 python3-networking-ovn-7.4.2-2.20220409154849.el8ost [heat-admin@controller-0 ~]$ sudo podman exec -it ovn_controller rpm -qa | grep ovn rhosp-ovn-host-2021-4.el8ost.1.noarch ovn-2021-21.12.0-46.el8fdp.x86_64 rhosp-ovn-2021-4.el8ost.1.noarch ovn-2021-host-21.12.0-46.el8fdp.x86_64 Reproduce: - Assigned VIP_FIP to VM. - Establish connection. - Capture the traffic on controller and computes. [1] https://code.engineering.redhat.com/gerrit/plugins/gitiles/rhos-qe-tests/tempest_neutron_plugin/+/dcefdfec23003790cb4a37ae56f07f7614dc98f0/neutron_plugin/tests/scenario/test_dvr_ovn.py#614
I had a look at the NB database ovsdb logs and I see: record 6840: 2022-07-04 21:20:54.420 [...] table Logical_Switch_Port insert row "1536b0bd-ccde-43b4-93ce-e86d5e797052" (b1d43df5): name="1536b0bd-ccde-43b4-93ce-e86d5e797052" addresses=["fa:16:3e:1a:ae:ea 192.168.0.145"] port_security=["fa:16:3e:1a:ae:ea 192.168.0.145"] options={mcast_flood_reports="true", requested-chassis=""} external_ids={"neutron:cidrs"="192.168.0.145/24", "neutron:device_id"="", "neutron:device_owner"="", "neutron:network_name"=neutron-436a7267-25b6-4681-af04-1eb8dce38f0b, "neutron:port_name"="", "neutron:project_id"=b059c55a3d734ad2bf0ac691d0fa824a, "neutron:revision_number"="1", "neutron:security_group_ids"="cb8dee07-0fc9-41db-8d83-8ec3bb0ddf18"} dhcpv4_options=17e8eadf-850e-4bd6-b37e-93c98ee8ac6b enabled=true # At this point the type for port "1536b0bd-ccde-43b4-93ce-e86d5e797052" is "", regular VIF. # Later this gets turned into a virtual port: record 6846: 2022-07-04 21:20:55.198 table Logical_Switch_Port row "1536b0bd-ccde-43b4-93ce-e86d5e797052" (b1d43df5) diff: options={virtual-ip="192.168.0.145", virtual-parents="9b49310a-6a59-49e7-91b4-a1c836ad650a"} type=virtual table Logical_Switch_Port row "9b49310a-6a59-49e7-91b4-a1c836ad650a" (4775b19e) diff: port_security=["fa:16:3e:8f:b0:21 192.168.0.72", "fa:16:3e:8f:b0:21 192.168.0.72 192.168.0.145"] external_ids={"neutron:revision_number"="5"} # And even later the virtual parents are set. record 6850: 2022-07-04 21:20:56.216 [...] table Logical_Switch_Port row "1536b0bd-ccde-43b4-93ce-e86d5e797052" (b1d43df5) diff: options={virtual-parents="7bf4442b-9585-4296-8897-36e23307b474,9b49310a-6a59-49e7-91b4-a1c836ad650a"} Changing from/to VIF/child port(container/virtual) ports is problematic. AFAICT we handled the case for container ports: https://github.com/ovn-org/ovn/commit/cd3b685043fa9758df3665bf3e3fc972048698a6 But it seems that we need the same fix applied for virtual ports. @mheib what do you think?
Hi, after applying the two patches [1][2] that handle the port type update in northd and updated the 3 controllers in panther23.lab.eng.tlv2.redhat.com setup the test seems to pass now, i tried to re-run it several times and it seems to be okay, @ralonsoh can you please do some more test execution in setup panther23.lab.eng.tlv2.redhat.com and see if you still have the issue or not. [1] https://github.com/ovn-org/ovn/commit/cd3b685043fa9758df3665bf3e3fc972048698a6 [2] https://patchwork.ozlabs.org/project/ovn/patch/20220719133328.2037257-1-mheib@redhat.com/ thanks.
ovn22.06 fast-datapath-rhel-8 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2115334 ovn22.06 fast-datapath-rhel-9 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2115335
using following script to test patch itself: systemctl start openvswitch systemctl start ovn-northd ovn-nbctl set-connection ptcp:6641 ovn-sbctl set-connection ptcp:6642 ovs-vsctl set open . external_ids:system-id=hv1 external_ids:ovn-remote=tcp:1.1.40.25:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=1.1.40.25 systemctl restart ovn-controller ovn-nbctl ls-add ls1 ovn-nbctl lsp-add ls1 ls1p1 ovn-nbctl lsp-set-addresses ls1p1 "00:00:00:01:01:01 192.168.1.1 2001::1" ovn-nbctl lsp-add ls1 ls1p2 ovn-nbctl lsp-set-addresses ls1p2 "00:00:00:01:01:02 192.168.1.2 2001::2" ovn-nbctl lsp-add ls1 ls1-vir ovn-nbctl lsp-set-type ls1-vir virtual ovn-nbctl lsp-set-addresses ls1-vir "00:00:00:01:01:11 192.168.1.11 2001::11" ovn-nbctl set logical_switch_port ls1-vir options:virtual-ip=192.168.1.11 ovn-nbctl set logical_switch_port ls1-vir options:virtual-parents="ls1p1,ls1p2" ovn-nbctl lr-add lr1 ovn-nbctl lrp-add lr1 lr1-ls1 00:00:00:00:00:01 192.168.1.254/24 2001::a/64 ovn-nbctl lsp-add ls1 ls1-lr1 ovn-nbctl lsp-set-addresses ls1-lr1 "00:00:00:00:00:01 192.168.1.254 2001::a" ovn-nbctl lsp-set-type ls1-lr1 router ovn-nbctl lsp-set-options ls1-lr1 router-port=lr1-ls1 ovs-vsctl add-port br-int ls1p1 -- set interface ls1p1 type=internal external_ids:iface-id=ls1p1 ip netns add ls1p1 ip link set ls1p1 netns ls1p1 ip netns exec ls1p1 ip link set ls1p1 address 00:00:00:01:01:01 ip netns exec ls1p1 ip link set ls1p1 up ip netns exec ls1p1 ip addr add 192.168.1.1/24 dev ls1p1 ip netns exec ls1p1 ip addr add 2001::1/64 dev ls1p1 ip netns exec ls1p1 ip route add default via 192.168.1.254 dev ls1p1 ip netns exec ls1p1 ip -6 route add default via 2001::a dev ls1p1 ovn-nbctl --wait=hv sync ovn-sbctl list port_binding ls1-vir ip netns exec ls1p1 ip addr add 192.168.1.11/24 dev ls1p1 ip netns exec ls1p1 arping -U -I ls1p1 -c 1 192.168.1.11 sleep 2 ovn-sbctl list port_binding ls1-vir ovn-nbctl set logical_switch_port ls1-vir type=\"\" ovn-nbctl list logical_switch_port ls1-vir ovn-nbctl --wait=hv sync ovn-sbctl list port_binding ls1-vir reproduced on ovn22.06-27: [root@dell-per740-69 bz2099288]# rpm -qa | grep -E "openvswitch2.15|ovn22.06" openvswitch2.15-2.15.0-38.el8fdp.x86_64 ovn22.06-22.06.0-27.el8fdp.x86_64 ovn22.06-host-22.06.0-27.el8fdp.x86_64 ovn22.06-central-22.06.0-27.el8fdp.x86_64 + ovn-sbctl list port_binding ls1-vir _uuid : 097cc76e-505b-4b67-b40e-9fb4eebde89f additional_chassis : [] additional_encap : [] chassis : [] datapath : 09c9e073-674a-457f-baae-b9d232fe2f4b encap : [] external_ids : {} gateway_chassis : [] ha_chassis_group : [] logical_port : ls1-vir mac : ["00:00:00:01:01:11 192.168.1.11 2001::11"] nat_addresses : [] options : {virtual-ip="192.168.1.11", virtual-parents="ls1p1,ls1p2"} parent_port : [] port_security : [] requested_additional_chassis: [] requested_chassis : [] tag : [] tunnel_key : 3 type : virtual up : false virtual_parent : [] + ip netns exec ls1p1 ip addr add 192.168.1.11/24 dev ls1p1 + ip netns exec ls1p1 arping -U -I ls1p1 -c 1 192.168.1.11 ARPING 192.168.1.11 from 192.168.1.11 ls1p1 Sent 1 probes (1 broadcast(s)) Received 0 response(s) + sleep 2 + ovn-sbctl list port_binding ls1-vir _uuid : 097cc76e-505b-4b67-b40e-9fb4eebde89f additional_chassis : [] additional_encap : [] chassis : b84f8aec-8ab6-427c-ac1b-05ef3dacd7a8 datapath : 09c9e073-674a-457f-baae-b9d232fe2f4b encap : [] external_ids : {} gateway_chassis : [] ha_chassis_group : [] logical_port : ls1-vir mac : ["00:00:00:01:01:11 192.168.1.11 2001::11"] nat_addresses : [] options : {virtual-ip="192.168.1.11", virtual-parents="ls1p1,ls1p2"} parent_port : [] port_security : [] requested_additional_chassis: [] requested_chassis : [] tag : [] tunnel_key : 3 type : virtual up : true virtual_parent : ls1p1 + ovn-nbctl set logical_switch_port ls1-vir 'type=""' + ovn-nbctl list logical_switch_port ls1-vir _uuid : c1a3aa9b-97d3-414c-bd34-c00df8bf6b09 addresses : ["00:00:00:01:01:11 192.168.1.11 2001::11"] dhcpv4_options : [] dhcpv6_options : [] dynamic_addresses : [] enabled : [] external_ids : {} ha_chassis_group : [] name : ls1-vir options : {virtual-ip="192.168.1.11", virtual-parents="ls1p1,ls1p2"} parent_name : [] port_security : [] tag : [] tag_request : [] type : "" up : false + ovn-nbctl --wait=hv sync + ovn-sbctl list port_binding ls1-vir _uuid : 097cc76e-505b-4b67-b40e-9fb4eebde89f additional_chassis : [] additional_encap : [] chassis : [] datapath : 09c9e073-674a-457f-baae-b9d232fe2f4b encap : [] external_ids : {} gateway_chassis : [] ha_chassis_group : [] logical_port : ls1-vir mac : ["00:00:00:01:01:11 192.168.1.11 2001::11"] nat_addresses : [] options : {virtual-ip="192.168.1.11", virtual-parents="ls1p1,ls1p2"} parent_port : [] port_security : [] requested_additional_chassis: [] requested_chassis : [] tag : [] tunnel_key : 3 type : "" up : true <=== up is still true even change the type to "" for virtual port virtual_parent : [] Verified on ovn22.06-57: [root@dell-per740-69 bz2099288]# rpm -qa | grep -E "openvswitch2.15|ovn22.06" openvswitch2.15-2.15.0-38.el8fdp.x86_64 ovn22.06-22.06.0-57.el8fdp.x86_64 ovn22.06-central-22.06.0-57.el8fdp.x86_64 ovn22.06-host-22.06.0-57.el8fdp.x86_64 + ip netns exec ls1p1 arping -U -I ls1p1 -c 1 192.168.1.11 ARPING 192.168.1.11 from 192.168.1.11 ls1p1 Sent 1 probes (1 broadcast(s)) Received 0 response(s) + sleep 2 + ovn-sbctl list port_binding ls1-vir _uuid : f1c2bef7-84eb-4969-aad5-272474850d5f additional_chassis : [] additional_encap : [] chassis : 1689782a-1e3f-47cc-9154-b69d5f2cfab5 datapath : 1189c03f-43f4-4711-b8d5-ee1d4ec44783 encap : [] external_ids : {} gateway_chassis : [] ha_chassis_group : [] logical_port : ls1-vir mac : ["00:00:00:01:01:11 192.168.1.11 2001::11"] nat_addresses : [] options : {virtual-ip="192.168.1.11", virtual-parents="ls1p1,ls1p2"} parent_port : [] port_security : [] requested_additional_chassis: [] requested_chassis : [] tag : [] tunnel_key : 3 type : virtual up : true virtual_parent : ls1p1 + ovn-nbctl set logical_switch_port ls1-vir 'type=""' + ovn-nbctl list logical_switch_port ls1-vir _uuid : 5632af16-2bb7-4e5a-9ee2-a7a60cf37400 addresses : ["00:00:00:01:01:11 192.168.1.11 2001::11"] dhcpv4_options : [] dhcpv6_options : [] dynamic_addresses : [] enabled : [] external_ids : {} ha_chassis_group : [] name : ls1-vir options : {virtual-ip="192.168.1.11", virtual-parents="ls1p1,ls1p2"} parent_name : [] port_security : [] tag : [] tag_request : [] type : "" up : false + ovn-nbctl --wait=hv sync + ovn-sbctl list port_binding ls1-vir _uuid : efefafbb-1ca4-4a99-9afe-d9125457be3f additional_chassis : [] additional_encap : [] chassis : [] datapath : 1189c03f-43f4-4711-b8d5-ee1d4ec44783 encap : [] external_ids : {} gateway_chassis : [] ha_chassis_group : [] logical_port : ls1-vir mac : ["00:00:00:01:01:11 192.168.1.11 2001::11"] nat_addresses : [] options : {virtual-ip="192.168.1.11", virtual-parents="ls1p1,ls1p2"} parent_port : [] port_security : [] requested_additional_chassis: [] requested_chassis : [] tag : [] tunnel_key : 5 type : "" up : false virtual_parent : []
reproduced on ovn22.03-22.03.0-69: [root@dell-per740-70 bz2099288]# rpm -qa | grep -E "openvswitch2.15|ovn22.03" openvswitch2.15-2.15.0-38.el8fdp.x86_64 ovn22.03-22.03.0-69.el8fdp.x86_64 ovn22.03-host-22.03.0-69.el8fdp.x86_64 ovn22.03-central-22.03.0-69.el8fdp.x86_64 + ovn-sbctl list port_binding ls1-vir _uuid : c82fcc5c-90f9-41be-846a-b94dc1de0bab chassis : [] datapath : 3f38a9a2-35b6-4a56-8c9f-01f04d7ccf76 encap : [] external_ids : {} gateway_chassis : [] ha_chassis_group : [] logical_port : ls1-vir mac : ["00:00:00:01:01:11 192.168.1.11 2001::11"] nat_addresses : [] options : {virtual-ip="192.168.1.11", virtual-parents="ls1p1,ls1p2"} parent_port : [] requested_chassis : [] tag : [] tunnel_key : 3 type : virtual up : false virtual_parent : [] + ip netns exec ls1p1 ip addr add 192.168.1.11/24 dev ls1p1 + ip netns exec ls1p1 arping -U -I ls1p1 -c 1 192.168.1.11 ARPING 192.168.1.11 from 192.168.1.11 ls1p1 Sent 1 probes (1 broadcast(s)) Received 0 response(s) + sleep 2 + ovn-sbctl list port_binding ls1-vir _uuid : c82fcc5c-90f9-41be-846a-b94dc1de0bab chassis : b4fb110d-914f-40bc-a3fb-1ce27672f5cf datapath : 3f38a9a2-35b6-4a56-8c9f-01f04d7ccf76 encap : [] external_ids : {} gateway_chassis : [] ha_chassis_group : [] logical_port : ls1-vir mac : ["00:00:00:01:01:11 192.168.1.11 2001::11"] nat_addresses : [] options : {virtual-ip="192.168.1.11", virtual-parents="ls1p1,ls1p2"} parent_port : [] requested_chassis : [] tag : [] tunnel_key : 3 type : virtual up : true virtual_parent : ls1p1 + ovn-nbctl set logical_switch_port ls1-vir 'type=""' + ovn-nbctl list logical_switch_port ls1-vir _uuid : f7fa2113-d65e-4f46-a030-88d4d420a1cd addresses : ["00:00:00:01:01:11 192.168.1.11 2001::11"] dhcpv4_options : [] dhcpv6_options : [] dynamic_addresses : [] enabled : [] external_ids : {} ha_chassis_group : [] name : ls1-vir options : {virtual-ip="192.168.1.11", virtual-parents="ls1p1,ls1p2"} parent_name : [] port_security : [] tag : [] tag_request : [] type : "" up : false + ovn-nbctl --wait=hv sync + ovn-sbctl list port_binding ls1-vir _uuid : c82fcc5c-90f9-41be-846a-b94dc1de0bab chassis : [] datapath : 3f38a9a2-35b6-4a56-8c9f-01f04d7ccf76 encap : [] external_ids : {} gateway_chassis : [] ha_chassis_group : [] logical_port : ls1-vir mac : ["00:00:00:01:01:11 192.168.1.11 2001::11"] nat_addresses : [] options : {virtual-ip="192.168.1.11", virtual-parents="ls1p1,ls1p2"} parent_port : [] requested_chassis : [] tag : [] tunnel_key : 3 type : "" up : true <== up is still true virtual_parent : [] Verified on ovn22.03-22.03.0-95: [root@dell-per740-70 bz2099288]# rpm -qa | grep -E "openvswitch2.15|ovn22.03" openvswitch2.15-2.15.0-38.el8fdp.x86_64 ovn22.03-22.03.0-95.el8fdp.x86_64 ovn22.03-central-22.03.0-95.el8fdp.x86_64 ovn22.03-host-22.03.0-95.el8fdp.x86_64 + ovn-sbctl list port_binding ls1-vir _uuid : 7995b874-dc5a-496b-a228-a71d583e7143 chassis : [] datapath : 34978aaf-e205-4c15-847a-7a8a7aadc803 encap : [] external_ids : {} gateway_chassis : [] ha_chassis_group : [] logical_port : ls1-vir mac : ["00:00:00:01:01:11 192.168.1.11 2001::11"] nat_addresses : [] options : {virtual-ip="192.168.1.11", virtual-parents="ls1p1,ls1p2"} parent_port : [] requested_chassis : [] tag : [] tunnel_key : 3 type : virtual up : false virtual_parent : [] + ip netns exec ls1p1 ip addr add 192.168.1.11/24 dev ls1p1 + ip netns exec ls1p1 arping -U -I ls1p1 -c 1 192.168.1.11 ARPING 192.168.1.11 from 192.168.1.11 ls1p1 Sent 1 probes (1 broadcast(s)) Received 0 response(s) + sleep 2 + ovn-sbctl list port_binding ls1-vir _uuid : 7995b874-dc5a-496b-a228-a71d583e7143 chassis : c0e1efe3-befc-4926-ac20-2a2c2d23f010 datapath : 34978aaf-e205-4c15-847a-7a8a7aadc803 encap : [] external_ids : {} gateway_chassis : [] ha_chassis_group : [] logical_port : ls1-vir mac : ["00:00:00:01:01:11 192.168.1.11 2001::11"] nat_addresses : [] options : {virtual-ip="192.168.1.11", virtual-parents="ls1p1,ls1p2"} parent_port : [] requested_chassis : [] tag : [] tunnel_key : 3 type : virtual up : true virtual_parent : ls1p1 + ovn-nbctl set logical_switch_port ls1-vir 'type=""' + ovn-nbctl list logical_switch_port ls1-vir _uuid : 8f294abf-9c15-4106-993e-c533c01b9814 addresses : ["00:00:00:01:01:11 192.168.1.11 2001::11"] dhcpv4_options : [] dhcpv6_options : [] dynamic_addresses : [] enabled : [] external_ids : {} ha_chassis_group : [] name : ls1-vir options : {virtual-ip="192.168.1.11", virtual-parents="ls1p1,ls1p2"} parent_name : [] port_security : [] tag : [] tag_request : [] type : "" up : false + ovn-nbctl --wait=hv sync + ovn-sbctl list port_binding ls1-vir _uuid : 9c02d2fd-8573-4c24-996b-161c006262b0 chassis : [] datapath : 34978aaf-e205-4c15-847a-7a8a7aadc803 encap : [] external_ids : {} gateway_chassis : [] ha_chassis_group : [] logical_port : ls1-vir mac : ["00:00:00:01:01:11 192.168.1.11 2001::11"] nat_addresses : [] options : {virtual-ip="192.168.1.11", virtual-parents="ls1p1,ls1p2"} parent_port : [] requested_chassis : [] tag : [] tunnel_key : 5 type : "" up : false <=== up is false virtual_parent : []
set VERIFIED per comment 23
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (ovn22.03 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:6870