Description of problem: When a virtual machine in Openstack is directly attached against a provider network, we see the MAC associated with the localport flapping across uplinks due to the fact the same interface is present on each compute node which hosts a virtual machine attached to that specific provider network. Version-Release number of selected component (if applicable): How reproducible: 100% of the times Steps to Reproduce: 1. Create a virtual machine and directly attach it to a provider network 2. Create another virtual machine and directly attach it to the same provider network as 1. 3. Fetch the localport used by the metadata service 4. Dump the traffic and verify the gARPs are being generated on each compute node hosting the localport causing Actual results: The localport traffic leaks traffic to the provider network it is attached to Expected results: The localport traffic should not leak any traffic to provider network Additional info:
I guess you mean localnet ports and not localports. Can you please confirm?
Daniel, no, I mean localport(s) [1], an example: port cc4bd2e8-f224-4a64-acba-d4d9c5e6e5cd type: localport addresses: ["fa:16:3e:3b:de:2f 10.x.x.x"] [1] https://patchwork.ozlabs.org/project/openvswitch/patch/1493118328-21311-1-git-send-email-dalvarez@redhat.com/
Thanks Andrea for the clarification. Those gARPs that you see on the provider network are sent with the source MAC address of the localport? Could you share some captures and match IP/MACs to the OVN ports involved?
Daniel, with the expectation the localport traffic should never leak outside of the localport network namespace: ARPing running while capturing traffic: [root@compute-az3-19 ~]# ip netns exec ovnmeta-1e483461-cf04-40f4-80cd-b1c8686742d4 arping -c 4 -A -I tap1e483461-c1 10.2.68.10 While it's running: [root@compute-az3-19 ~]# ip netns exec ovnmeta-1e483461-cf04-40f4-80cd-b1c8686742d4 tcpdump -i tap1e483461-c1 ether host fa:16:3e:f7:aa:fd or ether host f0:7c:c7:23:61:40 -vneeee tcpdump: listening on tap1e483461-c1, link-type EN10MB (Ethernet), capture size 262144 bytes 14:03:33.790269 fa:16:3e:f7:aa:fd > Broadcast, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Reply 10.2.68.10 is-at fa:16:3e:f7:aa:fd, length 28 14:03:34.790443 fa:16:3e:f7:aa:fd > Broadcast, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Reply 10.2.68.10 is-at fa:16:3e:f7:aa:fd, length 28 14:03:35.774551 fa:16:3e:f7:aa:fd > 94:f7:ad:b7:84:40, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Reply 10.2.68.10 is-at fa:16:3e:f7:aa:fd, length 28 14:03:35.790614 fa:16:3e:f7:aa:fd > Broadcast, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Reply 10.2.68.10 is-at fa:16:3e:f7:aa:fd, length 28 14:03:35.802690 f0:7c:c7:23:61:40 > Broadcast, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Request who-has 10.2.68.10 tell 10.2.68.252, length 42 14:03:35.802700 fa:16:3e:f7:aa:fd > f0:7c:c7:23:61:40, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Reply 10.2.68.10 is-at fa:16:3e:f7:aa:fd, length 28 14:03:36.449964 fa:16:3e:f7:aa:fd > Broadcast, ethertype Reverse ARP (0x8035), length 60: Ethernet (len 6), IPv4 (len 4), Request who-has 0.0.0.0 (Broadcast) tell 0.0.0.0, length 46 14:03:36.460246 fa:16:3e:f7:aa:fd > Broadcast, ethertype ARP (0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Request who-has 0.0.0.0 (Broadcast) tell 0.0.0.0, length 46 14:03:36.487212 fa:16:3e:f7:aa:fd > Broadcast, ethertype Reverse ARP (0x8035), length 60: Ethernet (len 6), IPv4 (len 4), Request who-has 0.0.0.0 (Broadcast) tell 0.0.0.0, length 46 14:03:36.497924 fa:16:3e:f7:aa:fd > Broadcast, ethertype ARP (0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Request who-has 0.0.0.0 (Broadcast) tell 0.0.0.0, length 46 14:03:36.700567 f0:7c:c7:23:61:40 > Broadcast, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Request who-has 10.2.68.10 tell 10.2.68.252, length 42 14:03:36.700578 fa:16:3e:f7:aa:fd > f0:7c:c7:23:61:40, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Reply 10.2.68.10 is-at fa:16:3e:f7:aa:fd, length 28 14:03:36.790689 fa:16:3e:f7:aa:fd > Broadcast, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Reply 10.2.68.10 is-at fa:16:3e:f7:aa:fd, length 28 14:03:37.300602 f0:7c:c7:23:61:40 > Broadcast, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Request who-has 10.2.68.10 tell 10.2.68.252, length 42 14:03:37.300611 fa:16:3e:f7:aa:fd > f0:7c:c7:23:61:40, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Reply 10.2.68.10 is-at fa:16:3e:f7:aa:fd, length 28 14:03:37.403623 fa:16:3e:f7:aa:fd > 94:f7:ad:b7:84:40, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Reply 10.2.68.10 is-at fa:16:3e:f7:aa:fd, length 28 14:03:37.459353 fa:16:3e:f7:aa:fd > Broadcast, ethertype Reverse ARP (0x8035), length 60: Ethernet (len 6), IPv4 (len 4), Request who-has 0.0.0.0 (Broadcast) tell 0.0.0.0, length 46 14:03:37.471158 fa:16:3e:f7:aa:fd > Broadcast, ethertype ARP (0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Request who-has 0.0.0.0 (Broadcast) tell 0.0.0.0, length 46 14:03:37.900185 f0:7c:c7:23:61:40 > Broadcast, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Request who-has 10.2.68.10 tell 10.2.68.252, length 42 14:03:37.900197 fa:16:3e:f7:aa:fd > f0:7c:c7:23:61:40, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Reply 10.2.68.10 is-at fa:16:3e:f7:aa:fd, length 28 14:03:38.103654 fa:16:3e:f7:aa:fd > 94:f7:ad:b7:84:40, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Reply 10.2.68.10 is-at fa:16:3e:f7:aa:fd, length 28 14:03:38.349935 f0:7c:c7:23:61:40 > Broadcast, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Request who-has 10.2.68.10 tell 10.2.68.252, length 42 14:03:38.349945 fa:16:3e:f7:aa:fd > f0:7c:c7:23:61:40, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Reply 10.2.68.10 is-at fa:16:3e:f7:aa:fd, length 28 14:03:38.402983 fa:16:3e:f7:aa:fd > 94:f7:ad:b7:84:40, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Reply 10.2.68.10 is-at fa:16:3e:f7:aa:fd, length 28 14:03:39.449457 f0:7c:c7:23:61:40 > Broadcast, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Request who-has 10.2.68.180 tell 10.2.68.252, length 42 14:03:44.376384 f0:7c:c7:23:61:40 > Broadcast, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Request who-has 10.2.68.222 tell 10.2.68.252, length 42 14:03:44.551100 f0:7c:c7:23:61:40 > Broadcast, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Request who-has 10.2.68.72 tell 10.2.68.252, length 42 14:03:47.151511 f0:7c:c7:23:61:40 > Broadcast, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Request who-has 10.2.68.52 tell 10.2.68.252, length 42 f0:7c:c7:23:61:40 being the router NIC MAC, fa:16:3e:f7:aa:fd being the localport MAC: port 16ef1dd5-42df-4579-949a-b087cdaef2a3 type: localport addresses: ["fa:16:3e:f7:aa:fd 10.2.68.10"] ()[root@controller-1 ~]# ovn-sbctl list port_binding 16ef1dd5-42df-4579-949a-b087cdaef2a3 _uuid : 6c57af29-6c47-44d3-b196-f057d126723f chassis : fc5e634d-2db0-431e-806a-f8f7ed97a9e3 datapath : 1e483461-cf04-40f4-80cd-b1c8686742d4 encap : [] external_ids : {"neutron:cidrs"="10.2.68.10/24", "neutron:device_id"=ovnmeta-36d02a00-988c-431d-a389-e24edb9edbe1, "neutron:device_owner"="network:dhcp", "neutron:network_name"=neutron-36d02a00-988c-431d-a389-e24edb9edbe1, "neutron:port_name"="", "neutron:project_id"=d2b4103f9246414193891c61f4482897, "neutron:revision_number"="2", "neutron:security_group_ids"=""} gateway_chassis : [] ha_chassis_group : [] logical_port : "16ef1dd5-42df-4579-949a-b087cdaef2a3" mac : ["fa:16:3e:f7:aa:fd 10.2.68.10"] nat_addresses : [] options : {requested-chassis=""} parent_port : [] tag : [] tunnel_key : 2 type : localport up : false virtual_parent : [] Please mark this comment as confidential.
I sent this patch that seems to work, would require review from the core OVN team and backports: http://patchwork.ozlabs.org/project/ovn/patch/20210323135559.14959-1-dalvarez@redhat.com/ Unfortunately I won't have the time to work on the delivery of the fix. Please core OVN team take over as time permits based on the urgency. Thanks!
Can we please get confirmation that this issue will be fixed in 16.1.5?
tested with following script: #!/bin/bash systemctl start openvswitch systemctl start ovn-northd ovn-nbctl set-connection ptcp:6641 ovn-sbctl set-connection ptcp:6642 ovs-vsctl set open . external_ids:system-id=hv1 external_ids:ovn-remote=tcp:20.0.175.25:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=20.0.175.25 systemctl restart ovn-controller ovs-vsctl add-br br-phys ip link set br-phys up ovs-vsctl set open . external-ids:ovn-bridge-mappings=phys:br-phys ovn-nbctl ls-add ls \ -- lsp-add ls lp \ -- lsp-set-type lp localport \ -- lsp-set-addresses lp "00:00:00:00:00:01 10.0.0.1" \ -- lsp-add ls ln \ -- lsp-set-type ln localnet \ -- lsp-set-options ln network_name=phys \ -- lsp-add ls lsp \ -- lsp-set-addresses lsp "00:00:00:00:00:02 10.0.0.2" ovs-vsctl add-port br-int lp -- set interface lp type=internal external_ids:iface-id=lp ip netns add lp ip link set lp netns lp ip netns exec lp ip link set lp address 00:00:00:00:00:01 ip netns exec lp ip link set lp up ip netns exec lp tcpdump -i lp -w lp.pcap & ip netns exec lp ip addr add 10.0.0.1/24 dev lp ovn-nbctl --wait=hv sync ovs-vsctl add-port br-int lsp -- set interface lsp type=internal external_ids:iface-id=lsp options:tx_pcap=lsp.pcap options:rxq_pcap=lsp-rx.pcap ip netns add lsp ip link set lsp netns lsp ip netns exec lsp ip link set lsp address 00:00:00:00:00:02 ip netns exec lsp ip link set lsp up ip netns exec lsp tcpdump -i lsp -w lsp.pcap & ip netns exec lsp ip addr add 10.0.0.2/24 dev lsp ovn-nbctl --wait=hv sync sleep 30 pkill tcpdump sleep 2 tcpdump -r lsp.pcap -nnle -v arp tcpdump -r lp.pcap -nnle -v arp reproduced on 20.12.0-95: [root@wsfd-advnetlab21 bz1939470]# rpm -qa | grep ovn2.13 ovn2.13-20.12.0-95.el8fdp.x86_64 ovn2.13-central-20.12.0-95.el8fdp.x86_64 ovn2.13-host-20.12.0-95.el8fdp.x86_64 + tcpdump -r lsp.pcap -nnle -v arp reading from file lsp.pcap, link-type EN10MB (Ethernet) dropped privs to tcpdump 03:40:27.354845 00:00:00:00:00:01 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.1 tell 10.0.0.1, length 28 03:40:29.357062 00:00:00:00:00:01 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.1 tell 10.0.0.1, length 28 03:40:33.361292 00:00:00:00:00:01 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.1 tell 10.0.0.1, length 28 03:40:41.369500 00:00:00:00:00:01 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.1 tell 10.0.0.1, length 28 <=== garp for localport + tcpdump -r lp.pcap -nnle -v arp reading from file lp.pcap, link-type EN10MB (Ethernet) dropped privs to tcpdump 03:40:27.354697 00:00:00:00:00:02 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.2 tell 10.0.0.2, length 28 03:40:29.356921 00:00:00:00:00:02 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.2 tell 10.0.0.2, length 28 03:40:33.361109 00:00:00:00:00:02 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.2 tell 10.0.0.2, length 28 03:40:41.369320 00:00:00:00:00:02 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.2 tell 10.0.0.2, length 28 Verified on 20.12.0-104: [root@wsfd-advnetlab21 bz1939470]# rpm -qa | grep ovn2.13 ovn2.13-central-20.12.0-104.el8fdp.x86_64 ovn2.13-20.12.0-104.el8fdp.x86_64 ovn2.13-host-20.12.0-104.el8fdp.x86_64 + tcpdump -r lsp.pcap -nnle -v arp reading from file lsp.pcap, link-type EN10MB (Ethernet) dropped privs to tcpdump <=== no garp for localport + tcpdump -r lp.pcap -nnle -v arp reading from file lp.pcap, link-type EN10MB (Ethernet) dropped privs to tcpdump 03:43:06.081836 00:00:00:00:00:02 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.2 tell 10.0.0.2, length 28 03:43:08.083924 00:00:00:00:00:02 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.2 tell 10.0.0.2, length 28 03:43:12.088099 00:00:00:00:00:02 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.2 tell 10.0.0.2, length 28 03:43:20.096282 00:00:00:00:00:02 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.2 tell 10.0.0.2, length 28
also Verified on ovn-2021-21.03.0-21.el8fdp.x86_64: [root@wsfd-advnetlab21 bz1939470]# rpm -qa | grep ovn-2021 ovn-2021-host-21.03.0-21.el8fdp.x86_64 ovn-2021-central-21.03.0-21.el8fdp.x86_64 ovn-2021-21.03.0-21.el8fdp.x86_64 + tcpdump -r lsp.pcap -nnle -v arp reading from file lsp.pcap, link-type EN10MB (Ethernet) dropped privs to tcpdump <=== no garp for localport + tcpdump -r lp.pcap -nnle -v arp reading from file lp.pcap, link-type EN10MB (Ethernet) dropped privs to tcpdump 03:45:16.871705 00:00:00:00:00:02 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.2 tell 10.0.0.2, length 28 03:45:18.873820 00:00:00:00:00:02 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.2 tell 10.0.0.2, length 28 03:45:22.878045 00:00:00:00:00:02 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.2 tell 10.0.0.2, length 28 03:45:30.886215 00:00:00:00:00:02 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.2 tell 10.0.0.2, length 28
also verified on ovn2.13-host-20.12.0-104.el7fdp.x86_64: + tcpdump -r lsp.pcap -nnle -v arp reading from file lsp.pcap, link-type EN10MB (Ethernet) <=== no garp for localport 10.0.0.1 + tcpdump -r lp.pcap -nnle -v arp reading from file lp.pcap, link-type EN10MB (Ethernet) 03:52:57.987562 00:00:00:00:00:02 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.2 tell 10.0.0.2, length 28 03:52:59.989752 00:00:00:00:00:02 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.2 tell 10.0.0.2, length 28 03:53:03.993977 00:00:00:00:00:02 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.2 tell 10.0.0.2, length 28 03:53:12.001281 00:00:00:00:00:02 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.2 tell 10.0.0.2, length 28
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (ovn bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:2080