Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
The FDP team is no longer accepting new bugs in Bugzilla. Please report your issues under FDP project in Jira. Thanks.

Bug 1939470

Summary: localports leak gARP when virtual machines are directly attached to a provider network
Product: Red Hat Enterprise Linux Fast Datapath Reporter: Andrea Veri <averi>
Component: ovn2.13Assignee: Daniel Alvarez Sanchez <dalvarez>
Status: CLOSED ERRATA QA Contact: Jianlin Shi <jishi>
Severity: urgent Docs Contact:
Priority: high    
Version: FDP 21.ACC: ctrautma, dalvarez, ihrachys, jishi, nusiddiq, ovnteam, pveiga, ralongi
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-05-20 19:28:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Andrea Veri 2021-03-16 13:08:23 UTC 
Description of problem:

When a virtual machine in Openstack is directly attached against a provider network, we see the MAC associated with the localport flapping across uplinks due to the fact the same interface is present on each compute node which hosts a virtual machine attached to that specific provider network.

Version-Release number of selected component (if applicable):


How reproducible:
100% of the times

Steps to Reproduce:
1. Create a virtual machine and directly attach it to a provider network
2. Create another virtual machine and directly attach it to the same provider network as 1.
3. Fetch the localport used by the metadata service
4. Dump the traffic and verify the gARPs are being generated on each compute node hosting the localport causing

Actual results:

The localport traffic leaks traffic to the provider network it is attached to

Expected results:

The localport traffic should not leak any traffic to provider network

Additional info:
Comment 1 Daniel Alvarez Sanchez 2021-03-16 14:54:57 UTC 
I guess you mean localnet ports and not localports. Can you please confirm?
Comment 2 Andrea Veri 2021-03-16 15:03:29 UTC 
Daniel,

no, I mean localport(s) [1], an example:

    port cc4bd2e8-f224-4a64-acba-d4d9c5e6e5cd
        type: localport
        addresses: ["fa:16:3e:3b:de:2f 10.x.x.x"]


[1] https://patchwork.ozlabs.org/project/openvswitch/patch/1493118328-21311-1-git-send-email-dalvarez@redhat.com/
Comment 3 Daniel Alvarez Sanchez 2021-03-17 08:38:29 UTC 
Thanks Andrea for the clarification.

Those gARPs that you see on the provider network are sent with the source MAC address of the localport? Could you share some captures and match IP/MACs to the OVN ports involved?
Comment 4 Andrea Veri 2021-03-17 14:23:17 UTC 
Daniel,

with the expectation the localport traffic should never leak outside of the localport network namespace:

ARPing running while capturing traffic:

[root@compute-az3-19 ~]# ip netns exec ovnmeta-1e483461-cf04-40f4-80cd-b1c8686742d4 arping -c 4 -A -I tap1e483461-c1 10.2.68.10

While it's running:

[root@compute-az3-19 ~]# ip netns exec ovnmeta-1e483461-cf04-40f4-80cd-b1c8686742d4 tcpdump -i tap1e483461-c1 ether host fa:16:3e:f7:aa:fd or ether host f0:7c:c7:23:61:40 -vneeee
tcpdump: listening on tap1e483461-c1, link-type EN10MB (Ethernet), capture size 262144 bytes
14:03:33.790269 fa:16:3e:f7:aa:fd > Broadcast, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Reply 10.2.68.10 is-at fa:16:3e:f7:aa:fd, length 28
14:03:34.790443 fa:16:3e:f7:aa:fd > Broadcast, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Reply 10.2.68.10 is-at fa:16:3e:f7:aa:fd, length 28
14:03:35.774551 fa:16:3e:f7:aa:fd > 94:f7:ad:b7:84:40, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Reply 10.2.68.10 is-at fa:16:3e:f7:aa:fd, length 28
14:03:35.790614 fa:16:3e:f7:aa:fd > Broadcast, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Reply 10.2.68.10 is-at fa:16:3e:f7:aa:fd, length 28
14:03:35.802690 f0:7c:c7:23:61:40 > Broadcast, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Request who-has 10.2.68.10 tell 10.2.68.252, length 42
14:03:35.802700 fa:16:3e:f7:aa:fd > f0:7c:c7:23:61:40, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Reply 10.2.68.10 is-at fa:16:3e:f7:aa:fd, length 28
14:03:36.449964 fa:16:3e:f7:aa:fd > Broadcast, ethertype Reverse ARP (0x8035), length 60: Ethernet (len 6), IPv4 (len 4), Request who-has 0.0.0.0 (Broadcast) tell 0.0.0.0, length 46
14:03:36.460246 fa:16:3e:f7:aa:fd > Broadcast, ethertype ARP (0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Request who-has 0.0.0.0 (Broadcast) tell 0.0.0.0, length 46
14:03:36.487212 fa:16:3e:f7:aa:fd > Broadcast, ethertype Reverse ARP (0x8035), length 60: Ethernet (len 6), IPv4 (len 4), Request who-has 0.0.0.0 (Broadcast) tell 0.0.0.0, length 46
14:03:36.497924 fa:16:3e:f7:aa:fd > Broadcast, ethertype ARP (0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Request who-has 0.0.0.0 (Broadcast) tell 0.0.0.0, length 46
14:03:36.700567 f0:7c:c7:23:61:40 > Broadcast, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Request who-has 10.2.68.10 tell 10.2.68.252, length 42
14:03:36.700578 fa:16:3e:f7:aa:fd > f0:7c:c7:23:61:40, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Reply 10.2.68.10 is-at fa:16:3e:f7:aa:fd, length 28
14:03:36.790689 fa:16:3e:f7:aa:fd > Broadcast, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Reply 10.2.68.10 is-at fa:16:3e:f7:aa:fd, length 28
14:03:37.300602 f0:7c:c7:23:61:40 > Broadcast, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Request who-has 10.2.68.10 tell 10.2.68.252, length 42
14:03:37.300611 fa:16:3e:f7:aa:fd > f0:7c:c7:23:61:40, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Reply 10.2.68.10 is-at fa:16:3e:f7:aa:fd, length 28
14:03:37.403623 fa:16:3e:f7:aa:fd > 94:f7:ad:b7:84:40, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Reply 10.2.68.10 is-at fa:16:3e:f7:aa:fd, length 28
14:03:37.459353 fa:16:3e:f7:aa:fd > Broadcast, ethertype Reverse ARP (0x8035), length 60: Ethernet (len 6), IPv4 (len 4), Request who-has 0.0.0.0 (Broadcast) tell 0.0.0.0, length 46
14:03:37.471158 fa:16:3e:f7:aa:fd > Broadcast, ethertype ARP (0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Request who-has 0.0.0.0 (Broadcast) tell 0.0.0.0, length 46
14:03:37.900185 f0:7c:c7:23:61:40 > Broadcast, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Request who-has 10.2.68.10 tell 10.2.68.252, length 42
14:03:37.900197 fa:16:3e:f7:aa:fd > f0:7c:c7:23:61:40, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Reply 10.2.68.10 is-at fa:16:3e:f7:aa:fd, length 28
14:03:38.103654 fa:16:3e:f7:aa:fd > 94:f7:ad:b7:84:40, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Reply 10.2.68.10 is-at fa:16:3e:f7:aa:fd, length 28
14:03:38.349935 f0:7c:c7:23:61:40 > Broadcast, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Request who-has 10.2.68.10 tell 10.2.68.252, length 42
14:03:38.349945 fa:16:3e:f7:aa:fd > f0:7c:c7:23:61:40, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Reply 10.2.68.10 is-at fa:16:3e:f7:aa:fd, length 28
14:03:38.402983 fa:16:3e:f7:aa:fd > 94:f7:ad:b7:84:40, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Reply 10.2.68.10 is-at fa:16:3e:f7:aa:fd, length 28
14:03:39.449457 f0:7c:c7:23:61:40 > Broadcast, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Request who-has 10.2.68.180 tell 10.2.68.252, length 42
14:03:44.376384 f0:7c:c7:23:61:40 > Broadcast, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Request who-has 10.2.68.222 tell 10.2.68.252, length 42
14:03:44.551100 f0:7c:c7:23:61:40 > Broadcast, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Request who-has 10.2.68.72 tell 10.2.68.252, length 42
14:03:47.151511 f0:7c:c7:23:61:40 > Broadcast, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Request who-has 10.2.68.52 tell 10.2.68.252, length 42

f0:7c:c7:23:61:40 being the router NIC MAC, fa:16:3e:f7:aa:fd being the localport MAC:

    port 16ef1dd5-42df-4579-949a-b087cdaef2a3
        type: localport
        addresses: ["fa:16:3e:f7:aa:fd 10.2.68.10"]

()[root@controller-1 ~]# ovn-sbctl list port_binding 16ef1dd5-42df-4579-949a-b087cdaef2a3
_uuid               : 6c57af29-6c47-44d3-b196-f057d126723f
chassis             : fc5e634d-2db0-431e-806a-f8f7ed97a9e3
datapath            : 1e483461-cf04-40f4-80cd-b1c8686742d4
encap               : []
external_ids        : {"neutron:cidrs"="10.2.68.10/24", "neutron:device_id"=ovnmeta-36d02a00-988c-431d-a389-e24edb9edbe1, "neutron:device_owner"="network:dhcp", "neutron:network_name"=neutron-36d02a00-988c-431d-a389-e24edb9edbe1, "neutron:port_name"="", "neutron:project_id"=d2b4103f9246414193891c61f4482897, "neutron:revision_number"="2", "neutron:security_group_ids"=""}
gateway_chassis     : []
ha_chassis_group    : []
logical_port        : "16ef1dd5-42df-4579-949a-b087cdaef2a3"
mac                 : ["fa:16:3e:f7:aa:fd 10.2.68.10"]
nat_addresses       : []
options             : {requested-chassis=""}
parent_port         : []
tag                 : []
tunnel_key          : 2
type                : localport
up                  : false
virtual_parent      : []

Please mark this comment as confidential.
Comment 7 Daniel Alvarez Sanchez 2021-03-23 13:58:21 UTC 
I sent this patch that seems to work, would require review from the core OVN team and backports:

http://patchwork.ozlabs.org/project/ovn/patch/20210323135559.14959-1-dalvarez@redhat.com/

Unfortunately I won't have the time to work on the delivery of the fix. Please core OVN team take over as time permits based on the urgency.

Thanks!
Comment 10 Priscila 2021-04-06 14:15:08 UTC 
Can we please get confirmation that this issue will be fixed in 16.1.5?
Comment 14 Jianlin Shi 2021-04-25 07:44:39 UTC 
tested with following script:

#!/bin/bash

systemctl start openvswitch                                                
systemctl start ovn-northd                                
ovn-nbctl set-connection ptcp:6641                                                          
ovn-sbctl set-connection ptcp:6642                                    
ovs-vsctl set open . external_ids:system-id=hv1 external_ids:ovn-remote=tcp:20.0.175.25:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=20.0.175.25
systemctl restart ovn-controller                                      
                                                                
ovs-vsctl add-br br-phys
ip link set br-phys up
                                                                   

ovs-vsctl set open . external-ids:ovn-bridge-mappings=phys:br-phys
ovn-nbctl ls-add ls \
    -- lsp-add ls lp \
    -- lsp-set-type lp localport \
    -- lsp-set-addresses lp "00:00:00:00:00:01 10.0.0.1" \
    -- lsp-add ls ln \
    -- lsp-set-type ln localnet \
    -- lsp-set-options ln network_name=phys \
    -- lsp-add ls lsp \
    -- lsp-set-addresses lsp "00:00:00:00:00:02 10.0.0.2"


ovs-vsctl add-port br-int lp -- set interface lp type=internal external_ids:iface-id=lp
ip netns add lp
ip link set lp netns lp
ip netns exec lp ip link set lp address 00:00:00:00:00:01
ip netns exec lp ip link set lp up
ip netns exec lp tcpdump -i lp -w lp.pcap &
ip netns exec lp ip addr add 10.0.0.1/24 dev lp

ovn-nbctl --wait=hv sync

ovs-vsctl add-port br-int lsp -- set interface lsp type=internal external_ids:iface-id=lsp options:tx_pcap=lsp.pcap options:rxq_pcap=lsp-rx.pcap
ip netns add lsp
ip link set lsp netns lsp
ip netns exec lsp ip link set lsp address 00:00:00:00:00:02
ip netns exec lsp ip link set lsp up
ip netns exec lsp tcpdump -i lsp -w lsp.pcap &
ip netns exec lsp ip addr add 10.0.0.2/24 dev lsp

ovn-nbctl --wait=hv sync
sleep 30
pkill tcpdump
sleep 2
tcpdump -r lsp.pcap -nnle -v arp
tcpdump -r lp.pcap -nnle -v arp

reproduced on 20.12.0-95:

[root@wsfd-advnetlab21 bz1939470]# rpm -qa | grep ovn2.13
ovn2.13-20.12.0-95.el8fdp.x86_64
ovn2.13-central-20.12.0-95.el8fdp.x86_64
ovn2.13-host-20.12.0-95.el8fdp.x86_64

+ tcpdump -r lsp.pcap -nnle -v arp
reading from file lsp.pcap, link-type EN10MB (Ethernet)
dropped privs to tcpdump          
03:40:27.354845 00:00:00:00:00:01 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.1 tell 10.0.0.1, length 28
03:40:29.357062 00:00:00:00:00:01 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.1 tell 10.0.0.1, length 28
03:40:33.361292 00:00:00:00:00:01 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.1 tell 10.0.0.1, length 28
03:40:41.369500 00:00:00:00:00:01 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.1 tell 10.0.0.1, length 28

<=== garp for localport

+ tcpdump -r lp.pcap -nnle -v arp
reading from file lp.pcap, link-type EN10MB (Ethernet)   
dropped privs to tcpdump
03:40:27.354697 00:00:00:00:00:02 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.2 tell 10.0.0.2, length 28
03:40:29.356921 00:00:00:00:00:02 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.2 tell 10.0.0.2, length 28
03:40:33.361109 00:00:00:00:00:02 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.2 tell 10.0.0.2, length 28
03:40:41.369320 00:00:00:00:00:02 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.2 tell 10.0.0.2, length 28

Verified on 20.12.0-104:

[root@wsfd-advnetlab21 bz1939470]# rpm -qa | grep ovn2.13
ovn2.13-central-20.12.0-104.el8fdp.x86_64
ovn2.13-20.12.0-104.el8fdp.x86_64
ovn2.13-host-20.12.0-104.el8fdp.x86_64

+ tcpdump -r lsp.pcap -nnle -v arp
reading from file lsp.pcap, link-type EN10MB (Ethernet)
dropped privs to tcpdump

<=== no garp for localport

+ tcpdump -r lp.pcap -nnle -v arp
reading from file lp.pcap, link-type EN10MB (Ethernet)
dropped privs to tcpdump
03:43:06.081836 00:00:00:00:00:02 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.2 tell 10.0.0.2, length 28
03:43:08.083924 00:00:00:00:00:02 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.2 tell 10.0.0.2, length 28
03:43:12.088099 00:00:00:00:00:02 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.2 tell 10.0.0.2, length 28
03:43:20.096282 00:00:00:00:00:02 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.2 tell 10.0.0.2, length 28
Comment 15 Jianlin Shi 2021-04-25 07:47:35 UTC 
also Verified on ovn-2021-21.03.0-21.el8fdp.x86_64:

[root@wsfd-advnetlab21 bz1939470]# rpm -qa | grep ovn-2021
ovn-2021-host-21.03.0-21.el8fdp.x86_64
ovn-2021-central-21.03.0-21.el8fdp.x86_64
ovn-2021-21.03.0-21.el8fdp.x86_64

+ tcpdump -r lsp.pcap -nnle -v arp
reading from file lsp.pcap, link-type EN10MB (Ethernet)
dropped privs to tcpdump

<=== no garp for localport

+ tcpdump -r lp.pcap -nnle -v arp
reading from file lp.pcap, link-type EN10MB (Ethernet)
dropped privs to tcpdump
03:45:16.871705 00:00:00:00:00:02 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.2 tell 10.0.0.2, length 28
03:45:18.873820 00:00:00:00:00:02 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.2 tell 10.0.0.2, length 28
03:45:22.878045 00:00:00:00:00:02 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.2 tell 10.0.0.2, length 28
03:45:30.886215 00:00:00:00:00:02 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.2 tell 10.0.0.2, length 28
Comment 16 Jianlin Shi 2021-04-25 07:54:50 UTC 
also verified on ovn2.13-host-20.12.0-104.el7fdp.x86_64:

+ tcpdump -r lsp.pcap -nnle -v arp
reading from file lsp.pcap, link-type EN10MB (Ethernet)

<=== no garp for localport 10.0.0.1

+ tcpdump -r lp.pcap -nnle -v arp
reading from file lp.pcap, link-type EN10MB (Ethernet)
03:52:57.987562 00:00:00:00:00:02 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.2 tell 10.0.0.2, length 28
03:52:59.989752 00:00:00:00:00:02 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.2 tell 10.0.0.2, length 28
03:53:03.993977 00:00:00:00:00:02 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.2 tell 10.0.0.2, length 28
03:53:12.001281 00:00:00:00:00:02 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.0.2 tell 10.0.0.2, length 28
Comment 18 errata-xmlrpc 2021-05-20 19:28:16 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (ovn bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:2080