Bug 2168087

Summary: it takes too long for mirror to take effect after change filter for mirror
Product: Red Hat Enterprise Linux Fast Datapath Reporter: Jianlin Shi <jishi>
Component: openvswitch2.17Assignee: Kevin Traynor <ktraynor>
Status: ASSIGNED --- QA Contact: Hekai Wang <hewang>
Severity: medium Docs Contact:
Priority: medium    
Version: FDP 22.ACC: amusil, ctrautma, fleitner, jhsiao, jiji, ralongi
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
reproducer.sh none

Description Jianlin Shi 2023-02-08 07:27:20 UTC
Description of problem:
set filter as to-lport for mirror, then change filter to from-lport, but it takes too long after change the filter

Version-Release number of selected component (if applicable):
ovn22.12-22.12.0-20.el8

How reproducible:
Always

Steps to Reproduce:
1. setup ovn on one server
systemctl start openvswitch
systemctl start ovn-northd 
ovn-nbctl set-connection ptcp:6641                                       
ovn-sbctl set-connection ptcp:6642
ovs-vsctl set open . external_ids:system-id=hv1 external_ids:ovn-remote=tcp:20.0.204.25:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=20.0.204.25
systemctl restart ovn-controller
                                                                                                
ovn-nbctl ls-add ls1                                                                   
ovn-nbctl lsp-add ls1 ls1p1                                                                     
ovn-nbctl lsp-set-addresses ls1p1 "00:00:00:01:01:01 192.168.1.1 2001::1"
ovn-nbctl lsp-add ls1 ls1p2 
ovn-nbctl lsp-set-addresses ls1p2 "00:00:00:01:01:02 192.168.1.2 2001::2"
ovn-nbctl lsp-add ls1 lp                                       
ovn-nbctl lsp-set-type lp localport     
ovn-nbctl lsp-set-addresses lp "00:00:00:01:01:11 192.168.1.11 2001::11"
                                                    
ovn-nbctl lr-add lr1                                                
ovn-nbctl lrp-add lr1 lr1-ls1 00:00:00:00:00:01 192.168.1.254/24 2001::a/64
ovn-nbctl lsp-add ls1 ls1-lr1
ovn-nbctl lsp-set-addresses ls1-lr1 "00:00:00:00:00:01 192.168.1.254 2001::a"
ovn-nbctl lsp-set-type ls1-lr1 router
ovn-nbctl lsp-set-options ls1-lr1 router-port=lr1-ls1    
                                  
ovn-nbctl lrp-add lr1 lr1-ls2 00:00:00:00:00:02 192.168.2.254/24 2002::a/64
                                               
ovn-nbctl ls-add ls2                                          
ovn-nbctl lsp-add ls2 ls2-lr1                              
ovn-nbctl lsp-set-addresses ls2-lr1 "00:00:00:00:00:02 192.168.2.254 2002::a"
ovn-nbctl lsp-set-type ls2-lr1 router
ovn-nbctl lsp-set-options ls2-lr1 router-port=lr1-ls2
                                                               
ovn-nbctl lsp-add ls2 ls2p1             
ovn-nbctl lsp-set-addresses ls2p1 "00:00:00:01:02:01 192.168.2.1 2002::1"
ovn-nbctl lsp-add ls2 ls2p2                         
ovn-nbctl lsp-set-addresses ls2p2 "00:00:00:01:02:02 192.168.2.2 2002::2"
                                                                 
ovs-vsctl add-port br-int ls1p1 -- set interface ls1p1 type=internal external_ids:iface-id=ls1p1
ovs-vsctl add-port br-int lp -- set interface lp type=internal external_ids:iface-id=lp
ovs-vsctl add-port br-int ls2p1 -- set interface ls2p1 type=internal external_ids:iface-id=ls2p1
                                                                                
ip netns add ls1p1                     
ip link set ls1p1 netns ls1p1              
ip netns exec ls1p1 ip link set ls1p1 address 00:00:00:01:01:01
ip netns exec ls1p1 ip link set ls1p1 up
ip netns exec ls1p1 ip addr add 192.168.1.1/24 dev ls1p1
ip netns exec ls1p1 ip addr add 2001::1/64 dev ls1p1
ip netns exec ls1p1 ip route add default via 192.168.1.254 dev ls1p1
ip netns exec ls1p1 ip -6 route add default via 2001::a dev ls1p1

ip netns add lp
ip link set lp netns lp
ip netns exec lp ip link set lp address 00:00:00:01:01:11
ip netns exec lp ip link set lp up
ip netns exec lp ip addr add 192.168.1.11/24 dev lp
ip netns exec lp ip addr add 2001::11/64 dev lp
ip netns exec lp ip route add default via 192.168.1.254 dev lp
ip netns exec lp ip -6 route add default via 2001::a dev lp 

ip netns add ls2p1
ip link set ls2p1 netns ls2p1
ip netns exec ls2p1 ip link set ls2p1 address 00:00:00:01:02:01
ip netns exec ls2p1 ip link set ls2p1 up
ip netns exec ls2p1 ip addr add 192.168.2.1/24 dev ls2p1
ip netns exec ls2p1 ip addr add 2002::1/64 dev ls2p1
ip netns exec ls2p1 ip route add default via 192.168.2.254 dev ls2p1
ip netns exec ls2p1 ip -6 route add default via 2002::a dev ls2p1

2. add mirror
ovn-nbctl mirror-add mirror0 gre 123 to-lport 20.0.204.26
ovn-nbctl lsp-attach-mirror ls1p1 mirror0
ovn-nbctl --wait=hv sync

3. add gretap on the other server
ip link add gretap1 type gretap local 20.0.204.26 remote 20.0.204.25 key 123
ip link set gretap1 up

4. check mirror
ip netns exec ls1p1 ping 192.168.1.11 -c 1 
tcpdump  -i gretap1 -nnle -v icmp
02:17:28.371922 00:00:00:01:01:11 > 00:00:00:01:01:01, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 60537, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.1.11 > 192.168.1.1: ICMP echo reply, id 2665, seq 1, length 64 

5. change filter for mirror to from-lport
mirror_uuid=$(ovn-nbctl find mirror name=mirror0 | awk '/_uuid/{print $3}')                           
ovn-nbctl set mirror $mirror_uuid filter=from-lport                                                   
for i in {1..20}                                                                                      
do                                                                                                    
ip netns exec ls1p1 ping 192.168.1.11 -c 1                                                            
sleep 5                                                                                               
done 

6. check packet with tcpdump
[root@wsfd-advnetlab18 mirror]# tcpdump  -i gretap1 -nnle -v icmp                                     
dropped privs to tcpdump                                                                                                                                                                                    
tcpdump: listening on gretap1, link-type EN10MB (Ethernet), capture size 262144 bytes                 
02:17:28.371922 00:00:00:01:01:11 > 00:00:00:01:01:01, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 60537, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.1.11 > 192.168.1.1: ICMP echo reply, id 2665, seq 1, length 64                            
02:17:29.250477 00:00:00:01:01:11 > 00:00:00:01:01:01, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 61118, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.1.11 > 192.168.1.1: ICMP echo reply, id 2666, seq 1, length 64                            
02:17:32.546959 00:00:00:01:01:11 > 00:00:00:01:01:01, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 62853, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.1.11 > 192.168.1.1: ICMP echo reply, id 2672, seq 1, length 64                            
02:17:37.584544 00:00:00:01:01:11 > 00:00:00:01:01:01, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 65412, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.1.11 > 192.168.1.1: ICMP echo reply, id 2676, seq 1, length 64                            
02:17:42.614511 00:00:00:01:01:11 > 00:00:00:01:01:01, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 142, offset 0, flags [none], proto ICMP (1), length 84)  
    192.168.1.11 > 192.168.1.1: ICMP echo reply, id 2680, seq 1, length 64                            
02:17:47.646966 00:00:00:01:01:11 > 00:00:00:01:01:01, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 617, offset 0, flags [none], proto ICMP (1), length 84)  
    192.168.1.11 > 192.168.1.1: ICMP echo reply, id 2684, seq 1, length 64                            
02:17:52.681054 00:00:00:01:01:11 > 00:00:00:01:01:01, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 2012, offset 0, flags [none], proto ICMP (1), length 84) 
    192.168.1.11 > 192.168.1.1: ICMP echo reply, id 2688, seq 1, length 64                            
02:17:57.709008 00:00:00:01:01:11 > 00:00:00:01:01:01, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 2347, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.1.11 > 192.168.1.1: ICMP echo reply, id 2692, seq 1, length 64                            
02:18:02.747034 00:00:00:01:01:11 > 00:00:00:01:01:01, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 3062, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.1.11 > 192.168.1.1: ICMP echo reply, id 2696, seq 1, length 64                            
02:18:07.784011 00:00:00:01:01:11 > 00:00:00:01:01:01, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 5779, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.1.11 > 192.168.1.1: ICMP echo reply, id 2700, seq 1, length 64                            
02:18:12.817048 00:00:00:01:01:11 > 00:00:00:01:01:01, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 6141, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.1.11 > 192.168.1.1: ICMP echo reply, id 2704, seq 1, length 64                            
02:18:17.850033 00:00:00:01:01:11 > 00:00:00:01:01:01, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 9363, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.1.11 > 192.168.1.1: ICMP echo reply, id 2708, seq 1, length 64                            
02:18:22.884017 00:00:00:01:01:11 > 00:00:00:01:01:01, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 12161, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.1.11 > 192.168.1.1: ICMP echo reply, id 2712, seq 1, length 64                            
02:18:27.917016 00:00:00:01:01:11 > 00:00:00:01:01:01, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 14441, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.1.11 > 192.168.1.1: ICMP echo reply, id 2716, seq 1, length 64                            
02:18:32.951096 00:00:00:01:01:11 > 00:00:00:01:01:01, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 16715, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.1.11 > 192.168.1.1: ICMP echo reply, id 2720, seq 1, length 64                            
02:18:37.985458 00:00:00:01:01:11 > 00:00:00:01:01:01, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 17750, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.1.11 > 192.168.1.1: ICMP echo reply, id 2724, seq 1, length 64                            
02:18:43.013578 00:00:00:01:01:11 > 00:00:00:01:01:01, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 18950, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.1.11 > 192.168.1.1: ICMP echo reply, id 2728, seq 1, length 64                            
02:18:48.051965 00:00:00:01:01:11 > 00:00:00:01:01:01, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 23972, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.1.11 > 192.168.1.1: ICMP echo reply, id 2732, seq 1, length 64                            
02:18:53.081956 00:00:00:01:01:11 > 00:00:00:01:01:01, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 24981, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.1.11 > 192.168.1.1: ICMP echo reply, id 2736, seq 1, length 64                            
02:18:58.120043 00:00:00:01:01:11 > 00:00:00:01:01:01, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 25394, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.1.11 > 192.168.1.1: ICMP echo reply, id 2740, seq 1, length 64                            
02:19:03.151521 00:00:00:01:01:11 > 00:00:00:01:01:01, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 26137, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.1.11 > 192.168.1.1: ICMP echo reply, id 2744, seq 1, length 64                            
02:19:08.186065 00:00:00:01:01:11 > 00:00:00:01:01:01, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 31019, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.1.11 > 192.168.1.1: ICMP echo reply, id 2748, seq 1, length 64 

<=== still get the packet coming into ls1p1

Actual results:
still get the packet coming into ls1p1

Expected results:
should get the packet going out of ls1p1

Additional info:


[root@wsfd-advnetlab17 mirror]# rpm -qa | grep -E "openvswitch2.17|ovn22.12"
python3-openvswitch2.17-2.17.0-74.el8fdp.x86_64
ovn22.12-central-22.12.0-20.el8fdp.x86_64
ovn22.12-22.12.0-20.el8fdp.x86_64
ovn22.12-host-22.12.0-20.el8fdp.x86_64
openvswitch2.17-2.17.0-74.el8fdp.x86_64

Comment 1 Ales Musil 2023-04-18 11:32:09 UTC
Hi,

it is indeed an issue, however it seems that the issue is within OvS.
The OvS database is updated with the right information but the flows installed
are the old ones:

recirc_id(0),in_port(5),eth(src=00:00:00:01:01:01,dst=00:00:00:01:01:11),eth_type(0x0800),ipv4(tos=0/0x3,frag=no), packets:249, bytes:24402, used:0.333s, actions:set(tunnel(tun_id=0x7b,dst=10.88.100.71,ttl=64,flags(df|key))),7,6

vs.

recirc_id(0),in_port(6),eth(src=00:00:00:01:01:11,dst=00:00:00:01:01:01),eth_type(0x0800),ipv4(tos=0/0x3,frag=no), packets:6, bytes:588, used:0.472s, actions:5,set(tunnel(tun_id=0x7b,dst=10.88.100.71,ttl=64,flags(df|key))),7

switch the component to OvS.

Comment 2 Ales Musil 2023-04-18 11:33:07 UTC
Created attachment 1957958 [details]
reproducer.sh

Comment 3 Flavio Leitner 2023-04-18 14:43:08 UTC
Hi Kevin,

Perhaps OVS is missing to flush the datapath flows once the mirror config is changed.
fbl