Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
The FDP team is no longer accepting new bugs in Bugzilla. Please report your issues under FDP project in Jira. Thanks.

Bug 2094566

Summary: IPv6 related datapath flows are evicted and re-added by revalidator threads
Product: Red Hat Enterprise Linux Fast Datapath Reporter: OvS team <ovs-bugzilla>
Component: openvswitch2.17Assignee: Aaron Conole <aconole>
Status: CLOSED ERRATA QA Contact: Rick Alongi <ralongi>
Severity: urgent Docs Contact:
Priority: urgent    
Version: FDP 21.KCC: ctrautma, jhsiao, ralongi, yingwang
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openvswitch2.17-2.17.0-23.el8fdp Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-09-06 18:49:03 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2018001    

Description OvS team 2022-06-07 22:11:28 UTC 
+++ This bug was initially created as a clone of Bug #2081773 +++

Description of problem:

ovs-vswitchd is evicting and re-adding the IPv6 related datapath flows whenever revalidation is triggered.



Version-Release number of selected component (if applicable):


How reproducible:
All the time.


Steps to Reproduce:

# Reproducer script

Start openvswitch


--------------------------------------

# Program the below flows in br-int

cat << EOF > flows.txt
table=0, priority=120 ipv6,icmp6,nw_ttl=255,icmp_type=135,icmp_code=0,nd_target=1000::10 actions=NORMAL
table=0, priority=0 actions=NORMAL
EOF

ovs-vsctl del-br br-int || :
ovs-vsctl add-br br-int
ovs-ofctl add-flows br-int flows.txt

create_fake_vm() {
    iface_id=$1
    name=$2
    mac=$3
    ip=$4
    mask=$5
    gw=$6
    ipv6_addr=$7
    ipv6_gw=$8
    ip netns add $name
    ovs-vsctl       -- add-port br-int $name       -- set interface $name type=internal       -- set Interface $name external_ids:iface-id=$iface_id
    ip link set $name netns $name
    ip netns exec $name ip link set lo up
    [ -n "$mac" ] && ip netns exec $name ip link set $name address $mac
    ip netns exec $name ip addr add $ip/$mask dev $name
    ip netns exec $name ip addr add $ipv6_addr dev $name
    ip netns exec $name ip link set $name up
    ip netns exec $name ip route add default via $gw dev $name
    ip netns exec $name ip -6 route add default via $ipv6_gw dev $name
}

ip netns delete p1 || :
ip netns delete p2 || :

# Create namespaces for p1 and p2 fake VMs.
create_fake_vm sw0-port1 p1 50:54:00:00:00:03 10.0.0.3 24 10.0.0.1 1000::3/64 1000::1
create_fake_vm sw0-port2 p2 50:54:00:00:00:04 10.0.0.4 24 10.0.0.1 1000::4/64 1000::1

# Start nc server on p1
# ip netns exec p1 nc -l -k -v 1000::3 8080

# Connect to the server from p2
# ip netns exec p2 nc 1000::3 8080

# In a separate terminal, trigger re validation

cat << EOF > trigger_dp_issue.sh
#!/bin/bash

ovs-vsctl --may-exist add-br brtmp

while true
do
    ovs-ofctl add-flow brtmp "table=0,priority=100,ip,udp actions=NORMAL"
    sleep 5
    ovs-ofctl del-flows brtmp
done

EOF

chmod 0755 trigger_dp_issue.sh

# On another session monitor datapath flows.

cat << EOF > monitor_dp_flows.sh
#!/bin/bash

watch -n1 "ovs-dpctl dump-flows -m"
EOF

chmod 0755 monitor_dp_flows.sh

--------------------------------------

Keep sending the traffic from p1 and p2.  And you'd notice that the datapath flows gets evicted and re-added with the same match.

This results in performance issues as there will be an upcall whenever revalidation threads kick-in.



Actual results:


Expected results:


Additional info:
Comment 1 OvS team 2022-06-07 22:11:31 UTC 
* Tue Jun 07 2022 Open vSwitch CI <ovs-ci> - 2.17.0-23
- Merging upstream branch-2.17 [RH git: 58de71ec1f]
    Commit list:
    250e1a6dd2 ofproto-dpif-xlate: Fix internal CT state for non-recirc traffic.
    fe870ee072 classifier: Adjust segment boundary to execute prerequisite processing. (#2081773)
    ec0ec464ba ovs-tcpdump: Fix error when stopping ovs-tcpdump.
Comment 4 Rick Alongi 2022-08-08 16:04:09 UTC 
# Reproduce issue using openvswitch2.17-2.17.0-19.el8fdp:

[root@netqe9 ~]# rpm -q openvswitch2.17
openvswitch2.17-2.17.0-19.el8fdp.x86_64

# Terminal 1:

[root@netqe9 ~]# cat setup_dp_issue.sh 
dnf -y install $RPM_OVS

systemctl restart openvswitch
cat << EOF > flows.txt
table=0, priority=120 ipv6,icmp6,nw_ttl=255,icmp_type=135,icmp_code=0,nd_target=1000::10 actions=NORMAL
table=0, priority=0 actions=NORMAL
EOF

ovs-vsctl del-br br-int || :
ovs-vsctl add-br br-int
ovs-ofctl add-flows br-int flows.txt

create_fake_vm() {
    iface_id=$1
    name=$2
    mac=$3
    ip=$4
    mask=$5
    gw=$6
    ipv6_addr=$7
    ipv6_gw=$8
    ip netns add $name
    ovs-vsctl       -- add-port br-int $name       -- set interface $name type=internal       -- set Interface $name external_ids:iface-id=$iface_id
    ip link set $name netns $name
    ip netns exec $name ip link set lo up
    [ -n "$mac" ] && ip netns exec $name ip link set $name address $mac
    ip netns exec $name ip addr add $ip/$mask dev $name
    ip netns exec $name ip addr add $ipv6_addr dev $name
    ip netns exec $name ip link set $name up
    ip netns exec $name ip route add default via $gw dev $name
    ip netns exec $name ip -6 route add default via $ipv6_gw dev $name
}

ip netns delete p1 || :
ip netns delete p2 || :

# Create namespaces for p1 and p2 fake VMs.
create_fake_vm sw0-port1 p1 50:54:00:00:00:03 10.0.0.3 24 10.0.0.1 1000::3/64 1000::1
create_fake_vm sw0-port2 p2 50:54:00:00:00:04 10.0.0.4 24 10.0.0.1 1000::4/64 1000::1

sleep 10

ip netns exec p1 nc -l -k -v 1000::3 8080

# Execute: ./setup_dp_issue.sh

# In Terminal 2:

ip netns exec p2 nc 1000::3 8080

# Terminal 3:

[root@netqe9 ~]# cat trigger_dp_issue.sh 
#!/bin/bash

ovs-vsctl --may-exist add-br brtmp

while true
do
    ovs-ofctl add-flow brtmp "table=0,priority=100,ip,udp actions=NORMAL"
    sleep 1
    ovs-ofctl del-flows brtmp
done

# Execute: ./trigger_dp_issue.sh

# Terminal 4

[root@netqe9 ~]# cat monitor_dp_flows.sh 
#!/bin/bash

#watch -n1 "ovs-dpctl dump-flows -m" | tee -a $output_file
rm -f output.txt
while [ 1 ]; do sleep 1; ovs-dpctl dump-flows -m | tee -a output.txt; done

# Execute: ./monitor_dp_flows.sh

# Go back to Terminal 2 and enter any characters on console to send data onver nc connection 

[root@netqe9 ~]# ip netns exec p2 nc 1000::3 8080
zhgdfhg
dfhgdfh
dzhgfdfgh
zdfhdfh
dfzhdfghdgfh
zdfhdfhgdfgh

# Go back to Terminal 4 and kill monitor_dp_flows.sh

# Check for "packets" field to confirm that the packets are not incrementing (this is the problem):

[root@netqe9 ~]# grep 'packets:' output.txt 
ufid:39843c39-9ff2-4281-9c4d-1a39c6503a29, recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(p1),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=50:54:00:00:00:03,dst=50:54:00:00:00:04),eth_type(0x86dd),ipv6(src=::/::,dst=::/::,label=0/0,proto=6,tclass=0/0,hlimit=64,frag=no),tcp(src=0/0,dst=0/0),tcp_flags(0/0), packets:0, bytes:0, used:never, dp:ovs, actions:p2
ufid:2104c4c8-b4f8-4924-8d19-b9f70c3c6227, recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(p2),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=50:54:00:00:00:04,dst=50:54:00:00:00:03),eth_type(0x86dd),ipv6(src=::/::,dst=::/::,label=0/0,proto=6,tclass=0/0,hlimit=64,frag=no),tcp(src=0/0,dst=0/0),tcp_flags(0/0), packets:0, bytes:0, used:never, dp:ovs, actions:p1
ufid:39843c39-9ff2-4281-9c4d-1a39c6503a29, recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(p1),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=50:54:00:00:00:03,dst=50:54:00:00:00:04),eth_type(0x86dd),ipv6(src=::/::,dst=::/::,label=0/0,proto=6,tclass=0/0,hlimit=64,frag=no),tcp(src=0/0,dst=0/0),tcp_flags(0/0), packets:0, bytes:0, used:never, dp:ovs, actions:p2
ufid:2104c4c8-b4f8-4924-8d19-b9f70c3c6227, recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(p2),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=50:54:00:00:00:04,dst=50:54:00:00:00:03),eth_type(0x86dd),ipv6(src=::/::,dst=::/::,label=0/0,proto=6,tclass=0/0,hlimit=64,frag=no),tcp(src=0/0,dst=0/0),tcp_flags(0/0), packets:0, bytes:0, used:never, dp:ovs, actions:p1
ufid:69da6fdf-1e84-40f0-b311-735e6d555d8c, recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(p1),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=50:54:00:00:00:03,dst=50:54:00:00:00:04),eth_type(0x86dd),ipv6(src=::/::,dst=::/::,label=0/0,proto=58,tclass=0/0,hlimit=255,frag=no),icmpv6(type=0/0,code=0/0), packets:0, bytes:0, used:never, dp:ovs, actions:p2
ufid:84ae0734-330f-4944-9d4b-6325b719c714, recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(p2),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=50:54:00:00:00:04,dst=50:54:00:00:00:03),eth_type(0x86dd),ipv6(src=::/::,dst=::/::,label=0/0,proto=58,tclass=0/0,hlimit=255,frag=no),icmpv6(type=0/0,code=0/0), packets:0, bytes:0, used:never, dp:ovs, actions:p1

# Note that all of the packets show zero.

# Verify the fix:

# Update to openvswitch2.17-2.17.0-37.el8fdp
dnf -y update http://download-node-02.eng.bos.redhat.com/brewroot/packages/openvswitch2.17/2.17.0/37.el8fdp/x86_64/openvswitch2.17-2.17.0-37.el8fdp.x86_64.rpm

systemctl restart openvswitch

# Follow the steps from above

# Note that the packets are now incrementing as they should with the fix:

ufid:8e068d4c-21ef-46aa-b84f-f23e09b2cbcc, recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(p2),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=50:54:00:00:00:04,dst=50:54:00:00:00:03),eth_type(0x86dd),ipv6(src=::/::,dst=::/::,label=0/0,proto=58,tclass=0/0,hlimit=255,frag=no),icmpv6(type=136/0xf8,code=0/0),nd(target=::/::,sll=00:00:00:00:00:00/00:00:00:00:00:00,tll=00:00:00:00:00:00/00:00:00:00:00:00), packets:1, bytes:78, used:10.226s, dp:ovs, actions:p1
ufid:c9e04616-ac9b-441c-9df3-ad2410d146e9, recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(p1),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=50:54:00:00:00:03,dst=33:33:00:00:00:02),eth_type(0x86dd),ipv6(src=::/::,dst=::/::,label=0/0,proto=58,tclass=0/0,hlimit=255,frag=no),icmpv6(type=132/0xfe,code=0/0), packets:0, bytes:0, used:never, dp:ovs, actions:br-int,p2
ufid:ac074ff2-6fc2-40be-8dfa-36b82b80daba, recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(p1),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=50:54:00:00:00:03,dst=50:54:00:00:00:04),eth_type(0x86dd),ipv6(src=::/::,dst=::/::,label=0/0,proto=58,tclass=0/0,hlimit=255,frag=no),icmpv6(type=135,code=0),nd(target=::/::,sll=00:00:00:00:00:00/00:00:00:00:00:00,tll=00:00:00:00:00:00/00:00:00:00:00:00), packets:1, bytes:86, used:10.226s, dp:ovs, actions:p2
ufid:8d646148-494a-443c-82bc-e8fb166e6281, recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(p2),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=50:54:00:00:00:04,dst=50:54:00:00:00:03),eth_type(0x86dd),ipv6(src=::/::,dst=::/::,label=0/0,proto=6,tclass=0/0,hlimit=64,frag=no),tcp(src=0/0,dst=0/0),tcp_flags(0/0), packets:1, bytes:98, used:0.011s, flags:P., dp:ovs, actions:p1
ufid:61ef0a62-c67f-42d0-9acc-f6ac6ad30bbd, recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(p1),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=50:54:00:00:00:03,dst=50:54:00:00:00:04),eth_type(0x86dd),ipv6(src=::/::,dst=::/::,label=0/0,proto=6,tclass=0/0,hlimit=64,frag=no),tcp(src=0/0,dst=0/0),tcp_flags(0/0), packets:1, bytes:86, used:0.011s, flags:., dp:ovs, actions:p2
ufid:c9e04616-ac9b-441c-9df3-ad2410d146e9, recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(p1),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=50:54:00:00:00:03,dst=33:33:00:00:00:02),eth_type(0x86dd),ipv6(src=::/::,dst=::/::,label=0/0,proto=58,tclass=0/0,hlimit=255,frag=no),icmpv6(type=132/0xfe,code=0/0), packets:0, bytes:0, used:never, dp:ovs, actions:br-int,p2
ufid:8d646148-494a-443c-82bc-e8fb166e6281, recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(p2),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=50:54:00:00:00:04,dst=50:54:00:00:00:03),eth_type(0x86dd),ipv6(src=::/::,dst=::/::,label=0/0,proto=6,tclass=0/0,hlimit=64,frag=no),tcp(src=0/0,dst=0/0),tcp_flags(0/0), packets:2, bytes:188, used:0.011s, flags:P., dp:ovs, actions:p1
ufid:61ef0a62-c67f-42d0-9acc-f6ac6ad30bbd, recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(p1),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=50:54:00:00:00:03,dst=50:54:00:00:00:04),eth_type(0x86dd),ipv6(src=::/::,dst=::/::,label=0/0,proto=6,tclass=0/0,hlimit=64,frag=no),tcp(src=0/0,dst=0/0),tcp_flags(0/0), packets:2, bytes:172, used:0.011s, flags:., dp:ovs, actions:p2
ufid:c9e04616-ac9b-441c-9df3-ad2410d146e9, recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(p1),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=50:54:00:00:00:03,dst=33:33:00:00:00:02),eth_type(0x86dd),ipv6(src=::/::,dst=::/::,label=0/0,proto=58,tclass=0/0,hlimit=255,frag=no),icmpv6(type=132/0xfe,code=0/0), packets:0, bytes:0, used:never, dp:ovs, actions:br-int,p2
ufid:8d646148-494a-443c-82bc-e8fb166e6281, recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(p2),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=50:54:00:00:00:04,dst=50:54:00:00:00:03),eth_type(0x86dd),ipv6(src=::/::,dst=::/::,label=0/0,proto=6,tclass=0/0,hlimit=64,frag=no),tcp(src=0/0,dst=0/0),tcp_flags(0/0), packets:3, bytes:283, used:0.252s, flags:P., dp:ovs, actions:p1
ufid:61ef0a62-c67f-42d0-9acc-f6ac6ad30bbd, recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(p1),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=50:54:00:00:00:03,dst=50:54:00:00:00:04),eth_type(0x86dd),ipv6(src=::/::,dst=::/::,label=0/0,proto=6,tclass=0/0,hlimit=64,frag=no),tcp(src=0/0,dst=0/0),tcp_flags(0/0), packets:3, bytes:258, used:0.252s, flags:., dp:ovs, actions:p2
ufid:8d646148-494a-443c-82bc-e8fb166e6281, recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(p2),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=50:54:00:00:00:04,dst=50:54:00:00:00:03),eth_type(0x86dd),ipv6(src=::/::,dst=::/::,label=0/0,proto=6,tclass=0/0,hlimit=64,frag=no),tcp(src=0/0,dst=0/0),tcp_flags(0/0), packets:4, bytes:379, used:0.267s, flags:P., dp:ovs, actions:p1
ufid:61ef0a62-c67f-42d0-9acc-f6ac6ad30bbd, recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(p1),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=50:54:00:00:00:03,dst=50:54:00:00:00:04),eth_type(0x86dd),ipv6(src=::/::,dst=::/::,label=0/0,proto=6,tclass=0/0,hlimit=64,frag=no),tcp(src=0/0,dst=0/0),tcp_flags(0/0), packets:4, bytes:344, used:0.267s, flags:., dp:ovs, actions:p2
ufid:8d646148-494a-443c-82bc-e8fb166e6281, recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(p2),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=50:54:00:00:00:04,dst=50:54:00:00:00:03),eth_type(0x86dd),ipv6(src=::/::,dst=::/::,label=0/0,proto=6,tclass=0/0,hlimit=64,frag=no),tcp(src=0/0,dst=0/0),tcp_flags(0/0), packets:5, bytes:476, used:0.268s, flags:P., dp:ovs, actions:p1
ufid:61ef0a62-c67f-42d0-9acc-f6ac6ad30bbd, recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(p1),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=50:54:00:00:00:03,dst=50:54:00:00:00:04),eth_type(0x86dd),ipv6(src=::/::,dst=::/::,label=0/0,proto=6,tclass=0/0,hlimit=64,frag=no),tcp(src=0/0,dst=0/0),tcp_flags(0/0), packets:5, bytes:430, used:0.268s, flags:., dp:ovs, actions:p2
Comment 5 William Zhao 2022-08-18 12:13:41 UTC 
*** Bug 2027584 has been marked as a duplicate of this bug. ***
Comment 7 errata-xmlrpc 2022-09-06 18:49:03 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (openvswitch2.17 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:6368