Bug 1776973

Summary: OCP IPv6 - Packets dropped on upcall from kernel after SNAT - Failed to acquire udpif_key corresponding to unexpected flow
Product: Red Hat Enterprise Linux Fast Datapath Reporter: Mark Michelson <mmichels>
Component: ovn2.11Assignee: Open vSwitch development team <ovs-team>
Status: CLOSED CURRENTRELEASE QA Contact: Jianlin Shi <jishi>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: RHEL 8.0CC: ctrautma, dcbw, fleitner, jhsiao, jishi, markmc, mmichels, nusiddiq, ovs-qe, ovs-team, qding, ralongi
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1773605 Environment:
Last Closed: 2023-06-14 17:46:18 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1773605, 1776994    
Bug Blocks: 1775160, 1776969    

Comment 2 Jianlin Shi 2019-11-28 06:23:26 UTC 
reproduced on ovn2.11-2.11.1-19:

[root@dell-per740-12 ~]# ip netns exec server0 ping6 3000::1 -c 2
PING 3000::1(3000::1) 56 data bytes                                                                   

--- 3000::1 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 12ms                                        

[root@dell-per740-12 ~]# grep ofproto_dpif_upcall /var/log/openvswitch/ovs-vswitchd.log
2019-11-28T06:16:54.594Z|00001|ofproto_dpif_upcall(handler52)|INFO|received packet on unassociated datapath port 4294967295
2019-11-28T06:16:54.607Z|00001|ofproto_dpif_upcall(revalidator85)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (Invalid argument): ufid:38c928be-6f25-41e4-a84a-3f7cafa92c1c
2019-11-28T06:16:55.604Z|00002|ofproto_dpif_upcall(handler52)|INFO|received packet on unassociated datapath port 4294967295
2019-11-28T06:16:55.608Z|00002|ofproto_dpif_upcall(revalidator85)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (Invalid argument): ufid:64bb5f22-8c0c-43ef-8af7-1d35aa1dd891

<==== error in ovs log

[root@dell-per740-12 ~]# rpm -qa | grep -E "openvswitch|ovn"
ovn2.11-2.11.1-19.el8fdp.x86_64                                                                       
openvswitch2.11-2.11.0-26.el8fdp.x86_64                                                               
ovn2.11-central-2.11.1-19.el8fdp.x86_64                                                               
openvswitch-selinux-extra-policy-1.0-19.el8fdp.noarch
ovn2.11-host-2.11.1-19.el8fdp.x86_64

[root@dell-per740-12 bz1776973]# ip netns exec server1 tcpdump -i any -nnle
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
^C                                                                                                    
0 packets captured                                                                                    
0 packets received by filter                                                                          
0 packets dropped by kernel

<==== no NS received

Verified on ovn2.11-2.11.1-20:

[root@dell-per740-12 ~]# ip netns exec server0 ping6 3000::1 -c 2
PING 3000::1(3000::1) 56 data bytes
64 bytes from 3000::1: icmp_seq=2 ttl=63 time=2.75 ms

--- 3000::1 ping statistics ---
2 packets transmitted, 1 received, 50% packet loss, time 44ms                                         
rtt min/avg/max/mdev = 2.745/2.745/2.745/0.000 ms

<==== passed

[root@dell-per740-12 ~]# grep ofproto_dpif_upcall /var/log/openvswitch/ovs-vswitchd.log               
2019-11-28T06:16:54.594Z|00001|ofproto_dpif_upcall(handler52)|INFO|received packet on unassociated datapath port 4294967295
2019-11-28T06:16:54.607Z|00001|ofproto_dpif_upcall(revalidator85)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (Invalid argument): ufid:38c928be-6f25-41e4-a84a-3f7cafa92c1c
2019-11-28T06:16:55.604Z|00002|ofproto_dpif_upcall(handler52)|INFO|received packet on unassociated datapath port 4294967295
2019-11-28T06:16:55.608Z|00002|ofproto_dpif_upcall(revalidator85)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (Invalid argument): ufid:64bb5f22-8c0c-43ef-8af7-1d35aa1dd891
2019-11-28T06:20:54.767Z|00001|ofproto_dpif_upcall(handler66)|INFO|received packet on unassociated datapath port 4294967295
2019-11-28T06:20:54.768Z|00001|ofproto_dpif_upcall(revalidator97)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (Invalid argument): ufid:2d287088-c56f-47ab-a92b-5bd1a7121da6

<==== also get error in log

[root@dell-per740-12 ~]# rpm -qa | grep -E "openvswitch|ovn"
ovn2.11-central-2.11.1-20.el8fdp.x86_64
openvswitch2.11-2.11.0-26.el8fdp.x86_64                                                               
ovn2.11-2.11.1-20.el8fdp.x86_64
ovn2.11-host-2.11.1-20.el8fdp.x86_64
openvswitch-selinux-extra-policy-1.0-19.el8fdp.noarch

[root@dell-per740-12 bz1776973]# ip netns exec server1 tcpdump -i any -nnle                           
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes                       
01:20:54.754719   M 00:de:ad:ff:01:03 ethertype IPv6 (0x86dd), length 88: 2001::1 > ff02::1:ff00:1: ICMP6, neighbor solicitation, who has 3000::1, length 32

<=== NS received

01:20:54.754770 Out fa:9b:16:f8:23:a4 ethertype IPv6 (0x86dd), length 88: 3000::1 > 2001::1: ICMP6, neighbor advertisement, tgt is 3000::1, length 32
01:20:55.603582 Out fa:9b:16:f8:23:a4 ethertype IPv6 (0x86dd), length 72: fe80::f89b:16ff:fef8:23a4 > ff02::2: ICMP6, router solicitation, length 16
01:20:55.796673  In 00:de:ad:ff:01:03 ethertype IPv6 (0x86dd), length 120: 3000::55 > 3000::1: ICMP6, echo request, seq 2, length 64
01:20:55.796747 Out fa:9b:16:f8:23:a4 ethertype IPv6 (0x86dd), length 88: 3000::1 > ff02::1:ff00:55: ICMP6, neighbor solicitation, who has 3000::55, length 32
01:20:55.797813  In 00:de:ad:ff:01:03 ethertype IPv6 (0x86dd), length 88: 3000::55 > 3000::1: ICMP6, neighbor advertisement, tgt is 3000::55, length 32
01:20:55.797846 Out fa:9b:16:f8:23:a4 ethertype IPv6 (0x86dd), length 120: 3000::1 > 3000::55: ICMP6, echo reply, seq 2, length 64


set VERIFIED