Bug 1776994

Summary: OCP IPv6 - Packets dropped on upcall from kernel after SNAT - Failed to acquire udpif_key corresponding to unexpected flow
Product: Red Hat Enterprise Linux Fast Datapath Reporter: Mark Michelson <mmichels>
Component: ovn2.12Assignee: Open vSwitch development team <ovs-team>
Status: CLOSED ERRATA QA Contact: Jianlin Shi <jishi>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: RHEL 8.0CC: ctrautma, dcbw, fleitner, jhsiao, jishi, kfida, markmc, mmichels, nusiddiq, ovs-qe, ovs-team, qding, ralongi
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1773605 Environment:
Last Closed: 2019-12-11 12:19:53 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1773605    
Bug Blocks: 1775160, 1776969, 1776973    

Comment 2 Jianlin Shi 2019-11-28 06:38:03 UTC
reproduced on ovn2.12-2.12.0-14:

[root@dell-per740-12 ~]# rpm -qa | grep -E "openvswitch|ovn"                                          
ovn2.12-host-2.12.0-14.el8fdp.x86_64                                                                  
openvswitch2.12-2.12.0-4.el8fdp.x86_64                                                                
ovn2.12-central-2.12.0-14.el8fdp.x86_64                                                               
openvswitch-selinux-extra-policy-1.0-19.el8fdp.noarch                                                 
ovn2.12-2.12.0-14.el8fdp.x86_64

[root@dell-per740-12 ~]# ip netns exec server0 ping6 3000::1 -c 1                                     
PING 3000::1(3000::1) 56 data bytes                                                                   

--- 3000::1 ping statistics ---                                                                       
1 packets transmitted, 0 received, 100% packet loss, time 0ms                                         
                                                                                                      
[root@dell-per740-12 ~]# grep ofproto_dpif_upcall /var/log/openvswitch/ovs-vswitchd.log               
2019-11-28T06:35:12.363Z|00001|ofproto_dpif_upcall(handler51)|INFO|received packet on unassociated datapath port 4294967295
2019-11-28T06:35:12.376Z|00001|ofproto_dpif_upcall(revalidator85)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (Invalid argument): ufid:6988794e-04ad-447d-a32e-96aa780075b5
<==== error message in log

[root@dell-per740-12 bz1776994]# ip netns exec server1 tcpdump -i any -nnle
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
^C                                                                                                    
0 packets captured                                                                                    
0 packets received by filter                                                                          
0 packets dropped by kernel

<=== no NS received

Comment 3 Jianlin Shi 2019-11-28 06:39:03 UTC
no ovn2.12-2.12.0-15 for rhel8 version, numan, please help to check

Comment 4 Numan Siddique 2019-11-28 10:08:35 UTC
Triggered the build for RHEL 8- https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=25010753

Comment 5 Jianlin Shi 2019-11-29 01:18:07 UTC
Verified on ovn2.12.0-15:

[root@dell-per740-12 bz1773598]# ip netns exec server0 ping6 3000::1 -c 2
PING 3000::1(3000::1) 56 data bytes
64 bytes from 3000::1: icmp_seq=2 ttl=63 time=3.38 ms

--- 3000::1 ping statistics ---
2 packets transmitted, 1 received, 50% packet loss, time 22ms
rtt min/avg/max/mdev = 3.376/3.376/3.376/0.000 ms

<==== ping passed
                                                     
[root@dell-per740-12 bz1773598]# grep ofproto_dpif_upcall /var/log/openvswitch/ovs-vswitchd.log
2019-11-29T01:15:31.717Z|00001|ofproto_dpif_upcall(handler50)|INFO|received packet on unassociated datapath port 4294967295
2019-11-29T01:15:31.718Z|00001|ofproto_dpif_upcall(revalidator85)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (Invalid argument): ufid:d1bfea1f-f07f-4ade-ad32-57fe344199f1

<==== error message

[root@dell-per740-12 bz1773598]# rpm -qa | grep -E "openvswitch|ovn"                                  
ovn2.12-central-2.12.0-15.el8fdp.x86_64
openvswitch2.12-2.12.0-8.el8fdp.x86_64
ovn2.12-2.12.0-15.el8fdp.x86_64
ovn2.12-host-2.12.0-15.el8fdp.x86_64
openvswitch-selinux-extra-policy-1.0-19.el8fdp.noarch

[root@dell-per740-12 bz1776994]# ip netns exec server1 tcpdump -i any -nnle
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode                            
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
20:15:31.704626   M 00:de:ad:ff:01:03 ethertype IPv6 (0x86dd), length 88: 2001::1 > ff02::1:ff00:1: ICMP6, neighbor solicitation, who has 3000::1, length 32

<==== NS received

20:15:31.704670 Out f2:ce:2e:1b:37:80 ethertype IPv6 (0x86dd), length 88: 3000::1 > 2001::1: ICMP6, neighbor advertisement, tgt is 3000::1, length 32
20:15:32.659571 Out f2:ce:2e:1b:37:80 ethertype IPv6 (0x86dd), length 72: fe80::f0ce:2eff:fe1b:3780 > ff02::2: ICMP6, router solicitation, length 16
20:15:32.659915   M 0a:aa:07:c9:21:4a ethertype IPv6 (0x86dd), length 72: fe80::8aa:7ff:fec9:214a > ff02::2: ICMP6, router solicitation, length 16
20:15:32.724362  In 00:de:ad:ff:01:03 ethertype IPv6 (0x86dd), length 120: 3000::55 > 3000::1: ICMP6, echo request, seq 2, length 64
20:15:32.724420 Out f2:ce:2e:1b:37:80 ethertype IPv6 (0x86dd), length 88: 3000::1 > ff02::1:ff00:55: ICMP6, neighbor solicitation, who has 3000::55, length 32
20:15:32.726194  In 00:de:ad:ff:01:03 ethertype IPv6 (0x86dd), length 88: 3000::55 > 3000::1: ICMP6, neighbor advertisement, tgt is 3000::55, length 32
20:15:32.726228 Out f2:ce:2e:1b:37:80 ethertype IPv6 (0x86dd), length 120: 3000::1 > 3000::55: ICMP6, echo reply, seq 2, length 64


set VERIFIED based on https://bugzilla.redhat.com/show_bug.cgi?id=1773605#c11

Comment 7 errata-xmlrpc 2019-12-11 12:19:53 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:4209