Bug 1775778

Summary: OCP IPv6 - Kernel rejects flows due to unexpected flow key - Key type 30 is out of range max 29
Product: Red Hat Enterprise Linux Fast Datapath Reporter: Flavio Leitner <fleitner>
Component: openvswitch2.12Assignee: Open vSwitch development team <ovs-team>
Status: CLOSED ERRATA QA Contact: Jianlin Shi <jishi>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: RHEL 8.0CC: ctrautma, fleitner, jhsiao, kfida, markmc, ovs-qe, qding, ralongi
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1773598 Environment:
Last Closed: 2019-12-11 12:06:53 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1773598    
Bug Blocks:    

Comment 2 Jianlin Shi 2019-11-29 01:03:34 UTC 
reproducer (download db files from bz1773598):

#!/bin/bash                                                                                                     
add_phys_port() {
    name=$1
    mac=$2
    iface_id=$3
    ip netns add $name
    ovs-vsctl add-port br-int $name -- set interface $name type=internal
    ip link set $name netns $name
    ip netns exec $name ip link set $name address $mac
    ip netns exec $name ip link set $name up
    ovs-vsctl set Interface $name external_ids:iface-id=$iface_id
}
dmesg -C
systemctl restart openvswitch                                                                         
systemctl restart ovn-northd
ovn-nbctl set-connection ptcp:6641
ovn-sbctl set-connection ptcp:6642                                                                    
ovs-vsctl set Open_vSwitch . external-ids:system-id=hv1 external-ids:ovn-remote=tcp:20.0.29.25:6642 external-ids:ovn-encap-type=geneve external-ids:ovn-encap-ip=20.0.29.25
systemctl restart ovn-controller
cp /root/bz1773598/k88_ipv6_local_reproduce_dbs/* /var/lib/ovn -rf
systemctl restart ovn-northd
ovn-nbctl set-connection ptcp:6641
ovn-sbctl set-connection ptcp:6642                                                                    
ovn-nbctl show

add_phys_port fake_vm1 2a:e7:a3:00:00:05 openshift-kube-apiserver-operator_kube-apiserver-operator-5f4bd8fdc4-w7tk2
ip netns exec fake_vm1 ip a a fd01::28e7:a3ff:fe00:5/64 dev fake_vm1
add_phys_port fake_vm2 2a:e7:a3:00:00:08 openshift-operator-lifecycle-manager_catalog-operator-5f75df8dd8-59xz7
ip netns exec fake_vm2 ip a a fd01::28e7:a3ff:fe00:8/64 dev fake_vm2
sleep 5
ovn-sbctl show
ip netns exec fake_vm2 ping6 -I fake_vm2 fd01::28e7:a3ff:fe00:5 -c 5
dmesg

reproduced on openvswitch-2.12.0-4:

+ ip netns exec fake_vm2 ip a a fd01::28e7:a3ff:fe00:8/64 dev fake_vm2
+ sleep 5                                                                                             
+ ovn-sbctl show                                                                                      
Chassis hv1                                                                                           
    hostname: dell-per740-12.rhts.eng.pek2.redhat.com                                                 
    Encap geneve                                     
        ip: "20.0.29.25"                   
        options: {csum="true"}
    Port_Binding openshift-operator-lifecycle-manager_catalog-operator-5f75df8dd8-59xz7
    Port_Binding openshift-kube-apiserver-operator_kube-apiserver-operator-5f4bd8fdc4-w7tk2
+ ip netns exec fake_vm2 ping6 -I fake_vm2 fd01::28e7:a3ff:fe00:5 -c 5
PING fd01::28e7:a3ff:fe00:5(fd01::28e7:a3ff:fe00:5) from fd01::28e7:a3ff:fe00:8 fake_vm2: 56 data bytes
64 bytes from fd01::28e7:a3ff:fe00:5: icmp_seq=1 ttl=64 time=16.4 ms
64 bytes from fd01::28e7:a3ff:fe00:5: icmp_seq=2 ttl=64 time=0.959 ms
64 bytes from fd01::28e7:a3ff:fe00:5: icmp_seq=3 ttl=64 time=0.104 ms
64 bytes from fd01::28e7:a3ff:fe00:5: icmp_seq=4 ttl=64 time=0.095 ms   
64 bytes from fd01::28e7:a3ff:fe00:5: icmp_seq=5 ttl=64 time=0.095 ms
                                                      
--- fd01::28e7:a3ff:fe00:5 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 45ms     
rtt min/avg/max/mdev = 0.095/3.539/16.443/6.460 ms
+ dmesg
[71933.003468] device br-int left promiscuous mode
[71933.136990] device br-int entered promiscuous mode
[71933.525234] device fake_vm1 entered promiscuous mode
[71933.721215] device fake_vm2 entered promiscuous mode
[71934.457443] openvswitch: netlink: Key type 30 is out of range max 29
[71934.585509] openvswitch: netlink: Key type 30 is out of range max 29
[71938.928889] openvswitch: netlink: Key type 30 is out of range max 29
[71938.937028] openvswitch: netlink: Key type 30 is out of range max 29    

<==== got error dmesg
                                                                                                                                
[root@dell-per740-12 bz1773598]# rpm -qa | grep -E "openvswitch|ovn"
ovn2.12-2.12.0-7.el8fdp.x86_64  
openvswitch2.12-2.12.0-4.el8fdp.x86_64
ovn2.12-central-2.12.0-7.el8fdp.x86_64                            
ovn2.12-host-2.12.0-7.el8fdp.x86_64
openvswitch-selinux-extra-policy-1.0-19.el8fdp.noarch

Verified on openvswitch-2.12.0-8:

+ ovn-sbctl show
Chassis hv1
    hostname: dell-per740-12.rhts.eng.pek2.redhat.com                                                 
    Encap geneve
        ip: "20.0.29.25"
        options: {csum="true"}
    Port_Binding openshift-kube-apiserver-operator_kube-apiserver-operator-5f4bd8fdc4-w7tk2
    Port_Binding openshift-operator-lifecycle-manager_catalog-operator-5f75df8dd8-59xz7
+ ip netns exec fake_vm2 ping6 -I fake_vm2 fd01::28e7:a3ff:fe00:5 -c 5
PING fd01::28e7:a3ff:fe00:5(fd01::28e7:a3ff:fe00:5) from fd01::28e7:a3ff:fe00:8 fake_vm2: 56 data bytes
64 bytes from fd01::28e7:a3ff:fe00:5: icmp_seq=1 ttl=64 time=2.93 ms
64 bytes from fd01::28e7:a3ff:fe00:5: icmp_seq=2 ttl=64 time=1.01 ms
64 bytes from fd01::28e7:a3ff:fe00:5: icmp_seq=3 ttl=64 time=0.099 ms
64 bytes from fd01::28e7:a3ff:fe00:5: icmp_seq=4 ttl=64 time=0.094 ms
64 bytes from fd01::28e7:a3ff:fe00:5: icmp_seq=5 ttl=64 time=0.093 ms
                                                                                                      
--- fd01::28e7:a3ff:fe00:5 ping statistics ---                                                        
5 packets transmitted, 5 received, 0% packet loss, time 52ms
rtt min/avg/max/mdev = 0.093/0.843/2.925/1.099 ms                                                     
+ dmesg                                                                                               
[72538.248471] device br-int left promiscuous mode                                                    
[72538.414805] device br-int entered promiscuous mode
[72538.641453] device fake_vm1 entered promiscuous mode
[72538.824122] device fake_vm2 entered promiscuous mode

<==== no error dmesg

[root@dell-per740-12 bz1773598]# rpm -qa | grep -E "openvswitch|ovn"
ovn2.12-2.12.0-7.el8fdp.x86_64                                                                        
ovn2.12-central-2.12.0-7.el8fdp.x86_64                                                                
openvswitch2.12-2.12.0-8.el8fdp.x86_64                                                                
ovn2.12-host-2.12.0-7.el8fdp.x86_64                                                                   
openvswitch-selinux-extra-policy-1.0-19.el8fdp.noarch
Comment 4 errata-xmlrpc 2019-12-11 12:06:53 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:4207