Description of problem:

This issue was found when testing cross node pod-to-pod traffic on openshift with ovn-kubernetes network type. OVS hw-offload was enabled on both nodes while running the test, tc-policy is the default "none".


Version-Release number of selected component (if applicable):

OS: 4.18.0-305.2.1.el8_4.x86_64
OVN: ovn2.13-20.12.0-115.el8fdp.x86_64
OVS: openvswitch2.15-2.15.0-15.el8fdp.x86_64
OVN-K8s master: 19fc45c2aad19065070c4622292b5f962a245357


The problematic flow in ovs datapath:

ufid:36aa7029-5d89-4729-a583-c8b30a0266ca, skb_priority(0/0),skb_mark(0/0),ct_state(0x22/0x3e),ct_zone(0/0),ct_mark(0/0),ct_label(0/0x3),recirc_id(0x1e7ee6),dp_hash(0/0),in_port(4e44b9b1e98615c),packet_type(ns=0/0,id=0/0),eth(src=0a:58:0a:80:02:09,dst=0a:58:0a:80:02:01),eth_type(0x0800),ipv4(src=10.128.2.8/255.255.255.248,dst=10.131.0.131,proto=6,tos=0/0x3,ttl=64,frag=no),tcp(src=0/0,dst=0/0), packets:0, bytes:0, used:0.910s, offloaded:yes, dp:tc, actions:ct_clear,ct_clear,set(tunnel(tun_id=0xc,dst=10.0.1.14,ttl=64,tp_dst=6081,key6(bad key length 1, expected 0)(01)geneve({class=0x102,type=0x80,len=4,0x10043}),flags(key))),set(eth(src=0a:58:0a:83:00:01,dst=0a:58:0a:83:00:83)),set(ipv4(ttl=63)),genev_sys_6081


In the action, it shows "bad key length 1, expected 0".


Additional info:

CX-5 ovs hardware offload:

[root@sriov-worker-0 core]# ethtool -i ens801f1
driver: mlx5e_rep
version: 4.18.0-305.2.1.el8_4.x86_64
firmware-version: 16.29.2002 (MT_0000000012)
expansion-rom-version: 
bus-info: 0000:b0:00.1
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
[root@sriov-worker-0 core]# lspci -vv -nn -mm -s 0000:b0:00.1
Slot:	b0:00.1
Class:	Ethernet controller [0200]
Vendor:	Mellanox Technologies [15b3]
Device:	MT27800 Family [ConnectX-5] [1017]
SVendor:	Mellanox Technologies [15b3]
SDevice:	Mellanox ConnectX®-5 MCX516A-CCAT [0007]
NUMANode: