Bug 2173387

Summary: [HWOL]tc(revalidator58)|ERR|failed to parse flower classifier terse options
Product: Red Hat Enterprise Linux Fast Datapath Reporter: mhou <mhou>
Component: openvswitch3.1Assignee: Eelco Chaudron <echaudro>
Status: CLOSED CURRENTRELEASE QA Contact: mhou <mhou>
Severity: medium Docs Contact:
Priority: medium    
Version: FDP 23.BCC: ctrautma, fleitner, jhsiao, kzhang, mleitner, ralongi
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-06-24 02:20:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2172622    

Description mhou 2023-02-26 09:42:38 UTC 
Description of problem:
run performance test and got err message on ovs-vswitchd.log
beaker job:https://beaker.engineering.redhat.com/jobs/7569059
ovs-vswitchd.log: https://beaker-archive.hosts.prod.psi.bos.redhat.com/beaker-logs/2023/02/75690/7569059/13454524/156739454/ovs-vswitchd.log


Version-Release number of selected component (if applicable):
kernel version: kernel-rt-5.14.0-70.43.1.rt21.114.el9_0 
ovs version: openvswitch3.1-3.1.0-2.el9fdp

How reproducible: only once
I run same test job on stock kernel but can't hit this issue.
kernel-5.14.0-70.43.1.el9_0+openvswitch3.1-3.1.0-2.el9fdp
https://beaker.engineering.redhat.com/jobs/7569995 --pass(no err message)
I change the distro from RHEL-9.0.0-updates-20230215.28 to RHEL-9.0.0-updates-20230224.29 and keep same rt kernel(kernel-rt-5.14.0-70.43.1.rt21.114.el9_0) but still no luck.
https://beaker.engineering.redhat.com/jobs/7570563 --pass(no err message)

Steps to Reproduce:
1. host cpu topo as below:
# numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54
node 0 size: 31260 MB
node 0 free: 2313 MB
node 1 cpus: 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55
node 1 size: 31177 MB
node 1 free: 2827 MB
node distances:
node   0   1 
  0:  10  20 
  1:  20  10 

2. tunning host as below:
# cat /proc/cmdline 
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.14.0-70.43.1.rt21.114.el9_0.x86_64 root=/dev/mapper/rhel_dell--per750--03-root ro ksdevice=bootif pci=realloc resume=/dev/mapper/rhel_dell--per750--03-swap rd.lvm.lv=rhel_dell-per750-03/root rd.lvm.lv=rhel_dell-per750-03/swap console=ttyS0,115200n81 skew_tick=1 isolcpus=managed_irq,domain,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54 intel_pstate=disable nosoftlockup tsc=nowatchdog nohz=on nohz_full=4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54 rcu_nocbs=4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54 irqaffinity=0,1,2,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55 crashkernel=1G-4G:384M,4G-16G:512M,16G-64G:1G,64G-128G:2G,128G-:4G default_hugepagesz=1G hugepagesz=1G hugepages=48 intel_iommu=on iommu=pt intel_idle.max_cstate=0 processor.max_cstate=0 intel_pstate=disable idle=poll rcu_nocb_poll

3. configure ovs hardware offload and guest testpmd as below
ovs-vsctl show
fd85b93a-4988-4a8d-b869-c01d6baa608d
    Bridge ovsbr0
        Port ens1f0np0
            Interface ens1f0np0
        Port eth0
            Interface eth0
        Port ovsbr0
            Interface ovsbr0
                type: internal
    Bridge ovsbr1
        Port eth1
            Interface eth1
        Port ens1f1np1
            Interface ens1f1np1
        Port ovsbr1
            Interface ovsbr1
                type: internal
    ovs_version: "3.1.1"
ovs-vsctl get Open_vSwitch . other_config
{hw-offload="true", max-idle="180000", tc-policy=none}

ovs-vsctl set Open_vSwitch . other_config:max-idle=3600000
ovs-vsctl set Open_vSwitch . other_config:max-revalidator=3600000
ovs-vsctl set bridge ovsbr0 other-config:mac-aging-time=3600
ovs-vsctl set bridge ovsbr0 other-config:mac-table-size=$((2*1024+8192))
ovs-vsctl set bridge ovsbr1 other-config:mac-aging-time=3600
ovs-vsctl set bridge ovsbr1 other-config:mac-table-size=$((2*1024+8192))

Actual results:
check error message on ovs-vswitchd.log
ovs_offload_pvp_flows1024_queue1_vcpus3_size9000_vlan0_hwoffload_on_tcpolicy_none_start
2023-02-24T12:50:42.388Z|00015|dpif_netlink(handler1)|WARN|system@ovs-system: lost packet on handler 0
2023-02-24T12:50:42.427Z|00016|dpif_netlink(handler1)|WARN|system@ovs-system: lost packet on handler 0
2023-02-24T12:52:06.793Z|00017|dpif_netlink(handler1)|WARN|system@ovs-system: lost packet on handler 0
2023-02-24T12:52:06.860Z|00018|dpif_netlink(handler1)|WARN|system@ovs-system: lost packet on handler 0
2023-02-24T12:52:06.934Z|00019|dpif_netlink(handler1)|WARN|system@ovs-system: lost packet on handler 0
2023-02-24T13:00:31.109Z|00045|ofproto_dpif_upcall(revalidator67)|INFO|Unexpected ukey transition from state 3 (last transitioned from thread 0 at ../ofproto/ofproto-dpif-upcall.c:2540)
2023-02-24T13:00:31.109Z|00042|dpif(revalidator65)|WARN|Dropped 235 log messages in last 604 seconds (most recently, 603 seconds ago) due to excessive rate
2023-02-24T13:00:31.109Z|00043|dpif(revalidator65)|WARN|system@ovs-system: failed to flow_get (No such file or directory) ufid:b663dbc8-68a2-4674-9791-0a5a7e85e87b <empty>, packets:0, bytes:0, used:never
2023-02-24T13:00:31.109Z|00044|ofproto_dpif_upcall(revalidator65)|WARN|Dropped 176 log messages in last 603 seconds (most recently, 603 seconds ago) due to excessive rate
2023-02-24T13:00:31.109Z|00045|ofproto_dpif_upcall(revalidator65)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:b663dbc8-68a2-4674-9791-0a5a7e85e87b
2023-02-24T13:00:31.111Z|00040|ofproto_dpif_upcall(revalidator66)|INFO|Unexpected ukey transition from state 3 (last transitioned from thread 0 at ../ofproto/ofproto-dpif-upcall.c:2540)
2023-02-24T13:00:31.112Z|00024|dpif(revalidator60)|WARN|system@ovs-system: failed to flow_get (No such file or directory) ufid:2417bbb5-24e0-4357-b8eb-5ce7d9c8c09f <empty>, packets:0, bytes:0, used:never
2023-02-24T13:00:31.112Z|00025|ofproto_dpif_upcall(revalidator60)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:2417bbb5-24e0-4357-b8eb-5ce7d9c8c09f
2023-02-24T13:00:31.112Z|00041|dpif(revalidator68)|WARN|system@ovs-system: failed to flow_get (No such file or directory) ufid:dbbf9465-cdcd-46a9-b2e8-65daea9715b7 <empty>, packets:0, bytes:0, used:never
2023-02-24T13:00:31.112Z|00042|ofproto_dpif_upcall(revalidator68)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:dbbf9465-cdcd-46a9-b2e8-65daea9715b7
2023-02-24T13:00:31.112Z|00043|dpif(revalidator68)|WARN|system@ovs-system: failed to flow_get (No such file or directory) ufid:6902839f-55a3-4a61-b8e8-ab194e8bb31f <empty>, packets:0, bytes:0, used:never
2023-02-24T13:00:31.112Z|00044|ofproto_dpif_upcall(revalidator68)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:6902839f-55a3-4a61-b8e8-ab194e8bb31f
2023-02-24T13:00:31.112Z|00045|dpif(revalidator68)|WARN|system@ovs-system: failed to flow_get (No such file or directory) ufid:8053d01b-45c6-486a-b807-4db5e08a2395 <empty>, packets:0, bytes:0, used:never
2023-02-24T13:00:31.112Z|00046|ofproto_dpif_upcall(revalidator68)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:8053d01b-45c6-486a-b807-4db5e08a2395
2023-02-24T13:00:31.113Z|00042|ofproto_dpif_upcall(revalidator70)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:4a7fc5e4-50ce-4a32-95e3-2e172ba369e2
2023-02-24T13:00:31.114Z|00039|ofproto_dpif_upcall(revalidator58)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:e89e2106-a58c-40d3-bda9-1db21ccd0c91
2023-02-24T13:00:31.115Z|00054|ofproto_dpif_upcall(revalidator63)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:08e94e98-7d87-407e-b325-37e4a110e715
2023-02-24T13:00:31.115Z|00055|ofproto_dpif_upcall(revalidator63)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:f78cb484-4ed6-4fb3-bc50-1cc3f54ff4cc
2023-02-24T13:00:31.115Z|00056|ofproto_dpif_upcall(revalidator63)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:f5bf9130-7272-427f-93d5-311aa5a8569d
2023-02-24T13:00:31.117Z|00046|ofproto_dpif_upcall(revalidator67)|INFO|Unexpected ukey transition from state 3 (last transitioned from thread 0 at ../ofproto/ofproto-dpif-upcall.c:2540)
2023-02-24T13:00:31.118Z|00046|ofproto_dpif_upcall(revalidator65)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:6c14b912-b665-4f69-96aa-56a2e0af9758
2023-02-24T13:00:31.118Z|00037|ofproto_dpif_upcall(revalidator61)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:2b226e6b-6075-45c0-9a9b-c6b8e644f489
2023-02-24T13:00:31.118Z|00038|ofproto_dpif_upcall(revalidator61)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:5c6aa84e-8d09-430d-933e-bd1875fb86b8
2023-02-24T13:00:31.119Z|00023|ofproto_dpif_upcall(revalidator59)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:3f523701-b994-4bb8-a7d4-159e40613da2
2023-02-24T13:00:31.120Z|00024|ofproto_dpif_upcall(revalidator59)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:b860e6d4-fa44-4c74-8cda-c5890a21df92
2023-02-24T13:00:31.120Z|00025|ofproto_dpif_upcall(revalidator59)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:3bd6ae28-ca7d-4781-bd5e-de431b3377ca
2023-02-24T13:00:31.121Z|00041|ofproto_dpif_upcall(revalidator66)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:356d8913-0525-4357-b477-3d2868416b6d
2023-02-24T13:00:31.122Z|00026|ofproto_dpif_upcall(revalidator60)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:d57a8d4d-83eb-45d4-9c4a-fc24b47e73f6
2023-02-24T13:00:31.122Z|00039|ofproto_dpif_upcall(revalidator71)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:50980ca5-329a-4746-8dc8-f8cc971e91d7
2023-02-24T13:00:31.122Z|00040|ofproto_dpif_upcall(revalidator71)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:c8379809-231a-43dc-9a09-9ef22673edcc
2023-02-24T13:00:31.122Z|00041|ofproto_dpif_upcall(revalidator71)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:000f44c1-37e5-4ba3-aba1-41158a707c04
2023-02-24T13:00:31.122Z|00042|ofproto_dpif_upcall(revalidator71)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:6aeef551-8779-4107-898f-9f2eb4947034
2023-02-24T13:00:31.122Z|00043|ofproto_dpif_upcall(revalidator64)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:e5f4d291-d41f-40de-89d0-4b30946cb3d4
2023-02-24T13:00:31.122Z|00044|ofproto_dpif_upcall(revalidator64)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:1490bf2f-b910-466f-b6fb-066565a059df
2023-02-24T13:00:31.122Z|00047|ofproto_dpif_upcall(revalidator68)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:28256d66-a9f6-42bd-8dde-9d21bfc09f80
2023-02-24T13:00:31.122Z|00048|ofproto_dpif_upcall(revalidator68)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:933b46fe-7ccb-4cd3-aa65-4c10be1b3dd1
2023-02-24T13:00:31.123Z|00049|ofproto_dpif_upcall(revalidator68)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:5dfb0b14-7a50-48d6-9762-74e8e362f44d
2023-02-24T13:00:31.123Z|00050|ofproto_dpif_upcall(revalidator68)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:13c69a5a-fa15-4503-992c-4f51c4a1c2e7
2023-02-24T13:00:31.124Z|00043|ofproto_dpif_upcall(revalidator70)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:159c99fd-26c5-4a87-814b-4779b0fbc476
2023-02-24T13:00:31.125Z|00044|ofproto_dpif_upcall(revalidator70)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:d212da82-1a43-4c47-8b18-2a1d36c3d35e
2023-02-24T13:00:31.125Z|00045|ofproto_dpif_upcall(revalidator70)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:a08fec1a-e201-4306-84fa-ff6a06156ee4
2023-02-24T13:00:31.125Z|00046|ofproto_dpif_upcall(revalidator70)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:34c06fd0-0ddc-4369-90b6-055ada1bc0c3
2023-02-24T13:00:31.125Z|00033|ofproto_dpif_upcall(revalidator62)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:abf95ede-2a9a-4be1-a23d-50ddd21c5e60
2023-02-24T13:00:31.125Z|00034|ofproto_dpif_upcall(revalidator62)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:e1eb3e25-4cc8-4055-a54b-68869af495ce
2023-02-24T13:00:31.126Z|00057|ofproto_dpif_upcall(revalidator63)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:a0057f9c-87a7-4db6-a61d-793e384abc89
2023-02-24T13:00:31.126Z|00058|ofproto_dpif_upcall(revalidator63)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:aad87004-836e-4582-be49-739ad3402700
2023-02-24T13:00:31.126Z|00059|ofproto_dpif_upcall(revalidator63)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:2fb10015-9f00-4879-a781-df5f9b38ef83
2023-02-24T13:00:31.126Z|00030|ofproto_dpif_upcall(revalidator69)|INFO|Unexpected ukey transition from state 3 (last transitioned from thread 0 at ../ofproto/ofproto-dpif-upcall.c:2540)
2023-02-24T13:00:31.127Z|00060|ofproto_dpif_upcall(revalidator63)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:881c04a7-aeb9-474a-b280-9b30b71506ea
2023-02-24T13:00:31.127Z|00039|ofproto_dpif_upcall(revalidator72)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:015c605f-4cef-4dc2-a456-635797a34980
2023-02-24T13:00:31.127Z|00040|ofproto_dpif_upcall(revalidator72)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:684ad8d7-037b-4c77-a88e-5fb0717c9717
2023-02-24T13:00:31.127Z|00041|ofproto_dpif_upcall(revalidator72)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:c1a81141-5422-4ebf-bcff-f1e74aef2d7b
2023-02-24T13:00:31.127Z|00042|ofproto_dpif_upcall(revalidator72)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:6b7e7791-1780-4257-ace5-27561aab0c51
2023-02-24T13:00:31.127Z|00043|ofproto_dpif_upcall(revalidator72)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:cafd0329-373d-4e7e-b453-87419bd9b4d8
2023-02-24T13:00:31.127Z|00044|ofproto_dpif_upcall(revalidator72)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:88f7be86-1a1a-4e9d-95b4-23839c7fbe78
2023-02-24T13:00:31.128Z|00061|ofproto_dpif_upcall(revalidator63)|INFO|Unexpected ukey transition from state 3 (last transitioned from thread 0 at ../ofproto/ofproto-dpif-upcall.c:2540)
2023-02-24T13:00:31.128Z|00040|ofproto_dpif_upcall(revalidator58)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:741cc7d3-9266-4e40-ac9f-a9f2b14d636f
2023-02-24T13:00:31.128Z|00041|ofproto_dpif_upcall(revalidator58)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:a3a9f64d-80df-48dc-8c3c-06f501b0958a
2023-02-24T13:00:31.129Z|00047|ofproto_dpif_upcall(revalidator67)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:6b7f6d26-0722-422d-9698-9b47809e3efd
2023-02-24T13:00:31.129Z|00042|ofproto_dpif_upcall(revalidator58)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:3aed4f6d-9a74-47e4-848d-0ad6ce215cbf
2023-02-24T13:00:31.129Z|00048|ofproto_dpif_upcall(revalidator67)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:48461d1a-6170-412c-abbc-a5932754ad8a
2023-02-24T13:00:31.129Z|00049|ofproto_dpif_upcall(revalidator67)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:93883be2-a647-4de2-91dc-4b3baa630dbb
2023-02-24T13:00:31.129Z|00050|ofproto_dpif_upcall(revalidator67)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:3e566165-a865-4de5-a73b-fa55ab8cb12f
2023-02-24T13:00:31.129Z|00051|ofproto_dpif_upcall(revalidator67)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:a9fe23dd-39bf-4cfa-a62e-1e95abc0a1d7
2023-02-24T13:00:31.129Z|00052|ofproto_dpif_upcall(revalidator67)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:d59bc318-cdc4-4443-a362-5d4a98ef0ac7
2023-02-24T13:00:31.130Z|00053|ofproto_dpif_upcall(revalidator67)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:97a9ea98-4528-4976-81d0-258f71ee3440
2023-02-24T13:00:31.130Z|00054|ofproto_dpif_upcall(revalidator67)|INFO|Unexpected ukey transition from state 3 (last transitioned from thread 0 at ../ofproto/ofproto-dpif-upcall.c:2540)
2023-02-24T13:00:31.130Z|00062|ofproto_dpif_upcall(revalidator63)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:09ee7476-6632-40c4-aec4-c411f9f0dc27
2023-02-24T13:00:31.130Z|00039|ofproto_dpif_upcall(revalidator61)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:4675fa2b-5094-4756-ae6f-ab6ab2235a39
2023-02-24T13:00:31.130Z|00043|ofproto_dpif_upcall(revalidator58)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:f0582e64-d8e1-4501-a033-3fa02548472b
2023-02-24T13:00:31.130Z|00047|ofproto_dpif_upcall(revalidator70)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:966d3987-ae43-40ce-953e-8bad8c7ccfa1
2023-02-24T13:00:31.131Z|00048|ofproto_dpif_upcall(revalidator70)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:34b80755-4327-4ffe-a1fe-ed14b1bcad5c
2023-02-24T13:00:31.131Z|00031|ofproto_dpif_upcall(revalidator69)|WARN|Failed to acquire udpif_key corresponding to unexpected flow (No such file or directory): ufid:a0bdba5c-b449-4cc6-a25a-1d5a44d54723
2023-02-24T13:00:31.135Z|00043|ofproto_dpif_upcall(revalidator71)|INFO|Unexpected ukey transition from state 3 (last transitioned from thread 0 at ../ofproto/ofproto-dpif-upcall.c:2540)
2023-02-24T13:00:31.135Z|00044|ofproto_dpif_upcall(revalidator71)|INFO|Unexpected ukey transition from state 3 (last transitioned from thread 0 at ../ofproto/ofproto-dpif-upcall.c:2540)
2023-02-24T13:00:31.136Z|00040|ofproto_dpif_upcall(revalidator61)|INFO|Unexpected ukey transition from state 3 (last transitioned from thread 0 at ../ofproto/ofproto-dpif-upcall.c:2540)
2023-02-24T13:00:31.137Z|00035|ofproto_dpif_upcall(revalidator62)|INFO|Unexpected ukey transition from state 3 (last transitioned from thread 0 at ../ofproto/ofproto-dpif-upcall.c:2540)
2023-02-24T13:00:31.138Z|00045|ofproto_dpif_upcall(revalidator71)|INFO|Unexpected ukey transition from state 3 (last transitioned from thread 0 at ../ofproto/ofproto-dpif-upcall.c:2540)
2023-02-24T13:00:31.139Z|00041|ofproto_dpif_upcall(revalidator61)|INFO|Unexpected ukey transition from state 3 (last transitioned from thread 0 at ../ofproto/ofproto-dpif-upcall.c:2540)
2023-02-24T13:00:31.145Z|00042|ofproto_dpif_upcall(revalidator66)|INFO|Unexpected ukey transition from state 3 (last transitioned from thread 0 at ../ofproto/ofproto-dpif-upcall.c:2540)
2023-02-24T13:00:31.160Z|00055|ofproto_dpif_upcall(revalidator67)|INFO|Unexpected ukey transition from state 3 (last transitioned from thread 0 at ../ofproto/ofproto-dpif-upcall.c:2540)
2023-02-24T13:00:31.163Z|00032|ofproto_dpif_upcall(revalidator69)|INFO|Unexpected ukey transition from state 3 (last transitioned from thread 0 at ../ofproto/ofproto-dpif-upcall.c:2540)
2023-02-24T13:00:31.167Z|00036|ofproto_dpif_upcall(revalidator62)|INFO|Unexpected ukey transition from state 3 (last transitioned from thread 0 at ../ofproto/ofproto-dpif-upcall.c:2540)
2023-02-24T13:00:31.181Z|00051|ofproto_dpif_upcall(revalidator68)|INFO|Unexpected ukey transition from state 3 (last transitioned from thread 0 at ../ofproto/ofproto-dpif-upcall.c:2540)
2023-02-24T13:00:31.181Z|00043|ofproto_dpif_upcall(revalidator66)|INFO|Unexpected ukey transition from state 3 (last transitioned from thread 0 at ../ofproto/ofproto-dpif-upcall.c:2540)
2023-02-24T13:00:31.185Z|00047|ofproto_dpif_upcall(revalidator65)|INFO|Unexpected ukey transition from state 3 (last transitioned from thread 0 at ../ofproto/ofproto-dpif-upcall.c:2540)
2023-02-24T13:00:31.193Z|00044|ofproto_dpif_upcall(revalidator58)|INFO|Unexpected ukey transition from state 3 (last transitioned from thread 0 at ../ofproto/ofproto-dpif-upcall.c:2540)
2023-02-24T13:00:31.193Z|00045|tc(revalidator58)|ERR|failed to parse flower classifier terse options
2023-02-24T13:00:31.195Z|00044|ofproto_dpif_upcall(revalidator66)|INFO|Unexpected ukey transition from state 3 (last transitioned from thread 0 at ../ofproto/ofproto-dpif-upcall.c:2540)
2023-02-24T13:00:31.197Z|00052|ofproto_dpif_upcall(revalidator68)|INFO|Unexpected ukey transition from state 3 (last transitioned from thread 0 at ../ofproto/ofproto-dpif-upcall.c:2540)
2023-02-24T13:00:31.198Z|00037|ofproto_dpif_upcall(revalidator62)|INFO|Unexpected ukey transition from state 3 (last transitioned from thread 0 at ../ofproto/ofproto-dpif-upcall.c:2540)
2023-02-24T13:00:31.202Z|00046|ofproto_dpif_upcall(revalidator58)|INFO|Unexpected ukey transition from state 3 (last transitioned from thread 0 at ../ofproto/ofproto-dpif-upcall.c:2540)
2023-02-24T13:00:31.202Z|00033|ofproto_dpif_upcall(revalidator69)|INFO|Unexpected ukey transition from state 3 (last transitioned from thread 0 at ../ofproto/ofproto-dpif-upcall.c:2540)
2023-02-24T13:00:31.202Z|00049|ofproto_dpif_upcall(revalidator70)|INFO|Unexpected ukey transition from state 3 (last transitioned from thread 0 at ../ofproto/ofproto-dpif-upcall.c:2540)
2023-02-24T13:00:31.202Z|00056|ofproto_dpif_upcall(revalidator67)|INFO|Unexpected ukey transition from state 3 (last transitioned from thread 0 at ../ofproto/ofproto-dpif-upcall.c:2540)
2023-02-24T13:00:31.202Z|00045|ofproto_dpif_upcall(revalidator64)|INFO|Unexpected ukey transition from state 3 (last transitioned from thread 0 at ../ofproto/ofproto-dpif-upcall.c:2540)
ovs_offload_pvp_xnuma_no_flows1024_queue1_vcpus3_size9000_vlan0_hwoffload_on_tcpolicy_none_end

Expected results:
no more error message on ovs-vswitchd.log

Additional info:
Comment 1 mhou 2023-03-09 07:49:36 UTC 
Run kernel-rt-5.14.0-283.rt14.283.el9 + openvswitch3.1-3.1.0-2.el9fdp hit very similar issue.
https://beaker.engineering.redhat.com/jobs/7602386
ovs log: https://beaker.engineering.redhat.com/recipes/13511072/tasks/157222558/logs/ovs-vswitchd.log
:: [ 06:05:05 ] :: [  BEGIN   ] :: Running 'grep -C 20 '|ERR|' /var/log/openvswitch/ovs-vswitchd.log'
2023-03-08T11:05:03.301Z|13313|dpif_netlink(revalidator4)|DBG|added flow
2023-03-08T11:05:03.302Z|13314|dpif_netlink(revalidator4)|DBG|added flow
2023-03-08T11:05:03.302Z|13315|dpif_netlink(revalidator4)|DBG|added flow
2023-03-08T11:05:03.302Z|13316|dpif_netlink(revalidator4)|DBG|added flow
2023-03-08T11:05:03.302Z|13317|dpif_netlink(revalidator4)|DBG|added flow
2023-03-08T11:05:03.302Z|13318|dpif_netlink(revalidator4)|DBG|added flow
2023-03-08T11:05:03.302Z|13319|dpif_netlink(revalidator4)|DBG|added flow
2023-03-08T11:05:03.302Z|13320|dpif_netlink(revalidator4)|DBG|added flow
2023-03-08T11:05:03.302Z|13321|dpif_netlink(revalidator4)|DBG|added flow
2023-03-08T11:05:03.302Z|13322|dpif_netlink(revalidator4)|DBG|added flow
2023-03-08T11:05:03.302Z|13323|dpif_netlink(revalidator4)|DBG|added flow
2023-03-08T11:05:03.302Z|13324|dpif_netlink(revalidator4)|DBG|added flow
2023-03-08T11:05:03.303Z|13325|dpif_netlink(revalidator4)|DBG|added flow
2023-03-08T11:05:03.303Z|13326|dpif_netlink(revalidator4)|DBG|added flow
2023-03-08T11:05:03.303Z|13327|dpif_netlink(revalidator4)|DBG|added flow
2023-03-08T11:05:03.303Z|13328|dpif_netlink(revalidator4)|DBG|added flow
2023-03-08T11:05:03.303Z|13329|ofproto_dpif_upcall(revalidator4)|INFO|Unexpected ukey transition from state 3 (last transitioned from thread 0 at ../ofproto/ofproto-dpif-upcall.c:2540)
2023-03-08T11:05:03.303Z|13330|netdev_offload_tc(revalidator4)|DBG|flow get (dev ens1f0np0 prio 2 handle 768 block_id 0)
2023-03-08T11:05:03.304Z|13331|dpif_netlink(revalidator4)|DBG|found flow from netdev, translating to dpif flow
2023-03-08T11:05:03.304Z|13332|netdev_offload_tc(revalidator4)|DBG|flow get (dev ens1f0np0 prio 2 handle 1338 block_id 0)
2023-03-08T11:05:03.304Z|13333|netdev_offload_tc(revalidator4)|ERR|flow get failed (dev ens1f0np0 prio 2 handle 1338): No such file or directory
2023-03-08T11:05:03.304Z|13334|netdev_offload_tc(revalidator4)|DBG|flow get (dev ovsbr1 prio 2 handle 1338 block_id 0)
2023-03-08T11:05:03.304Z|13335|netdev_offload_tc(revalidator4)|ERR|flow get failed (dev ovsbr1 prio 2 handle 1338): No such file or directory
2023-03-08T11:05:03.304Z|13336|netdev_offload_tc(revalidator4)|DBG|flow get (dev eth1 prio 2 handle 1338 block_id 0)
2023-03-08T11:05:03.304Z|13337|netdev_offload_tc(revalidator4)|ERR|flow get failed (dev eth1 prio 2 handle 1338): No such file or directory
2023-03-08T11:05:03.304Z|13338|netdev_offload_tc(revalidator4)|DBG|flow get (dev ens1f1np1 prio 2 handle 1338 block_id 0)
2023-03-08T11:05:03.304Z|13339|netdev_offload_tc(revalidator4)|ERR|flow get failed (dev ens1f1np1 prio 2 handle 1338): No such file or directory
2023-03-08T11:05:03.304Z|13340|netdev_offload_tc(revalidator4)|DBG|flow get (dev ovsbr0 prio 2 handle 1338 block_id 0)
2023-03-08T11:05:03.304Z|13341|netdev_offload_tc(revalidator4)|ERR|flow get failed (dev ovsbr0 prio 2 handle 1338): No such file or directory
2023-03-08T11:05:03.304Z|13342|netdev_offload_tc(revalidator4)|DBG|flow get (dev eth0 prio 2 handle 1338 block_id 0)
2023-03-08T11:05:03.304Z|13343|netdev_offload_tc(revalidator4)|DBG|flow get (dev ovs-system prio 2 handle 1338 block_id 0)
2023-03-08T11:05:03.304Z|13344|dpif_netlink(revalidator4)|DBG|added flow
2023-03-08T11:05:03.304Z|13345|dpif_netlink(revalidator4)|DBG|added flow
2023-03-08T11:05:03.305Z|13346|dpif_netlink(revalidator4)|DBG|added flow
2023-03-08T11:05:03.305Z|13347|dpif_netlink(revalidator4)|DBG|added flow
2023-03-08T11:05:03.305Z|13348|dpif_netlink(revalidator4)|DBG|added flow
2023-03-08T11:05:03.305Z|13349|dpif_netlink(revalidator4)|DBG|added flow
2023-03-08T11:05:03.305Z|13350|dpif_netlink(revalidator4)|DBG|added flow
2023-03-08T11:05:03.305Z|13351|dpif_netlink(revalidator4)|DBG|added flow
2023-03-08T11:05:03.305Z|13352|dpif_netlink(revalidator4)|DBG|added flow
2023-03-08T11:05:03.305Z|13353|dpif_netlink(revalidator4)|DBG|added flow
2023-03-08T11:05:03.305Z|13354|dpif_netlink(revalidator4)|DBG|added flow
2023-03-08T11:05:03.305Z|13355|dpif_netlink(revalidator4)|DBG|added flow
2023-03-08T11:05:03.305Z|13356|dpif_netlink(revalidator4)|DBG|added flow
2023-03-08T11:05:03.305Z|13357|dpif_netlink(revalidator4)|DBG|added flow
2023-03-08T11:05:03.305Z|13358|dpif_netlink(revalidator4)|DBG|added flow
2023-03-08T11:05:03.306Z|13359|dpif_netlink(revalidator4)|DBG|added flow
2023-03-08T11:05:03.306Z|13360|dpif_netlink(revalidator4)|DBG|added flow
2023-03-08T11:05:03.306Z|13361|dpif_netlink(revalidator4)|DBG|added flow
Comment 2 mhou 2023-03-16 08:22:51 UTC 
Run same test on same environment with kernel-rt-5.14.0-284.1.1.rt14.285.el9_2 can also hit this issue.

https://beaker.engineering.redhat.com/jobs/7628881
full of ovs-vswitchd.log: https://beaker-archive.hosts.prod.psi.bos.redhat.com/beaker-logs/2023/03/76288/7628881/13552994/157546339/ovs-vswitchd.log
Snippet of ovs-vswitchd.log
:: [ 22:22:57 ] :: [  BEGIN   ] :: Running 'grep -C 20 '|ERR|' /tmp/ovs_temp_log.log'
2023-03-16T02:22:36.970Z|08658|dpif_netlink(revalidator4)|DBG|added flow
2023-03-16T02:22:36.970Z|08659|dpif_netlink(revalidator4)|DBG|added flow
2023-03-16T02:22:36.970Z|08660|dpif_netlink(revalidator4)|DBG|added flow
2023-03-16T02:22:36.970Z|08661|ofproto_dpif_upcall(revalidator4)|INFO|Unexpected ukey transition from state 3 (last transitioned from thread 0 at ../ofproto/ofproto-dpif-upcall.c:2540)
2023-03-16T02:22:36.971Z|08662|ofproto_dpif_upcall(revalidator4)|INFO|Unexpected ukey transition from state 3 (last transitioned from thread 0 at ../ofproto/ofproto-dpif-upcall.c:2540)
2023-03-16T02:22:36.971Z|08663|ofproto_dpif_upcall(revalidator4)|INFO|Unexpected ukey transition from state 3 (last transitioned from thread 0 at ../ofproto/ofproto-dpif-upcall.c:2540)
2023-03-16T02:22:36.972Z|08664|ofproto_dpif_upcall(revalidator4)|INFO|Unexpected ukey transition from state 3 (last transitioned from thread 0 at ../ofproto/ofproto-dpif-upcall.c:2540)
2023-03-16T02:22:36.988Z|08665|netdev_offload_tc(revalidator4)|DBG|flow get (dev ovsbr1 prio 4 handle 22 block_id 0)
2023-03-16T02:22:36.988Z|08666|dpif_netlink(revalidator4)|DBG|found flow from netdev, translating to dpif flow
2023-03-16T02:22:36.988Z|08667|netdev_offload_tc(revalidator4)|DBG|flow get (dev ovsbr1 prio 4 handle 71 block_id 0)
2023-03-16T02:22:36.989Z|08668|dpif_netlink(revalidator4)|DBG|found flow from netdev, translating to dpif flow
2023-03-16T02:22:36.989Z|08669|netdev_offload_tc(revalidator4)|DBG|flow get (dev ovsbr1 prio 4 handle 81 block_id 0)
2023-03-16T02:22:36.989Z|08670|dpif_netlink(revalidator4)|DBG|found flow from netdev, translating to dpif flow
2023-03-16T02:22:36.989Z|08671|netdev_offload_tc(revalidator4)|DBG|flow get (dev ovsbr1 prio 4 handle 134 block_id 0)
2023-03-16T02:22:36.989Z|08672|dpif_netlink(revalidator4)|DBG|found flow from netdev, translating to dpif flow
2023-03-16T02:22:36.989Z|08673|netdev_offload_tc(revalidator4)|DBG|flow get (dev ovsbr1 prio 4 handle 208 block_id 0)
2023-03-16T02:22:36.989Z|08674|dpif_netlink(revalidator4)|DBG|found flow from netdev, translating to dpif flow
2023-03-16T02:22:36.989Z|08675|netdev_offload_tc(revalidator4)|DBG|flow get (dev ovsbr1 prio 4 handle 224 block_id 0)
2023-03-16T02:22:36.989Z|08676|dpif_netlink(revalidator4)|DBG|found flow from netdev, translating to dpif flow
2023-03-16T02:22:36.989Z|08677|netdev_offload_tc(revalidator4)|DBG|flow get (dev ovsbr1 prio 4 handle 876 block_id 0)
2023-03-16T02:22:36.989Z|08678|netdev_offload_tc(revalidator4)|ERR|flow get failed (dev ovsbr1 prio 4 handle 876): No such file or directory
2023-03-16T02:22:36.989Z|08679|netdev_offload_tc(revalidator4)|DBG|flow get (dev eth1 prio 4 handle 876 block_id 0)
2023-03-16T02:22:36.989Z|08680|netdev_offload_tc(revalidator4)|ERR|flow get failed (dev eth1 prio 4 handle 876): No such file or directory
2023-03-16T02:22:36.989Z|08681|netdev_offload_tc(revalidator4)|DBG|flow get (dev ens1f1np1 prio 4 handle 876 block_id 0)
2023-03-16T02:22:36.989Z|08682|netdev_offload_tc(revalidator4)|ERR|flow get failed (dev ens1f1np1 prio 4 handle 876): No such file or directory
2023-03-16T02:22:36.989Z|08683|netdev_offload_tc(revalidator4)|DBG|flow get (dev eth0 prio 4 handle 876 block_id 0)
2023-03-16T02:22:36.989Z|08684|netdev_offload_tc(revalidator4)|ERR|flow get failed (dev eth0 prio 4 handle 876): No such file or directory
2023-03-16T02:22:36.989Z|08685|netdev_offload_tc(revalidator4)|DBG|flow get (dev ovsbr0 prio 4 handle 876 block_id 0)
2023-03-16T02:22:36.989Z|08686|netdev_offload_tc(revalidator4)|ERR|flow get failed (dev ovsbr0 prio 4 handle 876): No such file or directory
2023-03-16T02:22:36.989Z|08687|netdev_offload_tc(revalidator4)|DBG|flow get (dev ens1f0np0 prio 4 handle 876 block_id 0)
2023-03-16T02:22:36.989Z|08688|netdev_offload_tc(revalidator4)|DBG|flow get (dev ovs-system prio 4 handle 876 block_id 0)
2023-03-16T02:22:36.989Z|08689|netdev_offload_tc(revalidator4)|DBG|flow get (dev ovsbr1 prio 4 handle 252 block_id 0)
2023-03-16T02:22:36.989Z|08690|dpif_netlink(revalidator4)|DBG|found flow from netdev, translating to dpif flow
2023-03-16T02:22:36.989Z|08691|netdev_offload_tc(revalidator4)|DBG|flow get (dev ovsbr1 prio 4 handle 287 block_id 0)
2023-03-16T02:22:36.989Z|08692|dpif_netlink(revalidator4)|DBG|found flow from netdev, translating to dpif flow
2023-03-16T02:22:36.989Z|08693|netdev_offload_tc(revalidator4)|DBG|flow get (dev ovsbr1 prio 4 handle 343 block_id 0)
2023-03-16T02:22:36.989Z|08694|dpif_netlink(revalidator4)|DBG|found flow from netdev, translating to dpif flow
2023-03-16T02:22:36.989Z|08695|netdev_offload_tc(revalidator4)|DBG|flow get (dev ovsbr1 prio 4 handle 353 block_id 0)
2023-03-16T02:22:36.989Z|08696|dpif_netlink(revalidator4)|DBG|found flow from netdev, translating to dpif flow
2023-03-16T02:22:36.989Z|08697|netdev_offload_tc(revalidator4)|DBG|flow get (dev ovsbr1 prio 4 handle 395 block_id 0)
2023-03-16T02:22:36.990Z|08698|dpif_netlink(revalidator4)|DBG|found flow from netdev, translating to dpif flow
2023-03-16T02:22:36.990Z|08699|netdev_offload_tc(revalidator4)|DBG|flow get (dev ovsbr1 prio 4 handle 455 block_id 0)
2023-03-16T02:22:36.990Z|08700|dpif_netlink(revalidator4)|DBG|found flow from netdev, translating to dpif flow
2023-03-16T02:22:36.990Z|08701|netdev_offload_tc(revalidator4)|DBG|flow get (dev ovsbr1 prio 4 handle 4 block_id 0)
2023-03-16T02:22:36.990Z|08702|dpif_netlink(revalidator4)|DBG|found flow from netdev, translating to dpif flow
2023-03-16T02:22:36.990Z|08703|netdev_offload_tc(revalidator4)|DBG|flow get (dev ovsbr1 prio 4 handle 5 block_id 0)
2023-03-16T02:22:36.990Z|08704|dpif_netlink(revalidator4)|DBG|found flow from netdev, translating to dpif flow
2023-03-16T02:22:36.990Z|08705|netdev_offload_tc(revalidator4)|DBG|flow get (dev ovsbr1 prio 4 handle 28 block_id 0)
2023-03-16T02:22:36.990Z|08706|dpif_netlink(revalidator4)|DBG|found flow from netdev, translating to dpif flow
Comment 3 mhou 2023-06-16 02:10:11 UTC 
This issue still can be reproduced on kernel-5.14.0-325.el9+rt + openvswitch3.1-3.1.0-24.el9fdp

beaker job:https://beaker.engineering.redhat.com/jobs/7975038

:: [ 14:55:18 ] :: [  BEGIN   ] :: Running 'check_error_message /tmp/ovs_temp_log.log'
2023-06-15T18:55:17.388Z|01588|tc|ERR|failed to parse flower classifier options
:: [ 14:55:18 ] :: [  BEGIN   ] :: Running 'grep -C 20 '|ERR|' /tmp/ovs_temp_log.log'
2023-06-15T18:55:17.386Z|41416|dpif_netlink(revalidator4)|DBG|added flow
2023-06-15T18:55:17.386Z|41417|dpif_netlink(revalidator4)|DBG|added flow
2023-06-15T18:55:17.386Z|41418|dpif_netlink(revalidator4)|DBG|added flow
2023-06-15T18:55:17.387Z|41419|dpif_netlink(revalidator4)|DBG|added flow
2023-06-15T18:55:17.387Z|41420|dpif_netlink(revalidator4)|DBG|added flow
2023-06-15T18:55:17.387Z|41421|dpif_netlink(revalidator4)|DBG|added flow
2023-06-15T18:55:17.387Z|41422|dpif_netlink(revalidator4)|DBG|added flow
2023-06-15T18:55:17.387Z|41423|dpif_netlink(revalidator4)|DBG|added flow
2023-06-15T18:55:17.387Z|41424|dpif_netlink(revalidator4)|DBG|added flow
2023-06-15T18:55:17.387Z|41425|dpif_netlink(revalidator4)|DBG|added flow
2023-06-15T18:55:17.387Z|41426|dpif_netlink(revalidator4)|DBG|added flow
2023-06-15T18:55:17.387Z|41427|dpif_netlink(revalidator4)|DBG|added flow
2023-06-15T18:55:17.388Z|41428|dpif_netlink(revalidator4)|DBG|added flow
2023-06-15T18:55:17.388Z|41429|dpif_netlink(revalidator4)|DBG|added flow
2023-06-15T18:55:17.388Z|41430|dpif_netlink(revalidator4)|DBG|added flow
2023-06-15T18:55:17.388Z|41431|dpif_netlink(revalidator4)|DBG|added flow
2023-06-15T18:55:17.388Z|41432|dpif_netlink(revalidator4)|DBG|added flow
2023-06-15T18:55:17.388Z|41433|dpif_netlink(revalidator4)|DBG|added flow
2023-06-15T18:55:17.388Z|41434|dpif_netlink(revalidator4)|DBG|added flow
2023-06-15T18:55:17.388Z|41435|dpif_netlink(revalidator4)|DBG|added flow
2023-06-15T18:55:17.388Z|01588|tc|ERR|failed to parse flower classifier options
2023-06-15T18:55:17.388Z|41436|dpif_netlink(revalidator4)|DBG|added flow
2023-06-15T18:55:17.388Z|41437|dpif_netlink(revalidator4)|DBG|added flow
2023-06-15T18:55:17.388Z|41438|dpif_netlink(revalidator4)|DBG|added flow
2023-06-15T18:55:17.388Z|41439|dpif_netlink(revalidator4)|DBG|added flow
2023-06-15T18:55:17.389Z|41440|dpif_netlink(revalidator4)|DBG|added flow
2023-06-15T18:55:17.389Z|41441|dpif_netlink(revalidator4)|DBG|added flow
2023-06-15T18:55:17.389Z|41442|dpif_netlink(revalidator4)|DBG|added flow
2023-06-15T18:55:17.389Z|41443|dpif_netlink(revalidator4)|DBG|added flow
2023-06-15T18:55:17.389Z|41444|dpif_netlink(revalidator4)|DBG|added flow
2023-06-15T18:55:17.389Z|41445|dpif_netlink(revalidator4)|DBG|added flow
2023-06-15T18:55:17.389Z|41446|dpif_netlink(revalidator4)|DBG|added flow
2023-06-15T18:55:17.389Z|41447|dpif_netlink(revalidator4)|DBG|added flow
2023-06-15T18:55:17.389Z|41448|dpif_netlink(revalidator4)|DBG|added flow
2023-06-15T18:55:17.389Z|41449|dpif_netlink(revalidator4)|DBG|added flow
2023-06-15T18:55:17.390Z|41450|dpif_netlink(revalidator4)|DBG|added flow
2023-06-15T18:55:17.390Z|41451|dpif_netlink(revalidator4)|DBG|added flow
2023-06-15T18:55:17.390Z|41452|dpif_netlink(revalidator4)|DBG|added flow
2023-06-15T18:55:17.390Z|41453|dpif_netlink(revalidator4)|DBG|added flow
2023-06-15T18:55:17.391Z|41454|dpif_netlink(revalidator4)|DBG|added flow
2023-06-15T18:55:17.391Z|41455|dpif_netlink(revalidator4)|DBG|added flow
:: [ 14:55:19 ] :: [   PASS   ] :: Command 'grep -C 20 '|ERR|' /tmp/ovs_temp_log.log' (Expected 0, got 0)
Comment 4 Eelco Chaudron 2023-06-19 14:27:26 UTC 
Hi @mhou, it looks like 2182942, 2173387, and 2143570 are related. Would it be possible to get access to a setup where I can replicate and further debug the issue?
Comment 5 mhou 2023-06-24 02:20:17 UTC 
Test on RHEL-9.3.0-20230621.0(kernel-5.14.0-325.el9+rt)+ openvswitch3.1-3.1.0-33.el9fdp and didn't hit this issue.
https://beaker.engineering.redhat.com/jobs/8002286