Created attachment 1945447 [details]
Flows and db dumps before and after the issue reproduces

Description of problem:
Detected in OpenStack Upstream CI[1][2]. Issue happens only on 22.03(22.03.0, 22.03.2, branch-22.03 etc). It's different than [3] as issue is seen with fix of it included. 

$ sudo ovs-vsctl list open .
_uuid               : 27746eeb-17bd-4d82-8efa-fc716594fda8
bridges             : [24bc6520-aa34-49e9-b4d0-ac6e09453672, 51a5686c-fcd7-4728-bb29-768e473d5826]
cur_cfg             : 634
datapath_types      : [netdev, system]
datapaths           : {system=3c3a1871-321f-450d-a3c6-8a8a86dd5d6c}
db_version          : []
dpdk_initialized    : false
dpdk_version        : none
external_ids        : {hostname=ykarel-jammytest, ovn-bridge=br-int, ovn-bridge-mappings="public:br-ex", ovn-cms-options=enable-chassis-as-gw, ovn-encap-ip="10.0.111.188", ovn-encap-type=geneve, ovn-remote="tcp:10.0.111.188:6642", system-id="c2a81ace-a1e3-491a-be11-97c8b6778374"}
iface_types         : [bareudp, erspan, geneve, gre, gtpu, internal, ip6erspan, ip6gre, lisp, patch, stt, system, tap, vxlan]
manager_options     : [b5aa86b7-b76b-4fc8-8716-2a1d86479d60]
next_cfg            : 634
other_config        : {vlan-limit="0"}
ovs_version         : []
ssl                 : []
statistics          : {}
system_type         : devstack
system_version      : []


Attached dbs, physical flows(with and without stats):-
- fresh(no tests runs)
- afterdummy resources create/cleanup
- beforerecompute flows miss detected
- after running recompute
- after cleanup of test resources

Version-Release number of selected component (if applicable):
ovn-22.03

How reproducible:
In CI[1][2] it's seen randomly. But with the reproducer below could reproduce consistently.

Steps to Reproduce:
1. Setup devstack with ovn22.03
2. Run dummy.sh shared below(reproduces only if logical_router "enabled=true" is set)
3. Run any of the tempest test relying on metadata flows, for ex:- tempest run --regex tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basic_ops
4. Check for flows of localport, those are missing
5. running recompute get backs the flows, ex:-

$ diff beforerecompute_flows_nostats.txt afterrecompute_flows_nostats.txt 
5a6
>  cookie=0x2233ffaa, priority=100,in_port=287 actions=load:0x10->NXM_NX_REG13[],load:0xa->NXM_NX_REG11[],load:0x9->NXM_NX_REG12[],load:0x5->OXM_OF_METADATA[],load:0x1->NXM_NX_REG14[],load:0x1->NXM_NX_REG10[10],resubmit(,8)
774a776
>  cookie=0x2233ffaa, table=37, priority=150,reg14=0x1,metadata=0x5 actions=resubmit(,38)
808d809
<  cookie=0x9cfac85f, table=38, priority=100,reg15=0x8000,metadata=0x5 actions=load:0xf->NXM_NX_REG13[],load:0x3->NXM_NX_REG15[],resubmit(,39),load:0x8000->NXM_NX_REG15[]
810c811,813
<  cookie=0xd295f58d, table=38, priority=100,reg15=0x8005,metadata=0x5 actions=load:0xf->NXM_NX_REG13[],load:0x3->NXM_NX_REG15[],resubmit(,39),load:0x8005->NXM_NX_REG15[]
---
>  cookie=0xd295f58d, table=38, priority=100,reg15=0x8005,metadata=0x5 actions=load:0x10->NXM_NX_REG13[],load:0xf->NXM_NX_REG13[],load:0x3->NXM_NX_REG15[],resubmit(,39),load:0x8005->NXM_NX_REG15[]
>  cookie=0x9cfac85f, table=38, priority=100,reg15=0x8000,metadata=0x5 actions=load:0x10->NXM_NX_REG13[],load:0xf->NXM_NX_REG13[],load:0x3->NXM_NX_REG15[],resubmit(,39),load:0x8000->NXM_NX_REG15[]
>  cookie=0x2233ffaa, table=38, priority=100,reg15=0x1,metadata=0x5 actions=load:0x10->NXM_NX_REG13[],load:0xa->NXM_NX_REG11[],load:0x9->NXM_NX_REG12[],resubmit(,39)
828a832
>  cookie=0x2233ffaa, table=39, priority=100,reg10=0/0x1,reg14=0x1,reg15=0x1,metadata=0x5 actions=drop
1082a1087
>  cookie=0x2233ffaa, table=64, priority=100,reg10=0x1/0x1,reg15=0x1,metadata=0x5 actions=push:NXM_OF_IN_PORT[],load:0xffff->NXM_OF_IN_PORT[],resubmit(,65),pop:NXM_OF_IN_PORT[]
1099a1105
>  cookie=0x2233ffaa, table=65, priority=100,reg15=0x1,metadata=0x5 actions=output:287
1103a1110
>  cookie=0xffc28b38, table=66, priority=100,reg0=0xac180401,reg15=0x1,metadata=0x6 actions=mod_dl_dst:22:65:bc:24:e9:49,load:0x1->NXM_NX_REG10[6]
1107a1115
>  cookie=0xffc28b38, table=67, priority=100,arp,reg0=0xac180401,reg14=0x1,metadata=0x6,dl_src=22:65:bc:24:e9:49 actions=load:0x1->NXM_NX_REG10[6]

6. rerunning the test again succeeds.

Actual results:
Flows missing for localport

Expected results:
Flows should exist for localport without running recompute, looks Incremental processing bug

Additional info:
=====dummy.sh=======
# cat repr.sh 
ovn-nbctl lr-add dummylr1 -- set logical_router dummylr1 enabled=true
ovn-nbctl ls-add dummyls1
ovn-nbctl lrp-add dummylr1 dummylr1-dummyls1 00:00:00:00:00:01 192.168.1.154/24
ovn-nbctl lsp-add dummyls1 dummyls1-dummylr1

# cleanup dummy resources
ovn-nbctl lsp-del dummyls1-dummylr1
ovn-nbctl lrp-del dummylr1-dummyls1
ovn-nbctl ls-del dummyls1
ovn-nbctl lr-del dummylr1


- ovn22.06 is not hitting this issue since [4], so some part of that commit fixes the issue.




[1] https://bugs.launchpad.net/neutron/+bug/2007166
[2] https://bugs.launchpad.net/neutron/+bug/2007565
[3] https://bugzilla.redhat.com/show_bug.cgi?id=2076604
[4] https://github.com/ovn-org/ovn/commit/eaf9832be248daca6d96202d504cc789d6dfec6d