Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

The FDP team is no longer accepting new bugs in Bugzilla. Please report your issues under FDP project in Jira. Thanks.

Bug 2071272

Summary:

Missing conjunction flows

Product:

Red Hat Enterprise Linux Fast Datapath

Reporter:

François Rigault <frigo>

Component:

ovn-2021

Assignee:

Numan Siddique <nusiddiq>

Status:

CLOSED ERRATA

QA Contact:

Ehsan Elahi <eelahi>

Severity:

medium

Docs Contact:

Priority:

high

Version:

RHEL 8.0

CC:

astupnik, ctrautma, dalvarez, jiji, jveiraca, mmichels, nusiddiq, yinxu, zhouhan

Target Milestone:

---

Target Release:

FDP 22.D

Hardware:

x86_64

OS:

Linux

Whiteboard:

Fixed In Version:

ovn-2021-21.12.0-46

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

Clones:

2079867 (view as bug list)

Environment:

Last Closed:

2022-05-27 18:14:06 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Bug Depends On:

Bug Blocks:

2079867

Attachments:

Description	Flags
ovn-controller	none

Description François Rigault 2022-04-02 15:07:11 UTC

Description of problem:
I'm investigating an issue running RHOSP16.2.2. Security group rules containing port ranges do not work reliably. Looking at the flows, some conjunctions flows are missing their "2/2" part.

Version-Release number of selected component (if applicable):
ovn-2021-host-21.12.0-11.el8fdp.x86_64, from https://access.redhat.com/containers/#/registry.access.redhat.com/rhosp16/openstack-ovn-controller/images/16.2.2-13

How reproducible:
intermittently

Steps to Reproduce: (hopefully I can simplify this)
1. deploy RHOSP 16.2.2
2. deploy OpenShift 4.10 on top
3. perform an upgrade or an update of machine config that triggers a reboot of all VMs.

Actual results:
security group are not working and traffic between some VMs on some port does not work.

Expected results:
security group should be applied properly

Additional info:
# some flows are missing the 2/2 part:

$ for i in `sudo ovs-ofctl -O OpenFlow14 dump-flows br-int | grep -o -P '(?<=conj_id=)\d*' | sort -u` ; do sudo ovs-ofctl -O OpenFlow14 dump-flows br-int | grep $i | grep 2/2 > /dev/null || echo missing for $i ;done
missing for 1577368725
missing for 2535284668
missing for 3637212797
missing for 4190879339
missing for 783999008

$ sudo ovs-ofctl -O OpenFlow14 dump-flows br-int | grep 1577368725
 cookie=0x0, duration=2761.553s, table=44, n_packets=0, n_bytes=0, priority=2002,udp,reg0=0x100/0x100,reg15=0x3,metadata=0x4,nw_src=10.0.0.0/16 actions=conjunction(1577368725,1/2)
 cookie=0x0, duration=2761.553s, table=44, n_packets=0, n_bytes=0, priority=2002,udp,reg0=0x100/0x100,reg15=0x5,metadata=0x4,nw_src=10.0.0.0/16 actions=conjunction(1577368725,1/2)
 cookie=0x5e04bc95, duration=2761.554s, table=44, n_packets=0, n_bytes=0, priority=2002,conj_id=1577368725,udp,reg0=0x100/0x100,metadata=0x4,nw_src=10.0.0.0/16 actions=resubmit(,45)

$ sudo ovs-ofctl -O OpenFlow14 dump-flows br-int | grep 2535284668
 cookie=0x0, duration=2814.312s, table=44, n_packets=0, n_bytes=0, priority=2002,tcp,reg0=0x80/0x80,reg15=0x5,metadata=0x4,nw_src=10.0.0.0/16 actions=conjunction(3702371357,1/2),conjunction(1120155173,1/2),conjunction(2535284668,1/2)
 cookie=0x0, duration=2814.312s, table=44, n_packets=0, n_bytes=0, priority=2002,tcp,reg0=0x80/0x80,reg15=0x3,metadata=0x4,nw_src=10.0.0.0/16 actions=conjunction(3702371357,1/2),conjunction(1120155173,1/2),conjunction(2535284668,1/2)
 cookie=0x971d5fbc, duration=2814.313s, table=44, n_packets=0, n_bytes=0, priority=2002,conj_id=2535284668,tcp,reg0=0x80/0x80,metadata=0x4,nw_src=10.0.0.0/16 actions=
load:0x1->NXM_NX_XXREG0[97],resubmit(,45)


# finding the associated logical flow from the ovn_controller:
# ovs-appctl -t /var/run/ovn/ovn-controller.7.ctl debug/dump-lflow-conj-ids | grep -P '1577368725|2535284668|3637212797|4190879339|783999008'
lflow: 2ebae020-1e27-4454-a39a-cb30e672adba, start: 783999008, n: 1
lflow: 5e04bc95-8091-4bb0-bf9a-57c091dc373a, start: 1577368725, n: 1
lflow: f9cbbe6b-48e6-49da-a612-f12545f5170a, start: 4190879339, n: 1
lflow: 971d5fbc-e1dd-4a6e-9eb0-295e126b4174, start: 2535284668, n: 1
lflow: d8cb767d-3ce4-4253-86e3-a748d1f7df70, start: 3637212797, n: 1

# ovs-appctl -t /var/run/ovn/ovn-controller.7.ctl lflow-cache/show-stats
Enabled: true
high-watermark  : 574
total           : 574
cache-expr      : 350
cache-matches   : 224
trim count      : 57
Mem usage (KB)  : 1263


# looking at the logical flow in southdb:
# ovn-sbctl list logical_flow 5e04bc95-8091-4bb0-bf9a-57c091dc373a
_uuid               : 5e04bc95-8091-4bb0-bf9a-57c091dc373a
actions             : "next;"
controller_meter    : []
external_ids        : {source="northd.c:6244", stage-hint=db80d5a3, stage-name=ls_out_acl}
logical_datapath    : ed696f15-2ae0-4ebc-ad9f-f4d8f1c358d0
logical_dp_group    : []
match               : "reg0[8] == 1 && (outport == @pg_b0371d80_7a8b_4a0d_af77_3b8ce5b85798 && ip4 && ip4.src == 10.0.0.0/16 && udp && udp.dst >= 9000 && udp.dst <= 9999)"
pipeline            : egress
priority            : 2002
table_id            : 4
tags                : {}
hash                : 0

# ovn-sbctl list logical_flow 971d5fbc-e1dd-4a6e-9eb0-295e126b4174
_uuid               : 971d5fbc-e1dd-4a6e-9eb0-295e126b4174
actions             : "reg0[1] = 1; next;"
controller_meter    : []
external_ids        : {source="northd.c:6220", stage-hint="8a905ef0", stage-name=ls_out_acl}
logical_datapath    : ed696f15-2ae0-4ebc-ad9f-f4d8f1c358d0
logical_dp_group    : []
match               : "reg0[7] == 1 && (outport == @pg_b0371d80_7a8b_4a0d_af77_3b8ce5b85798 && ip4 && ip4.src == 10.0.0.0/16 && tcp && tcp.dst >= 9000 && tcp.dst <= 9999)"
pipeline            : egress
priority            : 2002
table_id            : 4
tags                : {}
hash                : 0

restarting the ovn_controller container fixes it - should look similar to this:
# sudo ovs-ofctl -O OpenFlow14 dump-flows br-int | grep 2535284668
 cookie=0x0, duration=69.353s, table=44, n_packets=0, n_bytes=0, priority=2002,tcp,reg0=0x80/0x80,metadata=0x4,nw_src=10.0.0.0/16,tp_dst=0x270c/0xfffe actions=conjunction(342635760,2/2),conjunction(2535284668,2/2)
 cookie=0x0, duration=69.344s, table=44, n_packets=0, n_bytes=0, priority=2002,tcp,reg0=0x80/0x80,metadata=0x4,nw_src=10.0.0.0/16,tp_dst=0x232a/0xfffe actions=conjunction(342635760,2/2),conjunction(2535284668,2/2)
 cookie=0x971d5fbc, duration=69.354s, table=44, n_packets=33, n_bytes=2442, priority=2002,conj_id=2535284668,tcp,reg0=0x80/0x80,metadata=0x4,nw_src=10.0.0.0/16 actions=load:0x1->NXM_NX_XXREG0[97],resubmit(,45)
 cookie=0x0, duration=69.354s, table=44, n_packets=0, n_bytes=0, priority=2002,tcp,reg0=0x80/0x80,metadata=0x4,nw_src=10.0.0.0/16,tp_dst=9999 actions=conjunction(342635760,2/2),conjunction(2535284668,2/2)
 cookie=0x0, duration=69.352s, table=44, n_packets=0, n_bytes=0, priority=2002,tcp,reg0=0x80/0x80,metadata=0x4,nw_src=10.0.0.0/16,tp_dst=9001 actions=conjunction(342635760,2/2),conjunction(2535284668,2/2)
 cookie=0x0, duration=69.350s, table=44, n_packets=0, n_bytes=0, priority=2002,tcp,reg0=0x80/0x80,metadata=0x4,nw_src=10.0.0.0/16,tp_dst=9000 actions=conjunction(342635760,2/2),conjunction(2535284668,2/2)
 cookie=0x0, duration=69.345s, table=44, n_packets=0, n_bytes=0, priority=2002,tcp,reg0=0x80/0x80,metadata=0x4,nw_src=10.0.0.0/16,tp_dst=9998 actions=conjunction(342635760,2/2),conjunction(2535284668,2/2)
 cookie=0x0, duration=69.352s, table=44, n_packets=0, n_bytes=0, priority=2002,tcp,reg0=0x80/0x80,reg15=0x6,metadata=0x4,nw_src=10.0.0.0/16 actions=conjunction(3702371357,1/2),conjunction(1120155173,1/2),conjunction(2535284668,1/2)
 cookie=0x0, duration=69.349s, table=44, n_packets=0, n_bytes=0, priority=2002,tcp,reg0=0x80/0x80,reg15=0x3,metadata=0x4,nw_src=10.0.0.0/16 actions=conjunction(3702371357,1/2),conjunction(1120155173,1/2),conjunction(2535284668,1/2)
 cookie=0x0, duration=69.352s, table=44, n_packets=0, n_bytes=0, priority=2002,tcp,reg0=0x80/0x80,metadata=0x4,nw_src=10.0.0.0/16,tp_dst=0x232c/0xfffc actions=conjunction(342635760,2/2),conjunction(2535284668,2/2)
 cookie=0x0, duration=69.347s, table=44, n_packets=0, n_bytes=0, priority=2002,tcp,reg0=0x80/0x80,metadata=0x4,nw_src=10.0.0.0/16,tp_dst=0x2708/0xfffc actions=conjunction(342635760,2/2),conjunction(2535284668,2/2)
 cookie=0x0, duration=69.351s, table=44, n_packets=0, n_bytes=0, priority=2002,tcp,reg0=0x80/0x80,metadata=0x4,nw_src=10.0.0.0/16,tp_dst=0x2380/0xff80 actions=conjunction(342635760,2/2),conjunction(2535284668,2/2)
 cookie=0x0, duration=69.350s, table=44, n_packets=0, n_bytes=0, priority=2002,tcp,reg0=0x80/0x80,metadata=0x4,nw_src=10.0.0.0/16,tp_dst=0x2700/0xfff8 actions=conjunction(342635760,2/2),conjunction(2535284668,2/2)
 cookie=0x0, duration=69.350s, table=44, n_packets=0, n_bytes=0, priority=2002,tcp,reg0=0x80/0x80,metadata=0x4,nw_src=10.0.0.0/16,tp_dst=0x2340/0xffc0 actions=conjunction(342635760,2/2),conjunction(2535284668,2/2)
 cookie=0x0, duration=69.348s, table=44, n_packets=0, n_bytes=0, priority=2002,tcp,reg0=0x80/0x80,metadata=0x4,nw_src=10.0.0.0/16,tp_dst=0x2600/0xff00 actions=conjunction(342635760,2/2),conjunction(2535284668,2/2)
 cookie=0x0, duration=69.348s, table=44, n_packets=0, n_bytes=0, priority=2002,tcp,reg0=0x80/0x80,metadata=0x4,nw_src=10.0.0.0/16,tp_dst=0x2400/0xfe00 actions=conjunction(342635760,2/2),conjunction(2535284668,2/2)
 cookie=0x0, duration=69.346s, table=44, n_packets=0, n_bytes=0, priority=2002,tcp,reg0=0x80/0x80,metadata=0x4,nw_src=10.0.0.0/16,tp_dst=0x2330/0xfff0 actions=conjunction(342635760,2/2),conjunction(2535284668,2/2)

Any help investigating this is appreciated!

Comment 1 François Rigault 2022-04-02 21:37:20 UTC

with debug logs on the ovn-controller, we see

$ grep 'tcp,reg0=0x80/0x80,metadata=0x4,nw_src=10.0.0.0/16,tp_dst=0x232c/0xfffc' ovn-controller.log

2022-04-02T20:44:37.814Z|28820|ofctrl|DBG|ofctrl_add_or_append_flow (append) flow: cookie=0, table_id=44, priority=2002, tcp,reg0=0x80/0x80,metadata=0x4,nw_src=10.0.0.0/16,tp_dst=0x232c/0xfffc, actions=conjunction(342635760,2/2),conjunction(2535284668,2/2)
2022-04-02T20:44:37.821Z|29248|ofctrl|DBG|flood remove flow: cookie=0, table_id=44, priority=2002, tcp,reg0=0x80/0x80,metadata=0x4,nw_src=10.0.0.0/16,tp_dst=0x232c/0xfffc, actions=conjunction(342635760,2/2),conjunction(2535284668,2/2)
2022-04-02T20:44:37.829Z|29671|ofctrl|DBG|ofctrl_add_or_append_flow (add) flow: cookie=0, table_id=44, priority=2002, tcp,reg0=0x80/0x80,metadata=0x4,nw_src=10.0.0.0/16,tp_dst=0x232c/0xfffc, actions=conjunction(342635760,2/2)
2022-04-02T20:44:37.832Z|29801|ofctrl|DBG|ofctrl_add_or_append_flow (append) flow: cookie=0, table_id=44, priority=2002, tcp,reg0=0x80/0x80,metadata=0x4,nw_src=10.0.0.0/16,tp_dst=0x232c/0xfffc, actions=conjunction(342635760,2/2),conjunction(2535284668,2/2)

for all the occurrences we have, the problem happens only for conjunctions that have been "appended" to existing flows. The first conjunction that was "added" is properly created, but the one that was "appended" is nowhere to be found.

$ sudo ovs-ofctl -O OpenFlow14 dump-flows br-int | grep 'tcp,reg0=0x80/0x80,metadata=0x4,nw_src=10.0.0.0/16,tp_dst=0x232c/0xfffc' 
 cookie=0x0, duration=3128.071s, table=44, n_packets=0, n_bytes=0, priority=2002,tcp,reg0=0x80/0x80,metadata=0x4,nw_src=10.0.0.0/16,tp_dst=0x232c/0xfffc actions=conjunction(342635760,2/2)

the "append" vanished...

Comment 2 François Rigault 2022-04-03 08:58:42 UTC

I've been able to reproduce trying this a few times in OVN sandbox
OVN a1001ce71 https://github.com/ovn-org/ovn/tree/v21.12.1 , OVS 498cedc


# step1:
ovn-nbctl ls-add sw

ovn-nbctl lsp-add sw p1
ovn-nbctl lsp-set-addresses p1 "00:00:00:00:00:02 192.168.0.2"

ovn-nbctl lsp-add sw p2
ovn-nbctl lsp-set-addresses p2 "00:00:00:00:00:03 192.168.0.3"

ovn-nbctl lsp-add sw p3
ovn-nbctl lsp-set-addresses p3 "00:00:00:00:00:04 192.168.0.4"

ovn-nbctl lsp-add sw p4
ovn-nbctl lsp-set-addresses p4 "00:00:00:00:00:05 192.168.0.5"

ovn-nbctl lsp-add sw p5
ovn-nbctl lsp-set-addresses p5 "00:00:00:00:00:06 192.168.0.6"

ovn-nbctl lsp-add sw p6
ovn-nbctl lsp-set-addresses p6 "00:00:00:00:00:07 192.168.0.7"


# Now bind the ports on the hypervisor

ovs-vsctl add-port br-int p1 -- set Interface p1 external_ids:iface-id=p1
ovs-vsctl add-port br-int p2 -- set Interface p2 external_ids:iface-id=p2
ovs-vsctl add-port br-int p3 -- set Interface p3 external_ids:iface-id=p3
ovs-vsctl add-port br-int p4 -- set Interface p4 external_ids:iface-id=p4
ovs-vsctl add-port br-int p5 -- set Interface p5 external_ids:iface-id=p5
ovs-vsctl add-port br-int p6 -- set Interface p6 external_ids:iface-id=p6


ovn-nbctl pg-add pg1 p1 p2 p3
ovn-nbctl pg-add pg2 p4 p5 p6

ovn-nbctl acl-add pg1 to-lport 2003 "inport==@pg1 && ip4 && ip4.src == 192.168.0.0/16 && udp && udp.dst >= 9000 && udp.dst <= 9999"  allow
ovn-nbctl acl-add pg2 to-lport 2003 "inport==@pg2 && ip4 && ip4.src == 192.168.0.0/16 && udp && udp.dst >= 9000 && udp.dst <= 9999"  allow


# step2, might need to run a few times or change the order of interfaces:
ovs-vsctl set Interface p1 external_ids:iface-id=p1x \
 -- set Interface p2 external_ids:iface-id=p2x \
 -- set Interface p3 external_ids:iface-id=p3x \
 -- set Interface p4 external_ids:iface-id=p4x \
 -- set Interface p5 external_ids:iface-id=p5x \
 -- set Interface p6 external_ids:iface-id=p6x

ovs-vsctl set Interface p4 external_ids:iface-id=p4
ovs-vsctl set Interface p5 external_ids:iface-id=p5
ovs-vsctl set Interface p6 external_ids:iface-id=p6
ovs-vsctl set Interface p1 external_ids:iface-id=p1
ovs-vsctl set Interface p2 external_ids:iface-id=p2
ovs-vsctl set Interface p3 external_ids:iface-id=p3

# step3: check the flows and see if both the conjunction 1/2 and 2/2 are present:

ovs-appctl -t `pwd`/sandbox/ovn-controller.1423.ctl debug/dump-lflow-conj-ids
ovs-ofctl -O OpenFlow14 dump-flows br-int | grep  1654221142
 cookie=0x4fcc661d, duration=88.967s, table=44, n_packets=0, n_bytes=0, priority=3003,conj_id=1654221142,udp,metadata=0x1,nw_src=192.168.0.0/16 actions=resubmit(,45)
 cookie=0x0, duration=88.967s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,reg14=0x2,metadata=0x1,nw_src=192.168.0.0/16 actions=conjunction(1654221142,1/2)
 cookie=0x0, duration=88.967s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,reg14=0x1,metadata=0x1,nw_src=192.168.0.0/16 actions=conjunction(1654221142,1/2)
 cookie=0x0, duration=88.954s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,reg14=0x3,metadata=0x1,nw_src=192.168.0.0/16 actions=conjunction(1654221142,1/2)

will send a case.

Comment 3 François Rigault 2022-04-03 10:48:36 UTC

bisected to 
https://github.com/ovn-org/ovn/commit/8994d4ac238ec9550e3d9de378f64215b9bcec9b
for the first fixed version.

Comment 6 Numan Siddique 2022-04-11 14:14:01 UTC

@François Rigault  - Thanks a lot for sharing the reproducer in #c4.  Looking into it.

Comment 8 Numan Siddique 2022-04-11 16:34:11 UTC

@François Rigault - Is it possible for you to enable ovn-controller debug logs in your sandbox setup and share the logs when the issue is reproduced ?

So far I'm not able to reproduce locally and hence the debug logs would be really helpful.

Thanks

Comment 9 François Rigault 2022-04-11 18:32:40 UTC

Created attachment 1871859 [details]
ovn-controller

sure. OVN a1001ce71, vagrant up debian-10, I use the reproducer steps above. The first try did not work, so I ran again with:

ovs-vsctl set Interface p1 external_ids:iface-id=p1x \
 -- set Interface p2 external_ids:iface-id=p2x \
 -- set Interface p3 external_ids:iface-id=p3x \
 -- set Interface p4 external_ids:iface-id=p4x \
 -- set Interface p5 external_ids:iface-id=p5x \
 -- set Interface p6 external_ids:iface-id=p6x

# create the conjunction for pg1 before pg2 this time
ovs-vsctl set Interface p1 external_ids:iface-id=p1
ovs-vsctl set Interface p2 external_ids:iface-id=p2
ovs-vsctl set Interface p3 external_ids:iface-id=p3
ovs-vsctl set Interface p4 external_ids:iface-id=p4
ovs-vsctl set Interface p5 external_ids:iface-id=p5
ovs-vsctl set Interface p6 external_ids:iface-id=p6

and reproduced. the order the interface get added matters.

Comment 10 François Rigault 2022-04-11 18:35:13 UTC

first tried that failed:
root@debian-10:/vagrant/ovn/tutorial# ovs-appctl -t `pwd`/sandbox/ovn-controller.*.ctl debug/dump-lflow-conj-ids
Conjunction IDs allocations:
lflow: bbeccea6-5c96-4927-a8b9-1b08eaf74e16, dp: 87e55c36-6c92-4d5b-be01-751d076aa4ca, start: 3453301836, n: 1
lflow: b6171a0c-4309-47ff-b018-ff952ad5ef0f, dp: 87e55c36-6c92-4d5b-be01-751d076aa4ca, start: 2304738621, n: 1
---
Total 2 IDs used.
root@debian-10:/vagrant/ovn/tutorial# ovs-ofctl -O OpenFlow14 dump-flows br-int | grep  3453301836
 cookie=0x0, duration=57.458s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,reg14=0x5,metadata=0x1,nw_src=192.168.0.0/16 actions=conjunction(3453301836,1/2)
 cookie=0x0, duration=57.458s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,reg14=0x6,metadata=0x1,nw_src=192.168.0.0/16 actions=conjunction(3453301836,1/2)
 cookie=0x0, duration=57.457s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,reg14=0x4,metadata=0x1,nw_src=192.168.0.0/16 actions=conjunction(3453301836,1/2)
 cookie=0xbbeccea6, duration=57.458s, table=44, n_packets=0, n_bytes=0, priority=3003,conj_id=3453301836,udp,metadata=0x1,nw_src=192.168.0.0/16 actions=resubmit(,45)
 cookie=0x0, duration=57.459s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x2340/0xffc0 actions=conjunction(2304738621,2/2),conjunction(3453301836,2/2)
 cookie=0x0, duration=57.459s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=9999 actions=conjunction(2304738621,2/2),conjunction(3453301836,2/2)
 cookie=0x0, duration=57.458s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=9000 actions=conjunction(2304738621,2/2),conjunction(3453301836,2/2)
 cookie=0x0, duration=57.458s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=9998 actions=conjunction(2304738621,2/2),conjunction(3453301836,2/2)
 cookie=0x0, duration=57.457s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=9001 actions=conjunction(2304738621,2/2),conjunction(3453301836,2/2)
 cookie=0x0, duration=57.459s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x2380/0xff80 actions=conjunction(2304738621,2/2),conjunction(3453301836,2/2)
 cookie=0x0, duration=57.458s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x270c/0xfffe actions=conjunction(2304738621,2/2),conjunction(3453301836,2/2)
 cookie=0x0, duration=57.458s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x232a/0xfffe actions=conjunction(2304738621,2/2),conjunction(3453301836,2/2)
 cookie=0x0, duration=57.458s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x2708/0xfffc actions=conjunction(2304738621,2/2),conjunction(3453301836,2/2)
 cookie=0x0, duration=57.458s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x232c/0xfffc actions=conjunction(2304738621,2/2),conjunction(3453301836,2/2)
 cookie=0x0, duration=57.458s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x2330/0xfff0 actions=conjunction(2304738621,2/2),conjunction(3453301836,2/2)
 cookie=0x0, duration=57.458s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x2700/0xfff8 actions=conjunction(2304738621,2/2),conjunction(3453301836,2/2)
 cookie=0x0, duration=57.457s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x2400/0xfe00 actions=conjunction(2304738621,2/2),conjunction(3453301836,2/2)
 cookie=0x0, duration=57.457s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x2600/0xff00 actions=conjunction(2304738621,2/2),conjunction(3453301836,2/2)
root@debian-10:/vagrant/ovn/tutorial# ovs-ofctl -O OpenFlow14 dump-flows br-int | grep  2304738621
 cookie=0x0, duration=63.569s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,reg14=0x2,metadata=0x1,nw_src=192.168.0.0/16 actions=conjunction(2304738621,1/2)
 cookie=0x0, duration=63.569s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,reg14=0x1,metadata=0x1,nw_src=192.168.0.0/16 actions=conjunction(2304738621,1/2)
 cookie=0x0, duration=63.114s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,reg14=0x3,metadata=0x1,nw_src=192.168.0.0/16 actions=conjunction(2304738621,1/2)
 cookie=0xb6171a0c, duration=63.569s, table=44, n_packets=0, n_bytes=0, priority=3003,conj_id=2304738621,udp,metadata=0x1,nw_src=192.168.0.0/16 actions=resubmit(,45)
 cookie=0x0, duration=63.569s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x2340/0xffc0 actions=conjunction(2304738621,2/2),conjunction(3453301836,2/2)
 cookie=0x0, duration=63.569s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=9999 actions=conjunction(2304738621,2/2),conjunction(3453301836,2/2)
 cookie=0x0, duration=63.568s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=9000 actions=conjunction(2304738621,2/2),conjunction(3453301836,2/2)
 cookie=0x0, duration=63.568s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=9998 actions=conjunction(2304738621,2/2),conjunction(3453301836,2/2)
 cookie=0x0, duration=63.567s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=9001 actions=conjunction(2304738621,2/2),conjunction(3453301836,2/2)
 cookie=0x0, duration=63.569s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x2380/0xff80 actions=conjunction(2304738621,2/2),conjunction(3453301836,2/2)
 cookie=0x0, duration=63.568s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x270c/0xfffe actions=conjunction(2304738621,2/2),conjunction(3453301836,2/2)
 cookie=0x0, duration=63.568s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x232a/0xfffe actions=conjunction(2304738621,2/2),conjunction(3453301836,2/2)
 cookie=0x0, duration=63.568s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x2708/0xfffc actions=conjunction(2304738621,2/2),conjunction(3453301836,2/2)
 cookie=0x0, duration=63.568s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x232c/0xfffc actions=conjunction(2304738621,2/2),conjunction(3453301836,2/2)
 cookie=0x0, duration=63.568s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x2330/0xfff0 actions=conjunction(2304738621,2/2),conjunction(3453301836,2/2)
 cookie=0x0, duration=63.568s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x2700/0xfff8 actions=conjunction(2304738621,2/2),conjunction(3453301836,2/2)
 cookie=0x0, duration=63.567s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x2400/0xfe00 actions=conjunction(2304738621,2/2),conjunction(3453301836,2/2)
 cookie=0x0, duration=63.567s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x2600/0xff00 actions=conjunction(2304738621,2/2),conjunction(3453301836,2/2)


after reproducing:
# ovs-ofctl -O OpenFlow14 dump-flows br-int | grep  2304738621
 cookie=0xb6171a0c, duration=14.812s, table=44, n_packets=0, n_bytes=0, priority=3003,conj_id=2304738621,udp,metadata=0x1,nw_src=192.168.0.0/16 actions=resubmit(,45)
 cookie=0x0, duration=14.812s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x2700/0xfff8 actions=conjunction(2304738621,2/2)
 cookie=0x0, duration=14.812s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x2600/0xff00 actions=conjunction(2304738621,2/2)
 cookie=0x0, duration=14.812s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x2400/0xfe00 actions=conjunction(2304738621,2/2)
 cookie=0x0, duration=14.813s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x2708/0xfffc actions=conjunction(2304738621,2/2)
 cookie=0x0, duration=14.812s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x232c/0xfffc actions=conjunction(2304738621,2/2)
 cookie=0x0, duration=14.812s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x2330/0xfff0 actions=conjunction(2304738621,2/2)
 cookie=0x0, duration=14.813s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x270c/0xfffe actions=conjunction(2304738621,2/2)
 cookie=0x0, duration=14.812s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x232a/0xfffe actions=conjunction(2304738621,2/2)
 cookie=0x0, duration=14.813s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x2380/0xff80 actions=conjunction(2304738621,2/2)
 cookie=0x0, duration=14.813s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x2340/0xffc0 actions=conjunction(2304738621,2/2)
 cookie=0x0, duration=14.813s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,reg14=0x2,metadata=0x1,nw_src=192.168.0.0/16 actions=conjunction(2304738621,1/2)
 cookie=0x0, duration=14.812s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,reg14=0x1,metadata=0x1,nw_src=192.168.0.0/16 actions=conjunction(2304738621,1/2)
 cookie=0x0, duration=14.812s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,reg14=0x3,metadata=0x1,nw_src=192.168.0.0/16 actions=conjunction(2304738621,1/2)
 cookie=0x0, duration=14.813s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=9999 actions=conjunction(2304738621,2/2)
 cookie=0x0, duration=14.812s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=9000 actions=conjunction(2304738621,2/2)
 cookie=0x0, duration=14.812s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=9998 actions=conjunction(2304738621,2/2)
 cookie=0x0, duration=14.812s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=9001 actions=conjunction(2304738621,2/2)

# ovs-ofctl -O OpenFlow14 dump-flows br-int | grep  3453301836
 cookie=0xbbeccea6, duration=16.264s, table=44, n_packets=0, n_bytes=0, priority=3003,conj_id=3453301836,udp,metadata=0x1,nw_src=192.168.0.0/16 actions=resubmit(,45)
 cookie=0x0, duration=16.264s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,reg14=0x5,metadata=0x1,nw_src=192.168.0.0/16 actions=conjunction(3453301836,1/2)
 cookie=0x0, duration=16.264s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,reg14=0x4,metadata=0x1,nw_src=192.168.0.0/16 actions=conjunction(3453301836,1/2)
 cookie=0x0, duration=15.759s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,reg14=0x6,metadata=0x1,nw_src=192.168.0.0/16 actions=conjunction(3453301836,1/2)

Comment 11 Numan Siddique 2022-04-11 19:33:06 UTC

Thanks a lot.  These logs are super useful.

Comment 12 Numan Siddique 2022-04-11 20:56:57 UTC

With the below script I can reproduce almost 100% of the time.  This script is almost the same as in #c2 except that this makes it deterministic.

------------------------------------------------
ovn-nbctl ls-add sw

ovn-nbctl lsp-add sw p1
ovn-nbctl lsp-set-addresses p1 "00:00:00:00:00:02 192.168.0.2"

ovn-nbctl lsp-add sw p2
ovn-nbctl lsp-set-addresses p2 "00:00:00:00:00:03 192.168.0.3"

ovn-nbctl lsp-add sw p3
ovn-nbctl lsp-set-addresses p3 "00:00:00:00:00:04 192.168.0.4"

ovn-nbctl lsp-add sw p4
ovn-nbctl lsp-set-addresses p4 "00:00:00:00:00:05 192.168.0.5"

ovn-nbctl lsp-add sw p5
ovn-nbctl lsp-set-addresses p5 "00:00:00:00:00:06 192.168.0.6"

ovn-nbctl lsp-add sw p6
ovn-nbctl lsp-set-addresses p6 "00:00:00:00:00:07 192.168.0.7"


# Now bind the ports on the hypervisor

ovs-vsctl add-port br-int p1 -- set Interface p1 external_ids:iface-id=p1
ovs-vsctl add-port br-int p2 -- set Interface p2 external_ids:iface-id=p2
ovs-vsctl add-port br-int p3 -- set Interface p3 external_ids:iface-id=p3
ovs-vsctl add-port br-int p4 -- set Interface p4 external_ids:iface-id=p4
ovs-vsctl add-port br-int p5 -- set Interface p5 external_ids:iface-id=p5
ovs-vsctl add-port br-int p6 -- set Interface p6 external_ids:iface-id=p6


ovn-nbctl pg-add pg1 p1 p2 p3
ovn-nbctl pg-add pg2 p4 p5 p6

ovn-nbctl acl-add pg1 to-lport 2003 "inport==@pg1 && ip4 && ip4.src == 192.168.0.0/16 && udp && udp.dst >= 9000 && udp.dst <= 9999"  allow
ovn-nbctl acl-add pg2 to-lport 2003 "inport==@pg2 && ip4 && ip4.src == 192.168.0.0/16 && udp && udp.dst >= 9000 && udp.dst <= 9999"  allow

ovn-nbctl --wait=hv sync

ovn-appctl -t ovn-controller vlog/set dbg


ovs-vsctl set Interface p1 external_ids:iface-id=p1x \
 -- set Interface p2 external_ids:iface-id=p2x \
 -- set Interface p3 external_ids:iface-id=p3x \
 -- set Interface p4 external_ids:iface-id=p4x \
 -- set Interface p5 external_ids:iface-id=p5x \
 -- set Interface p6 external_ids:iface-id=p6x


ovn-nbctl --wait=hv sync

ovs-vsctl set Interface p1 external_ids:iface-id=p1
ovs-vsctl set Interface p2 external_ids:iface-id=p2
ovs-vsctl set Interface p3 external_ids:iface-id=p3
ovs-vsctl set Interface p4 external_ids:iface-id=p4

ovn-nbctl --wait=hv sync

ovs-vsctl set Interface p5 external_ids:iface-id=p5

sleep 1

ovs-ofctl dump-flows br-int table=44 | grep conjunc

ovn-appctl -t ovn-controller recompute
ovn-nbctl --wait=hv sync
echo "------------ AFTER RECOMPUTE --------------------"
ovs-ofctl dump-flows br-int table=44 | grep conjunc


-------------------------------------------------------------


This issue is not seen with OVN master, ovn-22.03 and with branch-21.12 + this patch https://github.com/ovn-org/ovn/commit/8994d4ac238ec9550e3d9de378f64215b9bcec9b


Thanks

Comment 13 François Rigault 2022-04-12 11:15:10 UTC

Thanks. It's slightly out of context of this bug - while investigating this issue originally I saw something curious about acl meters, and wanted to try again with the patch applied, but now I notice something different. Would you mind trying these?


ovn-nbctl ls-add sw
ovn-nbctl lsp-add sw p1
ovn-nbctl lsp-set-addresses p1 "00:00:00:00:00:02 192.168.0.2"
ovn-nbctl lsp-add sw p2
ovn-nbctl lsp-set-addresses p2 "00:00:00:00:00:03 192.168.0.3"
ovn-nbctl pg-add pg1  p1 p2
ovn-nbctl acl-add pg1 to-lport 2003 "inport==@pg1 && ip4 && ip4.src == 192.168.0.0/16 && udp && udp.dst >= 9000 && udp.dst <= 9999"  allow
ovn-nbctl acl-add pg1 to-lport 2001 "inport==@pg1" reject

ovn-trace 'inport == "p2" && eth.src == 00:00:00:00:00:03 && eth.dst == 00:00:00:00:00:02 && ip.ttl==42 && ip4.src == 192.168.0.3'

I get a segfault, seems caused by some recursion of some sort.

Comment 14 François Rigault 2022-04-12 12:44:48 UTC

(please forget about #c13, opening https://bugzilla.redhat.com/show_bug.cgi?id=2074537 for that minor one)

Comment 15 Han Zhou 2022-04-13 06:44:11 UTC

Thanks François and Numan for the excellent debugging. I submitted a fix: https://patchwork.ozlabs.org/project/ovn/patch/20220413063412.2972298-1-hzhou@ovn.org/

Han

Comment 16 François Rigault 2022-04-13 07:39:41 UTC

I could not reproduce with patch applied. Thanks for the fix!
If you can provide beta version of ovn-2021-host rpms I can deploy that as well.

Comment 23 Ehsan Elahi 2022-05-03 18:52:36 UTC

[root@2071272 ~]# rpm -qa |grep -E 'ovn|openvswitch'
ovn-2021-central-21.12.0-42.el8fdp.x86_64
openvswitch-selinux-extra-policy-1.0-28.el8fdp.noarch
openvswitch2.15-2.15.0-93.el8fdp.x86_64
ovn-2021-21.12.0-42.el8fdp.x86_64
ovn-2021-host-21.12.0-42.el8fdp.x86_64

systemctl start ovn-northd
ovn-nbctl set-connection ptcp:6641
ovn-sbctl set-connection ptcp:6642
systemctl start openvswitch
ovs-vsctl set open . external_ids:system-id=hv1
ovs-vsctl set open . external_ids:ovn-remote=tcp:192.168.0.1:6642
ovs-vsctl set open . external_ids:ovn-encap-type=geneve
ovs-vsctl set open . external_ids:ovn-encap-ip=192.168.0.1
systemctl start ovn-controller

ovn-nbctl ls-add sw
ovn-nbctl lsp-add sw p1
ovn-nbctl lsp-set-addresses p1 "00:00:00:00:00:02 192.168.0.2"
ovn-nbctl lsp-add sw p2
ovn-nbctl lsp-set-addresses p2 "00:00:00:00:00:03 192.168.0.3"
ovn-nbctl lsp-add sw p3
ovn-nbctl lsp-set-addresses p3 "00:00:00:00:00:04 192.168.0.4"
ovn-nbctl lsp-add sw p4
ovn-nbctl lsp-set-addresses p4 "00:00:00:00:00:05 192.168.0.5"
ovn-nbctl lsp-add sw p5
ovn-nbctl lsp-set-addresses p5 "00:00:00:00:00:06 192.168.0.6"
ovn-nbctl lsp-add sw p6
ovn-nbctl lsp-set-addresses p6 "00:00:00:00:00:07 192.168.0.7"

# Now bind the ports on the hypervisor
ovs-vsctl add-port br-int p1 -- set interface p1 type=internal -- set Interface p1  external_ids:iface-id=p1
ovs-vsctl add-port br-int p2 -- set interface p2 type=internal -- set Interface p2 external_ids:iface-id=p2
ovs-vsctl add-port br-int p3 -- set interface p3 type=internal -- set Interface p3 external_ids:iface-id=p3
ovs-vsctl add-port br-int p4 -- set interface p4 type=internal -- set Interface p4 external_ids:iface-id=p4
ovs-vsctl add-port br-int p5 -- set interface p5 type=internal -- set Interface p5 external_ids:iface-id=p5
ovs-vsctl add-port br-int p6 -- set interface p6 type=internal -- set Interface p6 external_ids:iface-id=p6

ovn-nbctl pg-add pg1 p1 p2 p3
ovn-nbctl pg-add pg2 p4 p5 p6

ovn-nbctl acl-add pg1 to-lport 2003 "inport==@pg1 && ip4 && ip4.src == 192.168.0.0/16 && udp && udp.dst >= 9000 && udp.dst <= 9999"  allow
ovn-nbctl acl-add pg2 to-lport 2003 "inport==@pg2 && ip4 && ip4.src == 192.168.0.0/16 && udp && udp.dst >= 9000 && udp.dst <= 9999"  allow

ovn-nbctl --wait=hv sync

ovn-appctl -t ovn-controller vlog/set dbg

ovs-vsctl set Interface p1 external_ids:iface-id=p1x 
ovs-vsctl set Interface p2 external_ids:iface-id=p2x
ovs-vsctl set Interface p3 external_ids:iface-id=p3x
ovs-vsctl set Interface p4 external_ids:iface-id=p4x
ovs-vsctl set Interface p5 external_ids:iface-id=p5x
ovs-vsctl set Interface p6 external_ids:iface-id=p6x

ovn-nbctl --wait=hv sync

ovs-vsctl set Interface p1 external_ids:iface-id=p1
ovs-vsctl set Interface p2 external_ids:iface-id=p2
ovs-vsctl set Interface p3 external_ids:iface-id=p3
ovs-vsctl set Interface p4 external_ids:iface-id=p4
ovs-vsctl set Interface p5 external_ids:iface-id=p5
ovn-nbctl --wait=hv sync
sleep 1

[root@2071272 ~]# ovs-ofctl -O OpenFlow14 dump-flows br-int | grep -o -P '(?<=conj_id=)\d*'
3002461978
1490020941
[root@2071272 ~]# ovs-ofctl -O OpenFlow14 dump-flows br-int | grep 1490020941
 cookie=0xb1d6cbff, duration=4347.838s, table=44, n_packets=0, n_bytes=0, priority=3003,conj_id=1490020941,udp,metadata=0x1,nw_src=192.168.0.0/16 actions=resubmit(,45)
 cookie=0x0, duration=4347.838s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,reg14=0x5,metadata=0x1,nw_src=192.168.0.0/16 actions=conjunction(1490020941,1/2)
 cookie=0x0, duration=4347.838s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,reg14=0x4,metadata=0x1,nw_src=192.168.0.0/16 actions=conjunction(1490020941,1/2)

[root@2071272 ~]# ovs-ofctl -O OpenFlow14 dump-flows br-int | grep 3002461978
 cookie=0x6a8eb22d, duration=4363.936s, table=44, n_packets=0, n_bytes=0, priority=3003,conj_id=3002461978,udp,metadata=0x1,nw_src=192.168.0.0/16 actions=resubmit(,45)
 cookie=0x0, duration=4363.936s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x2700/0xfff8 actions=conjunction(3002461978,2/2)
 cookie=0x0, duration=4363.936s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x2600/0xff00 actions=conjunction(3002461978,2/2)
 cookie=0x0, duration=4363.936s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x2400/0xfe00 actions=conjunction(3002461978,2/2)
 cookie=0x0, duration=4363.936s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x2708/0xfffc actions=conjunction(3002461978,2/2)
 cookie=0x0, duration=4363.936s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x232c/0xfffc actions=conjunction(3002461978,2/2)
 cookie=0x0, duration=4363.936s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x2330/0xfff0 actions=conjunction(3002461978,2/2)
 cookie=0x0, duration=4363.936s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x270c/0xfffe actions=conjunction(3002461978,2/2)
 cookie=0x0, duration=4363.936s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x232a/0xfffe actions=conjunction(3002461978,2/2)
 cookie=0x0, duration=4363.936s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x2380/0xff80 actions=conjunction(3002461978,2/2)
 cookie=0x0, duration=4363.936s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x2340/0xffc0 actions=conjunction(3002461978,2/2)
 cookie=0x0, duration=4363.936s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,reg14=0x2,metadata=0x1,nw_src=192.168.0.0/16 actions=conjunction(3002461978,1/2)
 cookie=0x0, duration=4363.936s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,reg14=0x1,metadata=0x1,nw_src=192.168.0.0/16 actions=conjunction(3002461978,1/2)
 cookie=0x0, duration=4363.926s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,reg14=0x3,metadata=0x1,nw_src=192.168.0.0/16 actions=conjunction(3002461978,1/2)
 cookie=0x0, duration=4363.936s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=9999 actions=conjunction(3002461978,2/2)
 cookie=0x0, duration=4363.936s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=9000 actions=conjunction(3002461978,2/2)
 cookie=0x0, duration=4363.936s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=9998 actions=conjunction(3002461978,2/2)
 cookie=0x0, duration=4363.936s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=9001 actions=conjunction(3002461978,2/2)

<================ some conjunctions flows (1490020941) are missing their "2/2" part

Verified on:
[root@2071272 ~]# rpm -qa |grep -E 'ovn|openvswitch'
openvswitch2.15-2.15.0-93.el8fdp.x86_64
ovn-2021-central-21.12.0-46.el8fdp.x86_64
openvswitch-selinux-extra-policy-1.0-28.el8fdp.noarch
ovn-2021-host-21.12.0-46.el8fdp.x86_64
ovn-2021-21.12.0-46.el8fdp.x86_64

[root@2071272 ~]# ovs-ofctl -O OpenFlow14 dump-flows br-int | grep -o -P '(?<=conj_id=)\d*'
2879016453
2541519959
[root@2071272 ~]# ovs-ofctl -O OpenFlow14 dump-flows br-int | grep 2879016453
 cookie=0xf50f8d60, duration=6904.096s, table=44, n_packets=0, n_bytes=0, priority=3003,conj_id=2879016453,udp,metadata=0x1,nw_src=192.168.0.0/16 actions=resubmit(,45)
 cookie=0x0, duration=6904.059s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x2700/0xfff8 actions=conjunction(2541519959,2/2),conjunction(2879016453,2/2)
 cookie=0x0, duration=6904.059s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x2600/0xff00 actions=conjunction(2541519959,2/2),conjunction(2879016453,2/2)
 cookie=0x0, duration=6904.059s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x2400/0xfe00 actions=conjunction(2541519959,2/2),conjunction(2879016453,2/2)
 cookie=0x0, duration=6904.059s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x2708/0xfffc actions=conjunction(2541519959,2/2),conjunction(2879016453,2/2)
 cookie=0x0, duration=6904.059s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x232c/0xfffc actions=conjunction(2541519959,2/2),conjunction(2879016453,2/2)
 cookie=0x0, duration=6904.059s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x2330/0xfff0 actions=conjunction(2541519959,2/2),conjunction(2879016453,2/2)
 cookie=0x0, duration=6904.059s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x232a/0xfffe actions=conjunction(2541519959,2/2),conjunction(2879016453,2/2)
 cookie=0x0, duration=6904.059s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x270c/0xfffe actions=conjunction(2541519959,2/2),conjunction(2879016453,2/2)
 cookie=0x0, duration=6904.059s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x2380/0xff80 actions=conjunction(2541519959,2/2),conjunction(2879016453,2/2)
 cookie=0x0, duration=6904.059s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x2340/0xffc0 actions=conjunction(2541519959,2/2),conjunction(2879016453,2/2)
 cookie=0x0, duration=6904.096s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,reg14=0x2,metadata=0x1,nw_src=192.168.0.0/16 actions=conjunction(2879016453,1/2)
 cookie=0x0, duration=6904.096s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,reg14=0x1,metadata=0x1,nw_src=192.168.0.0/16 actions=conjunction(2879016453,1/2)
 cookie=0x0, duration=6904.088s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,reg14=0x3,metadata=0x1,nw_src=192.168.0.0/16 actions=conjunction(2879016453,1/2)
 cookie=0x0, duration=6904.059s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=9001 actions=conjunction(2541519959,2/2),conjunction(2879016453,2/2)
 cookie=0x0, duration=6904.059s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=9999 actions=conjunction(2541519959,2/2),conjunction(2879016453,2/2)
 cookie=0x0, duration=6904.059s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=9998 actions=conjunction(2541519959,2/2),conjunction(2879016453,2/2)
 cookie=0x0, duration=6904.059s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=9000 actions=conjunction(2541519959,2/2),conjunction(2879016453,2/2)

[root@2071272 ~]# ovs-ofctl -O OpenFlow14 dump-flows br-int | grep 2541519959
 cookie=0x4e676bcd, duration=6917.143s, table=44, n_packets=0, n_bytes=0, priority=3003,conj_id=2541519959,udp,metadata=0x1,nw_src=192.168.0.0/16 actions=resubmit(,45)
 cookie=0x0, duration=6917.143s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x2700/0xfff8 actions=conjunction(2541519959,2/2),conjunction(2879016453,2/2)
 cookie=0x0, duration=6917.143s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x2600/0xff00 actions=conjunction(2541519959,2/2),conjunction(2879016453,2/2)
 cookie=0x0, duration=6917.143s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x2400/0xfe00 actions=conjunction(2541519959,2/2),conjunction(2879016453,2/2)
 cookie=0x0, duration=6917.143s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x2708/0xfffc actions=conjunction(2541519959,2/2),conjunction(2879016453,2/2)
 cookie=0x0, duration=6917.143s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x232c/0xfffc actions=conjunction(2541519959,2/2),conjunction(2879016453,2/2)
 cookie=0x0, duration=6917.143s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x2330/0xfff0 actions=conjunction(2541519959,2/2),conjunction(2879016453,2/2)
 cookie=0x0, duration=6917.143s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x232a/0xfffe actions=conjunction(2541519959,2/2),conjunction(2879016453,2/2)
 cookie=0x0, duration=6917.143s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x270c/0xfffe actions=conjunction(2541519959,2/2),conjunction(2879016453,2/2)
 cookie=0x0, duration=6917.143s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x2380/0xff80 actions=conjunction(2541519959,2/2),conjunction(2879016453,2/2)
 cookie=0x0, duration=6917.143s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=0x2340/0xffc0 actions=conjunction(2541519959,2/2),conjunction(2879016453,2/2)
 cookie=0x0, duration=6917.143s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,reg14=0x5,metadata=0x1,nw_src=192.168.0.0/16 actions=conjunction(2541519959,1/2)
 cookie=0x0, duration=6917.143s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,reg14=0x4,metadata=0x1,nw_src=192.168.0.0/16 actions=conjunction(2541519959,1/2)
 cookie=0x0, duration=6917.143s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=9001 actions=conjunction(2541519959,2/2),conjunction(2879016453,2/2)
 cookie=0x0, duration=6917.143s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=9999 actions=conjunction(2541519959,2/2),conjunction(2879016453,2/2)
 cookie=0x0, duration=6917.143s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=9998 actions=conjunction(2541519959,2/2),conjunction(2879016453,2/2)
 cookie=0x0, duration=6917.143s, table=44, n_packets=0, n_bytes=0, priority=3003,udp,metadata=0x1,nw_src=192.168.0.0/16,tp_dst=9000 actions=conjunction(2541519959,2/2),conjunction(2879016453,2/2)

Also verified on: 
[root@2071272 ~]# rpm -qa |grep -E 'ovn|openvswitch'
openvswitch2.15-2.15.0-93.el8fdp.x86_64
ovn22.03-central-22.03.0-9.el8fdp.x86_64
openvswitch-selinux-extra-policy-1.0-28.el8fdp.noarch
ovn22.03-host-22.03.0-9.el8fdp.x86_64
ovn22.03-22.03.0-9.el8fdp.x86_64

Comment 25 errata-xmlrpc 2022-05-27 18:14:06 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (ovn bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:4784

Comment 26 Dumitru Ceara 2023-10-30 18:44:18 UTC

*** Bug 2244405 has been marked as a duplicate of this bug. ***