Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
The FDP team is no longer accepting new bugs in Bugzilla. Please report your issues under FDP project in Jira. Thanks.

Bug 2228162

Summary: Add incremental processing in ovn-northd for port groups.
Product: Red Hat Enterprise Linux Fast Datapath Reporter: Dumitru Ceara <dceara>
Component: ovn23.09Assignee: Dumitru Ceara <dceara>
Status: CLOSED ERRATA QA Contact: Ehsan Elahi <eelahi>
Severity: high Docs Contact:
Priority: high    
Version: FDP 23.GCC: ctrautma, jiji, jishi
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ovn23.09-23.09.0-alpha.151.el9fdp Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2024-01-24 11:18:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2228166    

Description Dumitru Ceara 2023-08-01 14:56:21 UTC 
Description of problem:

Currently NB DB port group changes trigger recomputes of the northd and lflow I-P nodes in ovn-northd.  That's costly and can be avoided if we split out the port groups into their separate I-P node.
Comment 1 Dumitru Ceara 2023-08-10 12:50:15 UTC 
v1 posted for review:
https://patchwork.ozlabs.org/project/ovn/list/?series=368245&state=*
Comment 2 Dumitru Ceara 2023-08-30 13:53:47 UTC 
v2 posted for review:
https://patchwork.ozlabs.org/project/ovn/list/?series=371000&state=*
Comment 5 Ehsan Elahi 2023-12-11 19:14:19 UTC 
Verified on:
[root@dell-per740-81 bz_2228162]# rpm -qa | grep -E 'ovn|openvswitch'
openvswitch-selinux-extra-policy-1.0-34.el9fdp.noarch
openvswitch2.17-2.17.0-125.el9fdp.x86_64
ovn23.09-23.09.0-87.el9fdp.x86_64
ovn23.09-host-23.09.0-87.el9fdp.x86_64
ovn23.09-central-23.09.0-87.el9fdp.x86_64

########### Here is the reproducer

systemctl start ovn-northd
ovn-nbctl set-connection ptcp:6641
ovn-sbctl set-connection ptcp:6642
systemctl start openvswitch
ovs-vsctl set open . external_ids:system-id=hv1
ifconfig ens1f0 192.168.20.1 netmask 255.255.255.0
ovs-vsctl set open . external_ids:ovn-remote=tcp:192.168.20.1:6642
ovs-vsctl set open . external_ids:ovn-encap-type=geneve
ovs-vsctl set open . external_ids:ovn-encap-ip=192.168.20.1
ovs-vsctl set open . external_ids:ovn-monitor-all=true
systemctl start ovn-controller

ovn-nbctl ls-add ls1
ovn-nbctl lsp-add ls1 lsp11
ovn-nbctl lsp-add ls1 lsp12
ovn-nbctl lsp-add ls1 lsp13
ovn-nbctl ls-add ls2
ovn-nbctl lsp-add ls2 lsp21
ovn-nbctl lsp-add ls2 lsp22
ovn-nbctl lsp-add ls2 lsp23
# ovn-nbctl lsp-set-addresses vif1 "00:00:01:01:01:01 192.168.20.1"
ovn-nbctl --wait=hv sync

ls1_key=$(ovn-sbctl  --bare --columns tunnel_key list datapath_binding ls1)
ls2_key=$(ovn-sbctl  --bare --columns tunnel_key list datapath_binding ls2)
echo ls1_key=$ls1_key
echo ls2_key=$ls2_key

########################## Create new PG1 and PG2
ovn-appctl -t ovn-northd inc-engine/clear-stats
ovn-nbctl --wait=sb sync
ovn-nbctl pg-add pg1 -- pg-add pg2
# The northd node should not recompute, it should handle nb_global update though, therefore "compute: 1"
ovn-appctl -t ovn-northd inc-engine/show-stats northd |grep 'recompute' |grep '0'
# The port_group node recomputes every time a NB port group is added/deleted.
ovn-appctl -t ovn-northd inc-engine/show-stats port_group |grep 'recompute' |awk '$3 > 0'
# The port_group node is an input for the lflow node. Port_group recompute/compute triggers lflow recompute.
ovn-appctl -t ovn-northd inc-engine/show-stats lflow |grep 'recompute' |awk '$3 > 0'

ovn-nbctl --wait=sb sync
ovn-nbctl acl-add pg1 from-lport 1 eth.src==41:41:41:41:41:41 allow
ovn-nbctl acl-add pg2 from-lport 1 eth.src==42:42:42:42:42:42 allow

################## Add one port from the two switches to PG1
ovn-appctl -t ovn-northd inc-engine/clear-stats
ovn-nbctl --wait=sb sync
ovn-nbctl pg-set-ports pg1 lsp11 lsp21

# The northd node should not recompute, it should handle nb_global update though, therefore "compute: 1"
ovn-appctl -t ovn-northd inc-engine/show-stats northd |grep 'recompute' |grep '0'

# The port_group node recomputes every time a NB port group is added/deleted.
ovn-appctl -t ovn-northd inc-engine/show-stats port_group |grep 'recompute' |awk '$3 > 0'

# The port_group node is an input for the lflow node. Port_group recompute/compute triggers lflow recompute.
ovn-appctl -t ovn-northd inc-engine/show-stats lflow |grep 'recompute' |awk '$3 > 0'

# should be 1
ovn-sbctl lflow-list ls1 | grep ls_in_acl_eval | grep eth.src==41:41:41:41:41:41 -c |grep 1
# should be 0
ovn-sbctl lflow-list ls1 | grep ls_in_acl_eval | grep eth.src==42:42:42:42:42:42 -c |grep 0
# should be 1
ovn-sbctl lflow-list ls2 | grep ls_in_acl_eval | grep eth.src==41:41:41:41:41:41 -c |grep 1
# should be 0
ovn-sbctl lflow-list ls2 | grep ls_in_acl_eval | grep eth.src==42:42:42:42:42:42 -c |grep 0

################### Add one port from the two switches to PG2
ovn-appctl -t ovn-northd inc-engine/clear-stats
ovn-nbctl --wait=sb sync
ovn-nbctl pg-set-ports pg2 lsp12 lsp22

# The northd node should not recompute, it should handle nb_global update though, therefore "compute: 1"
ovn-appctl -t ovn-northd inc-engine/show-stats northd |grep 'recompute' |grep '0'

# The port_group node recomputes every time a NB port group is added/deleted.
ovn-appctl -t ovn-northd inc-engine/show-stats port_group |grep 'recompute' |awk '$3 > 0'

# The port_group node is an input for the lflow node. Port_group recompute/compute triggers lflow recompute.
ovn-appctl -t ovn-northd inc-engine/show-stats lflow |grep 'recompute' |awk '$3 > 0'

# should be 1
ovn-sbctl lflow-list ls1 | grep ls_in_acl_eval | grep eth.src==41:41:41:41:41:41 -c |grep 1
# should be 1
ovn-sbctl lflow-list ls1 | grep ls_in_acl_eval | grep eth.src==42:42:42:42:42:42 -c |grep 1
# should be 1
ovn-sbctl lflow-list ls2 | grep ls_in_acl_eval | grep eth.src==41:41:41:41:41:41 -c |grep 1
# should be 1
ovn-sbctl lflow-list ls2 | grep ls_in_acl_eval | grep eth.src==42:42:42:42:42:42 -c |grep 1

################## Add one more port from the two switches to PG1 and PG2
ovn-appctl -t ovn-northd inc-engine/clear-stats
ovn-nbctl --wait=sb sync
ovn-nbctl pg-set-ports pg1 lsp11 lsp21 lsp13 lsp23
ovn-nbctl pg-set-ports pg2 lsp12 lsp22 lsp13 lsp23

# The northd node should not recompute, it should handle nb_global update though, therefore "compute: 1"
ovn-appctl -t ovn-northd inc-engine/show-stats northd |grep 'recompute' |grep '0'

# We did not change the set of switches a pg is applied to, there should be no recompute
ovn-appctl -t ovn-northd inc-engine/show-stats port_group |grep 'recompute' |grep '0'

# The port_group node is an input for the lflow node. Port_group recompute/compute triggers lflow recompute.
ovn-appctl -t ovn-northd inc-engine/show-stats lflow |grep 'recompute' |grep '0'

# should be 1
ovn-sbctl lflow-list ls1 | grep ls_in_acl_eval | grep eth.src==41:41:41:41:41:41 -c |grep 1
# should be 1
ovn-sbctl lflow-list ls1 | grep ls_in_acl_eval | grep eth.src==42:42:42:42:42:42 -c |grep 1
# should be 1
ovn-sbctl lflow-list ls2 | grep ls_in_acl_eval | grep eth.src==41:41:41:41:41:41 -c |grep 1
# should be 1
ovn-sbctl lflow-list ls2 | grep ls_in_acl_eval | grep eth.src==42:42:42:42:42:42 -c |grep 1

################### Remove the last port from PG1 and PG2
ovn-appctl -t ovn-northd inc-engine/clear-stats
ovn-nbctl --wait=sb sync
ovn-nbctl pg-set-ports pg1 lsp11 lsp21
ovn-nbctl pg-set-ports pg2 lsp12 lsp22

# The northd node should not recompute, it should handle nb_global update though, therefore "compute: 1"
ovn-appctl -t ovn-northd inc-engine/show-stats northd |grep 'recompute' |grep '0'

# We did not change the set of switches a pg is applied to, there should be no recompute
ovn-appctl -t ovn-northd inc-engine/show-stats port_group |grep 'recompute' |grep '0'

# We did not change the set of switches a pg is applied to, there should be no recompute
ovn-appctl -t ovn-northd inc-engine/show-stats lflow |grep 'recompute' |grep '0'

# should be 1
ovn-sbctl lflow-list ls1 | grep ls_in_acl_eval | grep eth.src==41:41:41:41:41:41 -c |grep 1
# should be 1
ovn-sbctl lflow-list ls1 | grep ls_in_acl_eval | grep eth.src==42:42:42:42:42:42 -c |grep 1
# should be 1
ovn-sbctl lflow-list ls2 | grep ls_in_acl_eval | grep eth.src==41:41:41:41:41:41 -c |grep 1
# should be 1
ovn-sbctl lflow-list ls2 | grep ls_in_acl_eval | grep eth.src==42:42:42:42:42:42 -c |grep 1

################### Remove the second port from PG2
ovn-appctl -t ovn-northd inc-engine/clear-stats
ovn-nbctl --wait=sb sync
ovn-nbctl pg-set-ports pg2 lsp12

# The northd node should not recompute, it should handle nb_global update though, therefore "compute: 1"
ovn-appctl -t ovn-northd inc-engine/show-stats northd |grep 'recompute' |grep '0'

# We changed the set of switches a pg is applied to, there should be a recompute
ovn-appctl -t ovn-northd inc-engine/show-stats port_group |grep 'recompute' |awk '$3 > 0'

# We changed the set of switches a pg is applied to, there should be a recompute (for ACLs).
ovn-appctl -t ovn-northd inc-engine/show-stats lflow |grep 'recompute' |awk '$3 > 0'

# should be 1
ovn-sbctl lflow-list ls1 | grep ls_in_acl_eval | grep eth.src==41:41:41:41:41:41 -c |grep 1
# should be 1
ovn-sbctl lflow-list ls1 | grep ls_in_acl_eval | grep eth.src==42:42:42:42:42:42 -c |grep 1
# should be 1
ovn-sbctl lflow-list ls2 | grep ls_in_acl_eval | grep eth.src==41:41:41:41:41:41 -c |grep 1
# should be 0
ovn-sbctl lflow-list ls2 | grep ls_in_acl_eval | grep eth.src==42:42:42:42:42:42 -c |grep 0

################### Remove the second port from PG1
ovn-appctl -t ovn-northd inc-engine/clear-stats
ovn-nbctl --wait=sb sync
ovn-nbctl pg-set-ports pg1 lsp11

# The northd node should not recompute, it should handle nb_global update though, therefore "compute: 1"
ovn-appctl -t ovn-northd inc-engine/show-stats northd |grep 'recompute' |grep '0'

# We changed the set of switches a pg is applied to, there should be a recompute
ovn-appctl -t ovn-northd inc-engine/show-stats port_group |grep 'recompute' |awk '$3 > 0'

# We changed the set of switches a pg is applied to, there should be a recompute (for ACLs).
ovn-appctl -t ovn-northd inc-engine/show-stats lflow |grep 'recompute' |awk '$3 > 0'

# should be 1
ovn-sbctl lflow-list ls1 | grep ls_in_acl_eval | grep eth.src==41:41:41:41:41:41 -c |grep 1
# should be 1
ovn-sbctl lflow-list ls1 | grep ls_in_acl_eval | grep eth.src==42:42:42:42:42:42 -c |grep 1
# should be 0
ovn-sbctl lflow-list ls2 | grep ls_in_acl_eval | grep eth.src==41:41:41:41:41:41 -c |grep 0
# should be 0
ovn-sbctl lflow-list ls2 | grep ls_in_acl_eval | grep eth.src==42:42:42:42:42:42 -c |grep 0

################### Add second port to both PGs
ovn-appctl -t ovn-northd inc-engine/clear-stats
ovn-nbctl --wait=sb sync
ovn-nbctl pg-set-ports pg1 lsp11 lsp21
ovn-nbctl pg-set-ports pg2 lsp12 lsp22

# The northd node should not recompute, it should handle nb_global update though, therefore "compute: 1"
ovn-appctl -t ovn-northd inc-engine/show-stats northd |grep 'recompute' |grep '0'

# We changed the set of switches a pg is applied to, there should be a recompute
ovn-appctl -t ovn-northd inc-engine/show-stats port_group |grep 'recompute' |awk '$3 > 0'

# We changed the set of switches a pg is applied to, there should be a recompute (for ACLs).
ovn-appctl -t ovn-northd inc-engine/show-stats lflow |grep 'recompute' |awk '$3 > 0'

# should be 1
ovn-sbctl lflow-list ls1 | grep ls_in_acl_eval | grep eth.src==41:41:41:41:41:41 -c |grep 1
# should be 1
ovn-sbctl lflow-list ls1 | grep ls_in_acl_eval | grep eth.src==42:42:42:42:42:42 -c |grep 1
# should be 1
ovn-sbctl lflow-list ls2 | grep ls_in_acl_eval | grep eth.src==41:41:41:41:41:41 -c |grep 1
# should be 1
ovn-sbctl lflow-list ls2 | grep ls_in_acl_eval | grep eth.src==42:42:42:42:42:42 -c |grep 1

################### Remove the second port from both PGs
ovn-appctl -t ovn-northd inc-engine/clear-stats
ovn-nbctl --wait=sb sync
ovn-nbctl pg-set-ports pg1 lsp11
ovn-nbctl pg-set-ports pg2 lsp12

# The northd node should not recompute, it should handle nb_global update though, therefore "compute: 1"
ovn-appctl -t ovn-northd inc-engine/show-stats northd |grep 'recompute' |grep '0'

# We changed the set of switches a pg is applied to, there should be a recompute
ovn-appctl -t ovn-northd inc-engine/show-stats port_group |grep 'recompute' |awk '$3 > 0'

# We changed the set of switches a pg is applied to, there should be a recompute (for ACLs).
ovn-appctl -t ovn-northd inc-engine/show-stats lflow |grep 'recompute' |awk '$3 > 0'

# should be 1
ovn-sbctl lflow-list ls1 | grep ls_in_acl_eval | grep eth.src==41:41:41:41:41:41 -c |grep 1
# should be 1
ovn-sbctl lflow-list ls1 | grep ls_in_acl_eval | grep eth.src==42:42:42:42:42:42 -c |grep 1
# should be 0
ovn-sbctl lflow-list ls2 | grep ls_in_acl_eval | grep eth.src==41:41:41:41:41:41 -c |grep 0
# should be 0
ovn-sbctl lflow-list ls2 | grep ls_in_acl_eval | grep eth.src==42:42:42:42:42:42 -c |grep 0
Comment 7 errata-xmlrpc 2024-01-24 11:18:05 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (ovn23.09 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2024:0392