Bug 2182223

Summary:	ovn database containers consume more memory when upgrading ovs version to 2.17-18 or higher
Product:	Red Hat Enterprise Linux Fast Datapath	Reporter:	OvS team <ovs-bugzilla>
Component:	openvswitch2.17	Assignee:	Timothy Redaelli <tredaelli>
Status:	CLOSED ERRATA	QA Contact:	Zhiqiang Fang <zfang>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	FDP 22.L	CC:	ctrautma, jhsiao, ralongi
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openvswitch2.17-2.17.0-74.el9fdp	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2023-07-06 19:17:41 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description OvS team 2023-03-27 23:46:24 UTC

+++ This bug was initially created as a clone of Bug #2159631 +++

Description of problem:


When deploying ovn-kubernetes on microshift inside VM, it is found that there is a memory jump (<10M to >15M) on ovn-kubernetes database containers when upgrading ovs version from 2.17.0-8 to 2.17.0-18 in ovn-kubernetes dockerfile. The issue also exists with latest ovs version 2.17.0-62.

The container memory are observed with metrics-server: https://github.com/kubernetes-sigs/metrics-server

VM spec: 4 vCPUs, 4Gi memory

Version-Release number of selected component (if applicable):

downstream ovn-kubernetes: dbdf0372b2f2be0c3413731eeae3084d924878b6
ovn: 22.09.0-25
ovs: ovs-2.17.0-62



Also run the same tests with different ovn/ovs versions:

ovn 22.09.0-25/ovs-2.17.0-8: low memory
ovn 22.09.0-25/ovs-2.17.0-18: high memory
ovn 22.09.0-25/ovs-2.17.0-22: high memory
ovn 22.09.0-25/ovs-2.17.0-50: high memory
ovn 22.09.0-25/ovs-2.17.0-62: high memory

ovn 22.09.0-22/ovs-2.17.0-8: low memory
ovn 22.09.0-11/ovs-2.17.0-8: low memory
ovn 22.09.0-10/ovs-2.17.0-8: low memory
ovn 22.09.0-5/ovs-2.17.0-8: low memory
ovn 22.09.0-4/ovs-2.17.0-8: low memory
ovn 22.06.0-82/ovs-2.17.0-8: low memory
ovn 22.06.0-preview.branched.38/ovs-2.17.0-8: low memory
ovn 22.09.0-25/ovs-2.17.0-62: high memory

Comment 1 OvS team 2023-03-27 23:46:27 UTC

* Mon Mar 27 2023 Open vSwitch CI <ovs-ci> - 2.17.0-74
- Merging upstream branch-2.17 [RH git: a86c3f2963]
    Commit list:
    d6d1cad6a7 dpif-netlink: Always create at least 1 handler.


* Mon Mar 20 2023 Timothy Redaelli <tredaelli> - 2.17.0-73
- Install shared linked binaries (all, but ovs-vswitchd) [RH git: 590d558943] (#2159631)
    Resolves: #2159631

Comment 4 Zhiqiang Fang 2023-06-12 16:29:58 UTC

~~~~~~~~ Reproducer

Tested on openvswitch2.17-2.17.0-72.el9fdp.x86_64 using image, quay.io/zshi/ovn-daemonset:microshift-2023-06-12-1-ovn22.12.0-18-ovs2.17.0-72.el9

[root@hpe-netqe-syn480g10-03 ~]# rpm -qa | grep openvswitch
openvswitch-selinux-extra-policy-1.0-31.el9fdp.noarch
openvswitch2.17-2.17.0-72.el9fdp.x86_64
python3-openvswitch2.17-2.17.0-72.el9fdp.x86_64

[microshift@hpe-netqe-syn480g10-03 rpmbuild]$  kubectl top  pods -n openshift-ovn-kubernetes --containers
POD                    NAME             CPU(cores)   MEMORY(bytes)   
ovnkube-master-lxnpr   nbdb             1m           26Mi            <<<----- big
ovnkube-master-lxnpr   northd           1m           4Mi             
ovnkube-master-lxnpr   ovnkube-master   7m           44Mi            
ovnkube-master-lxnpr   sbdb             1m           28Mi            <<<----- big
ovnkube-node-xczqn     ovn-controller   1m           5Mi    


~~~~~~~~ Verified the fix

Tested on openvswitch2.17-2.17.0-74.el9fdp.x86_64 using image quay.io/zshi/ovn-daemonset:microshift-2023-06-12-1-ovn22.12.0-18-ovs2.17.0-74.el9

[root@hpe-netqe-syn480g10-03 ~]# rpm -qa | grep openvswitch
openvswitch-selinux-extra-policy-1.0-31.el9fdp.noarch
openvswitch2.17-2.17.0-74.el9fdp.x86_64
python3-openvswitch2.17-2.17.0-74.el9fdp.x86_64


[microshift@hpe-netqe-syn480g10-03 rpmbuild]$ kubectl top  pods -n openshift-ovn-kubernetes --containers
POD                    NAME             CPU(cores)   MEMORY(bytes)   
ovnkube-master-mk9kp   nbdb             1m           3Mi             <<<----- small
ovnkube-master-mk9kp   northd           1m           5Mi             
ovnkube-master-mk9kp   ovnkube-master   3m           50Mi            
ovnkube-master-mk9kp   sbdb             2m           5Mi             <<<----- small
ovnkube-node-9zkns     ovn-controller   6m           5Mi  


~~~~~~~~ Also verified the current microshift default image has no such issue

It is on openvswitch3.1-3.1.0-10.el9fdp.x86_64, the image is quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0aa48c2544ec25b42a18958afc18acffca9130c34f0c2aa371a1baa992bd9a87

[root@hpe-netqe-syn480g10-03 ~]# rpm -qa | grep openvswitch
openvswitch-selinux-extra-policy-1.0-31.el9fdp.noarch
openvswitch3.1-3.1.0-10.el9fdp.x86_64
python3-openvswitch3.1-3.1.0-10.el9fdp.x86_64

[microshift@hpe-netqe-syn480g10-03 rpmbuild]$ kubectl top  pods -n openshift-ovn-kubernetes --containers
POD                    NAME             CPU(cores)   MEMORY(bytes)   
ovnkube-master-hx5dl   nbdb             1m           3Mi            <<<----- small
ovnkube-master-hx5dl   northd           1m           4Mi             
ovnkube-master-hx5dl   ovnkube-master   4m           49Mi            
ovnkube-master-hx5dl   sbdb             1m           4Mi           <<<-----  small
ovnkube-node-brbs6     ovn-controller   1m           6Mi

Comment 9 errata-xmlrpc 2023-07-06 19:17:41 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (openvswitch2.17 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:3988