+++ This bug was initially created as a clone of Bug #2159631 +++ Description of problem: When deploying ovn-kubernetes on microshift inside VM, it is found that there is a memory jump (<10M to >15M) on ovn-kubernetes database containers when upgrading ovs version from 2.17.0-8 to 2.17.0-18 in ovn-kubernetes dockerfile. The issue also exists with latest ovs version 2.17.0-62. The container memory are observed with metrics-server: https://github.com/kubernetes-sigs/metrics-server VM spec: 4 vCPUs, 4Gi memory Version-Release number of selected component (if applicable): downstream ovn-kubernetes: dbdf0372b2f2be0c3413731eeae3084d924878b6 ovn: 22.09.0-25 ovs: ovs-2.17.0-62 Also run the same tests with different ovn/ovs versions: ovn 22.09.0-25/ovs-2.17.0-8: low memory ovn 22.09.0-25/ovs-2.17.0-18: high memory ovn 22.09.0-25/ovs-2.17.0-22: high memory ovn 22.09.0-25/ovs-2.17.0-50: high memory ovn 22.09.0-25/ovs-2.17.0-62: high memory ovn 22.09.0-22/ovs-2.17.0-8: low memory ovn 22.09.0-11/ovs-2.17.0-8: low memory ovn 22.09.0-10/ovs-2.17.0-8: low memory ovn 22.09.0-5/ovs-2.17.0-8: low memory ovn 22.09.0-4/ovs-2.17.0-8: low memory ovn 22.06.0-82/ovs-2.17.0-8: low memory ovn 22.06.0-preview.branched.38/ovs-2.17.0-8: low memory ovn 22.09.0-25/ovs-2.17.0-62: high memory
* Mon Mar 27 2023 Open vSwitch CI <ovs-ci> - 2.17.0-74 - Merging upstream branch-2.17 [RH git: a86c3f2963] Commit list: d6d1cad6a7 dpif-netlink: Always create at least 1 handler. * Mon Mar 20 2023 Timothy Redaelli <tredaelli> - 2.17.0-73 - Install shared linked binaries (all, but ovs-vswitchd) [RH git: 590d558943] (#2159631) Resolves: #2159631
~~~~~~~~ Reproducer Tested on openvswitch2.17-2.17.0-72.el9fdp.x86_64 using image, quay.io/zshi/ovn-daemonset:microshift-2023-06-12-1-ovn22.12.0-18-ovs2.17.0-72.el9 [root@hpe-netqe-syn480g10-03 ~]# rpm -qa | grep openvswitch openvswitch-selinux-extra-policy-1.0-31.el9fdp.noarch openvswitch2.17-2.17.0-72.el9fdp.x86_64 python3-openvswitch2.17-2.17.0-72.el9fdp.x86_64 [microshift@hpe-netqe-syn480g10-03 rpmbuild]$ kubectl top pods -n openshift-ovn-kubernetes --containers POD NAME CPU(cores) MEMORY(bytes) ovnkube-master-lxnpr nbdb 1m 26Mi <<<----- big ovnkube-master-lxnpr northd 1m 4Mi ovnkube-master-lxnpr ovnkube-master 7m 44Mi ovnkube-master-lxnpr sbdb 1m 28Mi <<<----- big ovnkube-node-xczqn ovn-controller 1m 5Mi ~~~~~~~~ Verified the fix Tested on openvswitch2.17-2.17.0-74.el9fdp.x86_64 using image quay.io/zshi/ovn-daemonset:microshift-2023-06-12-1-ovn22.12.0-18-ovs2.17.0-74.el9 [root@hpe-netqe-syn480g10-03 ~]# rpm -qa | grep openvswitch openvswitch-selinux-extra-policy-1.0-31.el9fdp.noarch openvswitch2.17-2.17.0-74.el9fdp.x86_64 python3-openvswitch2.17-2.17.0-74.el9fdp.x86_64 [microshift@hpe-netqe-syn480g10-03 rpmbuild]$ kubectl top pods -n openshift-ovn-kubernetes --containers POD NAME CPU(cores) MEMORY(bytes) ovnkube-master-mk9kp nbdb 1m 3Mi <<<----- small ovnkube-master-mk9kp northd 1m 5Mi ovnkube-master-mk9kp ovnkube-master 3m 50Mi ovnkube-master-mk9kp sbdb 2m 5Mi <<<----- small ovnkube-node-9zkns ovn-controller 6m 5Mi ~~~~~~~~ Also verified the current microshift default image has no such issue It is on openvswitch3.1-3.1.0-10.el9fdp.x86_64, the image is quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0aa48c2544ec25b42a18958afc18acffca9130c34f0c2aa371a1baa992bd9a87 [root@hpe-netqe-syn480g10-03 ~]# rpm -qa | grep openvswitch openvswitch-selinux-extra-policy-1.0-31.el9fdp.noarch openvswitch3.1-3.1.0-10.el9fdp.x86_64 python3-openvswitch3.1-3.1.0-10.el9fdp.x86_64 [microshift@hpe-netqe-syn480g10-03 rpmbuild]$ kubectl top pods -n openshift-ovn-kubernetes --containers POD NAME CPU(cores) MEMORY(bytes) ovnkube-master-hx5dl nbdb 1m 3Mi <<<----- small ovnkube-master-hx5dl northd 1m 4Mi ovnkube-master-hx5dl ovnkube-master 4m 49Mi ovnkube-master-hx5dl sbdb 1m 4Mi <<<----- small ovnkube-node-brbs6 ovn-controller 1m 6Mi
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (openvswitch2.17 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2023:3988