Description of problem: The perfscale team ran cluster-density with 2000 projects at 100 node scale. The observation was that the ovn-controller memory grew to 10GB and stayed at 10GB even after the cleanup Version-Release number of selected component (if applicable): Cluster version is 4.8.0-0.nightly-2021-04-30-201824 sh-4.4# ovs-vswitchd --version ovs-vswitchd (Open vSwitch) 2.15.0 How reproducible: Always Steps to Reproduce: 1. Run kube-burner cluster-density with 2k projects at 100 node scale Actual results: ovn-controller memory not freed/released to OS Expected results: memory should be released/freed Additional info: Must-gathers requested (VPN required) http://dell-r510-01.perf.lab.eng.rdu2.redhat.com/msheth/may-7-must-gathers-ovn-controller-mem/
This might be related to malloc fastbins not being consolidated and not honoring M_TRIM_THRESHOLD (default 128KB) [0] [1]. One option would be to disable fast bins for ovn-controller although that needs to be scale tested properly as it might affect performance. This can be done at runtime without changing the OVN code by setting the GLIBC_TUNABLES before ovn-controller is started, e.g.: GLIBC_TUNABLES=glibc.malloc.mxfast=0 export GLIBC_TUNABLES [0] https://bugzilla.redhat.com/show_bug.cgi?id=921676 [1] https://sourceware.org/bugzilla/show_bug.cgi?id=14827 [2] https://sourceware.org/bugzilla/show_bug.cgi?id=14827#c7
We are observing the same behaviour in a 120 node baremetal cluster. Cluster density test was run on Friday and cleaned up on Friday. Since then, the cluster is in an idle state. Today Monday, the openshift-ovn-kubernetes memory usage is 730 GiB with a request of 78.2 GiB. CPU usage is 2.19 out of 5 requested. Env: - OCP 4.8.0-fc.9 - local gateway - ovn2.13-20.12.0-25.el8fdp.x86_64
Observed this on OCP 4.7.11 as well, after running some node-density workloads with 2000 services. The memory and CPU usage of OVN pods started increasing and new pod creation failed with error - CNI request timeout.
Fix is present in 21.12.0-24.el8fdp
On the build 4.10.0-0.nightly-2022-02-02-131023 I ran cluster density tests for 500 iterations on 100 worker nodes, and monitored the OVN-controller memory usage, it showed that the usage spiked for a bit before it went down to normal levels. Which makes this bz resolved.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0056