Bug 2117462 - [4.11 backport] percpu Memory leak CRIO due to no garbage collection in /run/crio/exits for exited containers
Summary: [4.11 backport] percpu Memory leak CRIO due to no garbage collection in /run/...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Node
Version: 4.11
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.11.z
Assignee: Peter Hunt
QA Contact: Sunil Choudhary
URL:
Whiteboard:
Depends On: 2116460
Blocks: 2117461
TreeView+ depends on / blocked
 
Reported: 2022-08-11 04:37 UTC by Pamela Escorza
Modified: 2022-09-07 20:49 UTC (History)
0 users

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-09-07 20:49:27 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github cri-o cri-o pull 6149 0 None open [release-1.24] server: remove exit file in exit monitor 2022-08-15 15:47:03 UTC
Red Hat Product Errata RHSA-2022:6287 0 None None None 2022-09-07 20:49:31 UTC

Description Pamela Escorza 2022-08-11 04:37:50 UTC
Description of problem:
This bug is opened to backport fix :
https://github.com/cri-o/cri-o/pull/5508

Version-Release number of selected component (if applicable):
OCP 4.11


Actual results:
Percpu memory usage is high:
$ cat proc/meminfo | awk '{print $2 "    " $1}'| sort -rn | awk '{print $1 " = " int($1/1024) "MB  -  "int($1/1024/1024)"GB   " $2}' | grep -E 'MemTotal|MemFree|Buffers|Cached|Percpu'
32897520 = 32126MB  -  31GB   MemTotal:
19215360 = 18765MB  -  18GB   Percpu:
4547680 = 4441MB  -  4GB   Cached:
449000 = 438MB  -  0GB   MemFree:
3108 = 3MB  -  0GB   Buffers:
0 = 0MB  -  0GB   SwapCached:


Expected results:
Not high usage of percpu memory.

Additional info:
child of bug https://bugzilla.redhat.com/show_bug.cgi?id=2004037#c121

Comment 6 Sunil Choudhary 2022-09-01 12:51:03 UTC
% oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.11.0-0.nightly-2022-08-31-193418   True        False         5h31m   Cluster version is 4.11.0-0.nightly-2022-08-31-193418

sh-4.4# while :; do podman run --name=test1 --replace centos /bin/echo 'running'; done
ae97e06331bb7ec9ed7c5cd9e018003f0a45b7cea189a634d6bced7e52051b31
running
….

sh-4.4# grep Per /proc/meminfo
Percpu:             6000 kB

sh-4.4# cat /proc/cgroups | column -t
#subsys_name  hierarchy  num_cgroups  enabled
cpuset        9          68           1
cpu           2          286          1
cpuacct       2          286          1
blkio         8          286          1
memory        5          368          1
devices       3          282          1
freezer       6          68           1
net_cls       4          68           1
perf_event    12         68           1
net_prio      4          68           1
hugetlb       7          68           1
pids          11         286          1
rdma          10         68           1

sh-4.4# date
Thu Sep  1 12:45:46 UTC 2022

sh-4.4# grep Per /proc/meminfo
Percpu:             6048 kB

sh-4.4# cat /proc/cgroups | column -t
#subsys_name  hierarchy  num_cgroups  enabled
cpuset        9          65           1
cpu           2          282          1
cpuacct       2          282          1
blkio         8          282          1
memory        5          370          1
devices       3          278          1
freezer       6          65           1
net_cls       4          65           1
perf_event    12         65           1
net_prio      4          65           1
hugetlb       7          65           1
pids          11         282          1
rdma          10         65           1

Comment 8 errata-xmlrpc 2022-09-07 20:49:27 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.11.3 packages and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:6287


Note You need to log in before you can comment on or make changes to this bug.