Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2117462

Summary:	[4.11 backport] percpu Memory leak CRIO due to no garbage collection in /run/crio/exits for exited containers
Product:	OpenShift Container Platform	Reporter:	Pamela Escorza <pescorza>
Component:	Node	Assignee:	Peter Hunt <pehunt>
Node sub component:	CRI-O	QA Contact:	Sunil Choudhary <schoudha>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	medium
Priority:	medium
Version:	4.11
Target Milestone:	---
Target Release:	4.11.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-09-07 20:49:27 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	2116460
Bug Blocks:	2117461

Description Pamela Escorza 2022-08-11 04:37:50 UTC

Description of problem:
This bug is opened to backport fix :
https://github.com/cri-o/cri-o/pull/5508

Version-Release number of selected component (if applicable):
OCP 4.11


Actual results:
Percpu memory usage is high:
$ cat proc/meminfo | awk '{print $2 "    " $1}'| sort -rn | awk '{print $1 " = " int($1/1024) "MB  -  "int($1/1024/1024)"GB   " $2}' | grep -E 'MemTotal|MemFree|Buffers|Cached|Percpu'
32897520 = 32126MB  -  31GB   MemTotal:
19215360 = 18765MB  -  18GB   Percpu:
4547680 = 4441MB  -  4GB   Cached:
449000 = 438MB  -  0GB   MemFree:
3108 = 3MB  -  0GB   Buffers:
0 = 0MB  -  0GB   SwapCached:


Expected results:
Not high usage of percpu memory.

Additional info:
child of bug https://bugzilla.redhat.com/show_bug.cgi?id=2004037#c121

Comment 6 Sunil Choudhary 2022-09-01 12:51:03 UTC

% oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.11.0-0.nightly-2022-08-31-193418   True        False         5h31m   Cluster version is 4.11.0-0.nightly-2022-08-31-193418

sh-4.4# while :; do podman run --name=test1 --replace centos /bin/echo 'running'; done
ae97e06331bb7ec9ed7c5cd9e018003f0a45b7cea189a634d6bced7e52051b31
running
….

sh-4.4# grep Per /proc/meminfo
Percpu:             6000 kB

sh-4.4# cat /proc/cgroups | column -t
#subsys_name  hierarchy  num_cgroups  enabled
cpuset        9          68           1
cpu           2          286          1
cpuacct       2          286          1
blkio         8          286          1
memory        5          368          1
devices       3          282          1
freezer       6          68           1
net_cls       4          68           1
perf_event    12         68           1
net_prio      4          68           1
hugetlb       7          68           1
pids          11         286          1
rdma          10         68           1

sh-4.4# date
Thu Sep  1 12:45:46 UTC 2022

sh-4.4# grep Per /proc/meminfo
Percpu:             6048 kB

sh-4.4# cat /proc/cgroups | column -t
#subsys_name  hierarchy  num_cgroups  enabled
cpuset        9          65           1
cpu           2          282          1
cpuacct       2          282          1
blkio         8          282          1
memory        5          370          1
devices       3          278          1
freezer       6          65           1
net_cls       4          65           1
perf_event    12         65           1
net_prio      4          65           1
hugetlb       7          65           1
pids          11         282          1
rdma          10         65           1

Comment 8 errata-xmlrpc 2022-09-07 20:49:27 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.11.3 packages and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:6287