Bug 2166889 - High memory consumption of virt-launcher process [NEEDINFO]
Summary: High memory consumption of virt-launcher process
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Container Native Virtualization (CNV)
Classification: Red Hat
Component: Virtualization
Version: 4.10.1
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ---
: 4.14.0
Assignee: sgott
QA Contact: Kedar Bidarkar
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-02-03 11:36 UTC by Juan Orti
Modified: 2023-07-11 12:52 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-07-11 12:52:45 UTC
Target Upstream Version:
Embargoed:
acardace: needinfo? (kbidarka)


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker CNV-25029 0 None None None 2023-02-03 11:39:18 UTC

Description Juan Orti 2023-02-03 11:36:36 UTC
Description of problem:
We have observed an event in which the virt-launcher process was consuming too much memory (+2 GiB) and was oom-killed because the VM has been configured with limits.

Version-Release number of selected component (if applicable):
Openshift 4.10
Openshift Virtualization 4.10.1
registry.redhat.io/container-native-virtualization/virt-launcher@sha256:e5270cccdbbc7c191f69b833d7b813ea20e0a29b0b1adf644ce3630127d4459e

How reproducible:
Only 1 event observed in customer's environment

Steps to Reproduce:
Not reproduced.

Actual results:
The virt-launcher process was killed with this message:

~~~
[414374.527154] virt-launcher invoked oom-killer: gfp_mask=0x6000c0(GFP_KERNEL), order=0, oom_score_adj=997
[414374.537771] CPU: 9 PID: 3541104 Comm: virt-launcher Not tainted 4.18.0-305.45.1.el8_4.x86_64 #1
[414374.547598] Hardware name: _REDACTED_
[414374.559656] Call Trace:
[414374.562495]  dump_stack+0x5c/0x80
[414374.566305]  dump_header+0x4a/0x1db
[414374.570305]  oom_kill_process.cold.32+0xb/0x10
[414374.575371]  out_of_memory+0x1ab/0x4a0
[414374.579654]  mem_cgroup_out_of_memory+0xe8/0x100
[414374.584917]  try_charge+0x65a/0x690
[414374.588916]  mem_cgroup_charge+0xca/0x220
[414374.593490]  do_anonymous_page+0x101/0x380
[414374.598170]  __handle_mm_fault+0x983/0xca0
[414374.602851]  ? __dentry_kill+0x121/0x170
[414374.607334]  handle_mm_fault+0xc2/0x1d0
[414374.611723]  __do_page_fault+0x1ed/0x4c0
[414374.616210]  do_page_fault+0x37/0x130
[414374.620404]  ? page_fault+0x8/0x30
[414374.624303]  page_fault+0x1e/0x30
[414374.628109] RIP: 0033:0x48e653
[414374.631621] Code: 00 48 81 eb 00 01 00 00 48 81 c7 00 01 00 00 48 81 fb 00 01 00 00 73 82 e9 07 ff ff ff c5 fd ef c0 48 81 fb 00 00 00 02 73 46 <c5> fe 7f 07 c5 fe 7f 47 20 c5 fe 7f 47 40 c5 fe 7f 47 60 48 81 eb
[414374.652720] RSP: 002b:000000c0323e47e0 EFLAGS: 00010202
[414374.658662] RAX: 0000000000000000 RBX: 0000000000004000 RCX: 000000c0873da000
[414374.666738] RDX: 000000000003cf4a RSI: 0000000000000020 RDI: 000000c0873e6000
[414374.674814] RBP: 000000c0323e4828 R08: 000000c0873da000 R09: 00007fb004012b90
[414374.682891] R10: 0000000000000008 R11: 00000000000019f4 R12: 0000000000000007
[414374.690966] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000001
[414374.699073] memory: usage 3366512kB, limit 3366512kB, failcnt 330
[414374.706075] memory+swap: usage 3366512kB, limit 9007199254740988kB, failcnt 0
[414374.714161] kmem: usage 64316kB, limit 9007199254740988kB, failcnt 0
[414374.721371] Memory cgroup stats for /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9558e503_dc06_4ba0_82b1_34ee6baae44c.slice:
[414374.721393] anon 3376525312
                file 540672
                kernel_stack 2838528
                slab 40289128
                percpu 522240
                sock 0
                shmem 0
                file_mapped 0
                file_dirty 811008
                file_writeback 0
                anon_thp 692060160
                inactive_anon 2828771328
                active_anon 547590144
                inactive_file 335872
                active_file 0
                unevictable 0
                slab_reclaimable 12228464
                slab_unreclaimable 28060664
                pgfault 635118
                pgmajfault 0
                workingset_refault_anon 0
                workingset_refault_file 0
                workingset_activate_anon 0
                workingset_activate_file 0
                workingset_restore_anon 0
                workingset_restore_file 0
                workingset_nodereclaim 0
                pgrefill 0
                pgscan 3001
                pgsteal 190
                pgactivate 134053
                pgdeactivate 0
                pglazyfree 0
                pglazyfreed 0
                thp_fault_alloc 297
                thp_collapse_alloc 0
[414374.848383] Tasks state (memory values in pages):
[414374.854048] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[414374.864096] [3532309]     0 3532309    35955      615   172032        0         -1000 conmon
[414374.873955] [3532321]     0 3532321  1293486    19745   978944        0           997 virt-launcher
[414374.884503] [3532372]     0 3532372  2449039   594970  5861376        0           997 virt-launcher
[414374.895060] [3532391]     0 3532391   371850     7557   503808        0           997 libvirtd
[414374.905108] [3532392]     0 3532392    36649     3670   315392        0           997 virtlogd
[414374.915142] [3532826]   107 3532826  1110789   229074  7213056        0           997 qemu-kvm
[414374.925199] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=crio-cb91a1d5720e2a8e56bd4476bb4b20044e778a3a8fd31e95862a85c4d5a86d42.scope,mems_allowed=0-1,oom_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9558e503_dc06_4ba0_82b1_34ee6baae44c.slice,task_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9558e503_dc06_4ba0_82b1_34ee6baae44c.slice/crio-cb91a1d5720e2a8e56bd4476bb4b20044e778a3a8fd31e95862a85c4d5a86d42.scope,task=virt-launcher,pid=3532372,uid=0
[414374.977305] Memory cgroup out of memory: Killed process 3532372 (virt-launcher) total-vm:9796156kB, anon-rss:2338832kB, file-rss:41112kB, shmem-rss:0kB, UID:0 pgtables:5724kB oom_score_adj:997
[414375.010625] oom_reaper: reaped process 3532372 (virt-launcher), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
~~~

Expected results:


Additional info:

Comment 9 Kedar Bidarkar 2023-03-01 13:41:22 UTC
Moving this bug to CNV 4.14 as we are unable to reproduce this bug currently.

Comment 10 Antonio Cardace 2023-06-20 10:15:43 UTC
@kbidarka were you able to reproduce this? I'd be inclined to close this if we're not able to repro it.

Comment 11 Antonio Cardace 2023-07-11 12:52:45 UTC
Closing as 'CURRENTRELEASE' as the bug was not reproduced on 4.14.


Note You need to log in before you can comment on or make changes to this bug.