Description of problem: We have observed an event in which the virt-launcher process was consuming too much memory (+2 GiB) and was oom-killed because the VM has been configured with limits. Version-Release number of selected component (if applicable): Openshift 4.10 Openshift Virtualization 4.10.1 registry.redhat.io/container-native-virtualization/virt-launcher@sha256:e5270cccdbbc7c191f69b833d7b813ea20e0a29b0b1adf644ce3630127d4459e How reproducible: Only 1 event observed in customer's environment Steps to Reproduce: Not reproduced. Actual results: The virt-launcher process was killed with this message: ~~~ [414374.527154] virt-launcher invoked oom-killer: gfp_mask=0x6000c0(GFP_KERNEL), order=0, oom_score_adj=997 [414374.537771] CPU: 9 PID: 3541104 Comm: virt-launcher Not tainted 4.18.0-305.45.1.el8_4.x86_64 #1 [414374.547598] Hardware name: _REDACTED_ [414374.559656] Call Trace: [414374.562495] dump_stack+0x5c/0x80 [414374.566305] dump_header+0x4a/0x1db [414374.570305] oom_kill_process.cold.32+0xb/0x10 [414374.575371] out_of_memory+0x1ab/0x4a0 [414374.579654] mem_cgroup_out_of_memory+0xe8/0x100 [414374.584917] try_charge+0x65a/0x690 [414374.588916] mem_cgroup_charge+0xca/0x220 [414374.593490] do_anonymous_page+0x101/0x380 [414374.598170] __handle_mm_fault+0x983/0xca0 [414374.602851] ? __dentry_kill+0x121/0x170 [414374.607334] handle_mm_fault+0xc2/0x1d0 [414374.611723] __do_page_fault+0x1ed/0x4c0 [414374.616210] do_page_fault+0x37/0x130 [414374.620404] ? page_fault+0x8/0x30 [414374.624303] page_fault+0x1e/0x30 [414374.628109] RIP: 0033:0x48e653 [414374.631621] Code: 00 48 81 eb 00 01 00 00 48 81 c7 00 01 00 00 48 81 fb 00 01 00 00 73 82 e9 07 ff ff ff c5 fd ef c0 48 81 fb 00 00 00 02 73 46 <c5> fe 7f 07 c5 fe 7f 47 20 c5 fe 7f 47 40 c5 fe 7f 47 60 48 81 eb [414374.652720] RSP: 002b:000000c0323e47e0 EFLAGS: 00010202 [414374.658662] RAX: 0000000000000000 RBX: 0000000000004000 RCX: 000000c0873da000 [414374.666738] RDX: 000000000003cf4a RSI: 0000000000000020 RDI: 000000c0873e6000 [414374.674814] RBP: 000000c0323e4828 R08: 000000c0873da000 R09: 00007fb004012b90 [414374.682891] R10: 0000000000000008 R11: 00000000000019f4 R12: 0000000000000007 [414374.690966] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000001 [414374.699073] memory: usage 3366512kB, limit 3366512kB, failcnt 330 [414374.706075] memory+swap: usage 3366512kB, limit 9007199254740988kB, failcnt 0 [414374.714161] kmem: usage 64316kB, limit 9007199254740988kB, failcnt 0 [414374.721371] Memory cgroup stats for /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9558e503_dc06_4ba0_82b1_34ee6baae44c.slice: [414374.721393] anon 3376525312 file 540672 kernel_stack 2838528 slab 40289128 percpu 522240 sock 0 shmem 0 file_mapped 0 file_dirty 811008 file_writeback 0 anon_thp 692060160 inactive_anon 2828771328 active_anon 547590144 inactive_file 335872 active_file 0 unevictable 0 slab_reclaimable 12228464 slab_unreclaimable 28060664 pgfault 635118 pgmajfault 0 workingset_refault_anon 0 workingset_refault_file 0 workingset_activate_anon 0 workingset_activate_file 0 workingset_restore_anon 0 workingset_restore_file 0 workingset_nodereclaim 0 pgrefill 0 pgscan 3001 pgsteal 190 pgactivate 134053 pgdeactivate 0 pglazyfree 0 pglazyfreed 0 thp_fault_alloc 297 thp_collapse_alloc 0 [414374.848383] Tasks state (memory values in pages): [414374.854048] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [414374.864096] [3532309] 0 3532309 35955 615 172032 0 -1000 conmon [414374.873955] [3532321] 0 3532321 1293486 19745 978944 0 997 virt-launcher [414374.884503] [3532372] 0 3532372 2449039 594970 5861376 0 997 virt-launcher [414374.895060] [3532391] 0 3532391 371850 7557 503808 0 997 libvirtd [414374.905108] [3532392] 0 3532392 36649 3670 315392 0 997 virtlogd [414374.915142] [3532826] 107 3532826 1110789 229074 7213056 0 997 qemu-kvm [414374.925199] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=crio-cb91a1d5720e2a8e56bd4476bb4b20044e778a3a8fd31e95862a85c4d5a86d42.scope,mems_allowed=0-1,oom_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9558e503_dc06_4ba0_82b1_34ee6baae44c.slice,task_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9558e503_dc06_4ba0_82b1_34ee6baae44c.slice/crio-cb91a1d5720e2a8e56bd4476bb4b20044e778a3a8fd31e95862a85c4d5a86d42.scope,task=virt-launcher,pid=3532372,uid=0 [414374.977305] Memory cgroup out of memory: Killed process 3532372 (virt-launcher) total-vm:9796156kB, anon-rss:2338832kB, file-rss:41112kB, shmem-rss:0kB, UID:0 pgtables:5724kB oom_score_adj:997 [414375.010625] oom_reaper: reaped process 3532372 (virt-launcher), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB ~~~ Expected results: Additional info:
Moving this bug to CNV 4.14 as we are unable to reproduce this bug currently.
@kbidarka were you able to reproduce this? I'd be inclined to close this if we're not able to repro it.
Closing as 'CURRENTRELEASE' as the bug was not reproduced on 4.14.