Bug 609030 - oom killer failed for kvm guest
oom killer failed for kvm guest
Status: CLOSED DUPLICATE of bug 596971
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: kernel (Show other bugs)
6.0
All Linux
low Severity medium
: rc
: ---
Assigned To: Red Hat Kernel Manager
Red Hat Kernel QE team
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2010-06-29 04:55 EDT by CAI Qian
Modified: 2010-11-11 14:30 EST (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-07-06 11:24:53 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
guest xml (2.16 KB, application/xml)
2010-06-29 04:55 EDT, CAI Qian
no flags Details

  None (edit)
Description CAI Qian 2010-06-29 04:55:48 EDT
Created attachment 427615 [details]
guest xml

Description of problem:
OOM killer did not trigger in KVM guest when malloc bigger memory than the system memory. The guest had 1G memory in this case.

overcommit_memory = 1: guest no response

I also tested in bare-metal and it works although it probably killed wrong processes.

Version-Release number of selected component (if applicable):
both host and guest:
RHEL6 20100622.1 tree
qemu-kvm-0.12.1.2-2.78.el6.x86_64
kernel-2.6.32-37.el6.x86_64

How reproducible:
always

Steps to Reproduce:
1. set overcommit_memory to 1
2. ./malloc
  
Actual results:
guest hung

Expected results:
oom killer killed malloc like in bare-metal.
Comment 1 RHEL Product and Program Management 2010-06-29 05:02:56 EDT
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux major release.  Product Management has requested further
review of this request by Red Hat Engineering, for potential inclusion in a Red
Hat Enterprise Linux Major release.  This request is not yet committed for
inclusion.
Comment 2 Rik van Riel 2010-06-29 10:03:17 EDT
The memory allocation, pageout and out of memory code are exactly the same in the guest and the host.  Either you got "lucky", or we have some strange interaction going on...

Is the OOM kill reliable in the host, and does it fail reliably in the guest?
Comment 3 CAI Qian 2010-06-29 10:11:40 EDT
> Is the OOM kill reliable in the host, and does it fail reliably in the guest?    
Pretty reliable in the host. Every time so far (tried around 10 times). It failed reliably in the guest (only killed once for around 10 times).
Comment 4 CAI Qian 2010-06-29 23:31:30 EDT
This happened sometimes.

<caiqian> sign... while this happened. virsh dump did not work either.
<caiqian> # virsh dump rhel6 rhel6.core
<caiqian> it hung there forever in the host.
<riel> great
<caiqian> the guest can't be destroyed either.
<caiqian> # virsh destroy rhel6
<caiqian> error: Failed to destroy domain rhel6
<caiqian> error: Timed out during operation: cannot acquire state change lock
<riel> ohhhh
<caiqian> I need to restart libvirtd to destroy it.
<riel> I wonder if you're running into another bug, then
<riel> and not an OOM
<riel> just kvm getting stuck when all of the guest memory is used
<riel> (no memory available for some critical driver?)
Comment 5 CAI Qian 2010-06-30 01:07:04 EDT
I have finally been able to get virsh dump from the guest. From the analysing results, looks like oom killer did kick in, and killed a probably wrong process (console-kit-daemon) and also triggered a soft lockup.

crash> log
...
rpcbind invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0
rpcbind cpuset=/ mems_allowed=0
Pid: 1162, comm: rpcbind Not tainted 2.6.32-37.el6.x86_64 #1
Call Trace:
 [<ffffffff810c2021>] ? cpuset_print_task_mems_allowed+0x91/0xb0
 [<ffffffff8110eb7b>] oom_kill_process+0xcb/0x2e0
 [<ffffffff8110f140>] ? select_bad_process+0xd0/0x110
 [<ffffffff8110f1d8>] __out_of_memory+0x58/0xc0
 [<ffffffff8110f3d9>] out_of_memory+0x199/0x210
 [<ffffffff8111c80c>] __alloc_pages_nodemask+0x7cc/0x7e0
 [<ffffffff8114dae4>] alloc_pages_vma+0x84/0x110
 [<ffffffff81250f3f>] ? cfq_set_request+0x18f/0x520
 [<ffffffff811422e9>] read_swap_cache_async+0xe9/0x140
 [<ffffffff81142cf9>] ? valid_swaphandles+0x69/0x160
 [<ffffffff811423c7>] swapin_readahead+0x87/0xc0
 [<ffffffff81133745>] handle_pte_fault+0x655/0x9a0
 [<ffffffff8101187e>] ? __switch_to+0x26e/0x320
 [<ffffffff81058d62>] ? finish_task_switch+0x42/0xd0
 [<ffffffff81133c7d>] handle_mm_fault+0x1ed/0x2b0
 [<ffffffff814dcfb3>] do_page_fault+0x123/0x3a0
 [<ffffffff814da9a5>] page_fault+0x25/0x30
 [<ffffffff8117f3e7>] ? do_sys_poll+0x347/0x520
 [<ffffffff8117f3cf>] ? do_sys_poll+0x32f/0x520
 [<ffffffff8117ef50>] ? __pollwait+0x0/0xf0
 [<ffffffff8117f040>] ? pollwake+0x0/0x60
 [<ffffffff8117f040>] ? pollwake+0x0/0x60
 [<ffffffff8117f040>] ? pollwake+0x0/0x60
 [<ffffffff8117f040>] ? pollwake+0x0/0x60
 [<ffffffff8117f040>] ? pollwake+0x0/0x60
 [<ffffffff8117f040>] ? pollwake+0x0/0x60
 [<ffffffff8117f040>] ? pollwake+0x0/0x60
 [<ffffffff811455cd>] ? swapcache_free+0x4d/0x70
 [<ffffffff81142538>] ? delete_from_swap_cache+0x58/0x70
 [<ffffffff8110bed7>] ? unlock_page+0x27/0x30
 [<ffffffff81133648>] ? handle_pte_fault+0x558/0x9a0
 [<ffffffff813febe9>] ? sock_destroy_inode+0x19/0x20
 [<ffffffff8103c908>] ? pvclock_clocksource_read+0x58/0xd0
 [<ffffffff8103ba41>] ? kvm_clock_read+0x21/0x30
 [<ffffffff8103ba5e>] ? kvm_clock_get_cycles+0xe/0x10
 [<ffffffff8109afa9>] ? ktime_get_ts+0xa9/0xe0
 [<ffffffff8117e74d>] ? poll_select_set_timeout+0x8d/0xa0
 [<ffffffff8117f7bc>] sys_poll+0x7c/0x110
 [<ffffffff81013172>] system_call_fastpath+0x16/0x1b
Mem-Info:
Node 0 DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
Node 0 DMA32 per-cpu:
CPU    0: hi:  186, btch:  31 usd: 159
active_anon:34984 inactive_anon:35084 isolated_anon:0
 active_file:14 inactive_file:0 isolated_file:0
 unevictable:1227 dirty:0 writeback:0 unstable:0
 free:12286 slab_reclaimable:2405 slab_unreclaimable:15169
 mapped:907 shmem:6 pagetables:1945 bounce:0
Node 0 DMA free:4640kB min:668kB low:832kB high:1000kB active_anon:3296kB inactive_anon:3444kB active_file:4kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15344kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:268kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 994 994 994
Node 0 DMA32 free:44504kB min:44384kB low:55480kB high:66576kB active_anon:136640kB inactive_anon:136892kB active_file:52kB inactive_file:0kB unevictable:4908kB isolated(anon):0kB isolated(file):0kB present:1018060kB mlocked:4908kB dirty:0kB writeback:0kB mapped:3628kB shmem:24kB slab_reclaimable:9620kB slab_unreclaimable:60672kB kernel_stack:1272kB pagetables:7512kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:13 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 0*4kB 2*8kB 1*16kB 4*32kB 0*64kB 1*128kB 1*256kB 0*512kB 2*1024kB 1*2048kB 0*4096kB = 4640kB
Node 0 DMA32: 338*4kB 130*8kB 154*16kB 85*32kB 29*64kB 36*128kB 51*256kB 20*512kB 5*1024kB 1*2048kB 0*4096kB = 44504kB
3363 total pagecache pages
2443 pages in swap cache
Swap cache stats: add 518093, delete 515650, find 189/421
Free swap  = 0kB
Total swap = 2064376kB
262138 pages RAM
72319 pages reserved
5958 pages shared
169102 pages non-shared
Out of memory: kill process 1512 (console-kit-dae) score 143702 or a child
Killed process 1512 (console-kit-dae) vsz:574808kB, anon-rss:0kB, file-rss:600kB
BUG: soft lockup - CPU#0 stuck for 61s! [kswapd0:26]
Modules linked in: sunrpc(U) xt_physdev(U) ip6t_REJECT(U) nf_conntrack_ipv6(U) ip6table_filter(U) ip6_tables(U) be2iscsi(U) bnx2i(U) cnic(U) uio(U) cxgb3i(U) cxgb3(U) mdio(U) ib_iser(U) rdma_cm(U) ib_cm(U) iw_cm(U) ib_sa(U) ib_mad(U) ib_core(U) ib_addr(U) ipv6(U) iscsi_tcp(U) libiscsi_tcp(U) libiscsi(U) scsi_transport_iscsi(U) dm_mirror(U) dm_region_hash(U) dm_log(U) virtio_net(U) virtio_balloon(U) snd_intel8x0(U) snd_ac97_codec(U) ac97_bus(U) snd_seq(U) snd_seq_device(U) snd_pcm(U) snd_timer(U) snd(U) soundcore(U) snd_page_alloc(U) i2c_piix4(U) i2c_core(U) ext4(U) mbcache(U) jbd2(U) virtio_blk(U) virtio_pci(U) virtio_ring(U) virtio(U) ata_generic(U) pata_acpi(U) ata_piix(U) dm_mod(U) [last unloaded: scsi_wait_scan]
CPU 0:
Modules linked in: sunrpc(U) xt_physdev(U) ip6t_REJECT(U) nf_conntrack_ipv6(U) ip6table_filter(U) ip6_tables(U) be2iscsi(U) bnx2i(U) cnic(U) uio(U) cxgb3i(U) cxgb3(U) mdio(U) ib_iser(U) rdma_cm(U) ib_cm(U) iw_cm(U) ib_sa(U) ib_mad(U) ib_core(U) ib_addr(U) ipv6(U) iscsi_tcp(U) libiscsi_tcp(U) libiscsi(U) scsi_transport_iscsi(U) dm_mirror(U) dm_region_hash(U) dm_log(U) virtio_net(U) virtio_balloon(U) snd_intel8x0(U) snd_ac97_codec(U) ac97_bus(U) snd_seq(U) snd_seq_device(U) snd_pcm(U) snd_timer(U) snd(U) soundcore(U) snd_page_alloc(U) i2c_piix4(U) i2c_core(U) ext4(U) mbcache(U) jbd2(U) virtio_blk(U) virtio_pci(U) virtio_ring(U) virtio(U) ata_generic(U) pata_acpi(U) ata_piix(U) dm_mod(U) [last unloaded: scsi_wait_scan]
Pid: 26, comm: kswapd0 Not tainted 2.6.32-37.el6.x86_64 #1 KVM
RIP: 0010:[<ffffffff81121b2c>]  [<ffffffff81121b2c>] shrink_slab+0xac/0x190
RSP: 0018:ffff88003ae1bcd0  EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff88003ae1bd20 RCX: 0000000000000002
RDX: 0000000000000000 RSI: 00000000000000d0 RDI: 0000000000000000
RBP: ffffffff81013c8e R08: 0000000000000000 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000996
R13: ffffffff8111da4f R14: ffff88003ae1bc70 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff880012200000(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000404490 CR3: 000000003b426000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
 [<ffffffff81124c1e>] ? balance_pgdat+0x53e/0x760
 [<ffffffff81125100>] ? isolate_pages_global+0x0/0x250
 [<ffffffff81124f5e>] ? kswapd+0x11e/0x2c0
 [<ffffffff81090a50>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81124e40>] ? kswapd+0x0/0x2c0
 [<ffffffff810906e6>] ? kthread+0x96/0xa0
 [<ffffffff810141ca>] ? child_rip+0xa/0x20
 [<ffffffff81090650>] ? kthread+0x0/0xa0
 [<ffffffff810141c0>] ? child_rip+0x0/0x20
Comment 8 Rik van Riel 2010-07-06 10:57:08 EDT
Cai, how many VCPUs does your KVM guest have?

It looks like this issue may be a duplicate of the issues raised here:

http://post-office.corp.redhat.com/archives/rhkernel-list/2010-June/msg01046.html
Comment 9 CAI Qian 2010-07-06 11:21:31 EDT
Thanks for pointing out. It was reproduced on a single CPU guest. 2-CPU guest has no such problem, so it looks like the problem you mentioned. I'll test out with a test kernel with the above patch.
Comment 10 CAI Qian 2010-07-06 11:24:53 EDT

*** This bug has been marked as a duplicate of bug 596971 ***

Note You need to log in before you can comment on or make changes to this bug.