Bug 236377 - [RHEL5 RT] This test failed because of memory pressure. It oom killed mem01
[RHEL5 RT] This test failed because of memory pressure. It oom killed mem01
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: realtime-kernel (Show other bugs)
1.0
All Linux
medium Severity medium
: ---
: ---
Assigned To: Larry Woodman
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2007-04-13 10:45 EDT by Jeff Burke
Modified: 2008-03-19 09:12 EDT (History)
1 user (show)

See Also:
Fixed In Version: 2.6.24
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2008-03-19 09:12:30 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Jeff Burke 2007-04-13 10:45:10 EDT
Description of problem:
On dl385-01.rhts.boston.redhat.com - x86_64 when running the following test
/kernel/distribution/ltp/20070228 the system will oom kill the test

System information:
http://rhts.lab.boston.redhat.com/cgi-bin/rhts/test_log.cgi?id=1240772
http://rhts.lab.boston.redhat.com/cgi-bin/rhts/test_log.cgi?id=1240853

This test failed because of memory pressure. It oom killed mem01

No available memory (MPOL_BIND): kill process 26685 (mem01) score 0 or a
child

Version-Release number of selected component (if applicable):
 2.6.20-12.el5rt

How reproducible:
 Unknown

Steps to Reproduce:
1. Run the /kernel/distribution/ltp/20070228 on 
dl385-01.rhts.boston.redhat.com
  
Actual results:

No available memory (MPOL_BIND): kill process 26685 (mem01) score 0 or a
child
Killed process 26685 (mem01)
 [<ffffffff81033b9b>] read_swap_cache_async+0x4b/0xde
 [<ffffffff810cfd6b>] swapin_readahead+0x61/0xcd
 [<ffffffff81008f23>] __handle_mm_fault+0x92f/0xd2b
 [<ffffffff81068901>] do_page_fault+0x46d/0x813
 [<ffffffff81066aed>] error_exit+0x0/0x84
 [<ffffffff81030b90>] do_sys_poll+0x323/0x3dc
 [<ffffffff8104db03>] sys_poll+0x34/0x3d
 [<ffffffff8105f29e>] tracesys+0xdc/0xe1
 [<00002b67e563f88f>]

Expected results:
 This test should pass.

Additional info:
 This does result in the same behavior with RHEL5 GA kernel.
Comment 1 Larry Woodman 2007-05-17 11:41:48 EDT
Jeff, I posted a RHEL4 patch to rhkernel-list to deal with swapin_readahead
under memory pressure on 2/23/07.  It looks like this exact same issue.  Can you
grab the show_mem() output from /var/log/messages when this OOM kill happened? 
I really dont have enough to go on with the above output.

Larry

Comment 2 Jeff Burke 2007-05-17 13:37:58 EDT
Larry,
System information:
http://armstrong.rhts.boston.redhat.com/cgi-bin/rhts/test_log.cgi?id=1240772
http://armstrong.rhts.boston.redhat.com/cgi-bin/rhts/test_log.cgi?id=1240853

Mem-info:
Node 0 DMA per-cpu:
CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    1: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    2: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    3: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Node 0 DMA32 per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:  31   Cold: hi:   62, btch:  15 usd:  56
CPU    1: Hot: hi:  186, btch:  31 usd:  31   Cold: hi:   62, btch:  15 usd:  14
CPU    2: Hot: hi:  186, btch:  31 usd: 158   Cold: hi:   62, btch:  15 usd:  40
CPU    3: Hot: hi:  186, btch:  31 usd:  59   Cold: hi:   62, btch:  15 usd:  56
Active:119851 inactive:119329 dirty:0 writeback:0 unstable:0 free:1992 slab:4528
mapped:657 pagetables:2194
Node 0 DMA free:3996kB min:60kB low:72kB high:88kB active:5836kB inactive:5952kB
present:15568kB pages_scanned:41505 all_unreclaimable? yes
lowmem_reserve[]: 0 984 984
Node 0 DMA32 free:3972kB min:3980kB low:4972kB high:5968kB active:473696kB
inactive:471364kB present:1007972kB pages_scanned:1545709 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0
Node 0 DMA: 1*4kB 1*8kB 1*16kB 0*32kB 0*64kB 1*128kB 1*256kB 1*512kB 1*1024kB
1*2048kB 0*4096kB = 3996kB
Node 0 DMA32: 17*4kB 0*8kB 0*16kB 0*32kB 1*64kB 0*128kB 1*256kB 1*512kB 1*1024kB
1*2048kB 0*4096kB = 3972kB
Swap cache: add 1444147, delete 1443927, find 56127/110668, race 0+0
Free swap  = 0kB
Total swap = 2031608kB
Free swap:            0kB
262136 pages of RAM
9131 reserved pages
4655 pages shared
220 pages swap cached
klogd invoked oom-killer: gfp_mask=0x200d2, order=0, oomkilladj=0

Call Trace:
 [<ffffffff8106d603>] dump_trace+0xb6/0x3ce
 [<ffffffff8106d962>] show_trace+0x47/0x68
 [<ffffffff8106d998>] dump_stack+0x15/0x17
 [<ffffffff810caa65>] out_of_memory+0x74/0x2c5
 [<ffffffff8100f37d>] __alloc_pages+0x238/0x2c5
 [<ffffffff810228ca>] alloc_page_vma+0x85/0x8e
 [<ffffffff81033b9b>] read_swap_cache_async+0x4b/0xde
 [<ffffffff810cfd6b>] swapin_readahead+0x61/0xcd
 [<ffffffff81008f23>] __handle_mm_fault+0x92f/0xd2b
 [<ffffffff81068901>] do_page_fault+0x46d/0x813
 [<ffffffff81066aed>] error_exit+0x0/0x84
 [<ffffffff81092790>] do_syslog+0x168/0x3af
 [<ffffffff8110d8d5>] kmsg_read+0x3b/0x46
 [<ffffffff8100b1e8>] vfs_read+0xcc/0x175
 [<ffffffff81011aa7>] sys_read+0x47/0x6f
 [<ffffffff8105f29e>] tracesys+0xdc/0xe1
 [<00002ac525d80a10>]

Mem-info:
Node 0 DMA per-cpu:
CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    1: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    2: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    3: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Node 0 DMA32 per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:  31   Cold: hi:   62, btch:  15 usd:  56
CPU    1: Hot: hi:  186, btch:  31 usd:  31   Cold: hi:   62, btch:  15 usd:  14
CPU    2: Hot: hi:  186, btch:  31 usd: 158   Cold: hi:   62, btch:  15 usd:  40
CPU    3: Hot: hi:  186, btch:  31 usd:  59   Cold: hi:   62, btch:  15 usd:  56
Active:119857 inactive:119323 dirty:0 writeback:0 unstable:0 free:1992 slab:4528
mapped:657 pagetables:2194
Node 0 DMA free:3996kB min:60kB low:72kB high:88kB active:5836kB inactive:5952kB
present:15568kB pages_scanned:41505 all_unreclaimable? yes
lowmem_reserve[]: 0 984 984
Node 0 DMA32 free:3972kB min:3980kB low:4972kB high:5968kB active:473592kB
inactive:471340kB present:1007972kB pages_scanned:1563399 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0
Node 0 DMA: 1*4kB 1*8kB 1*16kB 0*32kB 0*64kB 1*128kB 1*256kB 1*512kB 1*1024kB
1*2048kB 0*4096kB = 3996kB
Node 0 DMA32: 17*4kB 0*8kB 0*16kB 0*32kB 1*64kB 0*128kB 1*256kB 1*512kB 1*1024kB
1*2048kB 0*4096kB = 3972kB
Swap cache: add 1444147, delete 1443927, find 56127/110668, race 0+0
Free swap  = 0kB
Total swap = 2031608kB
Free swap:            0kB
262136 pages of RAM
9131 reserved pages
4667 pages shared
220 pages swap cached
klogd invoked oom-killer: gfp_mask=0x200d2, order=0, oomkilladj=0

Call Trace:
 [<ffffffff8106d603>] dump_trace+0xb6/0x3ce
 [<ffffffff8106d962>] show_trace+0x47/0x68
 [<ffffffff8106d998>] dump_stack+0x15/0x17
 [<ffffffff810caa65>] out_of_memory+0x74/0x2c5
 [<ffffffff8100f37d>] __alloc_pages+0x238/0x2c5
 [<ffffffff810228ca>] alloc_page_vma+0x85/0x8e
 [<ffffffff81033b9b>] read_swap_cache_async+0x4b/0xde
 [<ffffffff810cfd6b>] swapin_readahead+0x61/0xcd
 [<ffffffff81008f23>] __handle_mm_fault+0x92f/0xd2b
 [<ffffffff81068901>] do_page_fault+0x46d/0x813
 [<ffffffff81066aed>] error_exit+0x0/0x84
 [<ffffffff81092790>] do_syslog+0x168/0x3af
 [<ffffffff8110d8d5>] kmsg_read+0x3b/0x46
 [<ffffffff8100b1e8>] vfs_read+0xcc/0x175
 [<ffffffff81011aa7>] sys_read+0x47/0x6f
 [<ffffffff8105f29e>] tracesys+0xdc/0xe1
 [<00002ac525d80a10>]

Comment 3 Larry Woodman 2007-05-17 13:45:25 EDT
Jeff in this case the system ran out of swap space and every zone's
all_unreclaimable is "yes" so its unlikely that there is any pagecache memory to
reclaim and it cant reclaim any anonymous memory.  Does this workload run on any
RHEL4, RHEL5 or RT kernel hased system without OOM killing?

Larry
Comment 4 Jeff Burke 2007-05-17 13:58:31 EDT
Larry,
   It does passed in RHEL5 GA.
Comment 5 Larry Woodman 2007-05-17 14:36:41 EDT
Jeff, its most likely that linux-2.6-mm-prevent-oom-fixes.patch needs to be
ported to the RT kernel.  I ported this patch from RHEL4 to RHEL5 to prevent
transient low memory combined with swap space exhaustion problems from causing
OOM kills.  I'll port it to the RT kernel as well so you can give it a try.

Larry
Comment 6 Larry Woodman 2007-05-18 07:38:46 EDT
Jeff, I ported this patch the the RT kernel.  Can you try this kernel:
>>>barstool.build:/mnt/brew/scratch/lwoodman/task_789013

Larry
Comment 7 Jeff Burke 2007-05-18 09:00:24 EDT
Larry,
    Thanks. I will try it now. In the future can you please add and identifier
to the kernel N-V-R. 

    Currently your and the "real"/original 2.6.21-6.el5rt have the same name.
When I run this through RHTS it will skew the results for the real
2.6.21-6.el5rt kernel.

Jeff
Comment 8 Jeff Burke 2007-05-18 19:44:05 EDT
file_rm.sh invoked oom-killer: gfp_mask=0xd0, order=1, oomkilladj=0
 [<c0405fec>] dump_trace+0x5f/0x107
 [<c04060ae>] show_trace_log_lvl+0x1a/0x2f
 [<c04066d0>] show_trace+0x12/0x14
 [<c0406763>] dump_stack+0x16/0x18
 [<c045fc20>] out_of_memory+0x68/0x1b0
 [<c046120e>] __alloc_pages+0x1f9/0x282
 [<c0475f06>] cache_alloc_refill+0x29d/0x4bf
 [<c0475c48>] kmem_cache_alloc+0x50/0x71
 [<c0426009>] copy_process+0x83/0x1176
 [<c04273bb>] do_fork+0x99/0x177
 [<c0403230>] sys_clone+0x33/0x39
 [<c0404fc0>] syscall_call+0x7/0xb
 [<ffffe410>] pg0+0x3f71f410/0xfffff7b8
 =======================
Mem-info:
DMA per-cpu:
CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    1: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    2: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    3: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:  12   Cold: hi:   62, btch:  15 usd:  49
CPU    1: Hot: hi:  186, btch:  31 usd:  49   Cold: hi:   62, btch:  15 usd:  50
CPU    2: Hot: hi:  186, btch:  31 usd:  34   Cold: hi:   62, btch:  15 usd:  58
CPU    3: Hot: hi:  186, btch:  31 usd:  77   Cold: hi:   62, btch:  15 usd:  60
HighMem per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:  41   Cold: hi:   62, btch:  15 usd:  37
CPU    1: Hot: hi:  186, btch:  31 usd: 163   Cold: hi:   62, btch:  15 usd:  53
CPU    2: Hot: hi:  186, btch:  31 usd:  73   Cold: hi:   62, btch:  15 usd:  54
CPU    3: Hot: hi:  186, btch:  31 usd: 150   Cold: hi:   62, btch:  15 usd:  48
Active:463681 inactive:333 dirty:151655 writeback:76 unstable:0
 free:192202 slab:207535 mapped:3050 pagetables:427 bounce:0
DMA free:3532kB min:64kB low:80kB high:96kB active:0kB inactive:0kB
present:16160kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 867 4024
Normal free:3576kB min:3736kB low:4668kB high:5604kB active:1204kB
inactive:916kB present:888800kB pages_scanned:6915 all_unreclaimable? yes
lowmem_reserve[]: 0 0 25250
HighMem free:761700kB min:512kB low:3908kB high:7304kB active:1853520kB
inactive:416kB present:3232000kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0
DMA: 15*4kB 2*8kB 0*16kB 0*32kB 0*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB
0*4096kB = 3532kB
Normal: 26*4kB 0*8kB 6*16kB 1*32kB 3*64kB 0*128kB 0*256kB 0*512kB 1*1024kB
1*2048kB 0*4096kB = 3496kB
HighMem: 21046*4kB 34981*8kB 11089*16kB 2373*32kB 613*64kB 764*128kB 14*256kB
5*512kB 1*1024kB 0*2048kB 0*4096kB = 761584kB
Swap cache: add 13, delete 13, find 0/0, race 0+0
Free swap  = 2031556kB
Total swap = 2031608kB
Free swap:       2031556kB
1048576 pages of RAM
819200 pages of HIGHMEM
180355 reserved pages
350054 pages shared
0 pages swap cached
151655 pages dirty
76 pages writeback
3050 pages mapped
207535 pages slab
427 pages pagetables
file_rm.sh invoked oom-killer: gfp_mask=0xd0, order=1, oomkilladj=0
 [<c0405fec>] dump_trace+0x5f/0x107
 [<c04060ae>] show_trace_log_lvl+0x1a/0x2f
 [<c04066d0>] show_trace+0x12/0x14
 [<c0406763>] dump_stack+0x16/0x18
 [<c045fc20>] out_of_memory+0x68/0x1b0
 [<c046120e>] __alloc_pages+0x1f9/0x282
 [<c0475f06>] cache_alloc_refill+0x29d/0x4bf
 [<c0475c48>] kmem_cache_alloc+0x50/0x71
 [<c0426009>] copy_process+0x83/0x1176
 [<c04273bb>] do_fork+0x99/0x177
 [<c0403230>] sys_clone+0x33/0x39
 [<c0404fc0>] syscall_call+0x7/0xb
 [<ffffe410>] pg0+0x3f71f410/0xfffff7b8
 =======================
Comment 9 Larry Woodman 2007-06-04 15:42:06 EDT

Jeff, this latest OOMkill is on a 32-bit x86 system.  Does the x86_64 system
running the kernel I built run any different?

Larry
Comment 10 Jeff Burke 2007-06-07 17:03:50 EDT
Larry,
    I don't have the results that you are looking for and the scratch build you
did is now gone. Can you please push through a new brew kernel. Please make sure
you give the kernel a unique name. i will re-run the tests. FYI the latest
kernel is 2.6.21-23.el5rt

Jeff
Comment 11 Larry Woodman 2007-06-11 15:53:59 EDT
Jeff, can you reproduce this in the latest RT kernel in an x86_64 system that
has swap space and get me the show_mem() output when the OOM kill occurs?  The
cause of the x86 failure is the normal zone is consumed by the slab cache and
x86_64 system are not limited by Lowmem.

slab:207535

Normal free:3576kB min:3736kB low:4668kB high:5604kB active:1204kB
inactive:916kB present:888800kB pages_scanned:6915 all_unreclaimable? yes


Larry
Comment 12 Larry Woodman 2007-06-14 08:24:02 EDT
So the OOM kill in comment #2 is an x86_64 system that exhausted swap space,
therefore this is expected behavior.  The OOM kill in comment #8 is an x86
system that the slabcache exhausted lowmem, this is not expected behavior but I
need a /proc/slabinfo output to see what data structures are sucking all of this
memory down.  How should we proceed???

Larry
 
Comment 13 Jeff Burke 2007-06-16 22:11:30 EDT
Call Trace:
klogd invoked oom-killer: gfp_mask=0x200d2, order=0, oomkilladj=0

Call Trace:
 [<ffffffff8028f9ee>] printk_ratelimit+0x15/0x17
 [<ffffffff802becf4>] out_of_memory+0x74/0x2c0
 [<ffffffff8020f67a>] __alloc_pages+0x238/0x2c5
 [<ffffffff8024764b>] try_to_wake_up+0x418/0x429
 [<ffffffff802226ef>] alloc_page_vma+0x85/0x8e
 [<ffffffff80233702>] read_swap_cache_async+0x4b/0xde
 [<ffffffff802c3de9>] swapin_readahead+0x61/0xcd
 [<ffffffff80264d8a>] _read_lock_irq+0x9/0xb
 [<ffffffff80208f15>] __handle_mm_fault+0x956/0xd52
 [<ffffffff80267549>] do_page_fault+0x457/0x7fd
 [<ffffffff8026575d>] error_exit+0x0/0x84
 [<ffffffff8028fb92>] do_syslog+0x168/0x3af
 [<ffffffff8028fb92>] do_syslog+0x168/0x3af
 [<ffffffff8029f1b1>] autoremove_wake_function+0x0/0x38
 [<ffffffff80301f69>] kmsg_read+0x3b/0x46
 [<ffffffff8020b39b>] vfs_read+0xcc/0x155
 [<ffffffff80211d58>] sys_read+0x47/0x6f
 [<ffffffff8025e29c>] tracesys+0xdc/0xe1

Mem-info:
Node 0 DMA per-cpu:
CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    1: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    2: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    3: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Node 0 DMA32 per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:  30   Cold: hi:   62, btch:  15 usd:  10
CPU    1: Hot: hi:  186, btch:  31 usd:  41   Cold: hi:   62, btch:  15 usd:  40
CPU    2: Hot: hi:  186, btch:  31 usd: 157   Cold: hi:   62, btch:  15 usd:  57
CPU    3: Hot: hi:  186, btch:  31 usd:  61   Cold: hi:   62, btch:  15 usd:  54
Active:112470 inactive:130484 dirty:0 writeback:0 unstable:0
 free:1983 slab:4276 mapped:644 pagetables:2209 bounce:0
Node 0 DMA free:4000kB min:28kB low:32kB high:40kB active:1976kB inactive:1800kB
present:7360kB pages_scanned:25260 all_unreclaimable? yes
lowmem_reserve[]: 0 994 994
Node 0 DMA32 free:3932kB min:4016kB low:5020kB high:6024kB active:447904kB
inactive:520136kB present:1018052kB pages_scanned:2078537 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0
Node 0 DMA: 0*4kB 0*8kB 4*16kB 1*32kB 1*64kB 0*128kB 1*256kB 1*512kB 1*1024kB
1*2048kB 0*4096kB = 4000kB
Node 0 DMA32: 1*4kB 1*8kB 1*16kB 4*32kB 1*64kB 1*128kB 0*256kB 1*512kB 1*1024kB
1*2048kB 0*4096kB = 3932kB
Swap cache: add 1448199, delete 1447979, find 67260/121194, race 0+0
Free swap  = 0kB
Total swap = 2031608kB
Free swap:            0kB
262136 pages of RAM
6251 reserved pages
3901 pages shared
220 pages swap cached
Out of memory: kill process 29203 (mem01) score 46330 or a child
Killed process 29203 (mem01)
syslogd invoked oom-killer: gfp_mask=0x200d2, order=0, oomkilladj=0
klogd invoked oom-killer: gfp_mask=0x200d2, order=0, oomkilladj=0

Call Trace:
 [<ffffffff8028f9ee>] printk_ratelimit+0x15/0x17
 [<ffffffff802becf4>] out_of_memory+0x74/0x2c0
 [<ffffffff8020f67a>] __alloc_pages+0x238/0x2c5
 [<ffffffff8024764b>] try_to_wake_up+0x418/0x429
 [<ffffffff802226ef>] alloc_page_vma+0x85/0x8e
 [<ffffffff80233702>] read_swap_cache_async+0x4b/0xde
 [<ffffffff802c3de9>] swapin_readahead+0x61/0xcd
 [<ffffffff80264d8a>] _read_lock_irq+0x9/0xb
 [<ffffffff80208f15>] __handle_mm_fault+0x956/0xd52
 [<ffffffff80267549>] do_page_fault+0x457/0x7fd
 [<ffffffff8026575d>] error_exit+0x0/0x84
 [<ffffffff8028fb92>] do_syslog+0x168/0x3af
 [<ffffffff8028fb92>] do_syslog+0x168/0x3af
 [<ffffffff8029f1b1>] autoremove_wake_function+0x0/0x38
 [<ffffffff80301f69>] kmsg_read+0x3b/0x46
 [<ffffffff8020b39b>] vfs_read+0xcc/0x155
 [<ffffffff80211d58>] sys_read+0x47/0x6f
 [<ffffffff8025e29c>] tracesys+0xdc/0xe1

Mem-info:
Node 0 DMA per-cpu:
CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    1: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    2: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    3: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Node 0 DMA32 per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:  30   Cold: hi:   62, btch:  15 usd:  10
CPU    1: Hot: hi:  186, btch:  31 usd:  41   Cold: hi:   62, btch:  15 usd:  40
CPU    2: Hot: hi:  186, btch:  31 usd: 157   Cold: hi:   62, btch:  15 usd:  57
CPU    3: Hot: hi:  186, btch:  31 usd:  61   Cold: hi:   62, btch:  15 usd:  54
Active:112459 inactive:130496 dirty:0 writeback:0 unstable:0
 free:1983 slab:4276 mapped:644 pagetables:2209 bounce:0
Node 0 DMA free:4000kB min:28kB low:32kB high:40kB active:1976kB inactive:1800kB
present:7360kB pages_scanned:25260 all_unreclaimable? yes
lowmem_reserve[]: 0 994 994
Node 0 DMA32 free:3932kB min:4016kB low:5020kB high:6024kB active:447860kB
inactive:520184kB present:1018052kB pages_scanned:2078777 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0
Node 0 DMA: 0*4kB 0*8kB 4*16kB 1*32kB 1*64kB 0*128kB 1*256kB 1*512kB 1*1024kB
1*2048kB 0*4096kB = 4000kB
Node 0 DMA32: 1*4kB 1*8kB 1*16kB 4*32kB 1*64kB 1*128kB 0*256kB 1*512kB 1*1024kB
1*2048kB 0*4096kB = 3932kB
Swap cache: add 1448199, delete 1447979, find 67260/121195, race 0+0
Free swap  = 0kB
Total swap = 2031608kB
Free swap:            0kB
262136 pages of RAM
6251 reserved pages
3901 pages shared
220 pages swap cached

Comment 14 Jeff Burke 2007-06-17 07:57:41 EDT
Here is another example
http://rhts.lab.boston.redhat.com/cgi-bin/rhts/test_log.cgi?id=180640
Comment 15 Jeff Burke 2007-06-19 13:36:17 EDT
Here is another example
http://rhts.lab.boston.redhat.com/cgi-bin/rhts/test_log.cgi?id=185336
Comment 16 Larry Woodman 2007-06-19 15:03:50 EDT
Jeff, in all of these examples the system has completely exhausted swap space. 
Since the swap cache adds and deletes were pretty much the same, the active and
inactive pages were not reclaimable without more swap space being available. 
Thats why we OOM killed.

Larry
Comment 17 Jeff Burke 2007-06-21 10:18:35 EDT
I386 system
 31871MB HIGHMEM available.
 896MB LOWMEM available.

http://rhts.lab.boston.redhat.com/cgi-bin/rhts/test_log.cgi?id=194401
Comment 18 Larry Woodman 2007-10-08 13:36:26 EDT
Jeff, have you tried the latest RT kernel I built with the changes to
out_of_memory() to postpone OOM kills until the system stays in that state for
several seconds???

Larry
Comment 19 Jeff Burke 2007-10-08 15:50:55 EDT
Larry,
  I did run the kernel through. Here is the job.
http://rhts.lab.boston.redhat.com/cgi-bin/rhts/jobs.cgi?id=8144
Comment 20 Larry Woodman 2007-10-09 10:23:59 EDT
Jeff, I'm still not sure how to interpret these results...

Larry
Comment 21 Jeff Burke 2007-11-08 16:49:24 EST
Larry,
   Are you not seeing the results or the results don't make sense.

Jeff
Comment 22 Jeff Burke 2008-03-19 09:12:30 EDT
This issue is no longer see with the 2.6.24 version of the kernel.

Note You need to log in before you can comment on or make changes to this bug.