Bug 183366

Summary:	WS, Dell 380N, VMware, out of memory, read only filesystem
Product:	Red Hat Enterprise Linux 4	Reporter:	Ahnjoan Amous <ahnjoan>
Component:	kernel	Assignee:	Larry Woodman <lwoodman>
Status:	CLOSED WONTFIX	QA Contact:	Brian Brock <bbrock>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	4.0	CC:	jbaron, rwheeler
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2012-06-20 13:17:35 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Ahnjoan Amous 2006-02-28 17:14:36 UTC

Description of problem:

Often, while working in VMware, I am presented with an error message stating
that my file system is read-only.  I am no longer to manipulate the VM, nor am I
able to recover on the host side.  I end up having to power off the machine and
fsck in single user mode to have a functioning host again.

Because the file system goes read only, the following error messages are
transported via syslog to another host.  I hope they will be enough to go on.

I have seen this page, and followed the instructions thinking the cause was
addressed there.  I have had two of the same crashes since following these
instructions. 
(http://www.vmware.com/community/thread.jspa?threadID=20690&messageID=248349#248349)
(I added vm.min_free_kbytes=5120 to the bottom of /etc/sysctl.conf)

Also, I have run the Dell diagnostics disk to verify that hardware is not a
problem.  I know Dell's tool is not the final word on hardware issues but its
all I have right now.

Here are the error messages when the problem happens.
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: X: page allocation failure.
order:0, mode:0x50
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: 
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Call
Trace:<ffffffff8015c842>{__alloc_pages+846}
<ffffffff80158c21>{find_or_create_page+53} 
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel:       
<ffffffff8017b3e2>{__getblk_slow+237} <ffffffff8017b539>{__getblk+60} 
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel:       
<ffffffff8017b54d>{__bread+6} <ffffffffa00580b8>{:ext3:read_block_bitmap+50} 
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel:       
<ffffffffa00591ac>{:ext3:ext3_new_block+629}
<ffffffffa005b3b6>{:ext3:ext3_alloc_block+7} 
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel:       
<ffffffffa005cf9b>{:ext3:ext3_get_block_handle+881} 
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel:       
<ffffffffa00463c4>{:jbd:start_this_handle+964}
<ffffffff8017a58f>{__block_write_full_page+198} 
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel:       
<ffffffffa005d40c>{:ext3:ext3_get_block+0}
<ffffffffa005bb46>{:ext3:ext3_ordered_writepage+245} 
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel:       
<ffffffff801638d7>{shrink_zone+3095} <ffffffff801314ab>{activate_task+124} 
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel:       
<ffffffff80163ec1>{try_to_free_pages+303} <ffffffff8015c748>{__alloc_pages+596} 
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel:       
<ffffffff801677b6>{do_no_page+620} <ffffffff80167cb3>{handle_mm_fault+343} 
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel:       
<ffffffff80123326>{do_page_fault+518} <ffffffff8010fee1>{sys_rt_sigreturn+532} 
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel:       
<ffffffff8010ff55>{sys_rt_sigreturn+648} <ffffffff80110b2d>{error_exit+0} 
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel:        
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Mem-info:
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 DMA per-cpu:
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 0 hot: low 2, high 6, batch 1
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 0 cold: low 0, high 2, batch 1
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 1 hot: low 2, high 6, batch 1
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 1 cold: low 0, high 2, batch 1
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 Normal per-cpu:
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 0 hot: low 32, high 96, batch 16
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 0 cold: low 0, high 32, batch 16
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 1 hot: low 32, high 96, batch 16
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 1 cold: low 0, high 32, batch 16
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 HighMem per-cpu: empty
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: 
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Free pages:       11928kB (0kB
HighMem)
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Active:183217 inactive:43432
dirty:610 writeback:0 unstable:0 free:2982 slab:11208 mapped:212017 pagetables:3444
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 DMA free:11928kB min:80kB
low:160kB high:240kB active:0kB inactive:0kB present:16384kB pages_scanned:5571
all_unrecl
aimable? yes
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: protections[]: 0 0 0
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 Normal free:0kB
min:5036kB low:10072kB high:15108kB active:732868kB inactive:173728kB
present:1030696kB pages_scanne
d:363 all_unreclaimable? no
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: protections[]: 0 0 0
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 HighMem free:0kB
min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB
pages_scanned:0 all_unreclaimabl
e? no
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: protections[]: 0 0 0
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 DMA: 2*4kB 6*8kB 2*16kB
2*32kB 2*64kB 3*128kB 2*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11928kB
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 Normal: 0*4kB 0*8kB
0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 HighMem: empty
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Swap cache: add 41004, delete
31945, find 6682/8144, race 0+0
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Free swap:       4075112kB
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: 261770 pages of RAM
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: 6670 reserved pages
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: 279029 pages shared
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: 9059 pages swap cached
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: X: page allocation failure.
order:0, mode:0x50
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: 
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Call
Trace:<ffffffff8015c842>{__alloc_pages+846}
<ffffffff80158c21>{find_or_create_page+53} 
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel:       
<ffffffff8017b3e2>{__getblk_slow+237} <ffffffff8017b539>{__getblk+60} 
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel:       
<ffffffff8017b54d>{__bread+6} <ffffffffa00580b8>{:ext3:read_block_bitmap+50} 
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel:       
<ffffffffa00591ac>{:ext3:ext3_new_block+629}
<ffffffffa005b3b6>{:ext3:ext3_alloc_block+7} 
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel:       
<ffffffffa005cf9b>{:ext3:ext3_get_block_handle+881} 
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel:       
<ffffffffa00463c4>{:jbd:start_this_handle+964}
<ffffffff8017a58f>{__block_write_full_page+198} 
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel:       
<ffffffffa005d40c>{:ext3:ext3_get_block+0}
<ffffffffa005bb46>{:ext3:ext3_ordered_writepage+245} 
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel:       
<ffffffff801638d7>{shrink_zone+3095} <ffffffff801314ab>{activate_task+124} 
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel:       
<ffffffff80163ec1>{try_to_free_pages+303} <ffffffff8015c748>{__alloc_pages+596} 
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel:       
<ffffffff801677b6>{do_no_page+620} <ffffffff80167cb3>{handle_mm_fault+343} 
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel:       
<ffffffff80123326>{do_page_fault+518} <ffffffff8010fee1>{sys_rt_sigreturn+532} 
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel:       
<ffffffff8010ff55>{sys_rt_sigreturn+648} <ffffffff80110b2d>{error_exit+0} 
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel:        
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Mem-info:
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 DMA per-cpu:
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 0 hot: low 2, high 6, batch 1
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 0 cold: low 0, high 2, batch 1
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 1 hot: low 2, high 6, batch 1
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 1 cold: low 0, high 2, batch 1
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 Normal per-cpu:
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 0 hot: low 32, high 96, batch 16
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 0 cold: low 0, high 32, batch 16
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 1 hot: low 32, high 96, batch 16
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 1 cold: low 0, high 32, batch 16
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 HighMem per-cpu: empty
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: 
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Free pages:       11928kB (0kB
HighMem)
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Active:181581 inactive:45041
dirty:1555 writeback:0 unstable:0 free:2982 slab:11243 mapped:210924 pagetables:3444
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 DMA free:11928kB min:80kB
low:160kB high:240kB active:0kB inactive:0kB present:16384kB pages_scanned:5571
all_unrecl
aimable? yes
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: protections[]: 0 0 0
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 Normal free:0kB
min:5036kB low:10072kB high:15108kB active:726324kB inactive:180164kB
present:1030696kB pages_scanne
d:0 all_unreclaimable? no
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: protections[]: 0 0 0
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 HighMem free:0kB
min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB
pages_scanned:0 all_unreclaimabl
e? no
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: protections[]: 0 0 0
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 DMA: 2*4kB 6*8kB 2*16kB
2*32kB 2*64kB 3*128kB 2*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11928kB
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 Normal: 0*4kB 0*8kB
0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 HighMem: empty
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Swap cache: add 41141, delete
31996, find 6682/8144, race 0+0
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Free swap:       4074564kB
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: 261770 pages of RAM
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: 6670 reserved pages
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: 278079 pages shared
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: 9145 pages swap cached
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: X: page allocation failure.
order:0, mode:0x850
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: 
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Call
Trace:<ffffffff8015c842>{__alloc_pages+846} <ffffffff8015c8d9>{__get_free_pages+11} 
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel:       
<ffffffff8015f850>{kmem_getpages+36} <ffffffff8015ffe5>{cache_alloc_refill+609} 
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel:       
<ffffffff8015fcb3>{__kmalloc+123} <ffffffffa004ba8c>{:jbd:__jbd_kmalloc+21} 
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel:       
<ffffffffa00477b0>{:jbd:journal_get_undo_access+96} 
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel:       
<ffffffffa0058aa9>{:ext3:ext3_try_to_allocate_with_rsv+84} 
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel:       
<ffffffffa00591df>{:ext3:ext3_new_block+680}
<ffffffffa005b3b6>{:ext3:ext3_alloc_block+7} 
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel:       
<ffffffffa005cf9b>{:ext3:ext3_get_block_handle+881} 
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel:       
<ffffffffa00463c4>{:jbd:start_this_handle+964}
<ffffffff8017a58f>{__block_write_full_page+198} 
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel:       
<ffffffffa005d40c>{:ext3:ext3_get_block+0}
<ffffffffa005bb46>{:ext3:ext3_ordered_writepage+245} 
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel:       
<ffffffff801638d7>{shrink_zone+3095} <ffffffff801314ab>{activate_task+124} 
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel:       
<ffffffff80163ec1>{try_to_free_pages+303} <ffffffff8015c748>{__alloc_pages+596} 
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel:       
<ffffffff801677b6>{do_no_page+620} <ffffffff80167cb3>{handle_mm_fault+343} 
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel:       
<ffffffff80123326>{do_page_fault+518} <ffffffff8010fee1>{sys_rt_sigreturn+532} 
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel:       
<ffffffff8010ff55>{sys_rt_sigreturn+648} <ffffffff80110b2d>{error_exit+0} 
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel:        
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Mem-info:
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 DMA per-cpu:
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 0 hot: low 2, high 6, batch 1
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 0 cold: low 0, high 2, batch 1
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 1 hot: low 2, high 6, batch 1
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 1 cold: low 0, high 2, batch 1
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 Normal per-cpu:
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 0 hot: low 32, high 96, batch 16
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 0 cold: low 0, high 32, batch 16
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 1 hot: low 32, high 96, batch 16
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 1 cold: low 0, high 32, batch 16
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 HighMem per-cpu: empty
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: 
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Free pages:       11928kB (0kB
HighMem)
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Active:181546 inactive:45058
dirty:1555 writeback:0 unstable:0 free:2982 slab:11259 mapped:210924 pagetables:3444
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 DMA free:11928kB min:80kB
low:160kB high:240kB active:0kB inactive:0kB present:16384kB pages_scanned:5571
all_unrecl
aimable? yes
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: protections[]: 0 0 0
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 Normal free:0kB
min:5036kB low:10072kB high:15108kB active:726184kB inactive:180232kB
present:1030696kB pages_scanne
d:0 all_unreclaimable? no
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: protections[]: 0 0 0
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 HighMem free:0kB
min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB
pages_scanned:0 all_unreclaimabl
e? no
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: protections[]: 0 0 0
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 DMA: 2*4kB 6*8kB 2*16kB
2*32kB 2*64kB 3*128kB 2*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11928kB
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 Normal: 0*4kB 0*8kB
0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 HighMem: empty
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Swap cache: add 41141, delete
32028, find 6682/8144, race 0+0
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Free swap:       4074564kB
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: 261770 pages of RAM
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: 6670 reserved pages
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: 278097 pages shared
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: 9113 pages swap cached
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: ext3_try_to_allocate_with_rsv:
aborting transaction: Out of memory in __ext3_journal_get_undo_access
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: EXT3-fs error (device md2) in
ext3_new_block: Out of memory
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Aborting journal on device md2.
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: ext3_abort called.
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: EXT3-fs error (device md2):
ext3_journal_start_sb: Detected aborted journal
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Remounting filesystem read-only
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: EXT3-fs error (device md2) in
ext3_ordered_writepage: Out of memory
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: EXT3-fs error (device md2) in
ext3_new_block: Journal has aborted
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: EXT3-fs error (device md2) in
ext3_ordered_writepage: Journal has aborted
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: EXT3-fs error (device md2) in
ext3_new_block: Journal has aborted
Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: EXT3-fs error (device md2) in
ext3_ordered_writepage: Journal has aborted


Version-Release number of selected component (if applicable):


How reproducible:
I'm sure it will happen again, just couldn't tell you how to make it happen.

Steps to Reproduce:
1. Wait
2.
3.
  
Actual results:


Expected results:


Additional info:

Comment 1 Ahnjoan Amous 2006-05-05 16:41:53 UTC

Here is another crash on the same machine.  Not sure if this information will
help, but I hope it will.

Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: kswapd0: page allocation
failure. order:0, mode:0x850
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: 
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: Call
Trace:<ffffffff8015c842>{__alloc_pages+846}
<ffffffff80171b47>{alloc_page_interleave+61} 
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel:       
<ffffffff8015c8d9>{__get_free_pages+11} <ffffffff8015f850>{kmem_getpages+36} 
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel:       
<ffffffff8015ffe5>{cache_alloc_refill+609} <ffffffff8015fcb3>{__kmalloc+123} 
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel:       
<ffffffffa004ba8c>{:jbd:__jbd_kmalloc+21}
<ffffffffa00477b0>{:jbd:journal_get_undo_access+96} 
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel:       
<ffffffffa0058aa9>{:ext3:ext3_try_to_allocate_with_rsv+84} 
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel:       
<ffffffffa00591df>{:ext3:ext3_new_block+680}
<ffffffffa005b3b6>{:ext3:ext3_alloc_block+7} 
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel:       
<ffffffffa005cf9b>{:ext3:ext3_get_block_handle+881} 
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel:       
<ffffffffa00463c4>{:jbd:start_this_handle+964}
<ffffffff8017a58f>{__block_write_full_page+198} 
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel:       
<ffffffffa005d40c>{:ext3:ext3_get_block+0}
<ffffffffa005bb46>{:ext3:ext3_ordered_writepage+245} 
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel:       
<ffffffff801638d7>{shrink_zone+3095} <ffffffff803037b4>{thread_return+42} 
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel:       
<ffffffff8013474a>{autoremove_wake_function+0}
<ffffffff801641ef>{balance_pgdat+506} 
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel:       
<ffffffff80164439>{kswapd+252} <ffffffff8013474a>{autoremove_wake_function+0} 
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel:       
<ffffffff80131c95>{finish_task_switch+55}
<ffffffff8013474a>{autoremove_wake_function+0} 
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel:       
<ffffffff80131ce4>{schedule_tail+11} <ffffffff80110ce3>{child_rip+8} 
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel:       
<ffffffff8016433d>{kswapd+0} <ffffffff80110cdb>{child_rip+0} 
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel:        
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: Mem-info:
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: Node 0 DMA per-cpu:
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: cpu 0 hot: low 2, high 6, batch 1
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: cpu 0 cold: low 0, high 2, batch 1
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: cpu 1 hot: low 2, high 6, batch 1
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: cpu 1 cold: low 0, high 2, batch 1
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: Node 0 Normal per-cpu:
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: cpu 0 hot: low 32, high 96, batch 16
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: cpu 0 cold: low 0, high 32, batch 16
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: cpu 1 hot: low 32, high 96, batch 16
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: cpu 1 cold: low 0, high 32, batch 16
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: Node 0 HighMem per-cpu: empty
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: 
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: Free pages:       11920kB (0kB
HighMem)
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: Active:203733 inactive:21400
dirty:0 writeback:0 unstable:0 free:2980 slab:13408 mapped:198999 pagetables
:3049
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: Node 0 DMA free:11920kB min:80kB
low:160kB high:240kB active:0kB inactive:0kB present:16384kB pages_scann
ed:1186 all_unreclaimable? yes
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: protections[]: 0 0 0
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: Node 0 Normal free:0kB
min:5036kB low:10072kB high:15108kB active:814932kB inactive:85600kB present:10306
96kB pages_scanned:99 all_unreclaimable? no
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: protections[]: 0 0 0
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: Node 0 HighMem free:0kB
min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:
0 all_unreclaimable? no
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: protections[]: 0 0 0
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: Node 0 DMA: 0*4kB 6*8kB 2*16kB
2*32kB 2*64kB 3*128kB 2*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11920kB
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: Node 0 Normal: 0*4kB 0*8kB
0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: Node 0 HighMem: empty
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: Swap cache: add 163758, delete
137114, find 105445/114390, race 0+0
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: Free swap:       3854764kB
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: 261770 pages of RAM
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: 6670 reserved pages
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: 215081 pages shared
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: 26644 pages swap cached
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: ext3_try_to_allocate_with_rsv:
aborting transaction: Out of memory in __ext3_journal_get_undo_access
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: EXT3-fs error (device md2) in
ext3_new_block: Out of memory
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: Aborting journal on device md2.
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: ext3_abort called.
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: EXT3-fs error (device md2):
ext3_journal_start_sb: Detected aborted journal
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: Remounting filesystem read-only
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: EXT3-fs error (device md2) in
ext3_ordered_writepage: Out of memory
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: EXT3-fs error (device md2) in
ext3_new_block: Journal has aborted
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: EXT3-fs error (device md2) in
ext3_ordered_writepage: Journal has aborted
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: EXT3-fs error (device md2) in
ext3_new_block: Journal has aborted
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: EXT3-fs error (device md2) in
ext3_ordered_writepage: Journal has aborted
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: EXT3-fs error (device md2) in
ext3_new_block: Journal has aborted
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: EXT3-fs error (device md2) in
ext3_ordered_writepage: Journal has aborted
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: EXT3-fs error (device md2) in
ext3_new_block: Journal has aborted
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: EXT3-fs error (device md2) in
ext3_ordered_writepage: Journal has aborted
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: EXT3-fs error (device md2) in
ext3_new_block: Journal has aborted
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: EXT3-fs error (device md2) in
ext3_ordered_writepage: Journal has aborted
Apr  4 16:28:01 sbusocwks01.esoc3.local kernel: EXT3-fs error (device md2) in
start_transaction: Journal has aborted

Comment 2 Larry Woodman 2006-11-27 18:43:13 UTC

Please try increasing /proc/sys/vm/min_free_kbytes to 8192.  This will prevent
the system from allowing memory from getting so exhausted before starting page
reclamation.  This will prevent the system form getting into this state in the
first place.  Let me know the results once you try this.

Thanks, Larry Woodman

Comment 3 Jiri Pallich 2012-06-20 13:17:35 UTC

Thank you for submitting this issue for consideration in Red Hat Enterprise Linux. The release for which you requested us to review is now End of Life. 
Please See https://access.redhat.com/support/policy/updates/errata/

If you would like Red Hat to re-consider your feature request for an active release, please re-open the request via appropriate support channels and provide additional supporting details about the importance of this issue.