Description of problem: Often, while working in VMware, I am presented with an error message stating that my file system is read-only. I am no longer to manipulate the VM, nor am I able to recover on the host side. I end up having to power off the machine and fsck in single user mode to have a functioning host again. Because the file system goes read only, the following error messages are transported via syslog to another host. I hope they will be enough to go on. I have seen this page, and followed the instructions thinking the cause was addressed there. I have had two of the same crashes since following these instructions. (http://www.vmware.com/community/thread.jspa?threadID=20690&messageID=248349#248349) (I added vm.min_free_kbytes=5120 to the bottom of /etc/sysctl.conf) Also, I have run the Dell diagnostics disk to verify that hardware is not a problem. I know Dell's tool is not the final word on hardware issues but its all I have right now. Here are the error messages when the problem happens. Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: X: page allocation failure. order:0, mode:0x50 Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Call Trace:<ffffffff8015c842>{__alloc_pages+846} <ffffffff80158c21>{find_or_create_page+53} Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: <ffffffff8017b3e2>{__getblk_slow+237} <ffffffff8017b539>{__getblk+60} Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: <ffffffff8017b54d>{__bread+6} <ffffffffa00580b8>{:ext3:read_block_bitmap+50} Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: <ffffffffa00591ac>{:ext3:ext3_new_block+629} <ffffffffa005b3b6>{:ext3:ext3_alloc_block+7} Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: <ffffffffa005cf9b>{:ext3:ext3_get_block_handle+881} Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: <ffffffffa00463c4>{:jbd:start_this_handle+964} <ffffffff8017a58f>{__block_write_full_page+198} Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: <ffffffffa005d40c>{:ext3:ext3_get_block+0} <ffffffffa005bb46>{:ext3:ext3_ordered_writepage+245} Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: <ffffffff801638d7>{shrink_zone+3095} <ffffffff801314ab>{activate_task+124} Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: <ffffffff80163ec1>{try_to_free_pages+303} <ffffffff8015c748>{__alloc_pages+596} Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: <ffffffff801677b6>{do_no_page+620} <ffffffff80167cb3>{handle_mm_fault+343} Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: <ffffffff80123326>{do_page_fault+518} <ffffffff8010fee1>{sys_rt_sigreturn+532} Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: <ffffffff8010ff55>{sys_rt_sigreturn+648} <ffffffff80110b2d>{error_exit+0} Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Mem-info: Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 DMA per-cpu: Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 0 hot: low 2, high 6, batch 1 Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 0 cold: low 0, high 2, batch 1 Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 1 hot: low 2, high 6, batch 1 Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 1 cold: low 0, high 2, batch 1 Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 Normal per-cpu: Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 0 hot: low 32, high 96, batch 16 Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 0 cold: low 0, high 32, batch 16 Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 1 hot: low 32, high 96, batch 16 Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 1 cold: low 0, high 32, batch 16 Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 HighMem per-cpu: empty Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Free pages: 11928kB (0kB HighMem) Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Active:183217 inactive:43432 dirty:610 writeback:0 unstable:0 free:2982 slab:11208 mapped:212017 pagetables:3444 Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 DMA free:11928kB min:80kB low:160kB high:240kB active:0kB inactive:0kB present:16384kB pages_scanned:5571 all_unrecl aimable? yes Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: protections[]: 0 0 0 Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 Normal free:0kB min:5036kB low:10072kB high:15108kB active:732868kB inactive:173728kB present:1030696kB pages_scanne d:363 all_unreclaimable? no Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: protections[]: 0 0 0 Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimabl e? no Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: protections[]: 0 0 0 Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 DMA: 2*4kB 6*8kB 2*16kB 2*32kB 2*64kB 3*128kB 2*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11928kB Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 HighMem: empty Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Swap cache: add 41004, delete 31945, find 6682/8144, race 0+0 Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Free swap: 4075112kB Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: 261770 pages of RAM Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: 6670 reserved pages Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: 279029 pages shared Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: 9059 pages swap cached Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: X: page allocation failure. order:0, mode:0x50 Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Call Trace:<ffffffff8015c842>{__alloc_pages+846} <ffffffff80158c21>{find_or_create_page+53} Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: <ffffffff8017b3e2>{__getblk_slow+237} <ffffffff8017b539>{__getblk+60} Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: <ffffffff8017b54d>{__bread+6} <ffffffffa00580b8>{:ext3:read_block_bitmap+50} Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: <ffffffffa00591ac>{:ext3:ext3_new_block+629} <ffffffffa005b3b6>{:ext3:ext3_alloc_block+7} Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: <ffffffffa005cf9b>{:ext3:ext3_get_block_handle+881} Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: <ffffffffa00463c4>{:jbd:start_this_handle+964} <ffffffff8017a58f>{__block_write_full_page+198} Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: <ffffffffa005d40c>{:ext3:ext3_get_block+0} <ffffffffa005bb46>{:ext3:ext3_ordered_writepage+245} Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: <ffffffff801638d7>{shrink_zone+3095} <ffffffff801314ab>{activate_task+124} Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: <ffffffff80163ec1>{try_to_free_pages+303} <ffffffff8015c748>{__alloc_pages+596} Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: <ffffffff801677b6>{do_no_page+620} <ffffffff80167cb3>{handle_mm_fault+343} Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: <ffffffff80123326>{do_page_fault+518} <ffffffff8010fee1>{sys_rt_sigreturn+532} Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: <ffffffff8010ff55>{sys_rt_sigreturn+648} <ffffffff80110b2d>{error_exit+0} Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Mem-info: Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 DMA per-cpu: Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 0 hot: low 2, high 6, batch 1 Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 0 cold: low 0, high 2, batch 1 Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 1 hot: low 2, high 6, batch 1 Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 1 cold: low 0, high 2, batch 1 Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 Normal per-cpu: Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 0 hot: low 32, high 96, batch 16 Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 0 cold: low 0, high 32, batch 16 Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 1 hot: low 32, high 96, batch 16 Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 1 cold: low 0, high 32, batch 16 Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 HighMem per-cpu: empty Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Free pages: 11928kB (0kB HighMem) Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Active:181581 inactive:45041 dirty:1555 writeback:0 unstable:0 free:2982 slab:11243 mapped:210924 pagetables:3444 Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 DMA free:11928kB min:80kB low:160kB high:240kB active:0kB inactive:0kB present:16384kB pages_scanned:5571 all_unrecl aimable? yes Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: protections[]: 0 0 0 Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 Normal free:0kB min:5036kB low:10072kB high:15108kB active:726324kB inactive:180164kB present:1030696kB pages_scanne d:0 all_unreclaimable? no Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: protections[]: 0 0 0 Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimabl e? no Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: protections[]: 0 0 0 Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 DMA: 2*4kB 6*8kB 2*16kB 2*32kB 2*64kB 3*128kB 2*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11928kB Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 HighMem: empty Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Swap cache: add 41141, delete 31996, find 6682/8144, race 0+0 Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Free swap: 4074564kB Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: 261770 pages of RAM Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: 6670 reserved pages Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: 278079 pages shared Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: 9145 pages swap cached Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: X: page allocation failure. order:0, mode:0x850 Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Call Trace:<ffffffff8015c842>{__alloc_pages+846} <ffffffff8015c8d9>{__get_free_pages+11} Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: <ffffffff8015f850>{kmem_getpages+36} <ffffffff8015ffe5>{cache_alloc_refill+609} Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: <ffffffff8015fcb3>{__kmalloc+123} <ffffffffa004ba8c>{:jbd:__jbd_kmalloc+21} Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: <ffffffffa00477b0>{:jbd:journal_get_undo_access+96} Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: <ffffffffa0058aa9>{:ext3:ext3_try_to_allocate_with_rsv+84} Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: <ffffffffa00591df>{:ext3:ext3_new_block+680} <ffffffffa005b3b6>{:ext3:ext3_alloc_block+7} Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: <ffffffffa005cf9b>{:ext3:ext3_get_block_handle+881} Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: <ffffffffa00463c4>{:jbd:start_this_handle+964} <ffffffff8017a58f>{__block_write_full_page+198} Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: <ffffffffa005d40c>{:ext3:ext3_get_block+0} <ffffffffa005bb46>{:ext3:ext3_ordered_writepage+245} Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: <ffffffff801638d7>{shrink_zone+3095} <ffffffff801314ab>{activate_task+124} Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: <ffffffff80163ec1>{try_to_free_pages+303} <ffffffff8015c748>{__alloc_pages+596} Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: <ffffffff801677b6>{do_no_page+620} <ffffffff80167cb3>{handle_mm_fault+343} Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: <ffffffff80123326>{do_page_fault+518} <ffffffff8010fee1>{sys_rt_sigreturn+532} Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: <ffffffff8010ff55>{sys_rt_sigreturn+648} <ffffffff80110b2d>{error_exit+0} Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Mem-info: Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 DMA per-cpu: Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 0 hot: low 2, high 6, batch 1 Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 0 cold: low 0, high 2, batch 1 Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 1 hot: low 2, high 6, batch 1 Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 1 cold: low 0, high 2, batch 1 Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 Normal per-cpu: Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 0 hot: low 32, high 96, batch 16 Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 0 cold: low 0, high 32, batch 16 Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 1 hot: low 32, high 96, batch 16 Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: cpu 1 cold: low 0, high 32, batch 16 Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 HighMem per-cpu: empty Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Free pages: 11928kB (0kB HighMem) Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Active:181546 inactive:45058 dirty:1555 writeback:0 unstable:0 free:2982 slab:11259 mapped:210924 pagetables:3444 Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 DMA free:11928kB min:80kB low:160kB high:240kB active:0kB inactive:0kB present:16384kB pages_scanned:5571 all_unrecl aimable? yes Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: protections[]: 0 0 0 Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 Normal free:0kB min:5036kB low:10072kB high:15108kB active:726184kB inactive:180232kB present:1030696kB pages_scanne d:0 all_unreclaimable? no Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: protections[]: 0 0 0 Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimabl e? no Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: protections[]: 0 0 0 Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 DMA: 2*4kB 6*8kB 2*16kB 2*32kB 2*64kB 3*128kB 2*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11928kB Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Node 0 HighMem: empty Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Swap cache: add 41141, delete 32028, find 6682/8144, race 0+0 Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Free swap: 4074564kB Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: 261770 pages of RAM Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: 6670 reserved pages Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: 278097 pages shared Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: 9113 pages swap cached Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: ext3_try_to_allocate_with_rsv: aborting transaction: Out of memory in __ext3_journal_get_undo_access Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: EXT3-fs error (device md2) in ext3_new_block: Out of memory Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Aborting journal on device md2. Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: ext3_abort called. Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: EXT3-fs error (device md2): ext3_journal_start_sb: Detected aborted journal Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: Remounting filesystem read-only Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: EXT3-fs error (device md2) in ext3_ordered_writepage: Out of memory Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: EXT3-fs error (device md2) in ext3_new_block: Journal has aborted Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: EXT3-fs error (device md2) in ext3_ordered_writepage: Journal has aborted Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: EXT3-fs error (device md2) in ext3_new_block: Journal has aborted Feb 27 22:56:53 sbusocwks01.esoc3.local kernel: EXT3-fs error (device md2) in ext3_ordered_writepage: Journal has aborted Version-Release number of selected component (if applicable): How reproducible: I'm sure it will happen again, just couldn't tell you how to make it happen. Steps to Reproduce: 1. Wait 2. 3. Actual results: Expected results: Additional info:
Here is another crash on the same machine. Not sure if this information will help, but I hope it will. Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: kswapd0: page allocation failure. order:0, mode:0x850 Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: Call Trace:<ffffffff8015c842>{__alloc_pages+846} <ffffffff80171b47>{alloc_page_interleave+61} Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: <ffffffff8015c8d9>{__get_free_pages+11} <ffffffff8015f850>{kmem_getpages+36} Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: <ffffffff8015ffe5>{cache_alloc_refill+609} <ffffffff8015fcb3>{__kmalloc+123} Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: <ffffffffa004ba8c>{:jbd:__jbd_kmalloc+21} <ffffffffa00477b0>{:jbd:journal_get_undo_access+96} Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: <ffffffffa0058aa9>{:ext3:ext3_try_to_allocate_with_rsv+84} Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: <ffffffffa00591df>{:ext3:ext3_new_block+680} <ffffffffa005b3b6>{:ext3:ext3_alloc_block+7} Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: <ffffffffa005cf9b>{:ext3:ext3_get_block_handle+881} Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: <ffffffffa00463c4>{:jbd:start_this_handle+964} <ffffffff8017a58f>{__block_write_full_page+198} Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: <ffffffffa005d40c>{:ext3:ext3_get_block+0} <ffffffffa005bb46>{:ext3:ext3_ordered_writepage+245} Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: <ffffffff801638d7>{shrink_zone+3095} <ffffffff803037b4>{thread_return+42} Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: <ffffffff8013474a>{autoremove_wake_function+0} <ffffffff801641ef>{balance_pgdat+506} Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: <ffffffff80164439>{kswapd+252} <ffffffff8013474a>{autoremove_wake_function+0} Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: <ffffffff80131c95>{finish_task_switch+55} <ffffffff8013474a>{autoremove_wake_function+0} Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: <ffffffff80131ce4>{schedule_tail+11} <ffffffff80110ce3>{child_rip+8} Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: <ffffffff8016433d>{kswapd+0} <ffffffff80110cdb>{child_rip+0} Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: Mem-info: Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: Node 0 DMA per-cpu: Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: cpu 0 hot: low 2, high 6, batch 1 Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: cpu 0 cold: low 0, high 2, batch 1 Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: cpu 1 hot: low 2, high 6, batch 1 Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: cpu 1 cold: low 0, high 2, batch 1 Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: Node 0 Normal per-cpu: Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: cpu 0 hot: low 32, high 96, batch 16 Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: cpu 0 cold: low 0, high 32, batch 16 Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: cpu 1 hot: low 32, high 96, batch 16 Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: cpu 1 cold: low 0, high 32, batch 16 Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: Node 0 HighMem per-cpu: empty Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: Free pages: 11920kB (0kB HighMem) Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: Active:203733 inactive:21400 dirty:0 writeback:0 unstable:0 free:2980 slab:13408 mapped:198999 pagetables :3049 Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: Node 0 DMA free:11920kB min:80kB low:160kB high:240kB active:0kB inactive:0kB present:16384kB pages_scann ed:1186 all_unreclaimable? yes Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: protections[]: 0 0 0 Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: Node 0 Normal free:0kB min:5036kB low:10072kB high:15108kB active:814932kB inactive:85600kB present:10306 96kB pages_scanned:99 all_unreclaimable? no Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: protections[]: 0 0 0 Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: Node 0 HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned: 0 all_unreclaimable? no Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: protections[]: 0 0 0 Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: Node 0 DMA: 0*4kB 6*8kB 2*16kB 2*32kB 2*64kB 3*128kB 2*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11920kB Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: Node 0 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: Node 0 HighMem: empty Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: Swap cache: add 163758, delete 137114, find 105445/114390, race 0+0 Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: Free swap: 3854764kB Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: 261770 pages of RAM Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: 6670 reserved pages Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: 215081 pages shared Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: 26644 pages swap cached Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: ext3_try_to_allocate_with_rsv: aborting transaction: Out of memory in __ext3_journal_get_undo_access Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: EXT3-fs error (device md2) in ext3_new_block: Out of memory Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: Aborting journal on device md2. Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: ext3_abort called. Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: EXT3-fs error (device md2): ext3_journal_start_sb: Detected aborted journal Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: Remounting filesystem read-only Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: EXT3-fs error (device md2) in ext3_ordered_writepage: Out of memory Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: EXT3-fs error (device md2) in ext3_new_block: Journal has aborted Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: EXT3-fs error (device md2) in ext3_ordered_writepage: Journal has aborted Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: EXT3-fs error (device md2) in ext3_new_block: Journal has aborted Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: EXT3-fs error (device md2) in ext3_ordered_writepage: Journal has aborted Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: EXT3-fs error (device md2) in ext3_new_block: Journal has aborted Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: EXT3-fs error (device md2) in ext3_ordered_writepage: Journal has aborted Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: EXT3-fs error (device md2) in ext3_new_block: Journal has aborted Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: EXT3-fs error (device md2) in ext3_ordered_writepage: Journal has aborted Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: EXT3-fs error (device md2) in ext3_new_block: Journal has aborted Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: EXT3-fs error (device md2) in ext3_ordered_writepage: Journal has aborted Apr 4 16:28:01 sbusocwks01.esoc3.local kernel: EXT3-fs error (device md2) in start_transaction: Journal has aborted
Please try increasing /proc/sys/vm/min_free_kbytes to 8192. This will prevent the system from allowing memory from getting so exhausted before starting page reclamation. This will prevent the system form getting into this state in the first place. Let me know the results once you try this. Thanks, Larry Woodman
Thank you for submitting this issue for consideration in Red Hat Enterprise Linux. The release for which you requested us to review is now End of Life. Please See https://access.redhat.com/support/policy/updates/errata/ If you would like Red Hat to re-consider your feature request for an active release, please re-open the request via appropriate support channels and provide additional supporting details about the importance of this issue.