Bug 143515

Summary: kernel: Out of Memory: Killed process...lowmem
Product: Red Hat Enterprise Linux 3 Reporter: Joe Thykattil <thykattil>
Component: kernelAssignee: Larry Woodman <lwoodman>
Status: CLOSED WORKSFORME QA Contact: Brian Brock <bbrock>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.0CC: mtonn, petrides, riel, rmuthu, rperkins, tao
Target Milestone: ---   
Target Release: ---   
Hardware: i686   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2005-10-04 01:17:05 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Out of memory error none

Description Joe Thykattil 2004-12-21 20:39:20 UTC
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.5)
Gecko/20041107 Firefox/1.0

Description of problem:
Kernel killing off processes including oracle procs when lowmem is
exhausted:

Sar data indicates that there is still 1.2 gigs of RAM unused and all
of 4 Gigs of swap unused when the event occurs.

System is HP DL 

Dec 12 04:02:05 fadvdb02 kernel: 197896 pages of slabcache
Dec 12 04:02:05 fadvdb02 kernel: 516 pages of kernel stacks
Dec 12 04:02:05 fadvdb02 kernel: 0 lowmem pagetables, 740 highmem
pagetables
Dec 12 04:02:05 fadvdb02 kernel: Free swap:       4092064kB
Dec 12 04:02:05 fadvdb02 kernel: 1310719 pages of RAM
Dec 12 04:02:05 fadvdb02 kernel: 1015801 pages of HIGHMEM
Dec 12 04:02:05 fadvdb02 kernel: 85979 reserved pages
Dec 12 04:02:05 fadvdb02 kernel: 51499 pages shared
Dec 12 04:02:05 fadvdb02 kernel: 0 pages swap cached
Dec 12 04:02:05 fadvdb02 kernel: Out of Memory: Killed process 28221
(oracle).
Dec 12 04:02:11 fadvdb02 kernel: Mem-info:
Dec 12 04:02:11 fadvdb02 kernel: Zone:DMA freepages:  2648 min:     0
low:     0 high:     0
Dec 12 04:02:11 fadvdb02 kernel: Zone:Normal freepages:  1103 min: 
1279 low:  4544 high:  6304
Dec 12 04:02:11 fadvdb02 kernel: Zone:HighMem freepages:874206 min:  
255 low: 16894 high: 25341


Version-Release number of selected component (if applicable):
2.4.21-15.ELsmp

How reproducible:
Didn't try


Additional info:

Event has occured multiples times on this hosts and others.

Comment 1 Rik van Riel 2004-12-21 21:32:56 UTC
A number of fixes related to this went into the RHEL3 U4 kernel, which was
released yesterday.  Could you please confirm whether or not this problem is
still present in kernel-2.4.21-27EL ?  I believe that the latest kernel should
fix your problem.

Comment 2 Ernie Petrides 2004-12-21 22:56:51 UTC
The RHEL3 U4 kernel advisory is RHBA-2004:550.

Comment 4 Larry Woodman 2005-04-04 17:38:51 UTC
Joe, we are still waiting for confirmation that this problem has been fixed
in then latest RHEL3 kernel.  Can you verify this so we can close this bug?

Thaank You, Larry Woodman


Comment 5 Larry Woodman 2005-04-07 14:15:02 UTC
Joe, we are still waiting for your verification that the RHEL3-U4 update fixed
this problem.  Your system exhausted lowmem in the slabcache: 

>>>197896 pages of slabcache

The previous version had a problem reclaiming inodes from the inode cache which
are allocated form the slabcache.  I fixed that problem in U4, can you verify
this so I can close this bug?

Thanks for your help on this Joe, Larry Woodman


Comment 6 Michael Tonn 2005-04-19 18:16:52 UTC
I have a server with 16gig of memeory running Oracle and Informatica.  The 
kernel is 2.4.21-27smp and I am experiencing the same memory issues.  The 
applications appear to be exhausting the lowmem.  I have over 10 gig of 
highmem available.

Comment 7 Larry Woodman 2005-04-19 18:35:57 UTC
Michael, please attach the show_mem() output that is written to
/var/log/messages when the OOM kill occurs so I can see where the lowmem on your
system is going.  While you are at it please get a "/proc/slabinfo" output just
incase your slabcache is consuming all of lowmem as well.

Thanks, Larry Woodman


Comment 8 Raja 2005-05-24 15:23:22 UTC
Created attachment 114775 [details]
Out of memory error

Comment 10 Larry Woodman 2005-05-25 17:05:55 UTC
The attched showmem is from 16GB system running the SMP kernel.  Lowmem is
totally exhausted with the mem_map and slabcache!!!  Please use the Hugemem
kernel and this problem will go away.

Larry Woodman




Comment 12 Ernie Petrides 2005-10-04 01:17:05 UTC
Closing due to lack of reporter's response to comment #1.