Bug 149762

Summary: Large portion of RAM taken - kernel starts killing processes
Product: [Fedora] Fedora Reporter: Stefan Neufeind <redhat>
Component: kernelAssignee: Dave Jones <davej>
Status: CLOSED NEXTRELEASE QA Contact:
Severity: high Docs Contact:
Priority: medium    
Version: 2CC: pfrields, wtogami
Target Milestone: ---   
Target Release: ---   
Hardware: i686   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2005-04-16 04:19:58 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Stefan Neufeind 2005-02-25 23:36:50 UTC
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.5)
Gecko/20041107 Firefox/1.0

Description of problem:
After upgrading from kernel-2.6.10-1.11_FC2smp to 1.14 we experienced
that under heavy load the system became unstable, because of too many
processes stacking up. In peak times this happened quite frequently,
resulting in urgent needs to e.g. shutdown the database to save the
machine from crashing.
The machine is running 2 processors in hyperthreading mode (meaning 4
cpus for smp). It's running both database and, for historical reasons,
NFS-server. Of 2GB RAM there was a bit less than 1GB taken for MySQL,
which was planned and worked fairly well before. But even after
shutting down MySQL, there was still 1GB of RAM taken for _something_.
We didn't find out where it went, actually. During rescue-operations
to the database even sshd-processes got killed and in
/var/log/messages kernel said it was "out of memory".
Since running out of memory the system did not have enough resources
for caching and the like, so it became quite slow and more processes
stacked up - boosting the system from 300/400 to about 900 processes
within minutes sometimes. Sometimes the system was able to run with
very low remaining RAM - but only while it was serving with very light
load.

These problems disappeared, as it seems until now, after rebooting
with kernel 1.11. Are there any known "memory-eaters" introduced
between 1.11 and 1.14?

Version-Release number of selected component (if applicable):
kernel-2.6.10-1.14_FC2smp

How reproducible:
Always

Steps to Reproduce:
1.Upgrade system from 1.11 to 1.14
2.Set up database and NFS.
3.Bring system under load.
    

Actual Results:  Even under medium load, after a few hours, we could
see memory "disappearing" somewhere.

Additional info:

Comment 1 Dave Jones 2005-02-25 23:44:23 UTC
yes, there is one known leak in the latest kernel, I'll be putting out
another fc2 soon fixing this and a few other problems.

Comment 2 Stefan Neufeind 2005-02-27 16:51:19 UTC
Does this also affect fc3?

If it was known, could you maybe send some warning through the
announcement-list for not too many additional people running into
similar problems?

Comment 3 Stefan Neufeind 2005-03-04 17:02:41 UTC
Dave, does the release of yesterday / today (1.770_FC2 / FC3) fix this
issue as well?

Comment 4 Dave Jones 2005-03-04 22:20:47 UTC
hopefully. if it doesn't, I'm unaware of any leaks right now.
if you still get OOM kills with that kernel, it could point to a more
fundamental problem with the VM.

Comment 5 Stefan Neufeind 2005-04-09 07:51:58 UTC
Dave, it worked for us so far. Afaik we can declare this one fixed.

Comment 6 Dave Jones 2005-04-16 04:19:58 UTC
Fedora Core 2 has now reached end of life, and no further updates will be
provided by Red Hat.  The Fedora legacy project will be producing further kernel
updates for security problems only.

If this bug has not been fixed in the latest Fedora Core 2 update kernel, please
try to reproduce it under Fedora Core 3, and reopen if necessary, changing the
product version accordingly.

Thank you.