Bug 118152 - swap usage when there's still free (although cached) memory
swap usage when there's still free (although cached) memory
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 3
Classification: Red Hat
Component: kernel (Show other bugs)
3.0
x86_64 Linux
medium Severity high
: ---
: ---
Assigned To: Larry Woodman
Brian Brock
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2004-03-12 11:09 EST by Joshua Baker-LePain
Modified: 2007-11-30 17:07 EST (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2004-09-02 00:31:10 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
vmstat, free, and ps output (6.94 KB, text/plain)
2004-03-12 11:10 EST, Joshua Baker-LePain
no flags Details
"AltSysrq M" output (3.42 KB, text/plain)
2004-03-15 11:20 EST, Joshua Baker-LePain
no flags Details
swap usage when there's still free (although cached) memory AltSysRq-M output for memory problem (1.99 KB, text/plain)
2004-03-19 15:18 EST, Darcy Bottaro
no flags Details
patch to evict page cache faster (1.15 KB, patch)
2004-07-06 08:24 EDT, Rik van Riel
no flags Details | Diff

  None (edit)
Description Joshua Baker-LePain 2004-03-12 11:09:04 EST
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.6) Gecko/20040115

Description of problem:
I am seeing significant swapping on systems even though there is
plenty of free (although used by cache) memory.  I have 6 dual Opteron
systems, 5 with 4GB of memory, and one with 8GB, all running the same
kernel, and all showing this behavior.  A user will start a couple of
simulations, each taking 1GB of RAM and writing lots of data over NFS.
  The system will then swap fairly heavily, having a very bad affect
on simulation performance.  I've reproduced the behavior with both
Matlab and LS-DYNA (commercial FEM code), so it's not application
specific. 

Version-Release number of selected component (if applicable):
kernel-smp-2.4.21-9.EL

How reproducible:
Always

Steps to Reproduce:
1.  Fill available memory with cached data.
2.  Start resource intensive application.
3.  Weep as performance plummets.

Actual Results:  System begins swapping.

Expected Results:  The cached memory should be tossed, and filled in
with actual application data.

Additional info:

I'll attach a file with some system output.
Comment 1 Joshua Baker-LePain 2004-03-12 11:10:13 EST
Created attachment 98489 [details]
vmstat, free, and ps output
Comment 2 Larry Woodman 2004-03-15 09:54:36 EST
Can you get me a quick "AltSysrq M" output when the system is in this
state?  You can do this by:

1.) login as root
2.) echo 1 > /proc/sys/kernel/sysrq
3.) echo m > /proc/sysrq-trigger
4.) attach the dmesg output to this bug


Larry Woodman
Comment 3 Joshua Baker-LePain 2004-03-15 11:20:17 EST
Created attachment 98534 [details]
"AltSysrq M" output

3 times -- let me know if you need more.
Comment 4 Larry Woodman 2004-03-15 13:58:53 EST
Ah, I think I see the problem.  Try setting pagecache.maxpercent to 15
via "echo 1 15 15 > /proc/sys/vm/pagecache" and see if this helps.

Larry
Comment 5 Joshua Baker-LePain 2004-03-15 14:32:12 EST
It does indeed.  I'm now seeing minimal swapping with the same
workload.  Thanks!
Comment 6 Gary Mansell 2004-03-17 12:05:31 EST
I have come across this same problem but even if I set the following
in vm.pagecache to 2 10 20, I still get more than 20% of memory used
as cache - 

Here is /proc/meminfo -

[grma@shane 59] ~ > cat /proc/meminfo        total:    used:    free:
 shared: buffers:  cached:Mem:  525836288 445038592 80797696        0
31404032 187092992Swap: 2146787328 30904320 2115883008
MemTotal:       513512 kB
MemFree:         78904 kB
MemShared:           0 kB
Buffers:         30668 kB
Cached:         181156 kB
SwapCached:       1552 kB
Active:         347540 kB
ActiveAnon:     204324 kB
ActiveCache:    143216 kB
Inact_dirty:     32452 kB
Inact_laundry:   27108 kB
Inact_clean:      5700 kB
Inact_target:    82560 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       513512 kB
LowFree:         78904 kB
SwapTotal:     2096472 kB
SwapFree:      2066292 kB
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     4096 kB

My system seems to grind to a halt due to their being no memory
available to the applications because it is all being used for cache.

Any ideas what I can do to prevent this?
Comment 7 Gary Mansell 2004-03-18 11:39:41 EST
Can someone please explain why Redhat seem to be doing nothing about a
MAJOR issue with their RHEL 3 flagship product.

I have been doing some research into this and it is definitely a
problem that people are coming across.

It appears that there is no work around so where is the updated kernel
- this problem has been around for a long time now. Where is the
support that we pay £000's for??
Comment 8 Larry Woodman 2004-03-18 12:18:49 EST
The pagecache parameter(specifically the third one: maxpercent) does
not stop the system from allocating more than that amount of memory in
the pagecache.  Instead, when the system runs out of memory and starts
reclaiming, it forces the pagecache to give up memory until it is down
below /proc/sus/vm/pagecache.maxpercent before it starts reclaiming
anonymous memory and therefore starts paging.  If your system is not
responding to lowering this value like Joshua's did, please get me
some AltSysrq-M outputs(like Joshua did) and attach them to this bug.
 I'm more than happy to figure out whats going on in your system.

Thanks, Larry
Comment 9 Darcy Bottaro 2004-03-19 15:18:41 EST
Created attachment 98693 [details]
swap usage when there's still free (although cached) memory

AltSysRq-M output for memory problem
Comment 10 Rik van Riel 2004-03-23 14:28:39 EST
grmansell, could you please try  "echo 30 >
/proc/sys/vm/inactive_clean_percent" ?

(the new default for RHEL3 U2)
Comment 11 Gary Mansell 2004-03-24 04:01:33 EST
I have added the new value for inactive_clean_percent and will
feedback if I get a problem again.

I must say that the machine has not locked up since I logged the call
report - typical !!!
Comment 12 Joshua Baker-LePain 2004-03-24 09:01:04 EST
For the record, I already had /proc/sys/vm/inactive_clean_percent set
to 30, due to bug 115438.
Comment 13 Larry Woodman 2004-03-30 09:55:59 EST
Joshua, is this problem now fixed with the inactive_cache_percent
set top 30 or have you still seen your system swap excessively?

Thanks, Larry Woodman
Comment 14 Joshua Baker-LePain 2004-03-30 10:01:53 EST
Sorry if I wasn't clear.  When I opened this bug (i.e. when I was
seeing the swapping), I already had inactive_clean_percent set to 30.
 On your advice in this bug, I set pagecache to "1 15 15", and that
fixed the problem.

Thanks again.
Comment 15 Ernie Petrides 2004-06-12 03:59:42 EDT
A fix for this problem has just been committed to the RHEL3 U3
patch pool this evening (in kernel version 2.4.21-15.10.EL).
Comment 16 Ernie Petrides 2004-07-05 20:44:35 EDT
*** Bug 127240 has been marked as a duplicate of this bug. ***
Comment 17 Rik van Riel 2004-07-06 08:24:14 EDT
Created attachment 101654 [details]
patch to evict page cache faster

If the patch that is currently applied to the tree isn't aggressive enough to
completely resolve your problem, this patch might help a bit more.  By evicting
the page cache more aggressively, more memory should be left over for
applications.
Comment 18 Mindaugas Riauba 2004-07-07 02:24:40 EDT
Setting vm.pagecache to "1 15 15" helped a bit but not completely. And I 
was unable to check Rik's patch because of some kernel compilation 
problems (bug #127365).
We cannot wait for U3. Those swapping issues give us some performance 
problems after update from Redhat 7.3 to RHES3.
Comment 19 Ernie Petrides 2004-07-07 13:39:23 EDT
In response to comment #18, the RHEL3 U3 beta will begin in a few
weeks.  You are welcome to try the fixed kernel during the beta
period.  From now until then, the fixed kernel is undergoing Q/A.
I wouldn't recommend running a kernel that has not yet been through
Q/A.

Also, Rik has answered bug #127365.
Comment 20 Mindaugas Riauba 2004-07-14 04:20:39 EDT
Applied Rik's patch. Swapping went back to normal. Thanks. Should I return 
vm.pagecache to something from "1 15 15"?
But one problem still persists. From time to time we are getting errors:
oracleMOON: error while loading shared 
libraries: /ora/product/9.2.0/lib/libjox9.so: cannot make segment writable for 
relocation: Cannot allocate memory
It looks like Linux not Oracle error.
To remind situation - kernel is hugemem, Oracle is relinked to be able to use 
2.7GB SGA.
Should I open new bugzilla entry about this bug?
Comment 21 Rik van Riel 2004-07-14 08:08:57 EDT
Good to hear that my patch brings swapping back to normal.  The Oracle
problem does indeed look like a bug, could you please open a bugzilla
entry about it ?
Comment 22 John Flanagan 2004-09-02 00:31:10 EDT
An errata has been issued which should help the problem 
described in this bug report. This report is therefore being 
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files, 
please follow the link below. You may reopen this bug report 
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2004-433.html

Note You need to log in before you can comment on or make changes to this bug.