Bug 127240

Summary: Swapping when there are plenty of cache available
Product: Red Hat Enterprise Linux 3 Reporter: Mindaugas Riauba <mindaugas>
Component: kernelAssignee: Rik van Riel <riel>
Status: CLOSED ERRATA QA Contact: Brian Brock <bbrock>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.0CC: petrides, riel, sct
Target Milestone: ---   
Target Release: ---   
Hardware: i686   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2004-09-02 04:31:53 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
echo m > /proc/sysrq-trigger
none
Patch to evict page cache sooner none

Description Mindaugas Riauba 2004-07-05 04:53:13 UTC 
Description of problem:
Server is too aggressively swapping. Even when there is plenty of cache 
memory available swap in/out activity is high.
Server is fully upgraded RHES 3 running Oracle 9i with 2.7GB SGA (that 
means 2.7GB of shm allocated).

Version-Release number of selected component (if applicable):
kernel-hugemem-2.4.21-15.0.2.EL

How reproducible:
Always

Steps to Reproduce:
Run Oracle with large SGA and do some load.

Additional info:
Comment 1 Mindaugas Riauba 2004-07-05 04:54:33 UTC 
# vmstat 1
procs                      memory      swap          io     system         cpu
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  3 1122100  16972  33028 3300540   24   29    37    65   58    44  9  2  7  
4
 0  4 1122100  16968  33028 3300532    0  296   312   908  479   164  3  2  
0 95
 0  3 1122100  17080  33080 3300004   88 3256 15440 10160  572   838 31 
10  0 58
 1  2 1122096  16924  33084 3299744  268 2836  8284  6200  442   647 
17  6  2 74
 0  4 1122096  17040  33088 3304132  372 2520 13252 17776 1782  3430 
31 12  0 58
 2  0 1122092  20176  33092 3305176  316  860  9276  5768  546   904 24 
11  6 58
 2  0 1122076  16976  33092 3310400   16    0  1608  7660  423   667 33  9 
42 16
 1  1 1122064  16904  33116 3312700   40  888  1480 10116  515   902 33 
11 36 20
 1  0 1122064  16988  33124 3318236    0  908   756 17488 1762  3425 19 
11 16 53
 0  1 1122040  17076  33124 3321696   24 1312  1432  9080  378   621 31 
11 38 20
 1  0 1122032  17060  33128 3324816    8 1776  1176 10476  396   659 36  
8 36 20
 1  2 1122016  16972  33140 3332440   16 2220  1048 25376 2427  4761 
31 12 22 35
 0  1 1122016  17032  33168 3334056    8  496  1016  4340  292   410 16  
5 44 35
 0  1 1121972  17088  33168 3336144   44 1724  1104  9060  350   517 25 
12 37 26
 0  3 1121972  16944  33172 3339488   44 1332  1268 11064  608  1084 
35  9 37 20
 0  1 1121968  17024  33180 3343396    4 2664   612 20320 1806  3414 20 
10 16 54
 1  0 1121968  18296  33180 3343896    0 1332  1084  9336  395   621 35  
9 34 22
 1  3 1121956  20620  33240 3346352   40 7104  1132 18528 1061  2008 
43 26 23  8
 0  4 1121956  17052  33320 3358128  208 3552   592 91124 1607  3758 
18 39  0 43
 0  5 1121956  17084  33320 3358356    0    0    20 26868  335    52  2  0  
0 98
 1  4 1121956  17096  33328 3358424   32  212   136  1040  380   111  1  
0  0 99
 0  1 1121956  17080  33332 3359304   32  232   592  4904  437   698 14  
5 35 46

# cat /proc/meminfo 
        total:    used:    free:  shared: buffers:  cached:
Mem:  4045217792 4021428224 23789568        0 28610560 3464491008
Swap: 4286529536 1179881472 3106648064
MemTotal:      3950408 kB
MemFree:         23232 kB
MemShared:           0 kB
Buffers:         27940 kB
Cached:        3340204 kB
SwapCached:      43088 kB
Active:        2799516 kB
ActiveAnon:    1974616 kB
ActiveCache:    824900 kB
Inact_dirty:    532084 kB
Inact_laundry:   85416 kB
Inact_clean:     74468 kB
Inact_target:   698296 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:      3950408 kB
LowFree:         23232 kB
SwapTotal:     4186064 kB
SwapFree:      3033836 kB
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     2048 kB
Comment 2 Mindaugas Riauba 2004-07-05 04:57:38 UTC 
Created attachment 101629 [details]
echo m > /proc/sysrq-trigger
Comment 3 Mindaugas Riauba 2004-07-05 05:24:57 UTC 
I even got the error from Oracle:
sqlplus: error while loading shared 
libraries: /ora/product/9.2.0/lib/libclntsh.so.9.0: cannot make segment 
writable for relocation: Cannot allocate memory

sysctl changes from default:
vm.pagecache = 1        15      30
vm.overcommit_ratio = 100
vm.overcommit_memory = 2
Comment 4 Rik van Riel 2004-07-05 13:31:17 UTC 
Created attachment 101638 [details]
Patch to evict page cache sooner

This patch should evict the page cache from memory faster, leaving more memory
available for applications.
Comment 5 Ernie Petrides 2004-07-06 00:44:32 UTC 
Hello.  Although the patch that Rik posted in commment #4 was not
incorporated in RHEL3 U3, other changes were made in U3 (most
relevant is Rik's change to __alloc_pages()) that should address
this problem.  Please try the U3 beta when it becomes available,
and if the problem still persists, then reopen this bug report.

For now, I'm closing this as a dup of 118152.  -ernie

*** This bug has been marked as a duplicate of 118152 ***
Comment 6 John Flanagan 2004-09-02 04:31:54 UTC 
An errata has been issued which should help the problem 
described in this bug report. This report is therefore being 
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files, 
please follow the link below. You may reopen this bug report 
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2004-433.html