Bug 110612 - System begins swapping when less than 2GB of memory is free causing memory to be underutilized
System begins swapping when less than 2GB of memory is free causing memory to...
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 3
Classification: Red Hat
Component: kernel (Show other bugs)
3.0
All Linux
medium Severity medium
: ---
: ---
Assigned To: Rik van Riel
Brian Brock
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2003-11-21 13:39 EST by Andrew Bond
Modified: 2007-11-30 17:06 EST (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2004-09-02 00:30:57 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
Patch from Intel to allow more memory to be utilized without triggering heavy swapping (1.99 KB, patch)
2003-11-21 13:41 EST, Andrew Bond
no flags Details | Diff
untested patch, to indicate the direction in which we're searching a fix (755 bytes, patch)
2003-11-21 14:15 EST, Rik van Riel
no flags Details | Diff
output files from system (5.97 KB, text/plain)
2003-11-21 14:32 EST, Andrew Bond
no flags Details
/var/log/messages from last boot (290.82 KB, text/plain)
2003-11-21 14:59 EST, Andrew Bond
no flags Details
sysrqm and meminfo output (2.35 KB, text/plain)
2003-11-25 11:08 EST, Andrew Bond
no flags Details
sysrqm and meminfo output (2.35 KB, text/plain)
2003-11-25 11:55 EST, Andrew Bond
no flags Details
better balancing in alloc_pages (2.70 KB, patch)
2004-06-08 10:14 EDT, Rik van Riel
no flags Details | Diff

  None (edit)
Description Andrew Bond 2003-11-21 13:39:19 EST
From Bugzilla Helper:
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET 
CLR 1.0.3705)

Description of problem:
The target system here is a 4P Itanium 2 box with 48GB of memory 
running Oracle 10g.  The system will start to swap if total free 
memory is pushed below 2GB and can get into an unusable state.  This 
causes us to have to leave over 2GB of memory unused when we could be 
using it to store Oracle data.  Not being able to use the memory 
lowers the overall database performance potential of the system.

Intel has a patch that allows the memory to be used without 
triggering high swap ativity.  This patch is attached here in the bug 
entry.

Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1.Have Oracle allocate enough memory so there is less than 2GB of 
free memory available on a 48GB system
2.
3.
    

Actual Results:  System begins heavy swapping and sometimes becomes 
unusable when free memory drops below 2GB out of 48GB.

Expected Results:  System should allow a lower threshold for free 
memory on large memory systems.

Additional info:
Comment 1 Andrew Bond 2003-11-21 13:41:40 EST
Created attachment 96122 [details]
Patch from Intel to allow more memory to be utilized without triggering heavy swapping
Comment 2 Rik van Riel 2003-11-21 13:45:22 EST
The patch is incompatible with the way memory usage is balanced
between the zones in the RHEL 3 VM.

Having said that, we would appreciate more details about the bug so we
can fix it in a way that is compatible with our VM.

I suspect you are right that the page cleaning thresholds are too
high, this is something to look into.  Do you have snapshots from
/proc/meminfo, info on application configuration, machine
configuration, etc. ?
Comment 3 Rik van Riel 2003-11-21 14:15:26 EST
Created attachment 96124 [details]
untested patch, to indicate the direction in which we're searching a fix
Comment 4 Andrew Bond 2003-11-21 14:30:32 EST
The system is an HP rx5670 server.  I am attaching an output file 
that has a current /proc/cpuinfo, /proc/meminfo, lspci, and a maps 
snapshot from one of the Oracle processes.  Oracle is not currently 
booted up for performance so you will see that not all of the 
hugepages are used right now, but during a performance run they would 
be.  This machine is a part of a cluster of identical systems.
Comment 5 Andrew Bond 2003-11-21 14:32:24 EST
Created attachment 96125 [details]
output files from system

The kernel here is the RHEL 3 AS source compiled with only the Intel swap patch
applied to it.
Comment 6 Rik van Riel 2003-11-21 14:35:47 EST
Could you please also post the boot dmesg output, so we can see
exactly how much memory is in the normal zone and how much in the dma
zone ?

(if the system in question has a dma zone, the HP systems have an
IOMMU don't they?)
Comment 7 Andrew Bond 2003-11-21 14:58:20 EST
Yes the HP system have an IOMMU.  I am attaching 
the /var/log/messages file from the last boot.  Dmesg gets blown out 
many times over when the Qlogic driver loads all the devices, so if 
there is something that does not get mirrored to the messages that 
you need to see we will have to arrange for a special boot later.

Here also is an lsmod

Module                  Size  Used by    Not tainted
qla2300               5111368354
tg3                   147432   3
nls_iso8859-1           5984   1  (autoclean)
nls_cp437               7680   1  (autoclean)
vfat                   30336   1  (autoclean)
fat                    85708   0  (autoclean) [vfat]
keybdev                 5976   0  (unused)
mousedev               14688   0  (unused)
hid                    49544   0  (unused)
input                  13536   0  [keybdev mousedev hid]
ehci-hcd               51584   0  (unused)
usb-ohci               57856   0  (unused)
usbcore               180928   1  [hid ehci-hcd usb-ohci]
ext3                  214024   1
jbd                   127680   1  [ext3]
sym53c8xx             147296   3
sd_mod                 3235216714
scsi_mod              229248   3  [qla2300 sym53c8xx sd_mod]
Comment 8 Andrew Bond 2003-11-21 14:59:12 EST
Created attachment 96130 [details]
/var/log/messages from last boot
Comment 9 Rik van Riel 2003-11-21 15:02:48 EST
Boy that's a lot of disks ... and like you suggested, it blew the
information I wanted right out of the ring buffer ;)

Could you please tell me what the kernel spits out when you "echo m >
/proc/sysrq_trigger" ?

Thank you.
Comment 10 Andrew Bond 2003-11-25 11:08:30 EST
Created attachment 96186 [details]
sysrqm and meminfo output

Here the database is up for the performance run.  All hugepages allocated are
being used.
Comment 11 Andrew Bond 2003-11-25 11:55:31 EST
Created attachment 96187 [details]
sysrqm and meminfo output

Here the database is up for the performance run.  All hugepages allocated are
being used.
Comment 12 Rik van Riel 2004-06-08 10:14:27 EDT
Created attachment 100966 [details]
better balancing in alloc_pages

This patch will probably be applied to RHEL3 U3 and should fix this problem (if
the system has multiple memory zones).
Comment 13 Ernie Petrides 2004-06-12 03:58:55 EDT
A fix for this problem has just been committed to the RHEL3 U3
patch pool this evening (in kernel version 2.4.21-15.10.EL).
Comment 14 John Flanagan 2004-09-02 00:30:57 EDT
An errata has been issued which should help the problem 
described in this bug report. This report is therefore being 
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files, 
please follow the link below. You may reopen this bug report 
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2004-433.html

Note You need to log in before you can comment on or make changes to this bug.