Bug 162417

Summary: (VM) Excessive swapping when free memory is ample
Product: Red Hat Enterprise Linux 3 Reporter: Daniel Ramsay <daniel.ramsay>
Component: kernelAssignee: Larry Woodman <lwoodman>
Status: CLOSED ERRATA QA Contact: Brian Brock <bbrock>
Severity: high Docs Contact:
Priority: medium    
Version: 3.0CC: k.georgiou, petrides
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: RHSA-2006-0144 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2006-03-15 16:11:53 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 168424    
Attachments:
Description Flags
/proc/meminfo dump
none
output from "vmstat 1" none

Description Daniel Ramsay 2005-07-04 11:44:35 UTC 
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.8) Gecko/20050511 Firefox/1.0.4

Description of problem:
Hi all, 

This report is very similar to:

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=124576

The system runs for a short time and begins using up swap space in preference to free memory.  Swap space used rises sharply with system activity, when free memory stays fairly constant.

We're running Oracle 9.2.0.6 (64-bit) on a cleanly installed Redhat AS 3.0 (64-bit update 5)

uname -a: 
Linux oracle2.redstardevelopment.com 2.4.21-32.ELsmp #1 SMP Fri Apr 15 21:03:28 EDT 2005 x86_64 x86_64 x86_64 GNU/Linux
The machine itself is a 4-way opteron with 16GB of ram.


Version-Release number of selected component (if applicable):
2.4.21-32.ELsmp

How reproducible:
Always

Steps to Reproduce:
1. system boots
2. system starts oracle 9.2.0.6 (64-bit) with SGA size 6.1 GB
3. system runs for an hour ...

[ sorry - quite straightforward ]

Actual Results:  Memory is swapped out after very little activity, despite an abundance of free memory.  Swap rate is proportional to system activity.  Without the skip_mapped_pages and pagecache tuning, the system would become unresponsive inside 2 hours.

Expected Results:  It should have used the free memory instead of swapping.  This was the case with the same database on the same machine running 32-bit Oracle & Linux.  It never touched the swapspace.

Additional info:

Attached is a vmstat 1 log for an affected time, plus meminfo dump.
This machine is our live database server.

We've stabilized the system with 

vm.skip_mapped_pages = 1
vm.pagecache = 1 10 10
vm.kswapd = 2048 64 64

gleaned from other bugzilla reports.

Currently it's running with a fluctuating amount of swap, hovering around 350MB, but can rise in seconds to 600MB.
It would be far preferable for it not to use the swap space at all.
Comment 1 Daniel Ramsay 2005-07-04 11:45:48 UTC 
Created attachment 116326 [details]
/proc/meminfo dump
Comment 2 Daniel Ramsay 2005-07-04 11:48:31 UTC 
Created attachment 116327 [details]
output from "vmstat 1"
Comment 3 Larry Woodman 2005-07-12 19:04:57 UTC 
This is most likely a NUMA issue.  Please try adding "numa=off" at the end of
the boot command line in /boot/grub/grub.conf and verify that the problem goes
away with acceptable system performance.  Please let me know how this works out.

Thank-you, Larry Woodman
Comment 4 Daniel Ramsay 2005-07-13 14:29:02 UTC 
This has worked out very well on our test system. Swap usage stays at zero even
after a greater workload has been thrown at it.  Thanks for your help!
Comment 5 Daniel Ramsay 2005-07-19 14:23:45 UTC 
Sorry for the delay, I'm currently waiting for the DBA team to schedule a
switchover to the box that's been rebooted with the numa=off setting.  This
should happen in the next couple of days.
Comment 6 Daniel Ramsay 2005-08-16 08:24:18 UTC 
Hi - 

We finally obtained a maintenance window for a server reboot.  The machine came
back perfectly, it's now using one very large memory region and no swap, even
under heavier load.  The throughput of the machine is better, as we'd expect
when it isn't swapping.   

Thanks very much for your help!
Comment 7 Ernie Petrides 2005-08-16 18:48:31 UTC 
Thanks for the verification, Daniel.  Since you now have an easily
applied work-around, I'm going to close this as WONTFIX (since we've
decided not to change the default when this issue came up during U5).

This is essentially a duplicate of bug 130387.
Comment 8 Ernie Petrides 2005-11-30 07:48:15 UTC 
A fix for this problem has just been committed to the RHEL3 U7
patch pool this evening (in kernel version 2.4.21-37.12.EL).

To enable an improved NUMA-friendly page allocation policy, please
set /proc/sys/vm/numa_memory_allocator via the "sysctl" command
(or put "vm.numa_memory_allocator = 1" in /etc/sysctl.conf).
Comment 12 Red Hat Bugzilla 2006-03-15 16:11:53 UTC 
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2006-0144.html