Red Hat Bugzilla – Bug 162417
(VM) Excessive swapping when free memory is ample
Last modified: 2007-11-30 17:07:07 EST
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.8) Gecko/20050511 Firefox/1.0.4
Description of problem:
This report is very similar to:
The system runs for a short time and begins using up swap space in preference to free memory. Swap space used rises sharply with system activity, when free memory stays fairly constant.
We're running Oracle 126.96.36.199 (64-bit) on a cleanly installed Redhat AS 3.0 (64-bit update 5)
Linux oracle2.redstardevelopment.com 2.4.21-32.ELsmp #1 SMP Fri Apr 15 21:03:28 EDT 2005 x86_64 x86_64 x86_64 GNU/Linux
The machine itself is a 4-way opteron with 16GB of ram.
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. system boots
2. system starts oracle 188.8.131.52 (64-bit) with SGA size 6.1 GB
3. system runs for an hour ...
[ sorry - quite straightforward ]
Actual Results: Memory is swapped out after very little activity, despite an abundance of free memory. Swap rate is proportional to system activity. Without the skip_mapped_pages and pagecache tuning, the system would become unresponsive inside 2 hours.
Expected Results: It should have used the free memory instead of swapping. This was the case with the same database on the same machine running 32-bit Oracle & Linux. It never touched the swapspace.
Attached is a vmstat 1 log for an affected time, plus meminfo dump.
This machine is our live database server.
We've stabilized the system with
vm.skip_mapped_pages = 1
vm.pagecache = 1 10 10
vm.kswapd = 2048 64 64
gleaned from other bugzilla reports.
Currently it's running with a fluctuating amount of swap, hovering around 350MB, but can rise in seconds to 600MB.
It would be far preferable for it not to use the swap space at all.
Created attachment 116326 [details]
Created attachment 116327 [details]
output from "vmstat 1"
This is most likely a NUMA issue. Please try adding "numa=off" at the end of
the boot command line in /boot/grub/grub.conf and verify that the problem goes
away with acceptable system performance. Please let me know how this works out.
Thank-you, Larry Woodman
This has worked out very well on our test system. Swap usage stays at zero even
after a greater workload has been thrown at it. Thanks for your help!
Sorry for the delay, I'm currently waiting for the DBA team to schedule a
switchover to the box that's been rebooted with the numa=off setting. This
should happen in the next couple of days.
We finally obtained a maintenance window for a server reboot. The machine came
back perfectly, it's now using one very large memory region and no swap, even
under heavier load. The throughput of the machine is better, as we'd expect
when it isn't swapping.
Thanks very much for your help!
Thanks for the verification, Daniel. Since you now have an easily
applied work-around, I'm going to close this as WONTFIX (since we've
decided not to change the default when this issue came up during U5).
This is essentially a duplicate of bug 130387.
A fix for this problem has just been committed to the RHEL3 U7
patch pool this evening (in kernel version 2.4.21-37.12.EL).
To enable an improved NUMA-friendly page allocation policy, please
set /proc/sys/vm/numa_memory_allocator via the "sysctl" command
(or put "vm.numa_memory_allocator = 1" in /etc/sysctl.conf).
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.