Red Hat Bugzilla – Bug 162417
(VM) Excessive swapping when free memory is ample
Last modified: 2007-11-30 17:07:07 EST
From Bugzilla Helper: User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.8) Gecko/20050511 Firefox/1.0.4 Description of problem: Hi all, This report is very similar to: https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=124576 The system runs for a short time and begins using up swap space in preference to free memory. Swap space used rises sharply with system activity, when free memory stays fairly constant. We're running Oracle 9.2.0.6 (64-bit) on a cleanly installed Redhat AS 3.0 (64-bit update 5) uname -a: Linux oracle2.redstardevelopment.com 2.4.21-32.ELsmp #1 SMP Fri Apr 15 21:03:28 EDT 2005 x86_64 x86_64 x86_64 GNU/Linux The machine itself is a 4-way opteron with 16GB of ram. Version-Release number of selected component (if applicable): 2.4.21-32.ELsmp How reproducible: Always Steps to Reproduce: 1. system boots 2. system starts oracle 9.2.0.6 (64-bit) with SGA size 6.1 GB 3. system runs for an hour ... [ sorry - quite straightforward ] Actual Results: Memory is swapped out after very little activity, despite an abundance of free memory. Swap rate is proportional to system activity. Without the skip_mapped_pages and pagecache tuning, the system would become unresponsive inside 2 hours. Expected Results: It should have used the free memory instead of swapping. This was the case with the same database on the same machine running 32-bit Oracle & Linux. It never touched the swapspace. Additional info: Attached is a vmstat 1 log for an affected time, plus meminfo dump. This machine is our live database server. We've stabilized the system with vm.skip_mapped_pages = 1 vm.pagecache = 1 10 10 vm.kswapd = 2048 64 64 gleaned from other bugzilla reports. Currently it's running with a fluctuating amount of swap, hovering around 350MB, but can rise in seconds to 600MB. It would be far preferable for it not to use the swap space at all.
Created attachment 116326 [details] /proc/meminfo dump
Created attachment 116327 [details] output from "vmstat 1"
This is most likely a NUMA issue. Please try adding "numa=off" at the end of the boot command line in /boot/grub/grub.conf and verify that the problem goes away with acceptable system performance. Please let me know how this works out. Thank-you, Larry Woodman
This has worked out very well on our test system. Swap usage stays at zero even after a greater workload has been thrown at it. Thanks for your help!
Sorry for the delay, I'm currently waiting for the DBA team to schedule a switchover to the box that's been rebooted with the numa=off setting. This should happen in the next couple of days.
Hi - We finally obtained a maintenance window for a server reboot. The machine came back perfectly, it's now using one very large memory region and no swap, even under heavier load. The throughput of the machine is better, as we'd expect when it isn't swapping. Thanks very much for your help!
Thanks for the verification, Daniel. Since you now have an easily applied work-around, I'm going to close this as WONTFIX (since we've decided not to change the default when this issue came up during U5). This is essentially a duplicate of bug 130387.
A fix for this problem has just been committed to the RHEL3 U7 patch pool this evening (in kernel version 2.4.21-37.12.EL). To enable an improved NUMA-friendly page allocation policy, please set /proc/sys/vm/numa_memory_allocator via the "sysctl" command (or put "vm.numa_memory_allocator = 1" in /etc/sysctl.conf).
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on the solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2006-0144.html