Escalated to Bugzilla from IssueTracker
I'll try the reproducer, but my opinion at this point is that this is a known
issue with RHEL3 NUMA that will not be fixed in an update. Basically, NUMA will
allocate all memory on a thread's home node before resorting to using other
nodes. This causes swapping to occur when we get over the swap threshold for a
given node, even if other nodes have plenty of free memory. Fixing this would
require either swap parameter changes or significant invasive VM changes that
are too invasive for a late update, in my opinion.
This is essentially a duplicate of bug 130387.
A fix for this problem has just been committed to the RHEL3 U7
patch pool this evening (in kernel version 2.4.21-37.12.EL).
To enable an improved NUMA-friendly page allocation policy, please
set /proc/sys/vm/numa_memory_allocator via the "sysctl" command
(or put "vm.numa_memory_allocator = 1" in /etc/sysctl.conf).
Bug fix, Devel ACK.
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.