I'll try the reproducer, but my opinion at this point is that this is a known
issue with RHEL3 NUMA that will not be fixed in an update.  Basically, NUMA will
allocate all memory on a thread's home node before resorting to using other
nodes.  This causes swapping to occur when we get over the swap threshold for a
given node, even if other nodes have plenty of free memory.  Fixing this would
require either swap parameter changes or significant invasive VM changes that
are too invasive for a late update, in my opinion.

This is essentially a duplicate of bug 130387.

A fix for this problem has just been committed to the RHEL3 U7
patch pool this evening (in kernel version 2.4.21-37.12.EL).

To enable an improved NUMA-friendly page allocation policy, please
set /proc/sys/vm/numa_memory_allocator via the "sysctl" command
(or put "vm.numa_memory_allocator = 1" in /etc/sysctl.conf).

Bug fix, Devel ACK.

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2006-0144.html