Description of problem:

issue (or symptom?): kswapd consumes 100% cpu and system slows to a crawl

rsyncing large filesystems (using rsyncd from remote redhat 9 system)
repeatably grinds to a snails pace on EL 3 ES.

this happens most notably when syncing large trees where it has to
create many files locally.  seems to get into this state after around
100GB or 1M files.  local rsync (and remote rsyncd) are around 1GB
process SIZE.  rsync run locally on machine that slows down.  local
box config is as follows

Red Hat Enterprise Linux ES release 3 (Taroon Update 3)
Linux mule-1 2.4.21-20.ELsmp #1 SMP Wed Aug 18 20:46:40 EDT 2004 i686
i686 i386 GNU/Linux
2 x 3.0GHz/512K L2/ Xeon w/ hyperthreading enabled
disk controller is 16-channel Adaptec SATA RAID (21610 ?)
16x250GB configured as 2 RAID-10 arrays
latest up2date patches have been applied
4GB physical RAM
2 x 8GB swap partitions

Version-Release number of selected component (if applicable):

probably the kernel or kernel-utils

How reproducible:

system runs for a few hours, then, after creating a million or so
files, as above, file creation rate drops dramatically (1-2 per
second) and kswapd appears extremely busy - racking up 100% cpu time
and cpu minutes nearly at wall-clock speed.  top reports about 2.2GB
in cached, but essentially no actual swap in use.  it can eventually
finish, and if the rsync is killed, kswapd nearly immediately stops
and overall system responsiveness seems to recover, but it feels like
the system is more susceptible to this condition and it comes back
quickly the next time - unless a full reboot is issued.

Steps to Reproduce:
1. rsync large filesystems from one system to another
2. 
3.
  
Actual results:

results are:

file creation rate drops dramatically (1-2 per second) and kswapd
appears extremely busy - racking up 100% cpu time and cpu minutes
nearly at wall-clock speed.  top reports about 2.2GB in cached, but
essentially no actual swap in use.

Expected results:

Additional info: