Bug 137513

Summary: kswapd consumes 100% cpu and system slows to a crawl
Product: Red Hat Enterprise Linux 3 Reporter: Jim Salter <jsalter>
Component: kernelAssignee: Larry Woodman <lwoodman>
Status: CLOSED ERRATA QA Contact: Brian Brock <bbrock>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.0CC: petrides, riel
Target Milestone: ---   
Target Release: ---   
Hardware: i686   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2004-10-29 23:46:47 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jim Salter 2004-10-28 23:44:03 UTC
Description of problem:

issue (or symptom?): kswapd consumes 100% cpu and system slows to a crawl

rsyncing large filesystems (using rsyncd from remote redhat 9 system)
repeatably grinds to a snails pace on EL 3 ES.

this happens most notably when syncing large trees where it has to
create many files locally.  seems to get into this state after around
100GB or 1M files.  local rsync (and remote rsyncd) are around 1GB
process SIZE.  rsync run locally on machine that slows down.  local
box config is as follows

Red Hat Enterprise Linux ES release 3 (Taroon Update 3)
Linux mule-1 2.4.21-20.ELsmp #1 SMP Wed Aug 18 20:46:40 EDT 2004 i686
i686 i386 GNU/Linux
2 x 3.0GHz/512K L2/ Xeon w/ hyperthreading enabled
disk controller is 16-channel Adaptec SATA RAID (21610 ?)
16x250GB configured as 2 RAID-10 arrays
latest up2date patches have been applied
4GB physical RAM
2 x 8GB swap partitions

Version-Release number of selected component (if applicable):

probably the kernel or kernel-utils

How reproducible:

system runs for a few hours, then, after creating a million or so
files, as above, file creation rate drops dramatically (1-2 per
second) and kswapd appears extremely busy - racking up 100% cpu time
and cpu minutes nearly at wall-clock speed.  top reports about 2.2GB
in cached, but essentially no actual swap in use.  it can eventually
finish, and if the rsync is killed, kswapd nearly immediately stops
and overall system responsiveness seems to recover, but it feels like
the system is more susceptible to this condition and it comes back
quickly the next time - unless a full reboot is issued.

Steps to Reproduce:
1. rsync large filesystems from one system to another
2. 
3.
  
Actual results:

results are:

file creation rate drops dramatically (1-2 per second) and kswapd
appears extremely busy - racking up 100% cpu time and cpu minutes
nearly at wall-clock speed.  top reports about 2.2GB in cached, but
essentially no actual swap in use.

Expected results:

Additional info:

Comment 1 Ernie Petrides 2004-10-29 23:46:47 UTC

*** This bug has been marked as a duplicate of 132639 ***

Comment 2 John Flanagan 2004-12-20 20:56:51 UTC
An errata has been issued which should help the problem 
described in this bug report. This report is therefore being 
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files, 
please follow the link below. You may reopen this bug report 
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2004-550.html