Bug 175140

Summary: Ext3 strange state during heavy writes
Product: Red Hat Enterprise Linux 4 Reporter: drew.middlesworth
Component: kernelAssignee: Stephen Tweedie <sct>
Status: CLOSED DUPLICATE QA Contact: Brian Brock <bbrock>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.0CC: jbaron, staubach
Target Milestone: ---   
Target Release: ---   
Hardware: i386   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2005-12-07 12:38:47 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Program to create slow writes state none

Description drew.middlesworth 2005-12-06 22:08:02 UTC
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12) Gecko/20050922 CentOS/1.7.12-1.4.1.centos4

Description of problem:
During writes to an ext3 filesystem after an amount of time the writes will slow almost to a halt with a kernel process eating up an entire cpu (25% of system time on a HT dual system). Stopping the process and starting it again fixes it for some time. Switching the filesystem to ext2 or running the latest from kernel.org (2.6.14.3) fixes this issue completely. We have created a program to duplicate this issue which I will attach, and seems to happen after about 40Gig has been written. This is on kernel-smp-2.6.9-22.0.1.EL.

iostat:

avg-cpu:  %user   %nice    %sys %iowait   %idle
          0.10    0.00   25.04    0.60   74.26

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda               1.80         0.00       682.40          0       3412
sdb               0.00         0.00         0.00          0          0

top:

top - 17:50:23 up 51 min,  4 users,  load average: 1.12, 4.49, 7.78
Tasks:  79 total,   2 running,  77 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0% us, 25.1% sy,  0.0% ni, 73.8% id,  1.1% wa,  0.0% hi,  0.0% si
Mem:   4148928k total,  4132900k used,    16028k free,    16368k buffers
Swap:  8193108k total,      128k used,  8192980k free,  3998196k cached

 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND          
3073 root      25   0  1888  428  360 R 99.9  0.0  13:00.88  make-fragmentat    
222 root      15   0     0    0    0 S  0.3  0.0   0:50.89 kjournald         

Version-Release number of selected component (if applicable):
kernel-smp-2.6.9-22.0.1.EL

How reproducible:
Always

Steps to Reproduce:
1. compile program make-fragmentation.c
2. run program make-fragmentation
3. iostat will show the writes slowing to almost nothing
  

Actual Results:  The process's writes would slow to about 600KB/s and system time would stick at 25% (1 of 4 virtual HT processors)

Expected Results:  System writes should have remained at a ~40MB/s level

Additional info:

Comment 1 drew.middlesworth 2005-12-06 22:09:03 UTC
Created attachment 121947 [details]
Program to create slow writes state

Comment 3 Stephen Tweedie 2005-12-07 12:38:47 UTC
Fixed in 2.6.9-22.12; will be in U3.



*** This bug has been marked as a duplicate of 156437 ***