Description of problem:

Disk I/O slows dramatically after continual heavy disk usage.  When 
transferring a large file (>3GB) Linux appears to introduce a huge amount of 
CPU overhead after transferring part of the file and slows the disk I/O to 
about one fourth of the prior transfer rate.  On some systems the a 100GB file 
transfers at full speed for about 80GB (a 160GB disk) and then the rest of the 
file is transfered at approximately one quarter of the original rate.  On 
another system (an 80GB disk) a 20GB file is transfered at full speed for 
about 3GB and then the ENTIRE remainder is transferred at the much lower rate 
(in this case about one eighth the orginal rate).  The application uses 
DIRECT_IO and this occurs in both SMP and NON-SMP.  It has also been reported 
by other users that a similar problem occurs when continually writing files 
that are only 100-200MB.  Occurs on RAIDs and single disks.


How reproducible:

The best way to replicate the slowing disk I/O issue is to compile the 
attached program using " gcc -o disk_test disk_test.c -D_GNU_SOURCE ".  The 
program transfers 512KB blocks and every 100 blocks (50MB) outputs a status 
message showing the time (in milli-seconds) that the last 100 blocks took to 
transfer.  If you do not see an increase in these times edit disk_test.c 
(attached) and change the line " block_count = 5000; " to a much higher number 
(80% of the disk space available is likely sufficient).

Steps to Reproduce:
1. download disk_test.c
2. gcc -o disk_test disk_test.c -D_GNU_SOURCE 
3. run disk_test; edit the number of blocks and recompile if the issue is not 
seen
  
Actual results:

On a single SATA drive I will see a sustained transfer rate of about 33MBps 
for up to 80GB (disk size is 160GB, disk cache is 8MB, system has 3GB RAM, 
using DIRECT_IO, ext3, default scheduler) when transferring a 100GB file.  The 
remaining 20GB is transferred at an unchanging rate of about 8MBps.