Created attachment 527794 [details]
data, statistics and graph showing regression, needs OpenOffice Calc to open

Description of problem:

There is a performance regression of as much as 50% from RHEL5.6 to RHEL5.7 for single threaded buffered write of medium-size files.  

What is peculiar about this regression is that it goes away when you do fsync() before close().  In fact RHEL5.7 is significantly faster than RHEL5.6 (as much as 25%) in this situation.

The problem goes away with a sufficiently large file.

It's too late for 5.7 but perhaps for 5.8?

 
Version-Release number of selected component (if applicable):

RHEL5.6, 2.6.18-238 kernel  =>  RHEL5.7, 2.6.18-274 kernel
NetApp OnTap 7.3.5.1 (Steve Dickson's filer may have it)
This is perf36.lab.bos.redhat.com (16 GB RAM) over 10-Gbps to a NetApp NFS export.  The configuration is identical except for the OS version in these tests.  

How reproducible:

Every time so far.  I tried 4 samples at every file size to verify that the variance in the result was well below the size of the regression (data in attachment)


Steps to Reproduce:
1. install both RHEL5.6 and RHEL5.7 kernels on same /boot, so everything else
is the same
2. mount -t nfs -o nfsvers=3 perf-na1-10ge:/vol/vol0/home /mnt/big
3. run the below script, set FSYNC env. var to "conv=fsync" to do fsync before close, delete/blank the environment variable to just do close at end.  Do this for both OS versions. 

script try-fsizes.sh:

for m in `seq 1 4 ` ; do 
  for n in 16 64 256 1024 4096 ; do 
    rm -fv /mnt/big/x.dd
    sync
    sleep 2
    cmd="dd $FSYNC if=/dev/zero of=/mnt/big/x.dd bs=1024k count=$n"
    echo "$cmd"
    $cmd
  done
done 2>&1 


Actual results:

The attached graph shows on a RHEL5.7 system how fsync() speeds up (huh?) performance of single-thread writes for an NFS file.  

Expected results:

This seems wrong to me.  I did some straces and without the fsync call, throughput was determined by the time it takes to close the file, whereas with the fsync call, throughput was determined by the elapsed time of the fsync call.  For example, using a 1 GB file, without an fsync(), the close() call took 7 seconds(!), whereas with fsync() the close call took 0 seconds and the fsync() call took 2.5 seconds.  I think writes without fsync at end should be no slower than writes with fsync at end.  For NFS, when you close a file, the NFS client has to flush all the dirty pages to the NFS server (not necessarily to disk), because of NFS close-to-open semantics.   But why should this process be slower with close() than with fsync?  

Additional info:

Unsure at this time whether it happens on a Linux NFS server, on 1-Gbps connection, etc.