From Bugzilla Helper:
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.4)
Gecko/20030624 Netscape/7.1 (ax)

Description of problem:
Slow IO performance was observed on RHEL3.0 (2.4.21-20). Below are
test results:

Test 1): copy a 515 MB file between 2 filesystems (/var and /var/qmail)
# df -k /var /var/qmail
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda5              4370688   1289188   2859476  32% /var
/dev/sda3              8749500    561540   7743508   7% /var/qmail

# time cp -p /var/log/testfile.log /var/qmail/
real    2m10.647s
user    0m0.070s
sys     0m6.780s

This translates to 3.93 MB/s.

==output from iostat command==
avg-cpu:  %user   %nice    %sys   %idle
           0.22    0.00    5.53   94.25

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s
avgrq-sz avgqu-sz   await  svctm  %util
/dev/sda5    0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00
    0.00     0.00    0.00   0.00   0.00
/dev/sda3    0.00 8876.79  0.00 7133.57    0.00 128082.86     0.00
64041.43    17.95    16.80    0.24   0.23 166.96

avg-cpu:  %user   %nice    %sys   %idle
           0.16    0.00    3.38   96.46

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s
avgrq-sz avgqu-sz   await  svctm  %util
/dev/sda5   38.47   0.00 13.20  0.00  413.35    0.00   206.68     0.00
   31.33     4.24   32.17   8.07  10.65
/dev/sda3    0.00 6229.57  0.00 5007.31    0.00 89893.80     0.00
44946.90    17.95    11.73    0.23   0.23 117.17

Test 2): copy a 515 MB file between the same filesystem (/var)

# time cp -p /var/log/testfile.log /var/log/testfile.log.bak
real    0m4.543s
user    0m0.070s
sys     0m3.510s

This translates to 114 MB/s, which is 40 times faster than transfer
rate between 2 separate filesystems.

==output from iostat command==
avg-cpu:  %user   %nice    %sys   %idle
           0.28    0.00   13.46   86.27

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s
avgrq-sz avgqu-sz   await  svctm  %util
/dev/sda5    0.00 12738.98  0.41 853.99    3.31 109377.41     1.65
54688.71   128.02  5029.57  555.09   1.09  93.25

We found that there is a huge difference in IO write rate (i.e. w/s -
the number of write requests that were issued to the device per second):
1) The average write rate is 7191.79 w/s when copying file between
separate filesystems
2) The average write rate is over 853.99 w/s when copying file between
separate filesystems

The same tests were run on RedHat 8.0 (2.4.20-28.7) and the transfer
rate between separate filesystems are the same as the rate between the
same filesystem.

Version-Release number of selected component (if applicable):
2.4.21-20.ELsmp

How reproducible:
Always

Steps to Reproduce:
1. copy file between 2 filesystems: "time cp /filesystem1/large_file
/filesystem2/".
2. "iostat -x /dev/sdaX -x /dev/sdaY 10" (where sdaX is the device
file for filesystem1 and sdaY for filesystem2).
3. Copy file on the same filesystem: "time cp /filesystem1/large_file
/filesystem1/large_file.bak".
4. "iostat -x /dev/sdaX" (where sdaX is the device file for filesystem1).

Actual Results:  Compared with result on the same filesystem,
significant IO degradation (40 slower) found when copying files cross
filesystems.

Expected Results:  The transfer rate should remain the same which is
the behave found on other version of Linux.

Additional info:

===HW information===
CPU: Dual Intel(R) Xeon(TM) CPU 3.06GHz (CPU: L2 cache: 512K)
Thread Mode: Hyper-thread enabled
RAM: 1.5 GB RAM

===SCSI related configuration - captured from dmesg===
SCSI subsystem driver Revision: 1.00
scsi0 : IBM PCI ServeRAID 7.00.15  Build 625 <ServeRAID 6i>
  Vendor: IBM       Model: SERVERAID         Rev: 1.00
  Type:   Direct-Access                      ANSI SCSI revision: 02
  Vendor: IBM       Model: SERVERAID         Rev: 1.00
  Type:   Processor                          ANSI SCSI revision: 02
  Vendor: IBM       Model: 32P0032a S320  1  Rev: 1
  Type:   Processor                          ANSI SCSI revision: 02
Attached scsi disk sda at scsi0, channel 0, id 0, lun 0
SCSI device sda: 71096320 512-byte hdwr sectors (36401 MB)
Partition check:
 sda: sda1 sda2 sda3 sda4 < sda5 sda6 sda7 >
Journalled Block Device driver loaded
kjournald starting.  Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.

===output from lsmod===
Module                  Size  Used by    Not tainted
audit                  90552   1
e1000                  76956   1
ipt_owner               2392   4  (autoclean)
ipt_state               1080   1  (autoclean)
ip_conntrack           29736   1  (autoclean) [ipt_state]
ipt_REJECT              4632   5  (autoclean)
iptable_filter          2412   1  (autoclean)
ip_tables              16544   4  [ipt_owner ipt_state ipt_REJECT
iptable_filter]
microcode               6848   0  (autoclean)
keybdev                 2976   0  (unused)
mousedev                5624   0  (unused)
hid                    22276   0  (unused)
input                   6144   0  [keybdev mousedev hid]
usb-ohci               23176   0  (unused)
usbcore                81056   1  [hid usb-ohci]
ext3                   89960   5
jbd                    55060   5  [ext3]
ips                    45156   6
sd_mod                 13712  12
scsi_mod              114344   2  [ips sd_mod]