Description of problem:
SCSI I/O Performance drops down after moving from RHAS 2.1 to RHEL3.
 
RHEL3 and RHAS2.1 updated to latest available errata running on dual
cpu DELL PowerEdge 2650.
ALL different kernels run on exactly same HW.
sda is HW RAID 5 device connected via DELL Perc3 card using aacraid
driver.
There are also 2 qla2300 cards connected via fiberchannel to EMC.

The performance difference between different kernels is similar for
qla2300 and aacraid devices. 
Customer is running oracle and complains about huge performance
decrease on RHEL3
 
How reproducible:
as test taken:
hdparm -t /dev/sda 
and 
time dd if=/dev/zero of=/tmp/dd-test bs=1M count=1000
  
Actual results:

2.4.9-e.40smp (RH AS 2.1)
  hdparm ~ 50 MB/sec

  dd test: 
     real    0m7.637s
     user    0m0.000s
     sys     0m7.630s

2.4.21-15.ELsmp

  hdparm ~ 24 MB/sec

  dd test:
    real    0m17.890s
    user    0m0.000s
    sys     0m6.190s
    

2.4.21-9.EL.noaffine2smp
  hdpartm ~24 MB/sec

2.4.22-1.2188.nptlsmp
  hdparm ~ 66MB /sec

Additional info:
lspci output

00:00.0 Host bridge: ServerWorks CNB20-HE Host Bridge (rev 33)
00:00.1 Host bridge: ServerWorks CNB20-HE Host Bridge
00:00.2 Host bridge: ServerWorks CNB20-HE Host Bridge
00:04.0 Class ff00: Dell Computer Corporation Remote Access
Controller:ERA or ERA/O
00:04.1 Class ff00: Dell Computer Corporation Remote Access Controller
00:04.2 Class ff00: Dell Computer Corporation BMC/SMIC device
00:0e.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27)
00:0f.0 Host bridge: ServerWorks CSB5 South Bridge (rev 93)
00:0f.1 IDE interface: ServerWorks CSB5 IDE Controller (rev 93)
00:0f.2 USB Controller: ServerWorks OSB4/CSB5 OHCI USB Controller (rev 05)
00:0f.3 ISA bridge: ServerWorks GCLE Host Bridge
00:10.0 Host bridge: ServerWorks: Unknown device 0101 (rev 05)
00:10.2 Host bridge: ServerWorks: Unknown device 0101 (rev 05)
00:11.0 Host bridge: ServerWorks: Unknown device 0101 (rev 05)
00:11.2 Host bridge: ServerWorks: Unknown device 0101 (rev 05)
01:06.0 Fibre Channel: QLogic Corp. QLA2312 Fibre Channel Adapter (rev 02)
01:08.0 Fibre Channel: QLogic Corp. QLA2312 Fibre Channel Adapter (rev 02)
02:06.0 Ethernet controller: Intel Corp. 82546EB Gigabit Ethernet
Controller (Copper) (rev 01)
02:06.1 Ethernet controller: Intel Corp. 82546EB Gigabit Ethernet
Controller (Copper) (rev 01)
03:06.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5703
Gigabit Ethernet (rev 02)
03:08.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5703
Gigabit Ethernet (rev 02)
04:08.0 PCI bridge: Intel Corp.: Unknown device 0309 (rev 01)
04:08.1 RAID bus controller: Dell Computer Corporation PowerEdge
Expandable RAID Controller 3 (rev 01)