From Bugzilla Helper: User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4.1) Gecko/20031114 Description of problem: Installed RHEL ES 3.0 on a Dell PE 2600 with a LSI raid-kontroller (4/Di). reading disks (dd if=/dev/sda of=/dev/null bs=8k) yields only 11MB/s. This is far better than kernel-smp-2.4.21-4.EL which only managed to read ~ 5MB/s. Write performance is ok, but far from great. About 25MB/s sustained. Sustained writing eats all CPU and makes the machine non-reponsive. Tried both the megaraid and the megaraid2 driver with no luck. Megaraid2 seems only to use more CPU. UP kernel behaves the same. 2.6.1-rc2 (from arjanv) performs very well. Read performance is 60MB/s. Version-Release number of selected component (if applicable): kernel-smp-2.4.21-4.0.2.EL How reproducible: Always Steps to Reproduce: 1. Read 200MB of disk time dd if=/dev/sda of=/dev/null bs=8k count=25600 Actual Results: real 0m17.843s Expected Results: real 0m5.002s Additional info:
doing /dev/sda IO is nor a realistic testcase since it avoids all filesystem optimisations that increase performance (eg no or little readahead etc etc etc)
Going through the filesystem yields 20MB/s which still is way below expected results. A mediocre IDE-drive yields approx. 45MB/s. Linux 2.6 outperforms 2.4.21-4.0.2.EL grossly. It should not do so.
I have also experienced the problem on a Dell 2650 with PERC 4 DC on a Power Vault. See http://marc.theaimsgroup.com/?l=linux-scsi&m=107408386620006&w=2 for some numbers georgios.no
I also am having the same problem on a customers box, IBM Server Raid 6m + raid 1 and raid 1e running as a database server, only getting about 10-15 MB/sec, will redhat fix this problem or will i just have to use the stock 2.6 kernel ????
I tried the new kernel from RHEL ES 3 OU1 (2.4.21-9.EL). Still same read performance with megaraid2 driver (~10MB/s). Results consistent for both bonnie++ and dd tests. plain-vanilla 2.4.24 kernel does not show this performance problem. Penguin Computing 225 Dual P/III-1266 4GB RAM MegaRAID Express 500 (RAID1: 2x Hitachi DK32DJ-18MC, RAID5: 4xSeagate ST336607LC) MegaRAID 320-2 in Dell PV220S (RAID5+1: 6x Maxtor Atlas10k4_73SCA) system up2date as of today. Is there a RH recommended work-around for this issue?
As Arjan said above, doing I/O to the device special file is not generally representative of real workloads. Consider these results, done on a Dell 1650 with a PERC3 and megaraid(1): # time dd if=/dev/sdc of=/dev/null bs=8k count=25600 25600+0 records in 25600+0 records out real 0m21.497s user 0m0.010s sys 0m2.120s => 9.5 MB/s - this reproduces your awful results, as above. Now assign the same device to raw1, and run the test again. # time dd if=/dev/raw/raw1 of=/dev/null bs=8k count=25600 25600+0 records in 25600+0 records out real 0m10.826s user 0m0.010s sys 0m0.290s => 18.8 MB/s - raw is twice as good, but still not great. Try it with larger I/Os: # time dd if=/dev/raw/raw1 of=/dev/null bs=64k count=3200 3200+0 records in 3200+0 records out real 0m4.544s user 0m0.000s sys 0m0.090s = > 44.4 MB/sec - not bad - larger I/O to the raw device produces reasonable results. Increase I/O size again: # time dd if=/dev/raw/raw1 of=/dev/null bs=256k count=800 800+0 records in 800+0 records out real 0m2.450s user 0m0.000s sys 0m0.120s => 82 MB/sec. The reason RHEL 3 is different from the stock 2.4 and 2.6 kernels is because of a performance tuning change that was done in RHEL 3. This change defers the combining of small I/Os into larger I/Os until later, when it can be done most efficiently (for some workloads). A disadvantage of this approach is that in some situations single-stream sequential I/O may not be combined at all. We are continuing to review the impact of this patch on real workloads. We will look at your iozone and bonnie++ results in more detail. If you have additional results for your actual workload we would be interested in reviewing them. Doug, this may be considered a duplicate of bug 104633. If you agree, you may want to mark it as such.
Tom, you are correct, this is a dup of 104633. Marking as such. *** This bug has been marked as a duplicate of 104633 ***
An errata has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on the solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2004-188.html