Red Hat Bugzilla – Bug 113171
lousy read performance on megaraid with 2.4.21-4.0.2.EL
Last modified: 2007-11-30 17:07:00 EST
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4.1)
Description of problem:
Installed RHEL ES 3.0 on a Dell PE 2600 with a LSI raid-kontroller
reading disks (dd if=/dev/sda of=/dev/null bs=8k) yields only 11MB/s.
This is far better than kernel-smp-2.4.21-4.EL which only managed to
read ~ 5MB/s.
Write performance is ok, but far from great. About 25MB/s sustained.
Sustained writing eats all CPU and makes the machine non-reponsive.
Tried both the megaraid and the megaraid2 driver with no luck.
Megaraid2 seems only to use more CPU. UP kernel behaves the same.
2.6.1-rc2 (from arjanv) performs very well. Read performance is 60MB/s.
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. Read 200MB of disk
time dd if=/dev/sda of=/dev/null bs=8k count=25600
Actual Results: real 0m17.843s
Expected Results: real 0m5.002s
doing /dev/sda IO is nor a realistic testcase since it avoids all
filesystem optimisations that increase performance (eg no or little
readahead etc etc etc)
Going through the filesystem yields 20MB/s which still is way below
expected results. A mediocre IDE-drive yields approx. 45MB/s.
Linux 2.6 outperforms 2.4.21-4.0.2.EL grossly. It should not do so.
I have also experienced the problem on a Dell 2650 with PERC 4 DC on a
for some numbers
I also am having the same problem on a customers box, IBM Server Raid
6m + raid 1 and raid 1e running as a database server, only getting
about 10-15 MB/sec, will redhat fix this problem or will i just have
to use the stock 2.6 kernel ????
I tried the new kernel from RHEL ES 3 OU1 (2.4.21-9.EL).
Still same read performance with megaraid2 driver (~10MB/s). Results
consistent for both bonnie++ and dd tests.
plain-vanilla 2.4.24 kernel does not show this performance problem.
Penguin Computing 225
MegaRAID Express 500 (RAID1: 2x Hitachi DK32DJ-18MC, RAID5: 4xSeagate
MegaRAID 320-2 in Dell PV220S (RAID5+1: 6x Maxtor Atlas10k4_73SCA)
system up2date as of today.
Is there a RH recommended work-around for this issue?
As Arjan said above, doing I/O to the device special file is
not generally representative of real workloads. Consider
these results, done on a Dell 1650 with a PERC3 and
# time dd if=/dev/sdc of=/dev/null bs=8k count=25600
25600+0 records in
25600+0 records out
=> 9.5 MB/s - this reproduces your awful results, as above.
Now assign the same device to raw1, and run the test again.
# time dd if=/dev/raw/raw1 of=/dev/null bs=8k count=25600
25600+0 records in
25600+0 records out
=> 18.8 MB/s - raw is twice as good, but still not great.
Try it with larger I/Os:
# time dd if=/dev/raw/raw1 of=/dev/null bs=64k count=3200
3200+0 records in
3200+0 records out
= > 44.4 MB/sec - not bad - larger I/O to the raw device
produces reasonable results. Increase I/O size again:
# time dd if=/dev/raw/raw1 of=/dev/null bs=256k count=800
800+0 records in
800+0 records out
=> 82 MB/sec.
The reason RHEL 3 is different from the stock 2.4 and
2.6 kernels is because of a performance tuning change that
was done in RHEL 3. This change defers the combining of
small I/Os into larger I/Os until later, when it can be done
most efficiently (for some workloads). A disadvantage of
this approach is that in some situations single-stream
sequential I/O may not be combined at all.
We are continuing to review the impact of this patch on
real workloads. We will look at your iozone and
bonnie++ results in more detail. If you have additional
results for your actual workload we would be interested in
Doug, this may be considered a duplicate of bug 104633.
If you agree, you may want to mark it as such.
Tom, you are correct, this is a dup of 104633. Marking as such.
*** This bug has been marked as a duplicate of 104633 ***
An errata has been issued which should help the problem described in this bug report.
This report is therefore being closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files, please follow the link below. You may reopen
this bug report if the solution does not work for you.