Bug 113171 - lousy read performance on megaraid with 2.4.21-4.0.2.EL
lousy read performance on megaraid with 2.4.21-4.0.2.EL
Product: Red Hat Enterprise Linux 3
Classification: Red Hat
Component: kernel (Show other bugs)
i686 Linux
medium Severity medium
: ---
: ---
Assigned To: Tom Coughlan
Brian Brock
Depends On:
  Show dependency treegraph
Reported: 2004-01-09 04:09 EST by Per Andreas Buer
Modified: 2007-11-30 17:07 EST (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2004-05-11 21:08:16 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Per Andreas Buer 2004-01-09 04:09:07 EST
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4.1)

Description of problem:
Installed RHEL ES 3.0 on a Dell PE 2600 with a LSI raid-kontroller

reading disks (dd if=/dev/sda of=/dev/null bs=8k) yields only 11MB/s.
This is far better than kernel-smp-2.4.21-4.EL which only managed to
read ~ 5MB/s. 

Write performance is ok, but far from great. About 25MB/s sustained. 
Sustained writing eats all CPU and makes the machine non-reponsive.

Tried both the megaraid and the megaraid2 driver with no luck.
Megaraid2 seems only to use more CPU. UP kernel behaves the same.

2.6.1-rc2 (from arjanv) performs very well. Read performance is 60MB/s.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. Read 200MB of disk
time dd if=/dev/sda of=/dev/null bs=8k count=25600

Actual Results:  real    0m17.843s

Expected Results:  real    0m5.002s

Additional info:
Comment 1 Arjan van de Ven 2004-01-09 08:29:50 EST
doing /dev/sda IO is nor a realistic testcase since it avoids all
filesystem optimisations that increase performance (eg no or little
readahead etc etc etc)
Comment 2 Per Andreas Buer 2004-01-09 09:14:37 EST
Going through the filesystem yields 20MB/s which still is way below
expected results. A mediocre IDE-drive yields approx. 45MB/s. 

Linux 2.6 outperforms 2.4.21-4.0.2.EL grossly. It should not do so. 
Comment 3 George B. Magklaras 2004-01-16 02:55:48 EST
I have also experienced the problem on a Dell 2650 with PERC 4 DC on a
Power Vault. 

See http://marc.theaimsgroup.com/?l=linux-scsi&m=107408386620006&w=2

for some numbers 

Comment 4 Davey Goode 2004-01-20 16:49:03 EST
I also am having the same problem on a customers box, IBM Server Raid 
6m + raid 1 and raid 1e running as a database server, only getting 
about 10-15 MB/sec, will redhat fix this problem or will i just have 
to use the stock 2.6 kernel ????
Comment 5 Bernhard Beck 2004-01-21 13:21:40 EST
I tried the new kernel from RHEL ES 3 OU1 (2.4.21-9.EL). 
Still same read performance with megaraid2 driver (~10MB/s). Results 
consistent for both bonnie++ and dd tests.
plain-vanilla 2.4.24 kernel does not show this performance problem. 

Penguin Computing 225
Dual P/III-1266
MegaRAID Express 500 (RAID1: 2x Hitachi DK32DJ-18MC, RAID5: 4xSeagate 
MegaRAID 320-2 in Dell PV220S (RAID5+1: 6x Maxtor Atlas10k4_73SCA)

system up2date as of today. 

Is there a RH recommended work-around for this issue?
Comment 6 Tom Coughlan 2004-01-22 10:00:00 EST
As Arjan said above, doing I/O to the device special file is
not generally representative of real workloads.  Consider
these results, done on a Dell 1650 with a PERC3 and

# time dd if=/dev/sdc of=/dev/null bs=8k count=25600
25600+0 records in
25600+0 records out

real    0m21.497s
user    0m0.010s
sys     0m2.120s

=> 9.5 MB/s - this reproduces your awful results, as above.

Now assign the same device to raw1, and run the test again.

# time dd if=/dev/raw/raw1 of=/dev/null bs=8k count=25600
25600+0 records in
25600+0 records out

real    0m10.826s
user    0m0.010s
sys     0m0.290s

=> 18.8 MB/s - raw is twice as good, but still not great.

Try it with larger I/Os:

# time dd if=/dev/raw/raw1 of=/dev/null bs=64k count=3200
3200+0 records in
3200+0 records out

real    0m4.544s
user    0m0.000s
sys     0m0.090s

= > 44.4 MB/sec - not bad - larger I/O to the raw device
produces reasonable results.  Increase I/O size again:

# time dd if=/dev/raw/raw1 of=/dev/null bs=256k count=800
800+0 records in
800+0 records out

real    0m2.450s
user    0m0.000s
sys     0m0.120s

=> 82 MB/sec.

The reason RHEL 3 is different from the stock 2.4 and 
2.6 kernels is because of a performance tuning change that
was done in RHEL 3.  This change defers the combining of
small I/Os into larger I/Os until later, when it can be done
most efficiently (for some workloads). A disadvantage of
this approach is that in some situations single-stream
sequential I/O may not be combined at all. 

We are continuing to review the impact of this patch on
real workloads.  We will look at your iozone and
bonnie++ results in more detail.  If you have additional
results for your actual workload we would be interested in
reviewing them.

Doug, this may be considered a duplicate of bug 104633. 
If you agree, you may want to mark it as such.
Comment 7 Doug Ledford 2004-01-22 11:21:06 EST
Tom, you are correct, this is a dup of 104633.  Marking as such.

*** This bug has been marked as a duplicate of 104633 ***
Comment 8 John Flanagan 2004-05-11 21:08:16 EDT
An errata has been issued which should help the problem described in this bug report. 
This report is therefore being closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files, please follow the link below. You may reopen 
this bug report if the solution does not work for you.


Note You need to log in before you can comment on or make changes to this bug.