Bug 113171 - lousy read performance on megaraid with 2.4.21-4.0.2.EL
lousy read performance on megaraid with 2.4.21-4.0.2.EL
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 3
Classification: Red Hat
Component: kernel (Show other bugs)
3.0
i686 Linux
medium Severity medium
: ---
: ---
Assigned To: Tom Coughlan
Brian Brock
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2004-01-09 04:09 EST by Per Andreas Buer
Modified: 2007-11-30 17:07 EST (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2004-05-11 21:08:16 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Per Andreas Buer 2004-01-09 04:09:07 EST
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4.1)
Gecko/20031114

Description of problem:
Installed RHEL ES 3.0 on a Dell PE 2600 with a LSI raid-kontroller
(4/Di). 

reading disks (dd if=/dev/sda of=/dev/null bs=8k) yields only 11MB/s.
This is far better than kernel-smp-2.4.21-4.EL which only managed to
read ~ 5MB/s. 

Write performance is ok, but far from great. About 25MB/s sustained. 
Sustained writing eats all CPU and makes the machine non-reponsive.

Tried both the megaraid and the megaraid2 driver with no luck.
Megaraid2 seems only to use more CPU. UP kernel behaves the same.

2.6.1-rc2 (from arjanv) performs very well. Read performance is 60MB/s.



Version-Release number of selected component (if applicable):
kernel-smp-2.4.21-4.0.2.EL

How reproducible:
Always

Steps to Reproduce:
1. Read 200MB of disk
time dd if=/dev/sda of=/dev/null bs=8k count=25600



Actual Results:  real    0m17.843s

Expected Results:  real    0m5.002s


Additional info:
Comment 1 Arjan van de Ven 2004-01-09 08:29:50 EST
doing /dev/sda IO is nor a realistic testcase since it avoids all
filesystem optimisations that increase performance (eg no or little
readahead etc etc etc)
Comment 2 Per Andreas Buer 2004-01-09 09:14:37 EST
Going through the filesystem yields 20MB/s which still is way below
expected results. A mediocre IDE-drive yields approx. 45MB/s. 

Linux 2.6 outperforms 2.4.21-4.0.2.EL grossly. It should not do so. 
Comment 3 George B. Magklaras 2004-01-16 02:55:48 EST
I have also experienced the problem on a Dell 2650 with PERC 4 DC on a
Power Vault. 

See http://marc.theaimsgroup.com/?l=linux-scsi&m=107408386620006&w=2

for some numbers 

georgios@biotek.uio.no
Comment 4 Davey Goode 2004-01-20 16:49:03 EST
I also am having the same problem on a customers box, IBM Server Raid 
6m + raid 1 and raid 1e running as a database server, only getting 
about 10-15 MB/sec, will redhat fix this problem or will i just have 
to use the stock 2.6 kernel ????
Comment 5 Bernhard Beck 2004-01-21 13:21:40 EST
I tried the new kernel from RHEL ES 3 OU1 (2.4.21-9.EL). 
Still same read performance with megaraid2 driver (~10MB/s). Results 
consistent for both bonnie++ and dd tests.
plain-vanilla 2.4.24 kernel does not show this performance problem. 

Penguin Computing 225
Dual P/III-1266
4GB RAM
MegaRAID Express 500 (RAID1: 2x Hitachi DK32DJ-18MC, RAID5: 4xSeagate 
ST336607LC)
MegaRAID 320-2 in Dell PV220S (RAID5+1: 6x Maxtor Atlas10k4_73SCA)

system up2date as of today. 

Is there a RH recommended work-around for this issue?
Comment 6 Tom Coughlan 2004-01-22 10:00:00 EST
As Arjan said above, doing I/O to the device special file is
not generally representative of real workloads.  Consider
these results, done on a Dell 1650 with a PERC3 and
megaraid(1):  

# time dd if=/dev/sdc of=/dev/null bs=8k count=25600
25600+0 records in
25600+0 records out

real    0m21.497s
user    0m0.010s
sys     0m2.120s

=> 9.5 MB/s - this reproduces your awful results, as above.

Now assign the same device to raw1, and run the test again.

# time dd if=/dev/raw/raw1 of=/dev/null bs=8k count=25600
25600+0 records in
25600+0 records out

real    0m10.826s
user    0m0.010s
sys     0m0.290s

=> 18.8 MB/s - raw is twice as good, but still not great.

Try it with larger I/Os:

# time dd if=/dev/raw/raw1 of=/dev/null bs=64k count=3200
3200+0 records in
3200+0 records out

real    0m4.544s
user    0m0.000s
sys     0m0.090s

= > 44.4 MB/sec - not bad - larger I/O to the raw device
produces reasonable results.  Increase I/O size again:

# time dd if=/dev/raw/raw1 of=/dev/null bs=256k count=800
800+0 records in
800+0 records out

real    0m2.450s
user    0m0.000s
sys     0m0.120s

=> 82 MB/sec.

The reason RHEL 3 is different from the stock 2.4 and 
2.6 kernels is because of a performance tuning change that
was done in RHEL 3.  This change defers the combining of
small I/Os into larger I/Os until later, when it can be done
most efficiently (for some workloads). A disadvantage of
this approach is that in some situations single-stream
sequential I/O may not be combined at all. 

We are continuing to review the impact of this patch on
real workloads.  We will look at your iozone and
bonnie++ results in more detail.  If you have additional
results for your actual workload we would be interested in
reviewing them.

Doug, this may be considered a duplicate of bug 104633. 
If you agree, you may want to mark it as such.
Comment 7 Doug Ledford 2004-01-22 11:21:06 EST
Tom, you are correct, this is a dup of 104633.  Marking as such.

*** This bug has been marked as a duplicate of 104633 ***
Comment 8 John Flanagan 2004-05-11 21:08:16 EDT
An errata has been issued which should help the problem described in this bug report. 
This report is therefore being closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files, please follow the link below. You may reopen 
this bug report if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2004-188.html

Note You need to log in before you can comment on or make changes to this bug.