Bug 109618 - 3ware raid extremely low throughput
3ware raid extremely low throughput
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 3
Classification: Red Hat
Component: kernel (Show other bugs)
3.0
athlon Linux
medium Severity medium
: ---
: ---
Assigned To: Tom Coughlan
Brian Brock
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2003-11-10 06:50 EST by Mario Lorenz
Modified: 2007-11-30 17:06 EST (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2004-05-11 21:07:45 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
tiobench output, enterprise beta kernel (3.35 KB, text/plain)
2003-11-11 06:50 EST, Mario Lorenz
no flags Details
tiobench output, home-built enterprise-final kernel (4.15 KB, text/plain)
2003-11-11 06:51 EST, Mario Lorenz
no flags Details
Tiobench output, Fedora kernel (3.32 KB, text/plain)
2003-11-11 06:51 EST, Mario Lorenz
no flags Details

  None (edit)
Description Mario Lorenz 2003-11-10 06:50:04 EST
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.0.2)
Gecko/20030708

Description of problem:

[ I meant to file this bug against the Public Beta since I do not
  (yet) have a "real" RHEL to test. Please relabel this bug as
  apropriate ]

To evaluate switching our systems to RHEL ES, I installed the public
beta (taroon) on one of our test machines (3 yr old ASUS MB, Athlon
700MHz, 512MB RAM) and a 3Ware 7410, running a RAID-1 over two 40GB
IBM IDE drives. As the system performed generally sluggish,
nonwithstanding that I run a beta version, I investigated.

Turns out that the reason is abysmally low throughput on said 3ware
RAID. This problem is specific to the enterprise kernel, putting a
RH9 kernel there, and the throughput is ok and the system doesnt
feel sluggish (same if I move the ide to the standard IDE port, even
tho thats only UDMA-66)


Version-Release number of selected component (if applicable):
kernel-2.4.21-1.1931.2.399.ent

How reproducible:
Always

Steps to Reproduce:
1.Install System on hardware
2.run hdparm -t /dev/hda
3.
    

Actual Results:  [root@taroon3 root]# /sbin/hdparm -t /dev/sda

/dev/sda:
 Timing buffered disk reads:   42 MB in  3.04 seconds =  13.82 MB/sec


Expected Results:  Throughput at around 35MB/sec, which is reachable
using the standard RH9 kernel on the same box.

Unless you tell me that there was a bug fixed in the release kernel
(which I obviously cannot yet test) this is a show-stopper here.


Additional info:

[root@taroon3 root]# uname -a
Linux taroon3.km3.de 2.4.21-1.1931.2.399.ent #1 Wed Aug 20 15:51:09
EDT 2003 i686 athlon i386 GNU/Linux

[root@taroon3 root]# cat /proc/cpuinfo 
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 6
model           : 2
model name      : AMD Athlon(tm) Processor
stepping        : 1
cpu MHz         : 704.953
cache size      : 512 KB
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 1
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 sep mtrr pge mca
cmov pat pse36 mmx fxsr syscall mmxext 3dnowext 3dnow
bogomips        : 1405.


[root@taroon3 root]# lspci
00:00.0 Host bridge: Advanced Micro Devices [AMD] AMD-751 [Irongate]
System Controller (rev 25)
00:01.0 PCI bridge: Advanced Micro Devices [AMD] AMD-751 [Irongate]
AGP Bridge (rev 01)
00:04.0 ISA bridge: VIA Technologies, Inc. VT82C686 [Apollo Super
South] (rev 1b)
00:04.1 IDE interface: VIA Technologies, Inc.
VT82C586A/B/VT82C686/A/B/VT8233/A/C/VT8235 PIPC Bus Master IDE (rev 06)
00:04.2 USB Controller: VIA Technologies, Inc. USB (rev 0e)
00:04.3 USB Controller: VIA Technologies, Inc. USB (rev 0e)
00:04.4 SMBus: VIA Technologies, Inc. VT82C686 [Apollo Super ACPI]
(rev 20)
00:0d.0 RAID bus controller: 3ware Inc 3ware 7000-series ATA-RAID (rev 01)
00:10.0 Ethernet controller: 3Com Corporation 3c905C-TX/TX-M [Tornado]
(rev 74)
01:05.0 VGA compatible controller: S3 Inc. 86c368 [Trio 3D/2X] (rev 02)
74
Comment 1 Arjan van de Ven 2003-11-10 13:31:03 EST
hdparm is the worst possible benchmarker; could you please use
tiobench (tiobech.sf.net) instead ?
Comment 2 Mario Lorenz 2003-11-11 06:50:14 EST
Created attachment 95898 [details]
tiobench output, enterprise beta kernel
Comment 3 Mario Lorenz 2003-11-11 06:51:01 EST
Created attachment 95899 [details]
tiobench output, home-built enterprise-final kernel
Comment 4 Mario Lorenz 2003-11-11 06:51:59 EST
Created attachment 95900 [details]
Tiobench output, Fedora kernel
Comment 5 Arjan van de Ven 2003-11-11 06:55:27 EST
they all seem to achieve 33Mbyte/sec-ish.....
Comment 6 Mario Lorenz 2003-11-11 07:01:26 EST
I have added 3 tiobench outputs, first of them is against the 
RHEL beta kernel, second is on a home-compiled RHEL final kernel,
third is the Fedora Core kernel (that showed 35MB/sec on hdparm -t)
Please be aware that I had to break the RAID, so its currently
degraded. But the comparison is still acurate (same condition  for all
3 tests)

I do agree that these look much better than the hdparm output, in
terms of throughput. However, CPU efficiency on bulk reads is much
better on the Fedora Kernel, and what is probably being the killer in
"subjectively felt system performance" is the exorbitant latency
on sequential writes, compared to the fedora one.
Any comments on that ?

[and I am impressed by the speed of the response - never before have
I seen a Bugzilla Mid-Air collision :]
Comment 7 Arjan van de Ven 2003-11-11 07:08:23 EST
well the average latency isn't that bad on first sight
could you experiment with elvtune a bit?
eg elvtune -r 8 -w 8 /dev/sda (or whatever device it is)
elvtune tunes the balance between throughput and latency, which for
RHEL is focused more on throughput by default
Comment 8 Mario Lorenz 2003-11-13 10:24:27 EST
elvtune did not really help. The worst case figures for latency are
still not much better.

Even tho tiobench does not show that much difference between
the various tests, whatever I do, the enterprise kernel looks to
be slower.
I do not know what impact the filesystem, caching, journaling etc
have when testing witht tiobench, so this might distort the tests.

And when it comes to transfer rate, I still do not see why
tiobench would be a better benchmark than the following (other
than Red Hat consider tiobench _the_ standard workload and optimized
the kernel to show up best with this bench^H^H^Hworkload):

(all repeated 3 times in a row, to account for caching,
 allways measured values varried only insignificantly)

#Enterprise  Kernel, 3Ware Raid:
# time dd if=/dev/sda of=/dev/null bs=4k count=100000
100000+0 records in
100000+0 records out

real    0m28.458s
user    0m0.030s
sys     0m4.870s

#Same Kernel, identical disk connected to on-board IDE-66
time dd if=/dev/hda of=/dev/null bs=4k count=100000
100000+0 records in
100000+0 records out

real    0m11.328s
user    0m0.040s
sys     0m5.420s

#Fedora Kernel, 3Ware disk
time dd if=/dev/sda of=/dev/null bs=4k count=100000
100000+0 records in
100000+0 records out

real    0m11.305s
user    0m0.110s
sys     0m2.820s



Comment 9 Arjan van de Ven 2003-11-13 10:26:41 EST
The main reason why I prefer tiobench over that dd is that your dd
goes to the raw device, which means the RHEL kernel won't try to do
things like readahead etc etc while it will do that for regular files
on a filesystem. 
Comment 10 Tom Coughlan 2004-01-23 10:13:52 EST

*** This bug has been marked as a duplicate of 104633 ***
Comment 11 John Flanagan 2004-05-11 21:07:45 EDT
An errata has been issued which should help the problem described in this bug report. 
This report is therefore being closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files, please follow the link below. You may reopen 
this bug report if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2004-188.html

Note You need to log in before you can comment on or make changes to this bug.