114052 – Raw device performance poor under WS 3 Dreamworks IT#29689

Bug 114052 - Raw device performance poor under WS 3 Dreamworks IT#29689

Summary: Raw device performance poor under WS 3 Dreamworks IT#29689

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 3
Classification:	Red Hat
Component:	kernel
Sub Component:
Version:	3.0
Hardware:	All
OS:	Linux
Priority:	high
Severity:	high
Target Milestone:	---
Assignee:	Doug Ledford
QA Contact:	Brian Brock
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2004-01-21 23:08 UTC by Mahesh Kunjal
Modified:	2007-11-30 22:07 UTC (History)
CC List:	3 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2004-05-12 01:08:22 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHSA-2004:188	0	normal	SHIPPED_LIVE	Important: Updated kernel packages available for Red Hat Enterprise Linux 3 Update 2	2004-05-11 04:00:00 UTC

Description Mahesh Kunjal 2004-01-21 23:08:16 UTC

SCSI performance using the raw device under RH WS 3 seems to be
degraded from what it was under RH 7.2.  It seems to peek around about
152 MB/s vs. 290 MB/s under RH 7.2, this is with the same hardware. 
They have tried both the on board LSI MPT Fusion controller and an
Adaptec 39160 controller with similar poor results.  They think it
might be the raw device, but are not sure and have probably eliminated
the SCSI drivers themselves.


Customer also tried compiling Linus-series 2.4.21 and 2.4.22 kernels
and confirmed that performance is fine against that baseline.

/etc/sysconfig/rawdevices looks like:

/dev/raw/raw1 /dev/md0


/etc/raidtab looks like:

raiddev             /dev/md0
raid-level                  0
nr-raid-disks               4
chunk-size                  64k
persistent-superblock       1
nr-spare-disks              0
   device          /dev/sdb
   raid-disk     0
   device          /dev/sdc
   raid-disk     1
   device          /dev/sdd
   raid-disk     2
   device          /dev/sde
   raid-disk     3

cat /proc/scsi/scsi looks like:

Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
 Vendor: SEAGATE  Model: ST318453LW       Rev: HPS1
 Type:   Direct-Access                    ANSI SCSI revision: 03
Host: scsi0 Channel: 00 Id: 01 Lun: 00
 Vendor: SEAGATE  Model: ST336753LW       Rev: 0003
 Type:   Direct-Access                    ANSI SCSI revision: 03
Host: scsi0 Channel: 00 Id: 02 Lun: 00
 Vendor: SEAGATE  Model: ST336753LW       Rev: 0003
 Type:   Direct-Access                    ANSI SCSI revision: 03
Host: scsi1 Channel: 00 Id: 01 Lun: 00
 Vendor: SEAGATE  Model: ST336753LW       Rev: 0003
 Type:   Direct-Access                    ANSI SCSI revision: 03
Host: scsi1 Channel: 00 Id: 02 Lun: 00
 Vendor: SEAGATE  Model: ST336753LW       Rev: 0003
 Type:   Direct-Access                    ANSI SCSI revision: 03

HOW THIS WAS TESTED - 
The test is available at:
http://tiobench.sourceforge.net/

You have to modify the makefile slightly to allow for large mem access
when building.
#DEFINES=-DLARGEFILES

A common test line is:
./tiotest -d /mnt/rawdevice  -t 7 -f 4096 -b 1048576

This can be tuned by several parameters which are available via the
--help command to optimize results.
The main idea here is to saturate the pipe of the raw device between
sends, which is the way to guage true
throughput. Obviously the larger your pipe the larger the packets you
want to send per transmission.

FOLLOWING is the result for 2.4.21 kernel

./tiotest -d /movie -t 1 -b 1048575 -f 4096
Tiotest results for 1 concurrent io threads:
,----------------------------------------------------------------------.
| Item                  | Time     | Rate         | Usr CPU  | Sys CPU |
+-----------------------+----------+--------------+----------+---------+
| Write        4096 MBs |   21.0 s | 195.103 MB/s |   0.1 %  |  49.3 % |
| Random Write 1000 MBs |   15.3 s |  65.285 MB/s |   0.0 %  |  15.9 % |
| Read         4096 MBs |   19.2 s | 213.670 MB/s |   0.2 %  |  75.5 % |
| Random Read  1000 MBs |    8.3 s | 119.860 MB/s |   0.1 %  |  37.8 % |
`----------------------------------------------------------------------'
Tiotest latency results:
,-------------------------------------------------------------------------.
| Item         | Average latency | Maximum latency | % >2 sec | % >10
sec |
+--------------+-----------------+-----------------+----------+-----------+
| Write        |        4.544 ms |      904.253 ms |  0.00000 |  
0.00000 |
| Random Write |       14.281 ms |       76.098 ms |  0.00000 |  
0.00000 |
| Read         |        4.676 ms |       23.379 ms |  0.00000 |  
0.00000 |
| Random Read  |        8.324 ms |       95.415 ms |  0.00000 |  
0.00000 |
|--------------+-----------------+-----------------+----------+-----------|
| Total        |        5.923 ms |      904.253 ms |  0.00000 |  
0.00000 |
`--------------+-----------------+-----------------+----------+-----------'

oakley [/var/tmp/tiobench-0.3.3](SHARK)(10)> ./tiotest -R -d /dev/md0
-t 1 -b 1048575 -f 4096
Tiotest results for 1 concurrent io threads:
,----------------------------------------------------------------------.
| Item                  | Time     | Rate         | Usr CPU  | Sys CPU |
+-----------------------+----------+--------------+----------+---------+
| Write        4096 MBs |   41.3 s |  99.128 MB/s |   0.0 %  |  22.6 % |
| Random Write 1000 MBs |   17.3 s |  57.768 MB/s |   0.1 %  |  12.8 % |
| Read         4096 MBs |   18.0 s | 227.463 MB/s |   0.2 %  |  73.1 % |
| Random Read  1000 MBs |    8.8 s | 113.064 MB/s |   0.1 %  |  34.1 % |
`----------------------------------------------------------------------'
Tiotest latency results:
,-------------------------------------------------------------------------.
| Item         | Average latency | Maximum latency | % >2 sec | % >10
sec |
+--------------+-----------------+-----------------+----------+-----------+
| Write        |        9.943 ms |       95.471 ms |  0.00000 |  
0.00000 |
| Random Write |       16.422 ms |      138.189 ms |  0.00000 |  
0.00000 |
| Read         |        4.392 ms |       16.970 ms |  0.00000 |  
0.00000 |
| Random Read  |        8.825 ms |       47.702 ms |  0.00000 |  
0.00000 |
|--------------+-----------------+-----------------+----------+-----------|
| Total        |        8.238 ms |      138.189 ms |  0.00000 |  
0.00000 |
`--------------+-----------------+-----------------+----------+-----------'

./tiotest -R -d /dev/raw/raw1 -t 1 -b 1048576 -f 4096
Tiotest results for 1 concurrent io threads:
,----------------------------------------------------------------------.
| Item                  | Time     | Rate         | Usr CPU  | Sys CPU |
+-----------------------+----------+--------------+----------+---------+
| Write        4096 MBs |   33.5 s | 122.205 MB/s |   0.0 %  |  13.2 % |
| Random Write 1000 MBs |   16.9 s |  59.202 MB/s |   0.0 %  |   5.2 % |
| Read         4096 MBs |   27.0 s | 151.801 MB/s |   0.0 %  |  12.8 % |
| Random Read  1000 MBs |   10.2 s |  98.126 MB/s |   0.0 %  |   8.8 % |
`----------------------------------------------------------------------'
Tiotest latency results:
,-------------------------------------------------------------------------.
| Item         | Average latency | Maximum latency | % >2 sec | % >10
sec |
+--------------+-----------------+-----------------+----------+-----------+
| Write        |        8.181 ms |      129.457 ms |  0.00000 |  
0.00000 |
| Random Write |       16.887 ms |      135.257 ms |  0.00000 |  
0.00000 |
| Read         |        6.586 ms |       13.119 ms |  0.00000 |  
0.00000 |
| Random Read  |       10.188 ms |       93.608 ms |  0.00000 |  
0.00000 |
|--------------+-----------------+-----------------+----------+-----------|
| Total        |        8.591 ms |      135.257 ms |  0.00000 |  
0.00000 |
`--------------+-----------------+-----------------+----------+-----------'

./tiotest -R -d /dev/md0 -t 1 -b 1048576 -f 4096
Tiotest results for 1 concurrent io threads:
,----------------------------------------------------------------------.
| Item                  | Time     | Rate         | Usr CPU  | Sys CPU |
+-----------------------+----------+--------------+----------+---------+
| Write        4096 MBs |   16.5 s | 248.086 MB/s |   0.3 %  |  55.3 % |
| Random Write 1000 MBs |    8.1 s | 123.448 MB/s |   0.1 %  |  25.6 % |
| Read         4096 MBs |   17.6 s | 232.478 MB/s |   0.2 %  |  76.3 % |
| Random Read  1000 MBs |    8.4 s | 119.276 MB/s |   0.0 %  |  37.2 % |
`----------------------------------------------------------------------'
Tiotest latency results:
,-------------------------------------------------------------------------.
| Item         | Average latency | Maximum latency | % >2 sec | % >10
sec |
+--------------+-----------------+-----------------+----------+-----------+
| Write        |        3.411 ms |      332.483 ms |  0.00000 |  
0.00000 |
| Random Write |        5.788 ms |      460.008 ms |  0.00000 |  
0.00000 |
| Read         |        4.297 ms |       14.304 ms |  0.00000 |  
0.00000 |
| Random Read  |        8.375 ms |       14.837 ms |  0.00000 |  
0.00000 |
|--------------+-----------------+-----------------+----------+-----------|
| Total        |        4.487 ms |      460.008 ms |  0.00000 |  
0.00000 |
`--------------+-----------------+-----------------+----------+-----------'

RESULTS FROM 2.6.0-test11 kernel

./tiotest -d /movie -t 1 -b 1048575 -f 4096
Tiotest results for 1 concurrent io threads:
,----------------------------------------------------------------------.
| Item                  | Time     | Rate         | Usr CPU  | Sys CPU |
+-----------------------+----------+--------------+----------+---------+
| Write        4096 MBs |   21.0 s | 195.424 MB/s |   0.1 %  |  44.4 % |
| Random Write 1000 MBs |   10.5 s |  95.033 MB/s |   0.0 %  |  16.0 % |
| Read         4096 MBs |   24.3 s | 168.458 MB/s |   0.1 %  |  38.9 % |
| Random Read  1000 MBs |    7.9 s | 126.029 MB/s |   0.1 %  |  26.7 % |
`----------------------------------------------------------------------'
Tiotest latency results:
,-------------------------------------------------------------------------.
| Item         | Average latency | Maximum latency | % >2 sec | % >10
sec |
+--------------+-----------------+-----------------+----------+-----------+
| Write        |        4.125 ms |      415.368 ms |  0.00000 |  
0.00000 |
| Random Write |        9.259 ms |       93.033 ms |  0.00000 |  
0.00000 |
| Read         |        5.933 ms |       30.779 ms |  0.00000 |  
0.00000 |
| Random Read  |        7.929 ms |       27.271 ms |  0.00000 |  
0.00000 |
|--------------+-----------------+-----------------+----------+-----------|
| Total        |        5.729 ms |      415.368 ms |  0.00000 |  
0.00000 |
`--------------+-----------------+-----------------+----------+-----------'

./tiotest -R -d /dev/md0 -t 1 -b 1048575 -f 4096
Tiotest results for 1 concurrent io threads:
,----------------------------------------------------------------------.
| Item                  | Time     | Rate         | Usr CPU  | Sys CPU |
+-----------------------+----------+--------------+----------+---------+
| Write        4096 MBs |   29.9 s | 136.782 MB/s |   0.0 %  |  24.0 % |
| Random Write 1000 MBs |   12.9 s |  77.349 MB/s |   0.1 %  |  14.6 % |
| Read         4096 MBs |   23.9 s | 171.733 MB/s |   0.1 %  |  41.6 % |
| Random Read  1000 MBs |    9.5 s | 105.334 MB/s |   0.1 %  |  26.4 % |
`----------------------------------------------------------------------'
Tiotest latency results:
,-------------------------------------------------------------------------.
| Item         | Average latency | Maximum latency | % >2 sec | % >10
sec |
+--------------+-----------------+-----------------+----------+-----------+
| Write        |        6.598 ms |      155.516 ms |  0.00000 |  
0.00000 |
| Random Write |        9.014 ms |      161.239 ms |  0.00000 |  
0.00000 |
| Read         |        5.820 ms |       23.064 ms |  0.00000 |  
0.00000 |
| Random Read  |        9.487 ms |       23.564 ms |  0.00000 |  
0.00000 |
|--------------+-----------------+-----------------+----------+-----------|
| Total        |        6.806 ms |      161.239 ms |  0.00000 |  
0.00000 |
`--------------+-----------------+-----------------+----------+-----------'

./tiotest -R -d /dev/raw/raw1 -t 1 -b 1048576 -f 4096
Tiotest results for 1 concurrent io threads:
,----------------------------------------------------------------------.
| Item                  | Time     | Rate         | Usr CPU  | Sys CPU |
+-----------------------+----------+--------------+----------+---------+
| Write        4096 MBs |   17.1 s | 240.096 MB/s |   0.0 %  |   6.0 % |
| Random Write 1000 MBs |   10.4 s |  95.943 MB/s |   0.0 %  |   2.5 % |
| Read         4096 MBs |   14.2 s | 289.157 MB/s |   0.0 %  |  10.0 % |
| Random Read  1000 MBs |    9.4 s | 106.827 MB/s |   0.0 %  |   3.4 % |
`----------------------------------------------------------------------'
Tiotest latency results:
,-------------------------------------------------------------------------.
| Item         | Average latency | Maximum latency | % >2 sec | % >10
sec |
+--------------+-----------------+-----------------+----------+-----------+
| Write        |        4.164 ms |       17.062 ms |  0.00000 |  
0.00000 |
| Random Write |       10.421 ms |       33.531 ms |  0.00000 |  
0.00000 |
| Read         |        3.457 ms |       11.822 ms |  0.00000 |  
0.00000 |
| Random Read  |        9.359 ms |       28.052 ms |  0.00000 |  
0.00000 |
|--------------+-----------------+-----------------+----------+-----------|
| Total        |        5.003 ms |       33.531 ms |  0.00000 |  
0.00000 |
`--------------+-----------------+-----------------+----------+-----------'

./tiotest -R -d /dev/md0 -t 1 -b 1048576 -f 4096
Tiotest results for 1 concurrent io threads:
,----------------------------------------------------------------------.
| Item                  | Time     | Rate         | Usr CPU  | Sys CPU |
+-----------------------+----------+--------------+----------+---------+
| Write        4096 MBs |   21.1 s | 193.962 MB/s |   0.0 %  |  41.6 % |
| Random Write 1000 MBs |    7.0 s | 142.856 MB/s |   0.1 %  |  31.4 % |
| Read         4096 MBs |   30.1 s | 136.219 MB/s |   0.1 %  |  30.8 % |
| Random Read  1000 MBs |   11.1 s |  89.860 MB/s |   0.0 %  |  21.3 % |
`----------------------------------------------------------------------'
Tiotest latency results:
,-------------------------------------------------------------------------.
| Item         | Average latency | Maximum latency | % >2 sec | % >10
sec |
+--------------+-----------------+-----------------+----------+-----------+
| Write        |        4.340 ms |      270.506 ms |  0.00000 |  
0.00000 |
| Random Write |        1.720 ms |        2.968 ms |  0.00000 |  
0.00000 |
| Read         |        7.277 ms |       26.267 ms |  0.00000 |  
0.00000 |
| Random Read  |       10.869 ms |       23.652 ms |  0.00000 |  
0.00000 |
|--------------+-----------------+-----------------+----------+-----------|
| Total        |        5.904 ms |      270.506 ms |  0.00000 |  
0.00000 |
`--------------+-----------------+-----------------+----------+-----------'


From the above results it looks like they are getting better
performance on all shares except raw, really
peaking out on the md's to the 230mb/s range. So this is specifically
a raw only issue. 

-----2.4.21----------------2.6.0-test11------------
/movie                 \     /movie
213.670 MB/s       \      168.458 MB/s
119.860 MB/s       \      126.029 MB/s
/dev/md0              \    /dev/md0
227.463 MB/s        \      171.733 MB/s
113.064 MB/s        \      105.334 MB/s
/dev/raw/raw1        \    /dev/raw/raw1
151.801 MB/s        \      289.157 MB/s
98.126 MB/s         \      106.827 MB/s
/dev/md0               \    /dev/md0
232.478 MB/s         \     136.219 MB/s
119.276 MB/s        \      89.860 MB/s

Comment 2 Doug Ledford 2004-01-26 16:41:34 UTC

I'm pretty sure this is a dup of 104633, marking it as such.

*** This bug has been marked as a duplicate of 104633 ***

Comment 3 Mahesh Kunjal 2004-01-26 16:52:08 UTC

The customer is not able to test out the fix you gave. Please see the
comments from them -

We tried installing both the kernel RPM as provided, and rolling a new
kernel RPM from the SRPM, and, in both cases, there are unresolved
symbols that prevent the system from booting from the SCSI drive (scsi
being one of the unresolved symbols).  The symbol is timer pending.

Do you have a new kernel? I gave them the kernel-smp from the url you
mentioned.

Comment 4 John Flanagan 2004-05-12 01:08:22 UTC

An errata has been issued which should help the problem described in this bug report. 
This report is therefore being closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files, please follow the link below. You may reopen 
this bug report if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2004-188.html

Note You need to log in before you can comment on or make changes to this bug.