Bug 513136 - [RHEL5.4 Snapshot1] File write performance degradation in RHEL5.4 Snapshot1 compared to RHEL5.3 GA
Summary: [RHEL5.4 Snapshot1] File write performance degradation in RHEL5.4 Snapshot1 c...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: kernel
Version: 5.4
Hardware: All
OS: Linux
high
high
Target Milestone: rc
: 5.5
Assignee: Josef Bacik
QA Contact: Red Hat Kernel QE team
URL:
Whiteboard:
Depends On:
Blocks: 499522 525215 533192
TreeView+ depends on / blocked
 
Reported: 2009-07-22 07:28 UTC by Lachlan McIlroy
Modified: 2018-10-27 15:20 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2010-03-30 07:16:43 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2010:0178 0 normal SHIPPED_LIVE Important: Red Hat Enterprise Linux 5.5 kernel security and bug fix update 2010-03-29 12:18:21 UTC

Description Lachlan McIlroy 2009-07-22 07:28:55 UTC
Issue:
Description of problem:
When file write performance is measure using iozone benchmark test, the
performance of RHEL5.4 GA Snapshot1 is about 40% lower than the performance of
RHEL5.3 GA in some cases.  File read performance of RHEL5.4 GA Snapshot1
is almost the same as RHEL5.3 GA.

The following table shows the iozone result on i386 architecture.

        |  Ratio[MB/sec]    |
size(kb)| RHEL5.3 GA | RHEL5.4 SP1 | 5.4/5.3
--------------------------------------
      1 |  177.52 |  165.51 |  93.2%
      2 |  300.21 |  263.13 |  87.6%
      4 |  518.14 |  374.06 |  72.2%
      8 |  694.12 |  454.43 |  65.5%
     16 |  825.29 |  526.04 |  63.7%
     32 |  930.80 |  574.00 |  61.7%
     64 | 1002.65 |  601.36 |  60.0%

This problem occurs on both i386 and x86_64, however i386 performance degradation seems to be worse compared to x86_64.

Version-Release number of selected component (if applicable):
RHEL5.4 GA Snapshot1
kernel-2.6.18-156.el5

How reproducible:
Always

Steps to Reproduce:
1. Install RHEL5.4 GA Snapshot1.
2. Boot RHEL5.4.
3. Execute iozone and measure the performance by following procedure.
3.1) Extract iozone contents in some directory (for example /home)
 ex) # cd /home
     # tar zxf iozone*.tar.gz
3.2) Change the directory as follows, and make.
     # cd iozone*/src/current
     # make linux (case of i386)
3.3) Prepare about 4G of space partition, make an optional partition and mount.
 ex) # mkdir -p /mnt/disk1
     # mount /dev/sdb1 /mnt/disk1
3.4) Input to command line as follows, and execute.
     # ./iozone -+n -Q -i 0 -i 1 -s 1024m -r 1 -r 2 -r 4 -r 8 -r 16 -r 32 -r 64 -f /mnt/disk1/testfile
4. The result of write and read performance is collected.
5. Install RHEL5.3 GA.
6. Boot RHEL5.3.
7. Execute iozone and measured result like Step 3.

Actual results:
The write performance of RHEL5.4 is lower than RHEL5.3.

Expected results:
The write performance of RHEL5.4 is same as RHEL5.3.

Business impact:
Customer running I/O intensive workload will have measurable performance degradation when they upgrade to RHEL5.4

Hardware info:
Machine : Express5800/120Rj-2
CPU     : Intel(R) Xeon(R) CPU E5405  @ 2.00GHz
RAID    : RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 1078 (rev 03)
Memory  : 16GB


Additional info:

This problem is already mentioned in the release note for Snapshot 1.
The bug for this problem (BZ#506511) only mentions x86_64 performance.
However, i386 performance degradation is much worse and may result in
customer issues if the problem isn't fixed.

Similar performance degradation was observed on another server with SATA disks too.

Comment 1 Lachlan McIlroy 2009-07-22 07:31:39 UTC
From BZ506511 (which was initially thought to be the same issue)

I'm sending summary information from Issue 316510.
Sorry for the rather long comment.

+ Write results from iozone shows large performance drop from RHEL5.3 to
  RHEL5.4 beta when writing 1GB files with iozone, especially with large
  record size (measured on server with 16GB memory).

  ./iozone -+n -Q -i 0 -i 1 -s 1024m -r 1 -r 2 -r 4 -r 8 -r 16 -r 32 -r 64 -f
/mnt/disk1/testfile

  The patch which fixes BZ#506511 does not seem to resolve this issue.
  Also, the performance degradation seems to be introduced between
  136.el5PAE and 138.el5PAE (The binary packages for these two kernels
  happened to be available when we tried analyzing the problem after
  beta release).

  [write result (MB/s)]
   record size    1kB     2kB     4kB     8kB    16kB    32kB    64kB
   128.el5PAE   177.52  300.21  518.14  694.12  825.29  930.80 1002.65
   136.el5PAE   168.83  287.14  520.10  695.58  829.76  931.53 1003.23
   138.el5PAE   158.69  268.08  384.04  463.74  531.64  576.59  602.55
   156.el5PAE   165.51  263.13  374.06  454.43  526.04  574.00  601.36
                 (-7%)  (-12%)  (-28%)  (-34%)  (-36%)  (-38%)  (-40%)
   158.el5PAE   152.60  266.89  380.05  457.33  527.84  573.06  598.81
                (-14%)  (-11%)  (-27%)  (-34%)  (-36%)  (-38%)  (-40%)

  A large number of mm and fs changes were introduced between 136 and
  138, so we think that one of the patches might have caused this
  performance drop (we are guessing
  linux-2.6-mm-fix-pagecache-write-deadlocks.patch, but it is
  accompanied by
  linux-2.6-mm-introduce-new-aops-write_begin-and-write_end.patch which
  is said to avoid the performance drop from this patch).

+ Below is the write and rewrite result from iozone (taken from "-a"
  output), measured with fixed record size of 64k and varying the file
  size from 1GB to 8GB (on a server with 4GB memory).

        [write performance]   [rewrite performance]
       128.el5PAE 156.el5PAE  128.el5PAE 156.el5PAE
  1GB   1072293    628430      1557671    1532492
  2GB    196562    186001       207761     206808
  4GB     58794     57962        64604      63951
  8GB     42273     41823        47480      48117

  It seems that writes to disk have almost the same performance between
  RHEL5.3 and RHEL5.4 beta.  Also from the rewrite result, writes to
  existing page cache have not slowed down.  If either the page cache
  gets written back to disk, or reused when containing hot data, the
  performance should be similar between RHEL5.3 and RHEL5.4 beta.

  So performance seems to drop only for writes to cold page cache.
  If such case is unlikey for customer's real workloads, then this
  problem may not have real customer impact.  However we think that
  small benchmark problems might be affected by this.

Comment 2 Lachlan McIlroy 2009-07-22 07:33:33 UTC
I have been able to reproduce this regression but only with ext2.

Results on ext3:

$ cat iozone.2.6.18-128.el5PAE.ext3
Iozone: Performance Test of File I/O
       Version $Revision: 3.326 $
Compiled for 32 bit mode.
Build: linux

Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
            Al Slater, Scott Rhine, Mike Wisner, Ken Goss
            Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
            Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
            Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy,
            Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root.

Run began: Thu Jul 23 14:19:27 2009

File size set to 131072 KB
File size set to 262144 KB
File size set to 524288 KB
File size set to 1048576 KB
File size set to 2097152 KB
Record Size 1024 KB
Command line used: iozone -i 0 -i 1 -s 128m -s 256m -s 512m -s 1024m -s 2048m -r 1024 -f /mnt/test/testfile
Output is in Kbytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
                                                           random  random    bkwd   record   stride                                   
             KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
         131072    1024  343447  563014   684502   688125
         262144    1024  347630  556745   683611   685766
         524288    1024  312985  534891   669929   685122
        1048576    1024  301260  529139   667233   683693
        2097152    1024   97178   97074   678880   683441

iozone test complete.

$ cat iozone.2.6.18-156.el5PAE.ext3
Iozone: Performance Test of File I/O
       Version $Revision: 3.326 $
Compiled for 32 bit mode.
Build: linux

Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
            Al Slater, Scott Rhine, Mike Wisner, Ken Goss
            Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
            Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
            Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy,
            Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root.

Run began: Thu Jul 23 14:12:08 2009

File size set to 131072 KB
File size set to 262144 KB
File size set to 524288 KB
File size set to 1048576 KB
File size set to 2097152 KB
Record Size 1024 KB
Command line used: iozone -i 0 -i 1 -s 128m -s 256m -s 512m -s 1024m -s 2048m -r 1024 -f /mnt/test/testfile
Output is in Kbytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
                                                           random  random    bkwd   record   stride                                   
             KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
         131072    1024  339285  561795   669216   672243
         262144    1024  343089  556807   677387   674799
         524288    1024  311530  540347   668187   669575
        1048576    1024  310352  516240   665432   670413
        2097152    1024   96927   96696   671158   667298

iozone test complete.

Results on ext2:

$ cat iozone.2.6.18-128.el5PAE.ext2
Iozone: Performance Test of File I/O
       Version $Revision: 3.326 $
Compiled for 32 bit mode.
Build: linux

Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
            Al Slater, Scott Rhine, Mike Wisner, Ken Goss
            Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
            Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
            Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy,
            Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root.

Run began: Thu Jul 23 14:29:13 2009

File size set to 131072 KB
File size set to 262144 KB
File size set to 524288 KB
File size set to 1048576 KB
File size set to 2097152 KB
Record Size 1024 KB
Command line used: iozone -i 0 -i 1 -s 128m -s 256m -s 512m -s 1024m -s 2048m -r 1024 -f /mnt/test/testfile
Output is in Kbytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
                                                           random  random    bkwd   record   stride                                   
             KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
         131072    1024  523260  648553   690245   693037
         262144    1024  534796  648376   687664   690919
         524288    1024  492928  635557   680341   679672
        1048576    1024  473663  602273   679033   688624
        2097152    1024   99995   97760   679571   681762

iozone test complete.

$ cat iozone.2.6.18-156.el5PAE.ext2
Iozone: Performance Test of File I/O
       Version $Revision: 3.326 $
Compiled for 32 bit mode.
Build: linux

Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
            Al Slater, Scott Rhine, Mike Wisner, Ken Goss
            Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
            Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
            Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy,
            Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root.

Run began: Thu Jul 23 14:40:10 2009

File size set to 131072 KB
File size set to 262144 KB
File size set to 524288 KB
File size set to 1048576 KB
File size set to 2097152 KB
Record Size 1024 KB
Command line used: iozone -i 0 -i 1 -s 128m -s 256m -s 512m -s 1024m -s 2048m -r 1024 -f /mnt/test/testfile
Output is in Kbytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
                                                           random  random    bkwd   record   stride                                   
             KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
         131072    1024  372346  609325   673771   682627
         262144    1024  374152  631191   677808   682638
         524288    1024  333760  611409   679149   676509
        1048576    1024  326648  609106   675600   677047
        2097152    1024   98168   95139   674830   678906

iozone test complete.

Comment 3 Kosuke TATSUKAWA 2009-07-22 08:10:14 UTC
> I have been able to reproduce this regression but only with ext2.

We've been able to reproduce this on ext3 (the sosreport in the IT#316510 doesn't contain disk sdb used for measurement, but it is an ext3 partition).

What is the memory size of the server you ran iozone on?

Comment 4 Lachlan McIlroy 2009-07-22 08:23:58 UTC
I ran those tests in a VM with 2 cores and 4GB RAM.  I also ran the same tests on a bare metal system with 8 cores and 8GB RAM and saw the same regression but still only on ext2 - ext3 results were consistent between -128 and -156.

Comment 5 Kosuke TATSUKAWA 2009-07-22 12:29:37 UTC
> I ran those tests in a VM with 2 cores and 4GB RAM.  I also ran the same tests
> on a bare metal system with 8 cores and 8GB RAM and saw the same regression but
> still only on ext2 - ext3 results were consistent between -128 and -156.  

I haven't tried running on a VM environment, but it's strange you cannot reproduce
it on the bare-metal system...

We found out that the following script also shows nearly 40% performance drop.
  RHEL5.3    40 sec
  RHEL5.4b   65 sec
Could you give it a try?

---------------------------------------------------------------
#!/bin/sh

sync
i=0
dd if=/dev/zero of=/tmp/foo$i bs=1024k count=1024 >/dev/null 2>&1
rm /tmp/foo$i

date
while { test $i -lt 5; }; do
  i=`expr $i + 1`
  dd if=/dev/zero of=/tmp/foo$i bs=1024k count=1024 >/dev/null 2>&1
  rm /tmp/foo$i
done
date

Comment 6 Lachlan McIlroy 2009-07-23 01:40:48 UTC
Well this problem just gets stranger.  I ran your dd script and the results show that -128 is slower than -158.  There was a bit of variation in the timings so I ran it several times and picked a result that was typical.

$ cat ddscript.2.6.18-128.el5 
Fri Jul 24 11:08:32 EST 2009
Fri Jul 24 11:08:57 EST 2009

real	0m26.821s
user	0m0.013s
sys	0m11.995s

$ cat ddscript.2.6.18-158.el5 
Fri Jul 24 10:59:04 EST 2009
Fri Jul 24 10:59:16 EST 2009

real	0m14.732s
user	0m0.016s
sys	0m12.280s

Comment 7 Lachlan McIlroy 2009-07-23 01:48:10 UTC
Here's some more results from running tests on a bare metal system.  This is with x86_64 kernels.  The ext2 results are clearly slower and the ext3 results do show a minor drop.

$ cat iozone.2.6.18-128.el5.ext3 
	Iozone: Performance Test of File I/O
	        Version $Revision: 3.326 $
		Compiled for 64 bit mode.
		Build: linux 

	Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
	             Al Slater, Scott Rhine, Mike Wisner, Ken Goss
	             Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
	             Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
	             Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy,
	             Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root.

	Run began: Fri Jul 24 10:34:38 2009

	File size set to 131072 KB
	File size set to 262144 KB
	File size set to 524288 KB
	File size set to 1048576 KB
	File size set to 2097152 KB
	Record Size 1024 KB
	Command line used: iozone -i 0 -i 1 -s 128m -s 256m -s 512m -s 1024m -s 2048m -r 1024 -f /mnt/test/testfile
	Output is in Kbytes/sec
	Time Resolution = 0.000001 seconds.
	Processor cache size set to 1024 Kbytes.
	Processor cache line size set to 32 bytes.
	File stride size set to 17 * record size.
                                                            random  random    bkwd   record   stride                                   
              KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
          131072    1024  639984 1476116  2704176  2782125
          262144    1024  657839 1478015  2706085  2775326
          524288    1024  656711 1475272  2711907  2774537
         1048576    1024  591328 1415218  2657691  2771488
         2097152    1024  400062 1430958  2657340  2767903

iozone test complete.

$ cat iozone.2.6.18-158.el5.ext3 
	Iozone: Performance Test of File I/O
	        Version $Revision: 3.326 $
		Compiled for 64 bit mode.
		Build: linux 

	Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
	             Al Slater, Scott Rhine, Mike Wisner, Ken Goss
	             Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
	             Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
	             Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy,
	             Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root.

	Run began: Fri Jul 24 10:27:03 2009

	File size set to 131072 KB
	File size set to 262144 KB
	File size set to 524288 KB
	File size set to 1048576 KB
	File size set to 2097152 KB
	Record Size 1024 KB
	Command line used: iozone -i 0 -i 1 -s 128m -s 256m -s 512m -s 1024m -s 2048m -r 1024 -f /mnt/test/testfile
	Output is in Kbytes/sec
	Time Resolution = 0.000001 seconds.
	Processor cache size set to 1024 Kbytes.
	Processor cache line size set to 32 bytes.
	File stride size set to 17 * record size.
                                                            random  random    bkwd   record   stride                                   
              KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
          131072    1024  629216 1407317  2716066  2765527
          262144    1024  644704 1385691  2713678  2760893
          524288    1024  644625 1408713  2723232  2761257
         1048576    1024  563560 1377179  2684800  2755088
         2097152    1024  387157 1350626  2693197  2754781

iozone test complete.

$ cat iozone.2.6.18-128.el5.ext2 
	Iozone: Performance Test of File I/O
	        Version $Revision: 3.326 $
		Compiled for 64 bit mode.
		Build: linux 

	Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
	             Al Slater, Scott Rhine, Mike Wisner, Ken Goss
	             Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
	             Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
	             Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy,
	             Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root.

	Run began: Fri Jul 24 10:38:04 2009

	File size set to 131072 KB
	File size set to 262144 KB
	File size set to 524288 KB
	File size set to 1048576 KB
	File size set to 2097152 KB
	Record Size 1024 KB
	Command line used: iozone -i 0 -i 1 -s 128m -s 256m -s 512m -s 1024m -s 2048m -r 1024 -f /mnt/test/testfile
	Output is in Kbytes/sec
	Time Resolution = 0.000001 seconds.
	Processor cache size set to 1024 Kbytes.
	Processor cache line size set to 32 bytes.
	File stride size set to 17 * record size.
                                                            random  random    bkwd   record   stride                                   
              KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
          131072    1024 1273237 1901520  2690531  2765471
          262144    1024 1343102 1901039  2724984  2772107
          524288    1024 1335667 1901054  2729289  2773306
         1048576    1024  938334 1852921  2735946  2767574
         2097152    1024  306059 1852305  2734330  2767144

iozone test complete.

$ cat iozone.2.6.18-158.el5.ext2 
	Iozone: Performance Test of File I/O
	        Version $Revision: 3.326 $
		Compiled for 64 bit mode.
		Build: linux 

	Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
	             Al Slater, Scott Rhine, Mike Wisner, Ken Goss
	             Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
	             Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
	             Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy,
	             Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root.

	Run began: Fri Jul 24 10:46:10 2009

	File size set to 131072 KB
	File size set to 262144 KB
	File size set to 524288 KB
	File size set to 1048576 KB
	File size set to 2097152 KB
	Record Size 1024 KB
	Command line used: iozone -i 0 -i 1 -s 128m -s 256m -s 512m -s 1024m -s 2048m -r 1024 -f /mnt/test/testfile
	Output is in Kbytes/sec
	Time Resolution = 0.000001 seconds.
	Processor cache size set to 1024 Kbytes.
	Processor cache line size set to 32 bytes.
	File stride size set to 17 * record size.
                                                            random  random    bkwd   record   stride                                   
              KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
          131072    1024 1081934 1812516  2738716  2775187
          262144    1024 1144913 1808440  2723580  2761677
          524288    1024 1144781 1812772  2709358  2757220
         1048576    1024  514363 1801494  2733349  2762208
         2097152    1024  216737 1780842  2720105  2757005

iozone test complete.

Comment 8 Lachlan McIlroy 2009-07-23 07:04:27 UTC
I noticed that results were being skewed when running multiple tests at once so I focused on one single test - 64KB writes to a 1GB file.  The first run after booting the system showed a significantly reduced throughput.  By the third run (and any subsequent runs) the throughput stabilized for both kernels and was about the same rate showing no regression.  The testfile was on a separate disk so this anomoly could be due to the disk spinning up.  If you run the -158 tests multiple times does the performance improve?

2.6.18-128.el5PAE:

	Iozone: Performance Test of File I/O
	        Version $Revision: 3.326 $
		Compiled for 32 bit mode.
		Build: linux 

	Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
	             Al Slater, Scott Rhine, Mike Wisner, Ken Goss
	             Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
	             Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
	             Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy,
	             Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root.

	Run began: Fri Jul 24 16:14:47 2009

	File size set to 1048576 KB
	Record Size 64 KB
	Command line used: iozone -i 0 -i 1 -s 1024m -r 64 -f /mnt/test/testfile
	Output is in Kbytes/sec
	Time Resolution = 0.000001 seconds.
	Processor cache size set to 1024 Kbytes.
	Processor cache line size set to 32 bytes.
	File stride size set to 17 * record size.
                                                            random  random    bkwd   record   stride                                   
              KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread

1st run
         1048576      64  285910 1351689  2124205  2139693

2nd run
         1048576      64  509590 1380855  2117959  2134857

3rd run
         1048576      64  533117 1363889  2119667  2138100


2.6.18-158.el5PAE:

	Iozone: Performance Test of File I/O
	        Version $Revision: 3.326 $
		Compiled for 32 bit mode.
		Build: linux 

	Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
	             Al Slater, Scott Rhine, Mike Wisner, Ken Goss
	             Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
	             Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
	             Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy,
	             Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root.

	Run began: Fri Jul 24 16:20:55 2009

	File size set to 1048576 KB
	Record Size 64 KB
	Command line used: iozone -i 0 -i 1 -s 1024m -r 64 -f /mnt/test/testfile
	Output is in Kbytes/sec
	Time Resolution = 0.000001 seconds.
	Processor cache size set to 1024 Kbytes.
	Processor cache line size set to 32 bytes.
	File stride size set to 17 * record size.
                                                            random  random    bkwd   record   stride                                   
              KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread

1st run
         1048576      64  363518 1330584  2115745  2129962

2nd run
         1048576      64  499372 1380488  2114806  2130004

3rd run
         1048576      64  529297 1340216  2121601  2126467

Comment 9 Kosuke TATSUKAWA 2009-07-23 12:34:40 UTC
> I noticed that results were being skewed when running multiple tests at once so
> I focused on one single test - 64KB writes to a 1GB file.  The first run after
> booting the system showed a significantly reduced throughput.  By the third run
> (and any subsequent runs) the throughput stabilized for both kernels and was
> about the same rate showing no regression.  The testfile was on a separate disk
> so this anomoly could be due to the disk spinning up.  If you run the -158
> tests multiple times does the performance improve?

thank you for the investigation.
I was out of office this afternoon.  We'll try running the test multiple times
on the -158 kernel tomorrow.

Comment 10 Kosuke TATSUKAWA 2009-07-24 01:37:36 UTC
Here's the result when running iozone multiple times on server with 4GB memory.

+ 158 kernel was still slower, but the performance improved after running
  the benchmark several times.
+ This time the performance difference was much smaller.  Previously, we
  were running iozone from our test script with another disk io benchmark,
  and the state of the page cache might had some influence on the result.
  Also, the performance was measured on the second run, so the disk spinning
  up shouldn't be a problem.
+ In comment #8, the performance of 158 kernel is better than 128 kernel.
  So there seems to be a difference here, even for the first run of iozone.


2.6.18-128.el5PAE:

        Iozone: Performance Test of File I/O
                Version $Revision: 3.263 $
                Compiled for 32 bit mode.
                Build: linux

        Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
                     Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                     Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
                     Randy Dunlap, Mark Montague, Dan Million,
                     Jean-Marc Zucconi, Jeff Blomberg,
                     Erik Habbinga, Kris Strecker, Walter Wong.

        Run began: Thu Jul 23 17:46:00 2009

        File size set to 1048576 KB
        Record Size 64 KB
        Command line used: ./iozone -i 0 -i 1 -s 1024m -r 64 -f /mnt/disk1/testfile
        Output is in Kbytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                            random  random    bkwd  record  stride
              KB  reclen   write rewrite    read    reread    read   write    read rewrite    read   fwrite frewrite   fread  freread
1st run
         1048576      64  506094 1171801  1894484  1912786

2nd run
         1048576      64  514863 1169955  1887363  1909303

3rd run
         1048576      64  513751 1015386  1890092  1905791


2.6.18-158.el5PAE:

        Iozone: Performance Test of File I/O
                Version $Revision: 3.263 $
                Compiled for 32 bit mode.
                Build: linux

        Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
                     Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                     Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
                     Randy Dunlap, Mark Montague, Dan Million,
                     Jean-Marc Zucconi, Jeff Blomberg,
                     Erik Habbinga, Kris Strecker, Walter Wong.

        Run began: Thu Jul 23 17:58:08 2009

        File size set to 1048576 KB
        Record Size 64 KB
        Command line used: ./iozone -i 0 -i 1 -s 1024m -r 64 -f /mnt/disk1/testfile
        Output is in Kbytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                            random  random    bkwd  record  stride
              KB  reclen   write rewrite    read    reread    read   write    read rewrite    read   fwrite frewrite   fread  freread
1st run
         1048576      64  486393 1122432  1897167  1909671

2nd run
         1048576      64  506193 1031835  1894292  1910687

3rd run
         1048576      64  507110 1025206  1885968  1909428

Comment 11 Kosuke TATSUKAWA 2009-07-26 11:49:00 UTC
> Here's the result when running iozone multiple times on server with 4GB memory.
>
> + 158 kernel was still slower, but the performance improved after running
>   the benchmark several times.
> + This time the performance difference was much smaller.  Previously, we
>   were running iozone from our test script with another disk io benchmark,
>   and the state of the page cache might had some influence on the result.
>   Also, the performance was measured on the second run, so the disk spinning
>   up shouldn't be a problem.

We found out that the performance drop in iozone was caused by our mistake.
One of the test programs running before iozone was rebuilding the target file
system as ext2.  I apologize for taking your time.

However, we double checked that the dd test script that showed performance
drop in RHEL5.4 beta compared to RHEL5.3 was indeed running on an ext3 file
system.  I know that Red Hat got the opposite result with this test, so I'd
like to analyze the result a little further to make the test result more
reproducable.

PS.
As for the performance drop with ext2, I don't think it will be a blocker
for RHEL5.4

Comment 12 Lachlan McIlroy 2009-07-27 01:25:27 UTC
Great, we're on the same page now - ext2 is affected and ext3 looks okay.

I found the results from the dd test script to be too unreliable to draw any concrete conclusions.  If there is a performance issue with ext3 then let's consider that a different bug.

Comment 13 Kosuke TATSUKAWA 2009-07-27 01:34:18 UTC
> Great, we're on the same page now - ext2 is affected and ext3 looks okay.
>
> I found the results from the dd test script to be too unreliable to draw any
> concrete conclusions.  If there is a performance issue with ext3 then let's
> consider that a different bug.  

I plan to add post some additional info on the dd test script, so please wait
a little more before closing this bug.

From there, we could either close this all together or continue discussion
(possibly on a different bug).

Comment 14 Lachlan McIlroy 2009-07-27 01:44:55 UTC
I don't intend to close this bug - we still have to find the source of the ext2 regression.

Comment 15 Kosuke TATSUKAWA 2009-07-27 01:53:11 UTC
linux-2.6-mm-introduce-new-aops-write_begin-and-write_end.patch introduces
new aops to compensate for the performance drop introduced in
linux-2.6-mm-fix-pagecache-write-deadlocks.patch.

linux-2.6-fs-ext3-convert-to-new-aops.patch modifies ext3 to use the new aops.
ext3, ext4, nfs, xfs, gfs2 have been converted, but I think the new aops change
is missing from ext2.

Comment 16 Kosuke TATSUKAWA 2009-07-27 10:29:40 UTC
Here's the additional information for the dd+rm script.

We modified the script to print out the date for each dd+rm step, and
measured the time for 70 iterations on a server with 4GB memory.

Below is the histogram for the time it took for each dd+rm step.
+ The average performance drop is about 25%.
+ The time for the fast case(2--4 seconds) seems almost the same for
  the two kernels and there seems to be difference in the slow case
  (possibly affected by journal write).

Could Red Hat reproduce similar result on their environment?

                -128    -158
                kernel  kernel
        0:00:01
        0:00:02 7       8
        0:00:03 31      29
        0:00:04 3
        0:00:05
        0:00:06
        0:00:07 1
        0:00:08 1       1
        0:00:09 1       1
        0:00:10 1
        0:00:11 5
        0:00:12 1
        0:00:13 4
        0:00:14 4       3
        0:00:15 3       2
        0:00:16 1       3
        0:00:17 3       2
        0:00:18 2       7
        0:00:19         4
        0:00:20 1       4
        0:00:21         3
        0:00:22
        0:00:23
        0:00:24         2
        0:00:25         1
        0:00:26 1

Comment 17 Josef Bacik 2009-07-27 16:08:33 UTC
yeah a drop on ext2 is to be expected, it was not converted to the new aops, we wanted to reduce the impact if there were severe problems found with the new aops.  Will just need to backport 

f34fb6eccc962e4137b77c4bbe1e9a31d27a30e6

at some point to fix the problem.

Comment 18 Kosuke TATSUKAWA 2009-07-30 10:12:41 UTC
Hi,

I've done additional tests with the dd+rm tests, and now I completely
agree that the result is too unstable.

I've measured the result several times on the same kernel version, but
the variance was large enough to hide the difference in comment #16.
Also the performance varied randomly when measuring consecutive kernel
versions (156,157,159,160),

Comment 19 RHEL Program Management 2009-09-25 17:42:41 UTC
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.

Comment 20 Jan Tluka 2009-10-14 13:40:04 UTC
Josef, is there any recommendation on how to reliably verify this bug?

Comment 21 Josef Bacik 2009-10-15 17:24:54 UTC
according to the other comments in this bug I assume iozone -i 0 -i 1 -s 1024m -r 64 will work just fine.

Comment 23 Don Zickus 2009-12-11 19:28:16 UTC
in kernel-2.6.18-179.el5
You can download this test kernel from http://people.redhat.com/dzickus/el5

Please update the appropriate value in the Verified field
(cf_verified) to indicate this fix has been successfully
verified. Include a comment with verification details.

Comment 25 Issue Tracker 2009-12-17 07:59:59 UTC
Event posted on 12-17-2009 04:09pm JST by tatsu.nec.com

NEC confirmed that the iozone result of 181.el5 kernel does not show
the performance degradation observed in RHEL5.4 GA kernels.

* i686
           Ratio[MB/sec]
 size(kb)| RHEL5.3 GA | RHEL5.4 GA | 181.el5 |181.el5/5.3
---------------------------------------------------------
       1 |     176.67 |     169.78 |  176.83 |     100.1%
       2 |     298.94 |     270.63 |  301.42 |     100.8%
       4 |     519.62 |     388.82 |  531.26 |     102.2%
       8 |     697.03 |     465.46 |  705.12 |     101.2%
      16 |     829.08 |     533.99 |  832.72 |     100.4%
      32 |     935.71 |     581.45 |  931.21 |      99.5%
      64 |    1008.69 |     607.51 | 1002.69 |      99.4%

* x86_64
           Ratio[MB/sec]
 size(kb)| RHEL5.3 GA | RHEL5.4 GA | 181.el5 |181.el5/5.3
---------------------------------------------------------
       1 |     250.29 |     260.79 |  264.87 |     105.8%
       2 |     399.23 |     397.35 |  413.99 |     103.7%
       4 |     601.57 |     555.26 |  619.80 |     103.0%
       8 |     784.57 |     692.98 |  798.60 |     101.8%
      16 |     915.50 |     787.35 |  920.95 |     100.6%
      32 |    1033.49 |     871.76 | 1027.69 |      99.4%
      64 |    1117.79 |     927.11 | 1103.88 |      98.8%



This event sent from IssueTracker by mfuruta 
 issue 316510

Comment 26 Jiri Hladky 2010-03-09 17:41:34 UTC
Hi,

I have verified that there is no write regression using RHEL 5.5 SNAP2 (kernel 2.6.18-189.el5)

We can close this BZ.

Thanks
Jirka

Comment 28 errata-xmlrpc 2010-03-30 07:16:43 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2010-0178.html


Note You need to log in before you can comment on or make changes to this bug.