Issue: Description of problem: When file write performance is measure using iozone benchmark test, the performance of RHEL5.4 GA Snapshot1 is about 40% lower than the performance of RHEL5.3 GA in some cases. File read performance of RHEL5.4 GA Snapshot1 is almost the same as RHEL5.3 GA. The following table shows the iozone result on i386 architecture. | Ratio[MB/sec] | size(kb)| RHEL5.3 GA | RHEL5.4 SP1 | 5.4/5.3 -------------------------------------- 1 | 177.52 | 165.51 | 93.2% 2 | 300.21 | 263.13 | 87.6% 4 | 518.14 | 374.06 | 72.2% 8 | 694.12 | 454.43 | 65.5% 16 | 825.29 | 526.04 | 63.7% 32 | 930.80 | 574.00 | 61.7% 64 | 1002.65 | 601.36 | 60.0% This problem occurs on both i386 and x86_64, however i386 performance degradation seems to be worse compared to x86_64. Version-Release number of selected component (if applicable): RHEL5.4 GA Snapshot1 kernel-2.6.18-156.el5 How reproducible: Always Steps to Reproduce: 1. Install RHEL5.4 GA Snapshot1. 2. Boot RHEL5.4. 3. Execute iozone and measure the performance by following procedure. 3.1) Extract iozone contents in some directory (for example /home) ex) # cd /home # tar zxf iozone*.tar.gz 3.2) Change the directory as follows, and make. # cd iozone*/src/current # make linux (case of i386) 3.3) Prepare about 4G of space partition, make an optional partition and mount. ex) # mkdir -p /mnt/disk1 # mount /dev/sdb1 /mnt/disk1 3.4) Input to command line as follows, and execute. # ./iozone -+n -Q -i 0 -i 1 -s 1024m -r 1 -r 2 -r 4 -r 8 -r 16 -r 32 -r 64 -f /mnt/disk1/testfile 4. The result of write and read performance is collected. 5. Install RHEL5.3 GA. 6. Boot RHEL5.3. 7. Execute iozone and measured result like Step 3. Actual results: The write performance of RHEL5.4 is lower than RHEL5.3. Expected results: The write performance of RHEL5.4 is same as RHEL5.3. Business impact: Customer running I/O intensive workload will have measurable performance degradation when they upgrade to RHEL5.4 Hardware info: Machine : Express5800/120Rj-2 CPU : Intel(R) Xeon(R) CPU E5405 @ 2.00GHz RAID : RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 1078 (rev 03) Memory : 16GB Additional info: This problem is already mentioned in the release note for Snapshot 1. The bug for this problem (BZ#506511) only mentions x86_64 performance. However, i386 performance degradation is much worse and may result in customer issues if the problem isn't fixed. Similar performance degradation was observed on another server with SATA disks too.
From BZ506511 (which was initially thought to be the same issue) I'm sending summary information from Issue 316510. Sorry for the rather long comment. + Write results from iozone shows large performance drop from RHEL5.3 to RHEL5.4 beta when writing 1GB files with iozone, especially with large record size (measured on server with 16GB memory). ./iozone -+n -Q -i 0 -i 1 -s 1024m -r 1 -r 2 -r 4 -r 8 -r 16 -r 32 -r 64 -f /mnt/disk1/testfile The patch which fixes BZ#506511 does not seem to resolve this issue. Also, the performance degradation seems to be introduced between 136.el5PAE and 138.el5PAE (The binary packages for these two kernels happened to be available when we tried analyzing the problem after beta release). [write result (MB/s)] record size 1kB 2kB 4kB 8kB 16kB 32kB 64kB 128.el5PAE 177.52 300.21 518.14 694.12 825.29 930.80 1002.65 136.el5PAE 168.83 287.14 520.10 695.58 829.76 931.53 1003.23 138.el5PAE 158.69 268.08 384.04 463.74 531.64 576.59 602.55 156.el5PAE 165.51 263.13 374.06 454.43 526.04 574.00 601.36 (-7%) (-12%) (-28%) (-34%) (-36%) (-38%) (-40%) 158.el5PAE 152.60 266.89 380.05 457.33 527.84 573.06 598.81 (-14%) (-11%) (-27%) (-34%) (-36%) (-38%) (-40%) A large number of mm and fs changes were introduced between 136 and 138, so we think that one of the patches might have caused this performance drop (we are guessing linux-2.6-mm-fix-pagecache-write-deadlocks.patch, but it is accompanied by linux-2.6-mm-introduce-new-aops-write_begin-and-write_end.patch which is said to avoid the performance drop from this patch). + Below is the write and rewrite result from iozone (taken from "-a" output), measured with fixed record size of 64k and varying the file size from 1GB to 8GB (on a server with 4GB memory). [write performance] [rewrite performance] 128.el5PAE 156.el5PAE 128.el5PAE 156.el5PAE 1GB 1072293 628430 1557671 1532492 2GB 196562 186001 207761 206808 4GB 58794 57962 64604 63951 8GB 42273 41823 47480 48117 It seems that writes to disk have almost the same performance between RHEL5.3 and RHEL5.4 beta. Also from the rewrite result, writes to existing page cache have not slowed down. If either the page cache gets written back to disk, or reused when containing hot data, the performance should be similar between RHEL5.3 and RHEL5.4 beta. So performance seems to drop only for writes to cold page cache. If such case is unlikey for customer's real workloads, then this problem may not have real customer impact. However we think that small benchmark problems might be affected by this.
I have been able to reproduce this regression but only with ext2. Results on ext3: $ cat iozone.2.6.18-128.el5PAE.ext3 Iozone: Performance Test of File I/O Version $Revision: 3.326 $ Compiled for 32 bit mode. Build: linux Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root. Run began: Thu Jul 23 14:19:27 2009 File size set to 131072 KB File size set to 262144 KB File size set to 524288 KB File size set to 1048576 KB File size set to 2097152 KB Record Size 1024 KB Command line used: iozone -i 0 -i 1 -s 128m -s 256m -s 512m -s 1024m -s 2048m -r 1024 -f /mnt/test/testfile Output is in Kbytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 Kbytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 131072 1024 343447 563014 684502 688125 262144 1024 347630 556745 683611 685766 524288 1024 312985 534891 669929 685122 1048576 1024 301260 529139 667233 683693 2097152 1024 97178 97074 678880 683441 iozone test complete. $ cat iozone.2.6.18-156.el5PAE.ext3 Iozone: Performance Test of File I/O Version $Revision: 3.326 $ Compiled for 32 bit mode. Build: linux Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root. Run began: Thu Jul 23 14:12:08 2009 File size set to 131072 KB File size set to 262144 KB File size set to 524288 KB File size set to 1048576 KB File size set to 2097152 KB Record Size 1024 KB Command line used: iozone -i 0 -i 1 -s 128m -s 256m -s 512m -s 1024m -s 2048m -r 1024 -f /mnt/test/testfile Output is in Kbytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 Kbytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 131072 1024 339285 561795 669216 672243 262144 1024 343089 556807 677387 674799 524288 1024 311530 540347 668187 669575 1048576 1024 310352 516240 665432 670413 2097152 1024 96927 96696 671158 667298 iozone test complete. Results on ext2: $ cat iozone.2.6.18-128.el5PAE.ext2 Iozone: Performance Test of File I/O Version $Revision: 3.326 $ Compiled for 32 bit mode. Build: linux Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root. Run began: Thu Jul 23 14:29:13 2009 File size set to 131072 KB File size set to 262144 KB File size set to 524288 KB File size set to 1048576 KB File size set to 2097152 KB Record Size 1024 KB Command line used: iozone -i 0 -i 1 -s 128m -s 256m -s 512m -s 1024m -s 2048m -r 1024 -f /mnt/test/testfile Output is in Kbytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 Kbytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 131072 1024 523260 648553 690245 693037 262144 1024 534796 648376 687664 690919 524288 1024 492928 635557 680341 679672 1048576 1024 473663 602273 679033 688624 2097152 1024 99995 97760 679571 681762 iozone test complete. $ cat iozone.2.6.18-156.el5PAE.ext2 Iozone: Performance Test of File I/O Version $Revision: 3.326 $ Compiled for 32 bit mode. Build: linux Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root. Run began: Thu Jul 23 14:40:10 2009 File size set to 131072 KB File size set to 262144 KB File size set to 524288 KB File size set to 1048576 KB File size set to 2097152 KB Record Size 1024 KB Command line used: iozone -i 0 -i 1 -s 128m -s 256m -s 512m -s 1024m -s 2048m -r 1024 -f /mnt/test/testfile Output is in Kbytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 Kbytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 131072 1024 372346 609325 673771 682627 262144 1024 374152 631191 677808 682638 524288 1024 333760 611409 679149 676509 1048576 1024 326648 609106 675600 677047 2097152 1024 98168 95139 674830 678906 iozone test complete.
> I have been able to reproduce this regression but only with ext2. We've been able to reproduce this on ext3 (the sosreport in the IT#316510 doesn't contain disk sdb used for measurement, but it is an ext3 partition). What is the memory size of the server you ran iozone on?
I ran those tests in a VM with 2 cores and 4GB RAM. I also ran the same tests on a bare metal system with 8 cores and 8GB RAM and saw the same regression but still only on ext2 - ext3 results were consistent between -128 and -156.
> I ran those tests in a VM with 2 cores and 4GB RAM. I also ran the same tests > on a bare metal system with 8 cores and 8GB RAM and saw the same regression but > still only on ext2 - ext3 results were consistent between -128 and -156. I haven't tried running on a VM environment, but it's strange you cannot reproduce it on the bare-metal system... We found out that the following script also shows nearly 40% performance drop. RHEL5.3 40 sec RHEL5.4b 65 sec Could you give it a try? --------------------------------------------------------------- #!/bin/sh sync i=0 dd if=/dev/zero of=/tmp/foo$i bs=1024k count=1024 >/dev/null 2>&1 rm /tmp/foo$i date while { test $i -lt 5; }; do i=`expr $i + 1` dd if=/dev/zero of=/tmp/foo$i bs=1024k count=1024 >/dev/null 2>&1 rm /tmp/foo$i done date
Well this problem just gets stranger. I ran your dd script and the results show that -128 is slower than -158. There was a bit of variation in the timings so I ran it several times and picked a result that was typical. $ cat ddscript.2.6.18-128.el5 Fri Jul 24 11:08:32 EST 2009 Fri Jul 24 11:08:57 EST 2009 real 0m26.821s user 0m0.013s sys 0m11.995s $ cat ddscript.2.6.18-158.el5 Fri Jul 24 10:59:04 EST 2009 Fri Jul 24 10:59:16 EST 2009 real 0m14.732s user 0m0.016s sys 0m12.280s
Here's some more results from running tests on a bare metal system. This is with x86_64 kernels. The ext2 results are clearly slower and the ext3 results do show a minor drop. $ cat iozone.2.6.18-128.el5.ext3 Iozone: Performance Test of File I/O Version $Revision: 3.326 $ Compiled for 64 bit mode. Build: linux Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root. Run began: Fri Jul 24 10:34:38 2009 File size set to 131072 KB File size set to 262144 KB File size set to 524288 KB File size set to 1048576 KB File size set to 2097152 KB Record Size 1024 KB Command line used: iozone -i 0 -i 1 -s 128m -s 256m -s 512m -s 1024m -s 2048m -r 1024 -f /mnt/test/testfile Output is in Kbytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 Kbytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 131072 1024 639984 1476116 2704176 2782125 262144 1024 657839 1478015 2706085 2775326 524288 1024 656711 1475272 2711907 2774537 1048576 1024 591328 1415218 2657691 2771488 2097152 1024 400062 1430958 2657340 2767903 iozone test complete. $ cat iozone.2.6.18-158.el5.ext3 Iozone: Performance Test of File I/O Version $Revision: 3.326 $ Compiled for 64 bit mode. Build: linux Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root. Run began: Fri Jul 24 10:27:03 2009 File size set to 131072 KB File size set to 262144 KB File size set to 524288 KB File size set to 1048576 KB File size set to 2097152 KB Record Size 1024 KB Command line used: iozone -i 0 -i 1 -s 128m -s 256m -s 512m -s 1024m -s 2048m -r 1024 -f /mnt/test/testfile Output is in Kbytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 Kbytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 131072 1024 629216 1407317 2716066 2765527 262144 1024 644704 1385691 2713678 2760893 524288 1024 644625 1408713 2723232 2761257 1048576 1024 563560 1377179 2684800 2755088 2097152 1024 387157 1350626 2693197 2754781 iozone test complete. $ cat iozone.2.6.18-128.el5.ext2 Iozone: Performance Test of File I/O Version $Revision: 3.326 $ Compiled for 64 bit mode. Build: linux Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root. Run began: Fri Jul 24 10:38:04 2009 File size set to 131072 KB File size set to 262144 KB File size set to 524288 KB File size set to 1048576 KB File size set to 2097152 KB Record Size 1024 KB Command line used: iozone -i 0 -i 1 -s 128m -s 256m -s 512m -s 1024m -s 2048m -r 1024 -f /mnt/test/testfile Output is in Kbytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 Kbytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 131072 1024 1273237 1901520 2690531 2765471 262144 1024 1343102 1901039 2724984 2772107 524288 1024 1335667 1901054 2729289 2773306 1048576 1024 938334 1852921 2735946 2767574 2097152 1024 306059 1852305 2734330 2767144 iozone test complete. $ cat iozone.2.6.18-158.el5.ext2 Iozone: Performance Test of File I/O Version $Revision: 3.326 $ Compiled for 64 bit mode. Build: linux Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root. Run began: Fri Jul 24 10:46:10 2009 File size set to 131072 KB File size set to 262144 KB File size set to 524288 KB File size set to 1048576 KB File size set to 2097152 KB Record Size 1024 KB Command line used: iozone -i 0 -i 1 -s 128m -s 256m -s 512m -s 1024m -s 2048m -r 1024 -f /mnt/test/testfile Output is in Kbytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 Kbytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 131072 1024 1081934 1812516 2738716 2775187 262144 1024 1144913 1808440 2723580 2761677 524288 1024 1144781 1812772 2709358 2757220 1048576 1024 514363 1801494 2733349 2762208 2097152 1024 216737 1780842 2720105 2757005 iozone test complete.
I noticed that results were being skewed when running multiple tests at once so I focused on one single test - 64KB writes to a 1GB file. The first run after booting the system showed a significantly reduced throughput. By the third run (and any subsequent runs) the throughput stabilized for both kernels and was about the same rate showing no regression. The testfile was on a separate disk so this anomoly could be due to the disk spinning up. If you run the -158 tests multiple times does the performance improve? 2.6.18-128.el5PAE: Iozone: Performance Test of File I/O Version $Revision: 3.326 $ Compiled for 32 bit mode. Build: linux Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root. Run began: Fri Jul 24 16:14:47 2009 File size set to 1048576 KB Record Size 64 KB Command line used: iozone -i 0 -i 1 -s 1024m -r 64 -f /mnt/test/testfile Output is in Kbytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 Kbytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 1st run 1048576 64 285910 1351689 2124205 2139693 2nd run 1048576 64 509590 1380855 2117959 2134857 3rd run 1048576 64 533117 1363889 2119667 2138100 2.6.18-158.el5PAE: Iozone: Performance Test of File I/O Version $Revision: 3.326 $ Compiled for 32 bit mode. Build: linux Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root. Run began: Fri Jul 24 16:20:55 2009 File size set to 1048576 KB Record Size 64 KB Command line used: iozone -i 0 -i 1 -s 1024m -r 64 -f /mnt/test/testfile Output is in Kbytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 Kbytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 1st run 1048576 64 363518 1330584 2115745 2129962 2nd run 1048576 64 499372 1380488 2114806 2130004 3rd run 1048576 64 529297 1340216 2121601 2126467
> I noticed that results were being skewed when running multiple tests at once so > I focused on one single test - 64KB writes to a 1GB file. The first run after > booting the system showed a significantly reduced throughput. By the third run > (and any subsequent runs) the throughput stabilized for both kernels and was > about the same rate showing no regression. The testfile was on a separate disk > so this anomoly could be due to the disk spinning up. If you run the -158 > tests multiple times does the performance improve? thank you for the investigation. I was out of office this afternoon. We'll try running the test multiple times on the -158 kernel tomorrow.
Here's the result when running iozone multiple times on server with 4GB memory. + 158 kernel was still slower, but the performance improved after running the benchmark several times. + This time the performance difference was much smaller. Previously, we were running iozone from our test script with another disk io benchmark, and the state of the page cache might had some influence on the result. Also, the performance was measured on the second run, so the disk spinning up shouldn't be a problem. + In comment #8, the performance of 158 kernel is better than 128 kernel. So there seems to be a difference here, even for the first run of iozone. 2.6.18-128.el5PAE: Iozone: Performance Test of File I/O Version $Revision: 3.263 $ Compiled for 32 bit mode. Build: linux Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Jean-Marc Zucconi, Jeff Blomberg, Erik Habbinga, Kris Strecker, Walter Wong. Run began: Thu Jul 23 17:46:00 2009 File size set to 1048576 KB Record Size 64 KB Command line used: ./iozone -i 0 -i 1 -s 1024m -r 64 -f /mnt/disk1/testfile Output is in Kbytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 Kbytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 1st run 1048576 64 506094 1171801 1894484 1912786 2nd run 1048576 64 514863 1169955 1887363 1909303 3rd run 1048576 64 513751 1015386 1890092 1905791 2.6.18-158.el5PAE: Iozone: Performance Test of File I/O Version $Revision: 3.263 $ Compiled for 32 bit mode. Build: linux Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Jean-Marc Zucconi, Jeff Blomberg, Erik Habbinga, Kris Strecker, Walter Wong. Run began: Thu Jul 23 17:58:08 2009 File size set to 1048576 KB Record Size 64 KB Command line used: ./iozone -i 0 -i 1 -s 1024m -r 64 -f /mnt/disk1/testfile Output is in Kbytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 Kbytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 1st run 1048576 64 486393 1122432 1897167 1909671 2nd run 1048576 64 506193 1031835 1894292 1910687 3rd run 1048576 64 507110 1025206 1885968 1909428
> Here's the result when running iozone multiple times on server with 4GB memory. > > + 158 kernel was still slower, but the performance improved after running > the benchmark several times. > + This time the performance difference was much smaller. Previously, we > were running iozone from our test script with another disk io benchmark, > and the state of the page cache might had some influence on the result. > Also, the performance was measured on the second run, so the disk spinning > up shouldn't be a problem. We found out that the performance drop in iozone was caused by our mistake. One of the test programs running before iozone was rebuilding the target file system as ext2. I apologize for taking your time. However, we double checked that the dd test script that showed performance drop in RHEL5.4 beta compared to RHEL5.3 was indeed running on an ext3 file system. I know that Red Hat got the opposite result with this test, so I'd like to analyze the result a little further to make the test result more reproducable. PS. As for the performance drop with ext2, I don't think it will be a blocker for RHEL5.4
Great, we're on the same page now - ext2 is affected and ext3 looks okay. I found the results from the dd test script to be too unreliable to draw any concrete conclusions. If there is a performance issue with ext3 then let's consider that a different bug.
> Great, we're on the same page now - ext2 is affected and ext3 looks okay. > > I found the results from the dd test script to be too unreliable to draw any > concrete conclusions. If there is a performance issue with ext3 then let's > consider that a different bug. I plan to add post some additional info on the dd test script, so please wait a little more before closing this bug. From there, we could either close this all together or continue discussion (possibly on a different bug).
I don't intend to close this bug - we still have to find the source of the ext2 regression.
linux-2.6-mm-introduce-new-aops-write_begin-and-write_end.patch introduces new aops to compensate for the performance drop introduced in linux-2.6-mm-fix-pagecache-write-deadlocks.patch. linux-2.6-fs-ext3-convert-to-new-aops.patch modifies ext3 to use the new aops. ext3, ext4, nfs, xfs, gfs2 have been converted, but I think the new aops change is missing from ext2.
Here's the additional information for the dd+rm script. We modified the script to print out the date for each dd+rm step, and measured the time for 70 iterations on a server with 4GB memory. Below is the histogram for the time it took for each dd+rm step. + The average performance drop is about 25%. + The time for the fast case(2--4 seconds) seems almost the same for the two kernels and there seems to be difference in the slow case (possibly affected by journal write). Could Red Hat reproduce similar result on their environment? -128 -158 kernel kernel 0:00:01 0:00:02 7 8 0:00:03 31 29 0:00:04 3 0:00:05 0:00:06 0:00:07 1 0:00:08 1 1 0:00:09 1 1 0:00:10 1 0:00:11 5 0:00:12 1 0:00:13 4 0:00:14 4 3 0:00:15 3 2 0:00:16 1 3 0:00:17 3 2 0:00:18 2 7 0:00:19 4 0:00:20 1 4 0:00:21 3 0:00:22 0:00:23 0:00:24 2 0:00:25 1 0:00:26 1
yeah a drop on ext2 is to be expected, it was not converted to the new aops, we wanted to reduce the impact if there were severe problems found with the new aops. Will just need to backport f34fb6eccc962e4137b77c4bbe1e9a31d27a30e6 at some point to fix the problem.
Hi, I've done additional tests with the dd+rm tests, and now I completely agree that the result is too unstable. I've measured the result several times on the same kernel version, but the variance was large enough to hide the difference in comment #16. Also the performance varied randomly when measuring consecutive kernel versions (156,157,159,160),
This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux maintenance release. Product Management has requested further review of this request by Red Hat Engineering, for potential inclusion in a Red Hat Enterprise Linux Update release for currently deployed products. This request is not yet committed for inclusion in an Update release.
Josef, is there any recommendation on how to reliably verify this bug?
according to the other comments in this bug I assume iozone -i 0 -i 1 -s 1024m -r 64 will work just fine.
in kernel-2.6.18-179.el5 You can download this test kernel from http://people.redhat.com/dzickus/el5 Please update the appropriate value in the Verified field (cf_verified) to indicate this fix has been successfully verified. Include a comment with verification details.
Event posted on 12-17-2009 04:09pm JST by tatsu.nec.com NEC confirmed that the iozone result of 181.el5 kernel does not show the performance degradation observed in RHEL5.4 GA kernels. * i686 Ratio[MB/sec] size(kb)| RHEL5.3 GA | RHEL5.4 GA | 181.el5 |181.el5/5.3 --------------------------------------------------------- 1 | 176.67 | 169.78 | 176.83 | 100.1% 2 | 298.94 | 270.63 | 301.42 | 100.8% 4 | 519.62 | 388.82 | 531.26 | 102.2% 8 | 697.03 | 465.46 | 705.12 | 101.2% 16 | 829.08 | 533.99 | 832.72 | 100.4% 32 | 935.71 | 581.45 | 931.21 | 99.5% 64 | 1008.69 | 607.51 | 1002.69 | 99.4% * x86_64 Ratio[MB/sec] size(kb)| RHEL5.3 GA | RHEL5.4 GA | 181.el5 |181.el5/5.3 --------------------------------------------------------- 1 | 250.29 | 260.79 | 264.87 | 105.8% 2 | 399.23 | 397.35 | 413.99 | 103.7% 4 | 601.57 | 555.26 | 619.80 | 103.0% 8 | 784.57 | 692.98 | 798.60 | 101.8% 16 | 915.50 | 787.35 | 920.95 | 100.6% 32 | 1033.49 | 871.76 | 1027.69 | 99.4% 64 | 1117.79 | 927.11 | 1103.88 | 98.8% This event sent from IssueTracker by mfuruta issue 316510
Hi, I have verified that there is no write regression using RHEL 5.5 SNAP2 (kernel 2.6.18-189.el5) We can close this BZ. Thanks Jirka
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2010-0178.html