Bug 506511

Summary: performance regression running Iozone with different I/O options on RHEL54 kernels
Product: Red Hat Enterprise Linux 5 Reporter: Tom Tracy <ttracy>
Component: kernelAssignee: Josef Bacik <jbacik>
Status: CLOSED ERRATA QA Contact: Red Hat Kernel QE team <kernel-qe>
Severity: high Docs Contact:
Priority: high    
Version: 5.4CC: bmarson, dshaks, dzickus, esandeen, jtluka, lmcilroy, rwheeler, syeghiay, tatsu-ab1
Target Milestone: rcKeywords: Regression
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2009-09-02 08:24:56 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Log results from Intel Iozone
none
patch to fix the problem none

Description Tom Tracy 2009-06-17 15:23:41 UTC 
Created attachment 348273 [details]
Log results from Intel Iozone 

Description of problem: Read and Initial Write performance regression running Iozone with different I/O options. Regressions of greater than 5% compared to RHEL53 128 kernel found in incache,fsync,outcache and directio tests. This happens on both Intel/AMD systems. Initial write issues are shown in the incache tests results.


Version-Release number of selected component (if applicable):
2.6.18-151.el5 and 2.6.18-152.el5

How reproducible: Run iozone incache test is the easiest to run and see regression in ReRead and FreRead results. Multiple tests on multiple platforms shows same regression


Steps to Reproduce:
1. run iozone
2. analysis results
3. generate reports
  
Actual results: One example is with Incache testing

IOZONE Analysis tool V1.4

FILE 1: results_rhel53U3_2.6.18-128.el5_perf4_4GB_taskset_cpu0/analysis/iozone_incache+fsync_default_analysis+rawdata.log
FILE 2: results_rhel5U4_2.6.18-152.el5_perf4_4GB_taskset_cpu0/analysis/iozone_incache+fsync_default_analysis+rawdata.log

TABLE:  SUMMARY of ALL FILE and RECORD SIZES                        Results in MB/sec

     FILE & REC      ALL  INIT   RE             RE   RANDOM RANDOM BACKWD  RECRE STRIDE  F      FRE     F      FRE 
FILE SIZES (KB)      IOS  WRITE  WRITE   READ   READ   READ  WRITE   READ  WRITE   READ  WRITE  WRITE   READ   READ
=====--------------------------------------------------------------------------------------------------------------
 1         ALL       832    129    189   2645   3069   2666    134   2481    985   2746    121    186   2794   3030 
 2         ALL       815    126    186   2709   2814   2586    133   2476    976   2758    118    185   2764   2812 
           ALL         .      .      .   +2.4   -8.3      .      .      .      .      .      .      .      .   -7.2 

Expected results:

All results should be within 5% 


Additional info: Full detailed analysis reports can be found http://perf1.lab.bos.redhat.com/perf4/bench_results/rhel5/U4/RHTS_iozone/full_analysis_rhel53_rhe54_152/
and 
http://perf1.lab.bos.redhat.com/perf7/bench_results/rhel5/U4/RHTS_iozone/full_analysis_rhel53_rhe54_152/
Comment 1 Josef Bacik 2009-06-17 17:20:32 UTC 
try -136 and -137 and see what the difference is.
Comment 2 Eric Sandeen 2009-06-17 17:33:30 UTC 
If no difference on 136 vs 137 can you guys do a binary chop on released kernels between 128 and 152 to see where the regression was?
Comment 3 Barry Marson 2009-06-17 17:42:26 UTC 
    In an attempt to reduce potential scheduler LCPU hopping issues, this version
    of the test binds the iozone process to LCPU 0.  This was done also due to
    issues with the the iozone runs for RHEL4.8.

    The above results are for an Intel based system but we see issues with AMD as
    well.

    I'll let ttracy add the AMD based information

    Barry
Comment 4 Tom Tracy 2009-06-17 18:48:29 UTC 
Links provided above, are for both Intel and AMD. What I am going to test the incache portion of iozone. This test show the following regression with these fields. I will test both the 136/137 kernels.

 AMD 

TABLE:  SUMMARY of ALL FILE and RECORD SIZES                        Results in MB/sec

     FILE & REC      ALL  INIT   RE             RE   RANDOM RANDOM BACKWD  RECRE STRIDE  F      FRE     F      FRE 
FILE SIZES (KB)      IOS  WRITE  WRITE   READ   READ   READ  WRITE   READ  WRITE   READ  WRITE  WRITE   READ   READ
=====--------------------------------------------------------------------------------------------------------------
 1         ALL      1029    292    752   1624   1751   1624    673   1546   1472   1676    288    731   1564   1701 
 2         ALL       976    274    728   1593   1596   1541    635   1495   1331   1623    270    706   1531   1556 
           ALL      -5.2   -6.3      .      .   -8.8   -5.1   -5.6      .   -9.5      .   -6.1      .      .   -8.5 

Intel
TABLE:  SUMMARY of ALL FILE and RECORD SIZES                        Results in MB/sec

     FILE & REC      ALL  INIT   RE             RE   RANDOM RANDOM BACKWD  RECRE STRIDE  F      FRE     F      FRE 
FILE SIZES (KB)      IOS  WRITE  WRITE   READ   READ   READ  WRITE   READ  WRITE   READ  WRITE  WRITE   READ   READ
=====--------------------------------------------------------------------------------------------------------------
 1         ALL      1625    478    923   2954   3155   2746    762   2609   2327   2810    464    924   2830   3085 
 2         ALL      1603    475    932   2964   2920   2697    750   2635   2267   2827    462    925   2848   2878 
           ALL         .      .      .      .   -7.4      .      .      .      .      .      .      .      .   -6.7
Comment 5 Tom Tracy 2009-06-18 12:25:31 UTC 
Incache testing shows no regression with 135 kernel. Having a hard time finding the 136 kernel so in the mean time, testing the 137 kernel.

IOZONE Analysis tool V1.4

FILE 1: results_rhel53U3_2.6.18-128.el5_perf4_4GB_taskset_cpu0/analysis/iozone_incache_default_analysis+rawdata.log
FILE 2: results_rhel53U3_2.6.18-135.el5_perf4_4GB_taskset_cpu0_incache/analysis/iozone_incache_default_analysis+rawdata.log

TABLE:  SUMMARY of ALL FILE and RECORD SIZES                        Results in MB/sec

     FILE & REC      ALL  INIT   RE             RE   RANDOM RANDOM BACKWD  RECRE STRIDE  F      FRE     F      FRE 
FILE SIZES (KB)      IOS  WRITE  WRITE   READ   READ   READ  WRITE   READ  WRITE   READ  WRITE  WRITE   READ   READ
=====--------------------------------------------------------------------------------------------------------------
 1         ALL      1625    478    923   2954   3155   2746    762   2609   2327   2810    464    924   2830   3085 
 2         ALL      1621    475    926   2897   3140   2738    769   2603   2305   2832    461    926   2824   3104 
           ALL         .      .      .      .      .      .      .      .      .      .      .      .      .      .
Comment 6 Tom Tracy 2009-06-18 15:38:51 UTC 
Ok, I found the regression in the 137 kernel. Still can not seem to find the 136 kernel. These results are in line with the 152 kernel. This was found in the Intel platform and will confirm it on the AMD platform

IOZONE Analysis tool V1.4

FILE 1: results_rhel53U3_2.6.18-128.el5_perf4_4GB_taskset_cpu0/analysis/iozone_incache_default_analysis+rawdata.log
FILE 2: results_rhel53U3_2.6.18-137.el5_perf4_4GB_taskset_cpu0_incache/analysis/iozone_incache_default_analysis+rawdata.log

TABLE:  SUMMARY of ALL FILE and RECORD SIZES                        Results in MB/sec

     FILE & REC      ALL  INIT   RE             RE   RANDOM RANDOM BACKWD  RECRE STRIDE  F      FRE     F      FRE 
FILE SIZES (KB)      IOS  WRITE  WRITE   READ   READ   READ  WRITE   READ  WRITE   READ  WRITE  WRITE   READ   READ
=====--------------------------------------------------------------------------------------------------------------
 1         ALL      1625    478    923   2954   3155   2746    762   2609   2327   2810    464    924   2830   3085 
 2         ALL      1591    478    924   2911   2861   2695    751   2612   2236   2810    457    921   2835   2858 
           ALL         .      .      .      .   -9.3      .      .      .      .      .      .      .      .   -7.3
Comment 7 Josef Bacik 2009-06-18 17:24:00 UTC 
sigh, thats what I was afraid of.  Its strange that the regression is on the read side since my patches didn't actually touch the read side.  Can I get the login for this box and the instructions on how to reproduce so I can start going through and figuring out which patch introduced the regression?  Thanks.
Comment 8 Tom Tracy 2009-06-18 17:37:04 UTC 
Josef
       Let me finish this test then I will hand over the keys. Incache tests shows the regression easily but it is not the only regression in the test suite. Another test which shows this behavior is the outcache portion of iozone. re write and FreWrite also shows this behavior with outcache test.
 I will send you mail when this test is complete and send you the information needed

Tom

IOZONE Analysis tool V1.4

FILE 1: results_rhel53U3_2.6.18-128.el5_perf4_4GB_taskset_cpu0/analysis/iozone_outcache_default_analysis+rawdata.log
FILE 2: results_rhel5U4_2.6.18-152.el5_perf4_4GB_taskset_cpu0/analysis/iozone_outcache_default_analysis+rawdata.log

TABLE:  SUMMARY of ALL FILE and RECORD SIZES                        Results in MB/sec

     FILE & REC      ALL  INIT   RE             RE   RANDOM RANDOM BACKWD  RECRE STRIDE  F      FRE     F      FRE 
FILE SIZES (KB)      IOS  WRITE  WRITE   READ   READ   READ  WRITE   READ  WRITE   READ  WRITE  WRITE   READ   READ
=====--------------------------------------------------------------------------------------------------------------
 1         ALL       313    337    343    525    499     83     34    273   3344    119    286    337    560    559 
 2         ALL       304    330    316    510    490     83     34    276   3219    121    285    292    548    523 
           ALL         .      .   -7.7      .      .      .      .      .      .      .      .  -13.3      .   -6.4
Comment 9 Josef Bacik 2009-06-18 17:50:24 UTC 
bugger, well write is more where i would expect the problem to happen, glad to see something makes sense at least.
Comment 10 Barry Marson 2009-06-18 19:24:06 UTC 
Tom,

Be careful on the out of cache test.  Remember the first file size is half of memory.  If the full analysis shows only that the half of memory file size regressed then you can ignore it.

Barry
Comment 11 Tom Tracy 2009-06-23 13:31:02 UTC 
Test AMD with the full suite of tests on both RHEL53 and RHEL54 152 kernel. Do not see the regression I am seeing now on the Intel box. Issue found was irqbalance was on. When turned off, there are actual performance gains depending on the type of tests. Analysis can be found on the link below. Still need to investigate Intel's system to find root cause of regression. 

http://perf1.lab.bos.redhat.com/perf7/bench_results/rhel5/U4/RHTS_iozone/?C=M;O=D
http://perf1.lab.bos.redhat.com/perf7/bench_results/rhel5/U4/RHTS_iozone/rhel53_rhel54_irqbalance_off_iozone_full_analysis.log
Comment 12 Tom Tracy 2009-06-26 14:07:27 UTC 
Rerun full set of tests on both AMD and Intel running the 155 kernel and the reread/freread issue still exists. This was been an issue since the 137 kernel. I tried all sorts of variables to see if the issue goes away on Intel box but nothing worked. I will send Josef the keys to the machines he requested. 

Full analysis is located here
http://perf1/perf7/bench_results/rhel5/U4/RHTS_iozone/results_ext3_perf7_rhel54_2.6.18-155.el5_full_analysis

[root@perf7 iozone]# ./analysis-iozone.pl -a results_ext3_perf7_rhel53_2.6.18-128.el5_taskset_cpu0/analysis/iozone_directio_default_analysis+rawdata.log results_ext3_perf7_rhel54_2.6.18-155.el5_taskset_cpu0/analysis/iozone_directio_default_analysis+rawdata.log

IOZONE Analysis tool V1.4

FILE 1: results_ext3_perf7_rhel53_2.6.18-128.el5_taskset_cpu0/analysis/iozone_directio_default_analysis+rawdata.log
FILE 2: results_ext3_perf7_rhel54_2.6.18-155.el5_taskset_cpu0/analysis/iozone_directio_default_analysis+rawdata.log

TABLE:  SUMMARY of ALL FILE and RECORD SIZES                        Results in MB/sec

     FILE & REC      ALL  INIT   RE             RE   RANDOM RANDOM BACKWD  RECRE STRIDE  F      FRE     F      FRE 
FILE SIZES (KB)      IOS  WRITE  WRITE   READ   READ   READ  WRITE   READ  WRITE   READ  WRITE  WRITE   READ   READ
=====--------------------------------------------------------------------------------------------------------------
 1         ALL       169     79     78     90     93     86     73     95     65     89    300    723   1474   1599 
 2         ALL       172     79     80     96    101     87     74     97     66     90    308    750   1483   1468 
           ALL         .      .   +2.4   +6.1   +8.0      .      .      .      .      .   +2.7   +3.8      .   -8.2 
[root@perf7 iozone]# ./analysis-iozone.pl -a results_ext3_perf7_rhel53_2.6.18-128.el5_taskset_cpu0/analysis/iozone_incache+fsync_default_analysis+rawdata.log results_ext3_perf7_rhel54_2.6.18-155.el5_taskset_cpu0/analysis/iozone_incache+fsync_default_analysis+rawdata.log

IOZONE Analysis tool V1.4

FILE 1: results_ext3_perf7_rhel53_2.6.18-128.el5_taskset_cpu0/analysis/iozone_incache+fsync_default_analysis+rawdata.log
FILE 2: results_ext3_perf7_rhel54_2.6.18-155.el5_taskset_cpu0/analysis/iozone_incache+fsync_default_analysis+rawdata.log

TABLE:  SUMMARY of ALL FILE and RECORD SIZES                        Results in MB/sec

     FILE & REC      ALL  INIT   RE             RE   RANDOM RANDOM BACKWD  RECRE STRIDE  F      FRE     F      FRE 
FILE SIZES (KB)      IOS  WRITE  WRITE   READ   READ   READ  WRITE   READ  WRITE   READ  WRITE  WRITE   READ   READ
=====--------------------------------------------------------------------------------------------------------------
 1         ALL       442     58     83   1552   1731   1591     63   1495    613   1635     56     82   1553   1697 
 2         ALL       440     60     87   1579   1600   1543     65   1496    584   1637     58     85   1578   1569 
           ALL         .   +2.4   +4.1      .   -7.5      .   +2.3      .      .      .   +2.9   +3.6      .   -7.5 
[root@perf7 iozone]# ./analysis-iozone.pl -a results_ext3_perf7_rhel53_2.6.18-128.el5_taskset_cpu0/analysis/iozone_outcache_default_analysis+rawdata.log results_ext3_perf7_rhel54_2.6.18-155.el5_taskset_cpu0/analysis/iozone_outcache_default_analysis+rawdata.log

IOZONE Analysis tool V1.4

FILE 1: results_ext3_perf7_rhel53_2.6.18-128.el5_taskset_cpu0/analysis/iozone_outcache_default_analysis+rawdata.log
FILE 2: results_ext3_perf7_rhel54_2.6.18-155.el5_taskset_cpu0/analysis/iozone_outcache_default_analysis+rawdata.log

TABLE:  SUMMARY of ALL FILE and RECORD SIZES                        Results in MB/sec

     FILE & REC      ALL  INIT   RE             RE   RANDOM RANDOM BACKWD  RECRE STRIDE  F      FRE     F      FRE 
FILE SIZES (KB)      IOS  WRITE  WRITE   READ   READ   READ  WRITE   READ  WRITE   READ  WRITE  WRITE   READ   READ
=====--------------------------------------------------------------------------------------------------------------
 1         ALL       185    174    169    254    247     75     33    192   1624    108    156    172    261    263 
 2         ALL       189    188    185    246    242     76     37    193   1597    104    166    179    265    258 
           ALL      +2.3   +8.2   +9.8      .      .      .  +13.0      .      .      .   +6.5   +3.9      .      . 
[root@perf7 iozone]# ./analysis-iozone.pl -a results_ext3_perf7_rhel53_2.6.18-128.el5_taskset_cpu0/analysis/iozone_incache_default_analysis+rawdata.log results_ext3_perf7_rhel54_2.6.18-155.el5_taskset_cpu0/analysis/iozone_incache_default_analysis+rawdata.log

IOZONE Analysis tool V1.4

FILE 1: results_ext3_perf7_rhel53_2.6.18-128.el5_taskset_cpu0/analysis/iozone_incache_default_analysis+rawdata.log
FILE 2: results_ext3_perf7_rhel54_2.6.18-155.el5_taskset_cpu0/analysis/iozone_incache_default_analysis+rawdata.log

TABLE:  SUMMARY of ALL FILE and RECORD SIZES                        Results in MB/sec

     FILE & REC      ALL  INIT   RE             RE   RANDOM RANDOM BACKWD  RECRE STRIDE  F      FRE     F      FRE 
FILE SIZES (KB)      IOS  WRITE  WRITE   READ   READ   READ  WRITE   READ  WRITE   READ  WRITE  WRITE   READ   READ
=====--------------------------------------------------------------------------------------------------------------
 1         ALL      1029    292    752   1624   1751   1624    673   1546   1472   1676    288    731   1564   1701 
 2         ALL      1012    290    779   1660   1616   1556    666   1533   1400   1642    285    750   1570   1563 
           ALL         .      .   +3.6   +2.2   -7.7      .      .      .      .      .      .   +2.6      .   -8.1 
[root@perf7 iozone]# ./analysis-iozone.pl -a results_ext3_perf7_rhel53_2.6.18-128.el5_taskset_cpu0/analysis/iozone_incache+mmap_default_analysis+rawdata.log results_ext3_perf7_rhel54_2.6.18-155.el5_taskset_cpu0/analysis/iozone_incache+mmap_default_analysis+rawdata.log

IOZONE Analysis tool V1.4

FILE 1: results_ext3_perf7_rhel53_2.6.18-128.el5_taskset_cpu0/analysis/iozone_incache+mmap_default_analysis+rawdata.log
FILE 2: results_ext3_perf7_rhel54_2.6.18-155.el5_taskset_cpu0/analysis/iozone_incache+mmap_default_analysis+rawdata.log

TABLE:  SUMMARY of ALL FILE and RECORD SIZES                        Results in MB/sec

     FILE & REC      ALL  INIT   RE             RE   RANDOM RANDOM BACKWD  RECRE STRIDE
FILE SIZES (KB)      IOS  WRITE  WRITE   READ   READ   READ  WRITE   READ  WRITE   READ
=====----------------------------------------------------------------------------------
 1         ALL      1276    592    917   1259   1315   1279    851   1273   4937   1457 
 2         ALL      1276    586    905   1259   1317   1273    842   1287   4995   1483 
           ALL         .      .      .      .      .      .      .      .      .      .
Comment 13 Josef Bacik 2009-06-26 19:54:07 UTC 
Ok, just to make sure I'm doing this right, with ./runtest -i i should be comparing those results against the 1'st row in c6 right?  If thats the case then fantastic, I've reproduced the regression.

  IOZONE             ALL  INIT   RE             RE   RANDOM RANDOM BACKWD  RECRE STRIDE  F      FRE     F      FRE
  RUN TYPE           IOS  WRITE  WRITE   READ   READ   READ  WRITE   READ  WRITE   READ  WRITE  WRITE   READ   READ
-------------------------------------------------------------------------------------------------------------------
In Cache            1590    481    921   2888   2870   2683    747   2586   2269   2797    464    926   2807   2843
Comment 14 Tom Tracy 2009-06-26 20:04:23 UTC 
Josef 
      You need to analyize your results comparing your kernel to the 128 kernel. 
After the run is complete, do the following. If you want to keep your results of the result, rename the results directory to something you can recognize 


Below compares your kernel to the 128 kernel. Note the regression is still there
./analysis-iozone.pl -a results_rhel53U3_2.6.18-128.el5_perf4_4GB_taskset_cpu0_incache_1/analysis/iozone_incache_default_analysis+rawdata.log results/analysis/iozone_incache_default_analysis+rawdata.log


IOZONE Analysis tool V1.4

FILE 1: results_rhel53U3_2.6.18-128.el5_perf4_4GB_taskset_cpu0_incache_1/analysis/iozone_incache_default_analysis+rawdata.log
FILE 2: results/analysis/iozone_incache_default_analysis+rawdata.log

Tom

TABLE:  SUMMARY of ALL FILE and RECORD SIZES                        Results in MB/sec

     FILE & REC      ALL  INIT   RE             RE   RANDOM RANDOM BACKWD  RECRE STRIDE  F      FRE     F      FRE 
FILE SIZES (KB)      IOS  WRITE  WRITE   READ   READ   READ  WRITE   READ  WRITE   READ  WRITE  WRITE   READ   READ
=====--------------------------------------------------------------------------------------------------------------
 1         ALL      1629    489    922   2944   3144   2743    762   2600   2339   2818    471    927   2830   3075 
 2         ALL      1590    481    921   2888   2870   2683    747   2586   2269   2797    464    926   2807   2843 
           ALL         .      .      .      .   -8.7      .      .      .      .      .      .      .      .   -7.5
Comment 15 Josef Bacik 2009-06-27 18:51:03 UTC 
Ok tested the -136 kernel, no regression.  I'm starting a git bisect now, there are only 25 patches to test, so it should take 4-5 runs to narrow down which patch introduced the regression.
Comment 16 Josef Bacik 2009-06-30 15:02:47 UTC 
ok found the problem, i'm testing a patch to make sure it fixes the problem.  hopefully i will be able to send this to rhkernel list this afternoon.
Comment 17 Josef Bacik 2009-06-30 18:24:02 UTC 
Created attachment 349997 [details]
patch to fix the problem

Ok here's a patch to fix the problem.  We weren't marking the page as accessed when writing to it, so we'd flush it out of pagecache too soon, so when we go back to read the page we'd have to read it from disk, which is obviously slower.  With this patch we mark the page as accessed so it doesn't get flushed out too soon so when we re-read the pages we've written to we are good.  With this patch applied, the test ran with no regressions

     FILE & REC      ALL  INIT   RE             RE   RANDOM RANDOM BACKWD  RECRE STRIDE  F      FRE     F      FRE
FILE SIZES (KB)      IOS  WRITE  WRITE   READ   READ   READ  WRITE   READ  WRITE   READ  WRITE  WRITE   READ   READ
=====--------------------------------------------------------------------------------------------------------------
 1         ALL      1629    489    922   2944   3144   2743    762   2600   2339   2818    471    927   2830   3075
 2         ALL      1624    486    911   2958   3162   2752    752   2621   2235   2829    472    912   2852   3135
           ALL         .      .      .      .      .      .      .      .      .      .      .      .      .      .
Comment 18 Tom Tracy 2009-06-30 20:10:10 UTC 
Josef
This is good news. I am starting a complete iozone tests on the box now which will run overnight. Would like to confirm this patch. Is there a way to get an kernel built so I can test it on AMD box

thanks
Tom
Comment 19 Josef Bacik 2009-06-30 20:31:23 UTC 
yeah i put something through brew tomorrow.
Comment 20 Tom Tracy 2009-07-01 13:54:37 UTC 
Did a complete set of runs last night and now I see regression with the following tests. Really only concerned with the outcache results since that shows the largest regression with theFreWrite. Will test AMD box once I get the new kernel 

Fsync  RECreWrite -5.1%  FreWrite -5.0%  
OutCache  ReWrite -7.1%  FReWrite -13.7%
DirectIO Initial Write -5.1%.

Summary analysis

http://perf1/perf4/bench_results/rhel5/U4/RHTS_iozone/analysis_results_2.6.18.4_josef_kernel

Full analysis

http://perf1/perf4/bench_results/rhel5/U4/RHTS_iozone/full_analysis_results_2.6.18.4_josef_kernel
Comment 21 Josef Bacik 2009-07-01 14:59:17 UTC 
here is the brew build

http://brewweb.devel.redhat.com/brew/taskinfo?taskID=1870132
Comment 22 Tom Tracy 2009-07-02 15:01:04 UTC 
No regression found on AMD system running full suite of tests. Going to double check Intel over the holiday. Only concern is the outcache results which is why I will rerun.

Results for AMD 

[root@perf7 iozone]# ./analysis-iozone.pl -a results_ext3_perf7_rhel53_2.6.18-128.el5_taskset_cpu0/analysis/iozone_directio_default_analysis+rawdata.log results_ext3_perf7_rhel54_2.6.18-156.el5/analysis/iozone_directio_default_analysis+rawdata.log

IOZONE Analysis tool V1.4

FILE 1: results_ext3_perf7_rhel53_2.6.18-128.el5_taskset_cpu0/analysis/iozone_directio_default_analysis+rawdata.log
FILE 2: results_ext3_perf7_rhel54_2.6.18-156.el5/analysis/iozone_directio_default_analysis+rawdata.log

TABLE:  SUMMARY of ALL FILE and RECORD SIZES                        Results in MB/sec

     FILE & REC      ALL  INIT   RE             RE   RANDOM RANDOM BACKWD  RECRE STRIDE  F      FRE     F      FRE 
FILE SIZES (KB)      IOS  WRITE  WRITE   READ   READ   READ  WRITE   READ  WRITE   READ  WRITE  WRITE   READ   READ
=====--------------------------------------------------------------------------------------------------------------
 1         ALL       169     79     78     90     93     86     73     95     65     89    300    723   1474   1599 
 2         ALL       173     80     80     94    100     87     74     97     65     90    319    732   1474   1567 
           ALL         .      .   +2.7   +3.9   +7.5      .      .      .      .      .   +6.3      .      .      . 
[root@perf7 iozone]# ./analysis-iozone.pl -a results_ext3_perf7_rhel53_2.6.18-128.el5_taskset_cpu0/analysis/iozone_incache+mmap_default_analysis+rawdata.log results_ext3_perf7_rhel54_2.6.18-156.el5/analysis/iozone_incache+mmap_default_analysis+rawdata.log

IOZONE Analysis tool V1.4

FILE 1: results_ext3_perf7_rhel53_2.6.18-128.el5_taskset_cpu0/analysis/iozone_incache+mmap_default_analysis+rawdata.log
FILE 2: results_ext3_perf7_rhel54_2.6.18-156.el5/analysis/iozone_incache+mmap_default_analysis+rawdata.log

TABLE:  SUMMARY of ALL FILE and RECORD SIZES                        Results in MB/sec

     FILE & REC      ALL  INIT   RE             RE   RANDOM RANDOM BACKWD  RECRE STRIDE
FILE SIZES (KB)      IOS  WRITE  WRITE   READ   READ   READ  WRITE   READ  WRITE   READ
=====----------------------------------------------------------------------------------
 1         ALL      1276    592    917   1259   1315   1279    851   1273   4937   1457 
 2         ALL      1274    579    898   1255   1302   1274    837   1303   5042   1481 
           ALL         .      .      .      .      .      .      .   +2.3   +2.1      . 
[root@perf7 iozone]# ./analysis-iozone.pl -a results_ext3_perf7_rhel53_2.6.18-128.el5_taskset_cpu0/analysis/iozone_incache_default_analysis+rawdata.log results_ext3_perf7_rhel54_2.6.18-156.el5/analysis/iozone_incache_default_analysis+rawdata.log

IOZONE Analysis tool V1.4

FILE 1: results_ext3_perf7_rhel53_2.6.18-128.el5_taskset_cpu0/analysis/iozone_incache_default_analysis+rawdata.log
FILE 2: results_ext3_perf7_rhel54_2.6.18-156.el5/analysis/iozone_incache_default_analysis+rawdata.log

TABLE:  SUMMARY of ALL FILE and RECORD SIZES                        Results in MB/sec

     FILE & REC      ALL  INIT   RE             RE   RANDOM RANDOM BACKWD  RECRE STRIDE  F      FRE     F      FRE 
FILE SIZES (KB)      IOS  WRITE  WRITE   READ   READ   READ  WRITE   READ  WRITE   READ  WRITE  WRITE   READ   READ
=====--------------------------------------------------------------------------------------------------------------
 1         ALL      1029    292    752   1624   1751   1624    673   1546   1472   1676    288    731   1564   1701 
 2         ALL      1021    297    764   1641   1729   1558    669   1503   1405   1622    294    742   1569   1672 
           ALL         .      .      .      .      .      .      .      .      .      .   +2.1      .      .      . 
[root@perf7 iozone]# ./analysis-iozone.pl -a results_ext3_perf7_rhel53_2.6.18-128.el5_taskset_cpu0/analysis/iozone_incache+fsync_default_analysis+rawdata.log results_ext3_perf7_rhel54_2.6.18-156.el5/analysis/iozone_incache+fsync_default_analysis+rawdata.log

IOZONE Analysis tool V1.4

FILE 1: results_ext3_perf7_rhel53_2.6.18-128.el5_taskset_cpu0/analysis/iozone_incache+fsync_default_analysis+rawdata.log
FILE 2: results_ext3_perf7_rhel54_2.6.18-156.el5/analysis/iozone_incache+fsync_default_analysis+rawdata.log

TABLE:  SUMMARY of ALL FILE and RECORD SIZES                        Results in MB/sec

     FILE & REC      ALL  INIT   RE             RE   RANDOM RANDOM BACKWD  RECRE STRIDE  F      FRE     F      FRE 
FILE SIZES (KB)      IOS  WRITE  WRITE   READ   READ   READ  WRITE   READ  WRITE   READ  WRITE  WRITE   READ   READ
=====--------------------------------------------------------------------------------------------------------------
 1         ALL       442     58     83   1552   1731   1591     63   1495    613   1635     56     82   1553   1697 
 2         ALL       446     61     88   1538   1686   1540     66   1457    613   1623     58     87   1560   1677 
           ALL         .   +3.7   +5.4      .      .      .   +4.6      .      .      .   +4.3   +6.3      .      . 
[root@perf7 iozone]# ./analysis-iozone.pl -a results_ext3_perf7_rhel53_2.6.18-128.el5_taskset_cpu0/analysis/iozone_incache+mmap_default_analysis+rawdata.log results_ext3_perf7_rhel54_2.6.18-156.el5/analysis/iozone_incache+mmap_default_analysis+rawdata.log

IOZONE Analysis tool V1.4

FILE 1: results_ext3_perf7_rhel53_2.6.18-128.el5_taskset_cpu0/analysis/iozone_incache+mmap_default_analysis+rawdata.log
FILE 2: results_ext3_perf7_rhel54_2.6.18-156.el5/analysis/iozone_incache+mmap_default_analysis+rawdata.log

TABLE:  SUMMARY of ALL FILE and RECORD SIZES                        Results in MB/sec

     FILE & REC      ALL  INIT   RE             RE   RANDOM RANDOM BACKWD  RECRE STRIDE
FILE SIZES (KB)      IOS  WRITE  WRITE   READ   READ   READ  WRITE   READ  WRITE   READ
=====----------------------------------------------------------------------------------
 1         ALL      1276    592    917   1259   1315   1279    851   1273   4937   1457 
 2         ALL      1274    579    898   1255   1302   1274    837   1303   5042   1481 
           ALL         .      .      .      .      .      .      .   +2.3   +2.1      .
Comment 24 Tom Tracy 2009-07-06 12:51:31 UTC 
Ok, reruns over the weekends confirms Josef's change fixes the regression. Will this change be in time for RHEL54 release

[root@perf4 iozone]# ./analysis-iozone.pl -a results_rhel53U3_2.6.18-128.el5_perf4_4GB_taskset_cpu0/analysis/iozone_directio_default_analysis+rawdata.log results_josef_2.6.18.4_repeat/analysis/iozone_directio_default_analysis+rawdata.log

IOZONE Analysis tool V1.4

FILE 1: results_rhel53U3_2.6.18-128.el5_perf4_4GB_taskset_cpu0/analysis/iozone_directio_default_analysis+rawdata.log
FILE 2: results_josef_2.6.18.4_repeat/analysis/iozone_directio_default_analysis+rawdata.log

TABLE:  SUMMARY of ALL FILE and RECORD SIZES                        Results in MB/sec

     FILE & REC      ALL  INIT   RE             RE   RANDOM RANDOM BACKWD  RECRE STRIDE  F      FRE     F      FRE 
FILE SIZES (KB)      IOS  WRITE  WRITE   READ   READ   READ  WRITE   READ  WRITE   READ  WRITE  WRITE   READ   READ
=====--------------------------------------------------------------------------------------------------------------
 1         ALL       337    154    149    223    246    188    123    211    109    198    549   1044   2973   3175 
 2         ALL       341    152    152    223    242    183    129    210    131    195    529   1032   2994   3178 
           ALL         .      .   +2.1      .      .      .   +5.2      .  +20.4      .      .      .      .      . 
[root@perf4 iozone]# ./analysis-iozone.pl -a results_rhel53U3_2.6.18-128.el5_perf4_4GB_taskset_cpu0/analysis/iozone_incache+fsync_default_analysis+rawdata.log results_josef_2.6.18.4_repeat/analysis/iozone_incache+fsync_default_analysis+rawdata.log

IOZONE Analysis tool V1.4

FILE 1: results_rhel53U3_2.6.18-128.el5_perf4_4GB_taskset_cpu0/analysis/iozone_incache+fsync_default_analysis+rawdata.log
FILE 2: results_josef_2.6.18.4_repeat/analysis/iozone_incache+fsync_default_analysis+rawdata.log

TABLE:  SUMMARY of ALL FILE and RECORD SIZES                        Results in MB/sec

     FILE & REC      ALL  INIT   RE             RE   RANDOM RANDOM BACKWD  RECRE STRIDE  F      FRE     F      FRE 
FILE SIZES (KB)      IOS  WRITE  WRITE   READ   READ   READ  WRITE   READ  WRITE   READ  WRITE  WRITE   READ   READ
=====--------------------------------------------------------------------------------------------------------------
 1         ALL       832    129    189   2645   3069   2666    134   2481    985   2746    121    186   2794   3030 
 2         ALL       832    129    190   2712   3076   2649    131   2501   1009   2743    117    184   2783   3064 
           ALL         .      .      .   +2.5      .      .      .      .   +2.4      .      .      .      .      . 
[root@perf4 iozone]# ./analysis-iozone.pl -a results_rhel53U3_2.6.18-128.el5_perf4_4GB_taskset_cpu0/analysis/iozone_outcache_default_analysis+rawdata.log results_josef_2.6.18.4_repeat/analysis/iozone_outcache_default_analysis+rawdata.log

IOZONE Analysis tool V1.4

FILE 1: results_rhel53U3_2.6.18-128.el5_perf4_4GB_taskset_cpu0/analysis/iozone_outcache_default_analysis+rawdata.log
FILE 2: results_josef_2.6.18.4_repeat/analysis/iozone_outcache_default_analysis+rawdata.log

TABLE:  SUMMARY of ALL FILE and RECORD SIZES                        Results in MB/sec

     FILE & REC      ALL  INIT   RE             RE   RANDOM RANDOM BACKWD  RECRE STRIDE  F      FRE     F      FRE 
FILE SIZES (KB)      IOS  WRITE  WRITE   READ   READ   READ  WRITE   READ  WRITE   READ  WRITE  WRITE   READ   READ
=====--------------------------------------------------------------------------------------------------------------
 1         ALL       313    337    343    525    499     83     34    273   3344    119    286    337    560    559 
 2         ALL       310    330    333    526    508     83     34    274   3276    124    280    309    545    554 
           ALL         .      .      .      .      .      .   +2.3      .      .   +4.3      .   -8.4      .      . 
[root@perf4 iozone]# ./analysis-iozone.pl -a results_rhel53U3_2.6.18-128.el5_perf4_4GB_taskset_cpu0/analysis/iozone_incache_default_analysis+rawdata.log results_josef_2.6.18.4_repeat/analysis/iozone_incache_default_analysis+rawdata.log

IOZONE Analysis tool V1.4

FILE 1: results_rhel53U3_2.6.18-128.el5_perf4_4GB_taskset_cpu0/analysis/iozone_incache_default_analysis+rawdata.log
FILE 2: results_josef_2.6.18.4_repeat/analysis/iozone_incache_default_analysis+rawdata.log

TABLE:  SUMMARY of ALL FILE and RECORD SIZES                        Results in MB/sec

     FILE & REC      ALL  INIT   RE             RE   RANDOM RANDOM BACKWD  RECRE STRIDE  F      FRE     F      FRE 
FILE SIZES (KB)      IOS  WRITE  WRITE   READ   READ   READ  WRITE   READ  WRITE   READ  WRITE  WRITE   READ   READ
=====--------------------------------------------------------------------------------------------------------------
 1         ALL      1625    478    923   2954   3155   2746    762   2609   2327   2810    464    924   2830   3085 
 2         ALL      1609    464    907   2942   3174   2733    753   2609   2243   2818    453    912   2833   3097 
           ALL         .      .      .      .      .      .      .      .      .      .      .      .      .      . 
[root@perf4 iozone]# ./analysis-iozone.pl -a results_rhel53U3_2.6.18-128.el5_perf4_4GB_taskset_cpu0/analysis/iozone_incache+mmap_default_analysis+rawdata.log results_josef_2.6.18.4_repeat/analysis/iozone_incache+mmap_default_analysis+rawdata.log

IOZONE Analysis tool V1.4

FILE 1: results_rhel53U3_2.6.18-128.el5_perf4_4GB_taskset_cpu0/analysis/iozone_incache+mmap_default_analysis+rawdata.log
FILE 2: results_josef_2.6.18.4_repeat/analysis/iozone_incache+mmap_default_analysis+rawdata.log

TABLE:  SUMMARY of ALL FILE and RECORD SIZES                        Results in MB/sec

     FILE & REC      ALL  INIT   RE             RE   RANDOM RANDOM BACKWD  RECRE STRIDE
FILE SIZES (KB)      IOS  WRITE  WRITE   READ   READ   READ  WRITE   READ  WRITE   READ
=====----------------------------------------------------------------------------------
 1         ALL      1924    904    998   2173   2298   2171    852   2134   7833   2591 
 2         ALL      1919    905   1000   2162   2249   2171    856   2126   7834   2596 
           ALL         .      .      .      .      .      .      .      .      .      .
Comment 26 RHEL Program Management 2009-07-06 15:03:16 UTC 
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.
Comment 28 Lachlan McIlroy 2009-07-13 07:34:49 UTC 
Adding issue 316510.
Comment 29 Don Zickus 2009-07-14 20:57:29 UTC 
in kernel-2.6.18-158.el5
You can download this test kernel from http://people.redhat.com/dzickus/el5

Please do NOT transition this bugzilla state to VERIFIED until our QE team
has sent specific instructions indicating when to do so.  However feel free
to provide a comment indicating that this fix has been verified.
Comment 32 Josef Bacik 2009-07-15 12:21:56 UTC 
Well thats not helpful at all.  What are these numbers for?  Reads?  Writes?  Re-reads?  Going to have to provide more information than a table of numbers.
Comment 33 Tom Tracy 2009-07-15 13:28:34 UTC 
Installed the 158 kernel and running iozone on both Intel/AMD. Will take 2 days to complete. Josef, what do you mean by the comment above? Iozone results?
Comment 34 Josef Bacik 2009-07-15 13:34:50 UTC 
It was in re c31.
Comment 35 Tom Tracy 2009-07-16 12:51:23 UTC 
Testing 158 kernel with Iozone on Intel shows none of the regression characteristics we were seeing in previous rhe54 kernels. Outcache results do show slight regression with rewrite and frewrite but this is completely different than previous test results.  Amd iozone testing has been started 

You see see the full results at the following

 http://perf1.lab.bos.redhat.com/perf4/bench_results/rhel5/U4/RHTS_iozone/analysis_results_2.6.18.158_kernel
Comment 36 Tom Tracy 2009-07-20 13:34:01 UTC 
No regressions running Iozone on AMD. In fact, performance gains in Reread running DirectIo of 10%. F-Writes also experience performance gains averaging 5% gains on all tests. Full results can be located in the link below

http://perf1.lab.bos.redhat.com/perf7/bench_results/rhel5/U4/RHTS_iozone/full_analysis_rhel53_128kernel_rhel54_158_kernel
Comment 37 Jan Tluka 2009-07-20 16:34:28 UTC 
Patch is in -158.el5. Adding SanityOnly.
Comment 38 Issue Tracker 2009-07-21 11:18:13 UTC 
Event posted on 07-21-2009 08:17pm JST by mfuruta

Tatsukawa-san,

Sorry for delayed response, but could you please elaborate your concern
about this as you would mention as below?

----
since it seems that BZ#506511 is different from this issue, could you open
a separate bug to track this issue? 
----

Thanks in advance,

Regards,
Masaki Furuta


Internal Status set to 'Waiting on Customer'
Status set to: Waiting on Client

This event sent from IssueTracker by mfuruta 
 issue 316510
Comment 39 Issue Tracker 2009-07-21 13:41:03 UTC 
Event posted on 07-16-2009 07:07am JST by tatsu.nec.com

> Well thats not helpful at all.  What are these numbers for?  Reads? 
Writes?  Re-reads?  Going to have to provide more information than a table
of numbers.

These are write performance.  Below is from IssueTracker comment.

Summary: File write performance degradation in RHEL5.4 Snapshot1 compared
to RHEL5.3 GA 
...
1. Install RHEL5.4 GA Snapshot1.
2. Boot RHEL5.4.
3. Execute iozone and measure the performance by following procedure.
3.1) Extract iozone contents in some directory (for example /home)
 ex) # cd /home
     # tar zxf iozone*.tar.gz
3.2) Change the directory as follows, and make.
     # cd iozone*/src/current
     # make linux (case of i386)
3.3) Prepare about 4G of space partition, make an optional partition and
mount.
 ex) # mkdir -p /mnt/disk1
     # mount /dev/sdb1 /mnt/disk1
3.4) Input to command line as follows, and execute.
     # ./iozone -+n -Q -i 0 -i 1 -s 1024m -r 1 -r 2 -r 4 -r 8 -r 16 -r 32
-r 64 -f /mnt/disk1/testfile
4. The result of write and read performance is collected.
5. Install RHEL5.3 GA.
6. Boot RHEL5.3.
7. Execute iozone and measured result like Step 3.



This event sent from IssueTracker by mfuruta 
 issue 316510
Comment 40 Kosuke TATSUKAWA 2009-07-22 07:13:40 UTC 
I'm sending summary information from Issue 316510.
Sorry for the rather long comment.

+ Write results from iozone shows large performance drop from RHEL5.3 to
  RHEL5.4 beta when writing 1GB files with iozone, especially with large
  record size (measured on server with 16GB memory).

  ./iozone -+n -Q -i 0 -i 1 -s 1024m -r 1 -r 2 -r 4 -r 8 -r 16 -r 32 -r 64 -f /mnt/disk1/testfile

  The patch which fixes BZ#506511 does not seem to resolve this issue.
  Also, the performance degradation seems to be introduced between
  136.el5PAE and 138.el5PAE (The binary packages for these two kernels
  happened to be available when we tried analyzing the problem after
  beta release).

  [write result (MB/s)]
   record size    1kB     2kB     4kB     8kB    16kB    32kB    64kB
   128.el5PAE   177.52  300.21  518.14  694.12  825.29  930.80 1002.65
   136.el5PAE   168.83  287.14  520.10  695.58  829.76  931.53 1003.23
   138.el5PAE   158.69  268.08  384.04  463.74  531.64  576.59  602.55
   156.el5PAE   165.51  263.13  374.06  454.43  526.04  574.00  601.36
                 (-7%)  (-12%)  (-28%)  (-34%)  (-36%)  (-38%)  (-40%)
   158.el5PAE   152.60  266.89  380.05  457.33  527.84  573.06  598.81
                (-14%)  (-11%)  (-27%)  (-34%)  (-36%)  (-38%)  (-40%)

  A large number of mm and fs changes were introduced between 136 and
  138, so we think that one of the patches might have caused this
  performance drop (we are guessing
  linux-2.6-mm-fix-pagecache-write-deadlocks.patch, but it is
  accompanied by
  linux-2.6-mm-introduce-new-aops-write_begin-and-write_end.patch which
  is said to avoid the performance drop from this patch).

+ Below is the write and rewrite result from iozone (taken from "-a"
  output), measured with fixed record size of 64k and varying the file
  size from 1GB to 8GB (on a server with 4GB memory).

        [write performance]   [rewrite performance]
       128.el5PAE 156.el5PAE  128.el5PAE 156.el5PAE
  1GB   1072293    628430      1557671    1532492
  2GB    196562    186001       207761     206808
  4GB     58794     57962        64604      63951
  8GB     42273     41823        47480      48117

  It seems that writes to disk have almost the same performance between
  RHEL5.3 and RHEL5.4 beta.  Also from the rewrite result, writes to
  existing page cache have not slowed down.  If either the page cache
  gets written back to disk, or reused when containing hot data, the
  performance should be similar between RHEL5.3 and RHEL5.4 beta.

  So performance seems to drop only for writes to cold page cache.
  If such case is unlikey for customer's real workloads, then this
  problem may not have real customer impact.  However we think that
  small benchmark problems might be affected by this.
Comment 41 Josef Bacik 2009-07-22 14:38:14 UTC 
Ok, so cold-cache performance has dropped.  This is not something that is blocker worthy, so create a new bugzilla for this issue and make sure I'm assigned to the new BZ and we'll set it for 5.5 and I'll start looking into it.  Please test this on x86_64 to make sure this isn't arch specific.  I need to know this to help narrow down where to look for the problem.  Thanks.
Comment 42 Barry Marson 2009-07-22 14:50:40 UTC 
I will shortly have iozone data from a system in RHTS that's testing the -158 kernel and also has collected data for RHEL5.3 run.  Preliminary analysis is not showing a problem with initial writes, but the data point tests large record sizes too.  Once I do the full analysis, I will be able to determine if we have recreated this.

Barry
Comment 43 Kosuke TATSUKAWA 2009-07-22 23:54:19 UTC 
> Ok, so cold-cache performance has dropped.  This is not something that is
> blocker worthy, so create a new bugzilla for this issue and make sure I'm
> assigned to the new BZ and we'll set it for 5.5 and I'll start looking into it.
>  Please test this on x86_64 to make sure this isn't arch specific.  I need to
> know this to help narrow down where to look for the problem.  Thanks.  

BZ#513136 is opened for this issue now.
It's currently assigned to kernel-mgr

Performance on x86_64 also drops, but shows better results compared to i386
[iozone write result for 1GB file with 64kB record size]
         -128    -156
i386    1002.65  601.36 (-40%)
x86_64  1115.91  935.55 (-16%)
Comment 44 Tom Tracy 2009-07-23 14:57:37 UTC 
Can you verify that your testing the with the 158 kernel which has the patch to this bugzilla. We have tested this kernel extensively and do not see the regression you are seeing. The 156 kernel does not have the patch to this bugzilla or is it a typo
Comment 45 Kosuke TATSUKAWA 2009-07-23 23:28:38 UTC 
> Can you verify that your testing the with the 158 kernel which has the patch to
> this bugzilla. We have tested this kernel extensively and do not see the
> regression you are seeing. The 156 kernel does not have the patch to this
> bugzilla or is it a typo

Please see Comment #40 and #43.  It's a different problem from this bug, and the
discussion is going on in bug #513136 now (except for the replies to the comments
here).
Comment 47 errata-xmlrpc 2009-09-02 08:24:56 UTC 
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2009-1243.html