Bug 852571

Summary:	iozone fails with Bad file descriptor - stripe-replicate
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Vidya Sakar <vinaraya>
Component:	glusterd	Assignee:	vpshastry <vshastry>
Status:	CLOSED ERRATA	QA Contact:	Ben Turner <bturner>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	2.0	CC:	amarts, bturner, gluster-bugs, jdarcy, lakshmipathi, nsathyan, nuaa_liuben, rfortier, rhs-bugs, rme, vbellur
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	glusterfs-3.4.0qa6	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:	GLUSTER-3108	Environment:
Last Closed:	2013-09-23 22:38:59 UTC	Type:	---
Regression:	RTP	Mount Type:	nfs
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	764840
Bug Blocks:

Description Vidya Sakar 2012-08-29 00:55:58 UTC

+++ This bug was initially created as a clone of Bug #764840 +++

While running iozone on nfs with following setup it fails -
# gluster volume info
 
Volume Name: dsr
Type: Striped-Replicate (RAID 01)
Status: Started
Number of Bricks: 2 x 3 = 6
Transport-type: tcp
Bricks:
Brick1: 10.1.12.172:/export/dsr1
Brick2: 10.1.12.172:/export/dsr2
Brick3: 10.1.12.170:/export/dsr1
Brick4: 10.1.12.170:/export/dsr2
Brick5: 10.1.12.173:/export/dsr1
Brick6: 10.1.12.173:/export/dsr2
Options Reconfigured:
diagnostics.client-log-level: TRACE
diagnostics.brick-log-level: TRACE


iozone 
-----
            128      16 2608629 2511022    79846  6406138 4759253 4104338 4596273 4267461 5847904  2140585  2326073   76785  5847904
             128      32 2784517 3657016    66086  6727225 3759450 4135958   41077 4267461 5545860  3359523  3359523   78092  6045455
             128      64 2770150 3199360    66834  7082197 4557257 3359523 5784891 3560017 5847904  1406136  4596273   77956  6406138
             128     128 2166499 2558895    89750  7176872 6406138 4104338   87483 3124872 4557257  3468030  3277486   78183  3982553
             256       4 2170027 2665657
Error reading block 32 2b0341500000
read: Bad file descriptor


nfslog-

[2011-06-29 05:22:04.58028] T [rpc-clnt.c:641:rpc_clnt_reply_init] 0-dsr-client-4: received rpc message (RPC XID: 0x1415x Program: GlusterFS 3.1, ProgVers: 310, Proc: 1) from rpc-transport (dsr-client-4)
[2011-06-29 05:22:04.58400] T [rpc-clnt.c:641:rpc_clnt_reply_init] 0-dsr-client-4: received rpc message (RPC XID: 0x1416x Program: GlusterFS 3.1, ProgVers: 310, Proc: 1) from rpc-transport (dsr-client-4)
[2011-06-29 05:22:04.58423] T [rpc-clnt.c:641:rpc_clnt_reply_init] 0-dsr-client-0: received rpc message (RPC XID: 0x3139x Program: GlusterFS 3.1, ProgVers: 310, Proc: 1) from rpc-transport (dsr-client-0)
[2011-06-29 05:22:04.58507] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: 7716401c, ACCESS: NFS: 0(Call completed successfully.), POSIX: 0(Success)
[2011-06-29 05:22:04.58530] T [rpcsvc.c:1675:nfs_rpcsvc_submit_generic] 0-nfsrpc: Tx message: 96
[2011-06-29 05:22:04.58546] T [rpcsvc.c:1456:nfs_rpcsvc_record_build_header] 0-nfsrpc: Reply fraglen 120, payload: 96, rpc hdr: 24
[2011-06-29 05:22:04.58554] T [rpcsvc.c:1493:nfs_rpcsvc_conn_submit] 0-nfsrpc: Tx Header: 28, payload: 96
[2011-06-29 05:22:04.58582] T [rpcsvc.c:2571:__nfs_rpcsvc_conn_data_poll_out] 0-nfsrpc: First Tx Buf
[2011-06-29 05:22:04.58613] T [rpcsvc.c:2582:__nfs_rpcsvc_conn_data_poll_out] 0-nfsrpc: conn: 0x5dff000, Tx request: 28, Tx sent: 28
[2011-06-29 05:22:04.58647] T [rpcsvc.c:2578:__nfs_rpcsvc_conn_data_poll_out] 0-nfsrpc: Last Tx Buf
[2011-06-29 05:22:04.58595] T [rpc-clnt.c:641:rpc_clnt_reply_init] 0-dsr-client-0: received rpc message (RPC XID: 0x3140x Program: GlusterFS 3.1, ProgVers: 310, Proc: 1) from rpc-transport (dsr-client-0)
[2011-06-29 05:22:04.58673] T [rpcsvc.c:2582:__nfs_rpcsvc_conn_data_poll_out] 0-nfsrpc: conn: 0x5dff000, Tx request: 96, Tx sent: 96
[2011-06-29 05:22:04.58709] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: 7816401c, ACCESS: NFS: 0(Call completed successfully.), POSIX: 0(Success)
[2011-06-29 05:22:04.58742] T [rpcsvc.c:1675:nfs_rpcsvc_submit_generic] 0-nfsrpc: Tx message: 96
[2011-06-29 05:22:04.58751] T [rpcsvc.c:1456:nfs_rpcsvc_record_build_header] 0-nfsrpc: Reply fraglen 120, payload: 96, rpc hdr: 24
[2011-06-29 05:22:04.58758] T [rpcsvc.c:1493:nfs_rpcsvc_conn_submit] 0-nfsrpc: Tx Header: 28, payload: 96
[2011-06-29 05:22:04.58779] T [rpcsvc.c:2571:__nfs_rpcsvc_conn_data_poll_out] 0-nfsrpc: First Tx Buf
[2011-06-29 05:22:04.58793] T [rpcsvc.c:2582:__nfs_rpcsvc_conn_data_poll_out] 0-nfsrpc: conn: 0x5dff000, Tx request: 28, Tx sent: 28
[2011-06-29 05:22:04.58802] T [rpcsvc.c:2578:__nfs_rpcsvc_conn_data_poll_out] 0-nfsrpc: Last Tx Buf
[2011-06-29 05:22:04.58820] T [rpcsvc.c:2582:__nfs_rpcsvc_conn_data_poll_out] 0-nfsrpc: conn: 0x5dff000, Tx request: 96, Tx sent: 96
[2011-06-29 05:22:16.220841] D [client-handshake.c:179:client_start_ping] 0-dsr-client-3: returning as transport is already disconnected OR there are no frames (0 || 0)
[2011-06-29 05:22:16.220885] D [client-handshake.c:179:client_start_ping] 0-dsr-client-1: returning as transport is already disconnected OR there are no frames (0 || 0)
[2011-06-29 05:22:16.220894] D [client-handshake.c:179:client_start_ping] 0-dsr-client-2: returning as transport is already disconnected OR there are no frames (0 || 0)
[2011-06-29 05:22:16.220902] D [client-handshake.c:179:client_start_ping] 0-dsr-client-5: returning as transport is already disconnected OR there are no frames (0 || 0)
[2011-06-29 05:22:24.233837] D [client-handshake.c:179:client_start_ping] 0-dsr-client-4: returning as transport is already disconnected OR there are no frames (0 || 0)
[2011-06-29 05:22:24.233891] D [client-handshake.c:179:client_start_ping] 0-dsr-client-0: returning as transport is already disconnected OR there are no frames (0 || 0)



glfs-log-
========

[2011-06-29 05:21:43.939025] T [rpcsvc.c:443:rpcsvc_handle_rpc_call] 0-rpcsvc: Client port: 1010
[2011-06-29 05:21:43.939044] T [rpcsvc-auth.c:302:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS
[2011-06-29 05:21:43.939052] T [rpcsvc.c:356:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0xc42, Ver: 2, Program: 1298437, ProgVers: 310, Proc: 12) from rpc-transport (tcp.dsr-server)
[2011-06-29 05:21:43.939060] T [auth-glusterfs.c:191:auth_glusterfs_authenticate] 0-rpc-service: Auth Info: pid: 1, uid: 0, gid: 0, owner: 1357
[2011-06-29 05:21:43.939067] T [rpcsvc.c:187:rpcsvc_program_actor] 0-rpc-service: Actor found: GlusterFS-3.1.0 - READ
[2011-06-29 05:21:43.939121] T [rpcsvc.c:1003:rpcsvc_submit_generic] 0-rpc-service: Tx message: 16496
[2011-06-29 05:21:43.939138] T [rpcsvc.c:640:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 16520, payload: 16496, rpc hdr: 24
[2011-06-29 05:21:43.939164] T [rpcsvc.c:1040:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x3138x, Program: GlusterFS-3.1.0, ProgVers: 310, Proc: 12) to rpc-transport (tcp.dsr-server)
[2011-06-29 05:22:04.58065] T [rpcsvc.c:443:rpcsvc_handle_rpc_call] 0-rpcsvc: Client port: 1010
[2011-06-29 05:22:04.58102] T [rpcsvc-auth.c:302:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS
[2011-06-29 05:22:04.58113] T [rpcsvc.c:356:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0xc43, Ver: 2, Program: 1298437, ProgVers: 310, Proc: 1) from rpc-transport (tcp.dsr-server)
[2011-06-29 05:22:04.58128] T [auth-glusterfs.c:191:auth_glusterfs_authenticate] 0-rpc-service: Auth Info: pid: 1, uid: 0, gid: 0, owner: 1358
[2011-06-29 05:22:04.58142] T [rpcsvc.c:187:rpcsvc_program_actor] 0-rpc-service: Actor found: GlusterFS-3.1.0 - STAT
[2011-06-29 05:22:04.58163] T [server-resolve.c:127:resolve_loc_touchup] 0-dsr-server: return value inode_path 1
[2011-06-29 05:22:04.58200] T [rpcsvc.c:443:rpcsvc_handle_rpc_call] 0-rpcsvc: Client port: 1010
[2011-06-29 05:22:04.58211] T [rpcsvc-auth.c:302:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS
[2011-06-29 05:22:04.58220] T [rpcsvc.c:356:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0xc44, Ver: 2, Program: 1298437, ProgVers: 310, Proc: 1) from rpc-transport (tcp.dsr-server)
[2011-06-29 05:22:04.58231] T [auth-glusterfs.c:191:auth_glusterfs_authenticate] 0-rpc-service: Auth Info: pid: 1, uid: 0, gid: 0, owner: 1359
[2011-06-29 05:22:04.58247] T [rpcsvc.c:187:rpcsvc_program_actor] 0-rpc-service: Actor found: GlusterFS-3.1.0 - STAT
[2011-06-29 05:22:04.58234] T [rpcsvc.c:1003:rpcsvc_submit_generic] 0-rpc-service: Tx message: 108
[2011-06-29 05:22:04.58279] T [rpcsvc.c:640:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 132, payload: 108, rpc hdr: 24
[2011-06-29 05:22:04.58260] T [server-resolve.c:127:resolve_loc_touchup] 0-dsr-server: return value inode_path 1
[2011-06-29 05:22:04.58298] T [rpcsvc.c:1040:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x3139x, Program: GlusterFS-3.1.0, ProgVers: 310, Proc: 1) to rpc-transport (tcp.dsr-server)
[2011-06-29 05:22:04.58322] T [rpcsvc.c:1003:rpcsvc_submit_generic] 0-rpc-service: Tx message: 108
[2011-06-29 05:22:04.58352] T [rpcsvc.c:640:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 132, payload: 108, rpc hdr: 24
[2011-06-29 05:22:04.58370] T [rpcsvc.c:1040:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x3140x, Program: GlusterFS-3.1.0, ProgVers: 310, Proc: 1) to rpc-transport (tcp.dsr-server)

--- Additional comment from shehjart on 2011-06-29 21:19:21 EDT ---

Lakshmi, do we have the complete log?

--- Additional comment from lakshmipathi on 2011-06-30 23:14:51 EDT ---

Created attachment 538 [details]
LSMOD

--- Additional comment from amarts on 2011-09-21 23:27:34 EDT ---

Please check if it is happening now, and take a resolution.

--- Additional comment from lakshmipathi on 2011-09-23 02:40:07 EDT ---

(In reply to comment #3)
> Please check if it is happening now, and take a resolution.

Yes,iozone still fails with stripe-replicated (tested with 3.3.qa10)

------
        Command line used: /opt/qa/tools/iozone -a -s 1m
        Output is in Kbytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                            random  random    bkwd   record   stride                                   
              KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
            1024       4 1389594 1999700
Error reading block 224 2b07cd500000
read: Bad file descriptor
------

--- Additional comment from lakshmipathi on 2011-09-28 01:47:51 EDT ---

note: seems like it starts to fail if size >= 132KB 
---------------
        File size set to 132 KB
        Command line used: /opt/qa/tools/iozone -a -s 132K
        Output is in Kbytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                            random  random    bkwd   record   stride                                   
              KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
             132       4 1166051 2757133
Error reading block 32 2b8195900000
read: Bad file descriptor
----------
with size 131KB
=======
       File size set to 131 KB
        Command line used: /opt/qa/tools/iozone -a -s 131K
        Output is in Kbytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                            random  random    bkwd   record   stride                                   
              KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
             131       4 1336145 2558895  3124872  3982553 2558895 2420455 2673584  2843510  2843510  2066432  1997245 3468030  3895854
             131       8 2969325 3277486  4934216  5122535 4135958 3124872 3759450  3982553  5122535  2558895  2409592 4717434  4934216
             131      16 2969325 3380677  5603747  5545860 4934216 2511022 4267461  3982553  5325799  2660335  3297617   37439  5122535
             131      32 3124872 3536566  2175277  6406138 4135958 2558895   42967  4267461  5847904  1967960  3657016   56436  5603747
             131      64 3199360 2843510  5847904  4557257 3867787 3759450 4934216  2905056  6114306  3277486  5122535 5545860  6114306
             131     128 3053774 3759450  6114306  6406138 6045455 3759450 3106789  3895854  5545860  3199360  3657016 5847904  6114306

iozone test complete.

Comment 2 Amar Tumballi 2012-12-21 08:06:53 UTC

moving the status to same as source bug

Comment 5 Ben Turner 2013-08-12 21:24:12 UTC

Verified on glusterfs-3.4.0.18rhs-1.el6rhs.x86_64:

Volume Name: testvol
Type: Striped-Replicate
Volume ID: 4f6ad18f-3c95-4f9b-9597-c256697a93d3
Status: Started
Number of Bricks: 1 x 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: beast.idm.lab.bos.redhat.com:/bricks/testvol_brick0
Brick2: cloud-qe-2.idm.lab.bos.redhat.com:/bricks/testvol_brick1
Brick3: beast.idm.lab.bos.redhat.com:/bricks/testvol_brick2
Brick4: cloud-qe-2.idm.lab.bos.redhat.com:/bricks/testvol_brick3


INFO: Currently in directory /gluster-mount/run6330_iozone
	Iozone: Performance Test of File I/O
	        Version $Revision: 3.394 $
		Compiled for 64 bit mode.
		Build: linux 

	Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
	             Al Slater, Scott Rhine, Mike Wisner, Ken Goss
	             Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
	             Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
	             Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
	             Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
	             Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer.
	             Ben England.

	Run began: Mon Aug 12 15:38:12 2013

	Auto Mode
	Command line used: iozone -a
	Output is in Kbytes/sec
	Time Resolution = 0.000001 seconds.
	Processor cache size set to 1024 Kbytes.
	Processor cache line size set to 32 bytes.
	File stride size set to 17 * record size.
                                                            random  random    bkwd   record   stride                                   
              KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
              64       4   29357   32752    29793   735831  140043   24207   11871    61726    29740    59588    31843   18233   315136
              64       8   41529   89873    30609   533875  158387   58612   14658    92890    21760    90145    46548   12725   310759
              64      16   64068   64191    39529   769584  338589  145745   15396   162019    29184   156084    84872   16048   345563
              64      32  128204  129565  4018152  4897948  447537  153848   15133   153408    21925   367817   230257   23871    36018
              64      64  225992  229274  4018152  5283570  450541  261173   30947   241440    34575   318502   231848   38141    34244
             128       4   61037   45245   515822   992724  150223   27563   32016    52847    42653    63747    60575   64189   526961
             128       8   64809   59866   537512   983630  230633   65306   38141   130758    50812   100862    66257   62261   538051
             128      16   73859   78001   715059   848397  273522  137772   44260   226546    47901   125384    92887   67303   511887
             128      32  157034  123908   719853   927549  460514  193341   35281   360363    42144   183119   162648   42052   581781
             128      64  234052  239695  4596273  5325799  722760  273522   49304   314362    43026   479432   448961   51967   753179
             128     128  441576  408949  4267461  4934216  718889  414314   65203   423798    71063   483316   385460   67194   841746
             256       4   59026   54758    79086  1306564  212419   30639   28381    82794    41097    59869    58141   48612   797250
             256       8  106799   83665    90330  1248819  277681   65040   39592   111063    50147    98087    83658   48502   797843
             256      16   84687  114547    90811   996229  445387  123378   36840   242445   101913   146111   115682   43573    66804
             256      32  155528  131880    83063  1152323  436870  226968   42083   324412   102282   220308   139959   44476   715443
             256      64  223985  233381   101470  1430119  653204  417505   39727   410325    79679   358534   496228   46997   822913
             256     128  400678  407058  1868008  4496299 1007446  495084   63982   527947    78941   709299   737558   83249  1190657
             256     256  550409  613985  4197489  4819184  991629  603630   78239   638445    78216   591330   530031   79946   973645
             512       4   73764   69726   109588  1575781  238481   30272   41141    63117    59259    76371    68669   77670  1160923
             512       8   91379  109560   108039  1546280  332489   67941   41905   146328    56585   118876   111350   59197  1129775
             512      16  185578  101402    78660   114796  420020  133187   40277   208984    50768   184637   124212   68356  1138158
             512      32  150861  168472    76784   114932  528962  239706   47930   349935   167043   212352   164901   61157  1040545
             512      64  221052  232743   107903  1645826  575300  286835   52340   447877   205543   309578   261479   70914  1152822
             512     128  398138  448719   212017  2357784 1053304  491324   53019   751707   140048   582478   547575   89981    96020
             512     256  481195  544106   264605  2275345 1350783  622855   71669   603767    83991   869143   937845   86967  1442420
             512     512  514144  631277  4838792  4927617 1533034  840566   88476   667582    85530   642419   680918   91985  1406520
            1024       4   68775   75522   114249  1744641  293897   37394   40003    82895    40894    84691    69410   91730  1395010
            1024       8  106821  115354   110930  1698417  429204   68297   44003   138302    80031   112936   113488   87266  1414304
            1024      16  159903  169789    95389  1738989  508472  125492   48967   258207    55564   184176   174892   65531    90910
            1024      32  165939  230889   110226  1628234  764255  211483   51995   400470    70503   245387   221929   86435  1475525
            1024      64  235240  242587   107665  1768345  670166  338298   61347   477175   400470   293155   318906   87814  1475525
            1024     128  368483  389361   125952  2133821  755649  625923   55627   707129   280097   523023   476276   97347  1701108
            1024     256  567981  565067   182929  2102484 1385560  665803   72387   710052   258720   787084   667458   96629  1741104
            1024     512  574977  694775   253842  2478620 1615374  749714   71744   703308   160855  1097669  1050420   96050  1698417
            1024    1024  544714  681329  4356809  4589593 1633188  765344   99292   722109   101144   830457   803728   96915  1668074
            2048       4   70409    1217   110185  1825390  230814    1266   41811      484    86057    71641     1963   92472  1662201
            2048       8   85697    1168   111948  1512364  500497    1154   51642      505    51234    78581     1965   96896  1671907
            2048      16   86122    1143   113595  1771556  670156    1178   52685      508    59877    71728     2517   98183  1645012
            2048      32   90651    1272   113412  1722532  756302   85629   57476      686    78476    86202     2527   97084  1784066
            2048      64   84398    1217   112422  1901771  894337   98609   59879      693    84975    83712     1150   94898  1654199
            2048     128  113374    1344   128902  2096227  895922  109571   61070      687   597943   133091     1748  107789  1820362
            2048     256  112998    1305   137191  2011785 1158327   86410   76139      706   513266   134497   157308  106054  2479196
            2048     512  102430    1572   160842   141445  450116    2588   61090     1224   327937   184686     2493  109313  1780738
            2048    1024  102744    1374   245385  2629459 1775951    2486   89148     2276   187051  1201926  1101018  106416  1681397
            2048    2048  115204    1265  4656281  4796687 1738569    1140  107449     2544   106245   108515   108279  105551  2050198
            4096       4   69987   72219   113550  1891055  396170    1892   41490     2022    92642    72454    67675  108128  1737129
            4096       8   73811    3222   112607  1783289  414034    1527   51568      901    82019    74677     3722  107348  1795028
            4096      16   75867    2964   113267  1810731  687145    2724   58452      944    68819    65227     3809  105578  1718192
            4096      32   75011    2561   113208   110217  891821    2522   61215     1062    80395    68043     2124  104749  1711686
            4096      64   75452    2400   113977  2179686 1403606    2853   62788     1401    88854    68959     5113  105959  2038958
            4096     128   73385    2615   122612  1943027  986509    2469   64830     1448   100480    79769     3797  109923  1818975
            4096     256   73928    2434   131543  1902362 1290864    2950   77009     1064   912667    80682     5102  111017  1815707
            4096     512   75576    3287   130878  1990298 1412956    2181   55340      867   614382     4579     2221  109480  1823609
            4096    1024   76035    3083   155492  2132342 1691963    2341   79372     2267   292319   111635     2306  109506  1790351
            4096    2048   77235    3019   286055  2555081 1750403   64474   97516    72395   171659   224979    73515  109355  1762434
            4096    4096   73310    2727  5489116  3886139 1743652    3949  110275     3866   109853     4699    72461  109700  1735549
            8192       4   63465   63445   114036  2335204  349636   41950   45249     4019    92320    63996    64029  111446  1749316
            8192       8   64786    6964   114317  1764317  376401    3052   50078     2216    81311    60908     6832  109107  1745938
            8192      16   65374    3986   113132  1691197  592676    7202   20214     2775    68002    60130     6986  109762  1745584
            8192      32   65425    4607   114285  1775898  793329    6653   36606     1972    88572    59635     6976  109759  1734131
            8192      64   65229    4376   113932  2386290 1511194    7396   63624     2137    95838    63329    11747  110202  2052086
            8192     128   65849    3687   117650  1869835 1102552    5088   65054     3045   103219    66918     3756  108190  1871567
            8192     256   65844    4011   121788  1855096 1336185    4800   77373     2263    90130    67980     4797  112259  1783920
            8192     512   65900    3848   122864  1933928 1466997    4666   54831     1962  1088476    71019     4744  112283  1792201
            8192    1024   65876    7494   131810  2369668 2379350    6728   73282     1478   565166     3759     4725  111790  1789867
            8192    2048   65873    3808   155095  2109667 1748871    4604   93227     2166   369774     3815     3837  111917  1769678
            8192    4096   65789    5685   230072  2430860 1748159    7205  106975     2363   208443     3949     4760  112300  1765949
            8192    8192   65750    5525  4911476  3420306 2048538    5373  111686     7425   111537     6072     8524  111898  1891034
           16384       4   60226   60422   113824  1765713  238066    4306   27350     4042    93173    60214    60213  112346  1974708
           16384       8   60680   10115   113804  1777405  369259   10085   53450     3788    80855    60694    12402  112091  1778555
           16384      16   60661   12594   114326  2167751  635380   12254   33425     4367    70249    57874     9977  112184  1764761
           16384      32   61076    6943   103160  1760602  817234    7045   63015     6491    90106    60876    10020  112028  2501363
           16384      64   61028   12379   113776  1828335 1012534   11111   30589     5462    97979    58028    37973  111837  2389760
           16384     128   60766    9371   112848  2486970 1300524   16005   67099     6233   103851    60889    28520  113523  1843788
           16384     256   61337    7023   115903  1881086 1528056    6728   79740     5397    85651    62149    10484  113155  2140006
           16384     512   61342    6892   118631  1854185 1547289    6590   53465     3847    60350    63173    10003  113070  2430499
           16384    1024   61293   13245   122105  2027493 2572914   13722   72659     3908  1264974     7104    10691  113042  1878874
           16384    2048   61352   12649   131884  1958667 1690149   14249   85731     3795   701822    70690    10198  113106  1770262
           16384    4096   61332   12570   156449  2996819 2605200   13374  103457     4228   401017    83186    10761  113199  1892118
           16384    8192   61271   13250   239113   238854 2554360   38667  110292     3907   214337   125535    68066  112858  1783586
           16384   16384   61272   11514  4386590  5249445 2726659   11573  113130    11362   112915    55156     9153  112808  2476037
           32768      64   59082   12382   114212   113733  137719   10765   66453    43217    95809    59155    12474  113433   113564
           32768     128   59251   59298   115196   115271  156276   59121   69066    11844   107235    59372    20564  113707   113990
           32768     256   59261   59282   116283   114139  170323   59116   78768    12169    90229    59619    12542  113078   111873
           32768     512   59250   12623   115296   106404  117931   11838   52402    59129    55558    38519    21831  112966   103691
           32768    1024   59262   12507   118198   118488  140734   11990   72634     6989    79748    61074    21765  113719   114009
           32768    2048   59268   12508   122027   122553  135828   12585   87387     6772  1301298    63332    22230  112704   113963
           32768    4096   59252   12510   132029   131392  103258   11798  100837     6897   769583    67925    22554  113797   113749
           32768    8192   59251   12549   152393   153221  150608   11904  108200     6954   413621    79486    23773  113787   113778
           32768   16384   59262   12515   231807   230358  113014   11536  111619     8817   217917   120389    27917  112859   113751
           65536      64   58155   16536   114210   113667   55425   58051   66998    36739    88267    58279    30196  113025   113992
           65536     128   58311   58314    83916   114595   76623   58247   68664    50224   105865    58365    30182   83825   114184
           65536     256   58310   58314   114729   114811   69815   58237   79473    58202    86905    58465    31632  113717   113794
           65536     512   58305   20578   114954   115490   61440   58234   54651    58237    53792    58708    15417  113769   113532
           65536    1024   58300   15826   116316   116053   73723   58241   73503    58260    72803    31977    31702  113805   114212
           65536    2048   58315   20631   117878   117953   88647   58242   88088    58319    95197    29428    22380  107370   114181
           65536    4096   58301   20466   122183   122308  100420   58237   99994    56197  1434803    24084    17450  113685   113621
           65536    8192   58301   20533   130020   121823  108679   58244  107190    58243   764152    25871    28336  113792    23179
           65536   16384   58301   58294   150997   153575  111771   58229  111501    58307   421906    33298    36274  113756    82086
          131072      64   57714   20458   114118   114014   45015   57665   67852    39368    64230    57731    19317  113883   113411
          131072     128   57842   26681   114393    96873   73835   44926   69864    48891   102475    57866    25766  113896   114097
          131072     256   57847   57840   112305   114321   68060   57807   78746    57769    88411    57921    19281  113384   113650
          131072     512   57843   57840   114682   114516   59015   57801   52385    57807    54464    58073    22419  113976    97134
          131072    1024   57838   20471   114998   114973   72292   57810   71519    57805    72936    40182    58298  113753   114055
          131072    2048   57844   20301   115757   116091   89290   57797   87619    57805    91217    43345    23020  113985   113993
          131072    4096   57841   20616    99171   118122  103487   57803   99930    57806   107277    34519    59700  113918   114013
          131072    8192   57831   21165   122016   121194  113186   57804  106166    57808  1361105    29260    61677  111861   114219
          131072   16384   57832   21980   130766   127819  110867   57804  110452    57807   797681    27852    28822  114051   114259
          262144      64   53239   57496   114066   114252   43671   49136   66950    39459    59666    57506    25234   96611   113870
          262144     128   57612   57613   114256   104805   74296   43863   69751    48883    95486    57623    26546  114191   114303
          262144     256   57608   28358   114399   114388   66875   57587   79065    57584    89123    57648    57665  114077   114138
          262144     512   57610   25954   114273   113365   59461   57601   52659    57595    55000    57724    25266  113431   113633
          262144    1024   57610   57613   113578   114616   70858   57590   71944    55600    72231    57818    27842  103417   114157
          262144    2048   57606   24555   115239   114908   89280   29596   87885    57589    88887    58060    27097  104235   114229
          262144    4096   57608   24419   106354   116148   99255   57587   99154    53040   102426    58495    25739  113727   114112
          262144    8192   57586   26827   118143   116951  109146   57589  106696    57591   113883    59451    28568  113816   114097
          262144   16384   57605   25266   121723   120519  110632   57588  110438    57588  1404899    28960    61416  104434   113786
          524288      64   43529   57387   114249   113654   42322   51992   67035    40781    58451    57391    31332  113609   108430
          524288     128   43584   57494   113865   113422   73451   41596   69221    48393    93334    57499    37949  113452   113887
          524288     256   44684   50695   114169   109191   67779   57485   80098    57478    87400    57514    36103  114088   113565
          524288     512   39372   34541   113834   113797   60070   57486   52756    57492    54544    57552    34663  114212   114012
          524288    1024   33971   57496   114442   114090   71298   57486   72021    57485    72142    57608    40930  109193   108698
          524288    2048   36882   34719   114559   114627   88128   57484   88293    57487    89157    57719    30959  114154   113436
          524288    4096   39984   36453   114113   115084   99612   57486   99289    55861   100924    57934    32122  114316   113737
          524288    8192   35731   57496   115956   116202  103471   57485  106451    57484   109596    58398    36582   96588   114098
          524288   16384   36887   57496   117572   117532  110305   57483  110289    57484   117733    59331    38132  114168   109107

iozone test complete.

real	64m19.660s
user	0m3.190s
sys	1m36.143s

Comment 6 Scott Haines 2013-09-23 22:38:59 UTC

Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. 

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1262.html

Comment 7 Scott Haines 2013-09-23 22:43:40 UTC

Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. 

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1262.html