Bug 980250

Summary: NFS: Poor bonnie++ performance.
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Ben Turner <bturner>
Component: glusterdAssignee: santosh pradhan <spradhan>
Status: CLOSED CURRENTRELEASE QA Contact: Ben Turner <bturner>
Severity: high Docs Contact:
Priority: medium    
Version: 2.1CC: rhs-bugs, vagarwal, vbellur
Target Milestone: ---Keywords: ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-09-05 07:19:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Ben Turner 2013-07-01 20:58:27 UTC
Description of problem:

Now that 976835 is resolved I am seeing very poor bonnie++ performance.  Normally on 1Gb NICs bonnie takes about 40 minutes, I am currently seeing:

INFO: Switching to /gluster-mount/run6267_bonnie
INFO: Currently in directory /gluster-mount/run6267_bonnie
Using uid:0, gid:0.
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
storage-qe06 31880M  1736  96 55575   3  1539   3  4596  97 109419   5 296.3   3
Latency              7170us   39326ms    1485ms   18543us   79663us     156ms
Version  1.96       ------Sequential Create------ --------Random Create--------
storage-qe06.lab.en -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16   339   3   507   3   460   4   330   2  1697   3  1043   4
Latency             43512us    2022ms    1636ms     426ms    1424us     854ms
1.96,1.96,storage-qe06.lab.eng.rdu2.redhat.com,1,1372372505,31880M,,1736,96,55575,3,1539,3,4596,97,109419,5,296.3,3,16,,,,,339,3,507,3,460,4,330,2,1697,3,1043,4,7170us,39326ms,1485ms,18543us,79663us,156ms,43512us,2022ms,1636ms,426ms,1424us,854ms

real	371m57.277s
user	0m15.361s
sys	12m54.482s

Version-Release number of selected component (if applicable):

glusterfs-3.4.0.12rhs-2.el6rhs.x86_64

How reproducible:

Every time I have run with NFS mounts in overnights.

Steps to Reproduce:
1.  Mount a gluster volume -t nfs
2.  Run bonnie++ on the NFS mount
3.

Actual results:

Bonnie took 371m57.277s to complete.

Expected results:

Bonnie to complete in under an hour.

Additional info:

Comment 2 Ben Turner 2013-07-03 18:22:50 UTC
I went to retest this on the latest bits today and I got:

INFO: Switching to /gluster-mount/run6343_bonnie
INFO: Currently in directory /gluster-mount/run6343_bonnie
Using uid:0, gid:0.
Writing a byte at a time...done
Writing intelligently...Can't write block.: Input/output error
Can't write block 3384330.

real	8m58.028s
user	0m0.654s
sys	0m17.336s

The glusterfs version is:

:: [ 13:48:10 ] ::  Package versions:
:: [ 13:48:10 ] ::    glusterfs-3.4.0.12rhs.beta1-1.el6rhs.x86_64

Comment 3 Vivek Agarwal 2013-07-18 17:13:43 UTC
Ben,

Is this issue seen with the latest build as well?

Comment 4 Ben Turner 2013-08-08 20:16:49 UTC
On the latest bits bonnie is still slower than in RHS2.0u4 but nothing like it was on the .12 version:

INFO: Switching to /gluster-mount/run6378_bonnie
INFO: Currently in directory /gluster-mount/run6378_bonnie
Using uid:0, gid:0.
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
storage-qe06 31880M  1833  96 56407   2 10866   1  3404  77 108288   5 236.4   5
Latency              8030us   34526ms      180s     114ms     889ms     124ms
Version  1.96       ------Sequential Create------ --------Random Create--------
storage-qe06.lab.en -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16   330   3   488   2   450   4   323   2  1631   3  1161   5
Latency             44095us    2141ms    1678ms     365ms    1435us     237ms
1.96,1.96,storage-qe06.lab.eng.rdu2.redhat.com,1,1375923171,31880M,,1833,96,56407,2,10866,1,3404,77,108288,5,236.4,5,16,,,,,330,3,488,2,450,4,323,2,1631,3,1161,5,8030us,34526ms,180s,114ms,889ms,124ms,44095us,2141ms,1678ms,365ms,1435us,237ms

real	68m39.913s
user	0m3.904s
sys	1m22.557s

Usually I expect bonnie to complete in ~40 mins in the fs sanity tests over NFS.  I would say that the reason this bug was opened is resolved(300+ minute bonnie runs) but performance is still slower than 2.0u4.

Comment 5 Vivek Agarwal 2013-09-05 07:19:15 UTC
Issue not seen the new builds as per Ben's comments