Bug 980250 - NFS: Poor bonnie++ performance.
NFS: Poor bonnie++ performance.
Status: CLOSED CURRENTRELEASE
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: glusterd (Show other bugs)
2.1
Unspecified Unspecified
medium Severity high
: ---
: ---
Assigned To: santosh pradhan
Ben Turner
: ZStream
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-07-01 16:58 EDT by Ben Turner
Modified: 2015-05-12 14:29 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-09-05 03:19:15 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Ben Turner 2013-07-01 16:58:27 EDT
Description of problem:

Now that 976835 is resolved I am seeing very poor bonnie++ performance.  Normally on 1Gb NICs bonnie takes about 40 minutes, I am currently seeing:

INFO: Switching to /gluster-mount/run6267_bonnie
INFO: Currently in directory /gluster-mount/run6267_bonnie
Using uid:0, gid:0.
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
storage-qe06 31880M  1736  96 55575   3  1539   3  4596  97 109419   5 296.3   3
Latency              7170us   39326ms    1485ms   18543us   79663us     156ms
Version  1.96       ------Sequential Create------ --------Random Create--------
storage-qe06.lab.en -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16   339   3   507   3   460   4   330   2  1697   3  1043   4
Latency             43512us    2022ms    1636ms     426ms    1424us     854ms
1.96,1.96,storage-qe06.lab.eng.rdu2.redhat.com,1,1372372505,31880M,,1736,96,55575,3,1539,3,4596,97,109419,5,296.3,3,16,,,,,339,3,507,3,460,4,330,2,1697,3,1043,4,7170us,39326ms,1485ms,18543us,79663us,156ms,43512us,2022ms,1636ms,426ms,1424us,854ms

real	371m57.277s
user	0m15.361s
sys	12m54.482s

Version-Release number of selected component (if applicable):

glusterfs-3.4.0.12rhs-2.el6rhs.x86_64

How reproducible:

Every time I have run with NFS mounts in overnights.

Steps to Reproduce:
1.  Mount a gluster volume -t nfs
2.  Run bonnie++ on the NFS mount
3.

Actual results:

Bonnie took 371m57.277s to complete.

Expected results:

Bonnie to complete in under an hour.

Additional info:
Comment 2 Ben Turner 2013-07-03 14:22:50 EDT
I went to retest this on the latest bits today and I got:

INFO: Switching to /gluster-mount/run6343_bonnie
INFO: Currently in directory /gluster-mount/run6343_bonnie
Using uid:0, gid:0.
Writing a byte at a time...done
Writing intelligently...Can't write block.: Input/output error
Can't write block 3384330.

real	8m58.028s
user	0m0.654s
sys	0m17.336s

The glusterfs version is:

:: [ 13:48:10 ] ::  Package versions:
:: [ 13:48:10 ] ::    glusterfs-3.4.0.12rhs.beta1-1.el6rhs.x86_64
Comment 3 Vivek Agarwal 2013-07-18 13:13:43 EDT
Ben,

Is this issue seen with the latest build as well?
Comment 4 Ben Turner 2013-08-08 16:16:49 EDT
On the latest bits bonnie is still slower than in RHS2.0u4 but nothing like it was on the .12 version:

INFO: Switching to /gluster-mount/run6378_bonnie
INFO: Currently in directory /gluster-mount/run6378_bonnie
Using uid:0, gid:0.
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
storage-qe06 31880M  1833  96 56407   2 10866   1  3404  77 108288   5 236.4   5
Latency              8030us   34526ms      180s     114ms     889ms     124ms
Version  1.96       ------Sequential Create------ --------Random Create--------
storage-qe06.lab.en -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16   330   3   488   2   450   4   323   2  1631   3  1161   5
Latency             44095us    2141ms    1678ms     365ms    1435us     237ms
1.96,1.96,storage-qe06.lab.eng.rdu2.redhat.com,1,1375923171,31880M,,1833,96,56407,2,10866,1,3404,77,108288,5,236.4,5,16,,,,,330,3,488,2,450,4,323,2,1631,3,1161,5,8030us,34526ms,180s,114ms,889ms,124ms,44095us,2141ms,1678ms,365ms,1435us,237ms

real	68m39.913s
user	0m3.904s
sys	1m22.557s

Usually I expect bonnie to complete in ~40 mins in the fs sanity tests over NFS.  I would say that the reason this bug was opened is resolved(300+ minute bonnie runs) but performance is still slower than 2.0u4.
Comment 5 Vivek Agarwal 2013-09-05 03:19:15 EDT
Issue not seen the new builds as per Ben's comments

Note You need to log in before you can comment on or make changes to this bug.