Description of problem: Smallfile performance regression run on fuse mount for RC1 build 3.6.0-24.1.el6rhs showed regression in creates and append in comparison with RHS 2.1 build 3.6.0-24.1.el6rhs


Version-Release number of selected component (if applicable): 3.6.0-24.1.el6rhs


How reproducible: Run as part of performance regression.


Steps to Reproduce:
1. Create a 2x2 Distributed-Replicate volume with 2 clients
2. Run smallfile test suite: https://github.com/bengland2/smallfile
3.

Actual results:

======= Throughput create =========

decision parameters:
  sample type = throughput
  confidence threshold =  95.00 %
  max. pct. deviation =  10.00 %
  regression threshold =  10.00 % 
sample stats for baseline:
  min = 1311.658531
  max = 1324.589308
  mean = 1320.117332
  sd = 7.329552
  pct.dev. =  0.56 %
sample stats for current:
  min = 654.529630
  max = 675.246078
  mean = 662.941517
  sd = 10.893002
  pct.dev. =  1.64 %
CHANGE -49.78 percent
magnitude of change is at least 48.14%
/usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean.
Please note that:
    - numpy.mean axis argument defaults to None, not 0
    - numpy.mean has a ddof argument to replace bias in a more general manner.
      scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x,
axis=0, ddof=1).
  axis=0, ddof=1).""", DeprecationWarning)
t-test t-statistic = 86.695987 probability = 0.000000
t-test says that mean of two sample sets differs with probability 100.00%
probability that sample sets have same mean = 0.0000
declaring a performance regression test FAILURE because of lower throughput
RESULT:10


======= Throughput append =========

decision parameters:
  sample type = throughput
  confidence threshold =  95.00 %
  max. pct. deviation =  10.00 %
  regression threshold =  10.00 % 
sample stats for baseline:
  min = 1799.352995
  max = 1821.136025
  mean = 1811.613934
  sd = 11.146796
  pct.dev. =  0.62 %
sample stats for current:
  min = 477.017769
  max = 522.316099
  mean = 496.810415
  sd = 23.183265
  pct.dev. =  4.67 %
CHANGE -72.58 percent
magnitude of change is at least 67.91%
/usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean.
Please note that:
    - numpy.mean axis argument defaults to None, not 0
    - numpy.mean has a ddof argument to replace bias in a more general manner.
      scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x,
axis=0, ddof=1).
  axis=0, ddof=1).""", DeprecationWarning)
t-test t-statistic = 88.529099 probability = 0.000000
t-test says that mean of two sample sets differs with probability 100.00%
probability that sample sets have same mean = 0.0000
declaring a performance regression test FAILURE because of lower throughput


Expected results:


Additional info:

run156 - glusterfs - 3.4.0.59rhs-1.el6rhs - distrep - (quota no) - Threads:8, files:10000, file-size:64
run171 - glusterfs - 3.6.0-24.1.el6rhs - distrep - (quota no) - Threads:8, files:10000, file-size:64

Operations                      RUN156  RUN171
-------------------------       ------- -------
create                          1319    662
append                          1811    496
read                            3120    4241
readdir                         8866    11767
ls-l                            8792    11579
mkdir                           912     938
rmdir                           946     753
stat                            6814    8185
chmod                           2479    3089
setxattr                        5557    6632
getxattr                        10793   12935
rename                          352     424
delete-renamed                  2179    1800


Volume info for RUN156
 
Volume Name: testvol
Type: Distributed-Replicate
Volume ID: 5596aa4d-7cd8-4481-9127-cfdbb5633e95
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: rhs-gp-srv2.lab.eng.blr.redhat.com:/data/datastore
Brick2: rhs-gp-srv5.lab.eng.blr.redhat.com:/data/datastore
Brick3: rhs-gp-srv6.lab.eng.blr.redhat.com:/data/datastore
Brick4: rhs-gp-srv16.lab.eng.blr.redhat.com:/data/datastore
Options Reconfigured:
performance.quick-read: on
performance.open-behind: on


Volume info for RUN171
 
Volume Name: testvol
Type: Distributed-Replicate
Volume ID: d8257ef3-b1a8-4cdb-969d-3db7990d0a1a
Status: Started
Snap Volume: no
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: rhs-gp-srv2.lab.eng.blr.redhat.com:/data/datastore
Brick2: rhs-gp-srv5.lab.eng.blr.redhat.com:/data/datastore
Brick3: rhs-gp-srv6.lab.eng.blr.redhat.com:/data/datastore
Brick4: rhs-gp-srv16.lab.eng.blr.redhat.com:/data/datastore
Options Reconfigured:
performance.quick-read: on
performance.open-behind: on
performance.readdir-ahead: on
auto-delete: disable
snap-max-soft-limit: 90
snap-max-hard-limit: 256