Bug 1121533 - Smallfile performance regression on nfs mount [NEEDINFO]
Summary: Smallfile performance regression on nfs mount
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: glusterfs
Version: rhgs-3.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: ---
Assignee: santosh pradhan
QA Contact: Anush Shetty
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-07-21 07:28 UTC by Anush Shetty
Modified: 2015-05-13 17:19 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-08-11 11:39:34 UTC
spradhan: needinfo? (ashetty)


Attachments (Terms of Use)

Description Anush Shetty 2014-07-21 07:28:11 UTC
Description of problem: We see performance regression in creates (-70.50%), append (-83.09%) and directory operations on NFS mount in comparison with RHS 2.1 build 3.4.0.59rhs-1.el6rhs


Version-Release number of selected component (if applicable): 3.6.0-24.1.el6rhs


How reproducible: Run as part of performance regression suite


Steps to Reproduce:
1. Create a 2x2 Distributed-Replicate volume with 2 clients
2. Run smallfile test suite: https://github.com/bengland2/smallfile


Actual results:


======= Throughput create =========

decision parameters:
  sample type = throughput
  confidence threshold =  95.00 %
  max. pct. deviation =  10.00 %
  regression threshold =  10.00 % 
sample stats for baseline:
  min = 493.648580
  max = 494.691803
  mean = 494.191710
  sd = 0.522941
  pct.dev. =  0.11 %
sample stats for current:
  min = 145.281489
  max = 146.432456
  mean = 145.805070
  sd = 0.582463
  pct.dev. =  0.40 %
CHANGE -70.50 percent
magnitude of change is at least 70.10%
/usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean.
Please note that:
    - numpy.mean axis argument defaults to None, not 0
    - numpy.mean has a ddof argument to replace bias in a more general manner.
      scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x,
axis=0, ddof=1).
  axis=0, ddof=1).""", DeprecationWarning)
t-test t-statistic = 770.881103 probability = 0.000000
t-test says that mean of two sample sets differs with probability 100.00%
probability that sample sets have same mean = 0.0000
declaring a performance regression test FAILURE because of lower throughput
RESULT:10


======= Throughput append =========

decision parameters:
  sample type = throughput
  confidence threshold =  95.00 %
  max. pct. deviation =  10.00 %
  regression threshold =  10.00 % 
sample stats for baseline:
  min = 1029.065275
  max = 1049.452872
  mean = 1038.682042
  sd = 10.242677
  pct.dev. =  0.99 %
sample stats for current:
  min = 175.168563
  max = 176.346088
  mean = 175.622468
  sd = 0.633404
  pct.dev. =  0.36 %
CHANGE -83.09 percent
magnitude of change is at least 82.11%
/usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean.
Please note that:
    - numpy.mean axis argument defaults to None, not 0
    - numpy.mean has a ddof argument to replace bias in a more general manner.
      scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x,
axis=0, ddof=1).
  axis=0, ddof=1).""", DeprecationWarning)
t-test t-statistic = 145.666311 probability = 0.000000
t-test says that mean of two sample sets differs with probability 100.00%
probability that sample sets have same mean = 0.0000
declaring a performance regression test FAILURE because of lower throughput
RESULT:10


======= Throughput read =========

decision parameters:
  sample type = throughput
  confidence threshold =  95.00 %
  max. pct. deviation =  10.00 %
  regression threshold =  10.00 % 
sample stats for baseline:
  min = 1652.487523
  max = 1711.558338
  mean = 1690.394255
  sd = 32.902546
  pct.dev. =  1.95 %
sample stats for current:
  min = 1441.697021
  max = 1492.139744
  mean = 1462.348419
  sd = 26.434273
  pct.dev. =  1.81 %
CHANGE -13.49 percent
magnitude of change is at least 11.54%
/usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean.
Please note that:
    - numpy.mean axis argument defaults to None, not 0
    - numpy.mean has a ddof argument to replace bias in a more general manner.
      scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x,
axis=0, ddof=1).
  axis=0, ddof=1).""", DeprecationWarning)
t-test t-statistic = 9.358545 probability = 0.000726
t-test says that mean of two sample sets differs with probability  99.93%
probability that sample sets have same mean = 0.0007
declaring a performance regression test FAILURE because of lower throughput
RESULT:10


======= Throughput readdir =========

decision parameters:
  sample type = throughput
  confidence threshold =  95.00 %
  max. pct. deviation =  10.00 %
  regression threshold =  10.00 % 
sample stats for baseline:
  min = 9243.819577
  max = 9267.905076
  mean = 9257.024533
  sd = 12.209832
  pct.dev. =  0.13 %
sample stats for current:
  min = 11028.228853
  max = 11368.462301
  mean = 11228.344092
  sd = 177.874766
  pct.dev. =  1.58 %
CHANGE 21.30 percent
magnitude of change is at least 19.71%
/usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean.
Please note that:
    - numpy.mean axis argument defaults to None, not 0
    - numpy.mean has a ddof argument to replace bias in a more general manner.
      scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x,
axis=0, ddof=1).
  axis=0, ddof=1).""", DeprecationWarning)
t-test t-statistic = -19.150607 probability = 0.000044
t-test says that mean of two sample sets differs with probability 100.00%
probability that sample sets have same mean = 0.0000
current sample set is statistically better than baseline sample set
RESULT:0


======= Throughput ls-l =========

decision parameters:
  sample type = throughput
  confidence threshold =  95.00 %
  max. pct. deviation =  10.00 %
  regression threshold =  10.00 % 
sample stats for baseline:
  min = 4518.678532
  max = 6671.560516
  mean = 5823.508967
  sd = 1146.826239
  pct.dev. = 19.69 %
sample stats for current:
  min = 3764.305345
  max = 3869.956399
  mean = 3832.686769
  sd = 59.300035
  pct.dev. =  1.55 %
ERROR: pct. deviation of 19.69 is too high for baseline samples
RESULT:11


======= Throughput mkdir =========

decision parameters:
  sample type = throughput
  confidence threshold =  95.00 %
  max. pct. deviation =  10.00 %
  regression threshold =  10.00 % 
sample stats for baseline:
  min = 733.029315
  max = 762.575098
  mean = 746.807376
  sd = 14.873042
  pct.dev. =  1.99 %
sample stats for current:
  min = 477.255789
  max = 503.437896
  mean = 490.027308
  sd = 13.102747
  pct.dev. =  2.67 %
CHANGE -34.38 percent
magnitude of change is at least 31.71%
/usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean.
Please note that:
    - numpy.mean axis argument defaults to None, not 0
    - numpy.mean has a ddof argument to replace bias in a more general manner.
      scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x,
axis=0, ddof=1).
  axis=0, ddof=1).""", DeprecationWarning)
t-test t-statistic = 22.438137 probability = 0.000023
t-test says that mean of two sample sets differs with probability 100.00%
probability that sample sets have same mean = 0.0000
declaring a performance regression test FAILURE because of lower throughput
RESULT:10


======= Throughput rmdir =========

decision parameters:
  sample type = throughput
  confidence threshold =  95.00 %
  max. pct. deviation =  10.00 %
  regression threshold =  10.00 % 
sample stats for baseline:
  min = 554.047717
  max = 566.721321
  mean = 558.589879
  sd = 7.058134
  pct.dev. =  1.26 %
sample stats for current:
  min = 360.613211
  max = 403.665958
  mean = 379.176939
  sd = 22.129540
  pct.dev. =  5.84 %
CHANGE -32.12 percent
magnitude of change is at least 26.28%
/usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean.
Please note that:
    - numpy.mean axis argument defaults to None, not 0
    - numpy.mean has a ddof argument to replace bias in a more general manner.
      scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x,
axis=0, ddof=1).
  axis=0, ddof=1).""", DeprecationWarning)
t-test t-statistic = 13.378427 probability = 0.000181
t-test says that mean of two sample sets differs with probability  99.98%
probability that sample sets have same mean = 0.0002
declaring a performance regression test FAILURE because of lower throughput
RESULT:10


======= Throughput stat =========

decision parameters:
  sample type = throughput
  confidence threshold =  95.00 %
  max. pct. deviation =  10.00 %
  regression threshold =  10.00 % 
sample stats for baseline:
  min = 4274.219088
  max = 4434.828514
  mean = 4335.970257
  sd = 86.495947
  pct.dev. =  1.99 %
sample stats for current:
  min = 3908.265340
  max = 4247.890344
  mean = 4129.556218
  sd = 191.797694
  pct.dev. =  4.64 %
CHANGE -4.76 percent
magnitude of change is at least  0.12%
/usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean.
Please note that:
    - numpy.mean axis argument defaults to None, not 0
    - numpy.mean has a ddof argument to replace bias in a more general manner.
      scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x,
axis=0, ddof=1).
  axis=0, ddof=1).""", DeprecationWarning)
t-test t-statistic = 1.699243 probability = 0.164501
t-test says that mean of two sample sets differs with probability  83.55%
probability that sample sets have same mean = 0.1645
sample sets are statistically indistinguishable for specified confidence level
RESULT:0


======= Throughput chmod =========

decision parameters:
  sample type = throughput
  confidence threshold =  95.00 %
  max. pct. deviation =  10.00 %
  regression threshold =  10.00 % 
sample stats for baseline:
  min = 1976.240010
  max = 2054.878358
  mean = 2019.279094
  sd = 39.843577
  pct.dev. =  1.97 %
sample stats for current:
  min = 1930.467085
  max = 2232.464141
  mean = 2041.222884
  sd = 166.309913
  pct.dev. =  8.15 %
CHANGE  1.09 percent
magnitude of change is less than standard deviation of samples
/usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean.
Please note that:
    - numpy.mean axis argument defaults to None, not 0
    - numpy.mean has a ddof argument to replace bias in a more general manner.
      scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x,
axis=0, ddof=1).
  axis=0, ddof=1).""", DeprecationWarning)
t-test t-statistic = -0.222247 probability = 0.835008
t-test says that mean of two sample sets differs with probability  16.50%
probability that sample sets have same mean = 0.8350
sample sets are statistically indistinguishable for specified confidence level
RESULT:0


======= Throughput setxattr =========

decision parameters:
  sample type = throughput
  confidence threshold =  95.00 %
  max. pct. deviation =  10.00 %
  regression threshold =  10.00 % 
sample stats for baseline:
  min = 2807.907130
  max = 2933.084229
  mean = 2870.635283
  sd = 62.589017
  pct.dev. =  2.18 %
sample stats for current:
  min = 2823.884808
  max = 2891.817321
  mean = 2859.885453
  sd = 34.148540
  pct.dev. =  1.19 %
CHANGE -0.37 percent
magnitude of change is less than standard deviation of samples
/usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean.
Please note that:
    - numpy.mean axis argument defaults to None, not 0
    - numpy.mean has a ddof argument to replace bias in a more general manner.
      scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x,
axis=0, ddof=1).
  axis=0, ddof=1).""", DeprecationWarning)
t-test t-statistic = 0.261144 probability = 0.806876
t-test says that mean of two sample sets differs with probability  19.31%
probability that sample sets have same mean = 0.8069
sample sets are statistically indistinguishable for specified confidence level
RESULT:0


======= Throughput getxattr =========

decision parameters:
  sample type = throughput
  confidence threshold =  95.00 %
  max. pct. deviation =  10.00 %
  regression threshold =  10.00 % 
sample stats for baseline:
  min = 11627.131972
  max = 11728.358597
  mean = 11677.918053
  sd = 50.614197
  pct.dev. =  0.43 %
sample stats for current:
  min = 12619.969594
  max = 12755.719380
  mean = 12671.562069
  sd = 73.500697
  pct.dev. =  0.58 %
CHANGE  8.51 percent
magnitude of change is at least  7.93%
/usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean.
Please note that:
    - numpy.mean axis argument defaults to None, not 0
    - numpy.mean has a ddof argument to replace bias in a more general manner.
      scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x,
axis=0, ddof=1).
  axis=0, ddof=1).""", DeprecationWarning)
t-test t-statistic = -19.285094 probability = 0.000043
t-test says that mean of two sample sets differs with probability 100.00%
probability that sample sets have same mean = 0.0000
current sample set is statistically better than baseline sample set
RESULT:0


======= Throughput rename =========

decision parameters:
  sample type = throughput
  confidence threshold =  95.00 %
  max. pct. deviation =  10.00 %
  regression threshold =  10.00 % 
sample stats for baseline:
  min = 182.264552
  max = 183.033306
  mean = 182.527871
  sd = 0.437848
  pct.dev. =  0.24 %
sample stats for current:
  min = 264.937331
  max = 271.017554
  mean = 268.500543
  sd = 3.172251
  pct.dev. =  1.18 %
CHANGE 47.10 percent
magnitude of change is at least 45.92%
/usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean.
Please note that:
    - numpy.mean axis argument defaults to None, not 0
    - numpy.mean has a ddof argument to replace bias in a more general manner.
      scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x,
axis=0, ddof=1).
  axis=0, ddof=1).""", DeprecationWarning)
t-test t-statistic = -46.500278 probability = 0.000001
t-test says that mean of two sample sets differs with probability 100.00%
probability that sample sets have same mean = 0.0000
current sample set is statistically better than baseline sample set
RESULT:0


======= Throughput delete-renamed =========

decision parameters:
  sample type = throughput
  confidence threshold =  95.00 %
  max. pct. deviation =  10.00 %
  regression threshold =  10.00 % 
sample stats for baseline:
  min = 1284.636443
  max = 1314.856695
  mean = 1299.809573
  sd = 15.110520
  pct.dev. =  1.16 %
sample stats for current:
  min = 811.306814
  max = 948.673979
  mean = 892.269959
  sd = 71.901304
  pct.dev. =  8.06 %
CHANGE -31.35 percent
magnitude of change is at least 23.30%
/usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean.
Please note that:
    - numpy.mean axis argument defaults to None, not 0
    - numpy.mean has a ddof argument to replace bias in a more general manner.
      scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x,
axis=0, ddof=1).
  axis=0, ddof=1).""", DeprecationWarning)
t-test t-statistic = 9.607469 probability = 0.000656
t-test says that mean of two sample sets differs with probability  99.93%
probability that sample sets have same mean = 0.0007
declaring a performance regression test FAILURE because of lower throughput
RESULT:10

run154 - nfs - 3.4.0.59rhs-1.el6rhs - distrep - (quota no) - Threads:8, files:10000, file-size:64
run170 - nfs - 3.6.0-24.1.el6rhs - distrep - (quota no) - Threads:8, files:10000, file-size:64

Operations                      RUN154  RUN170
-------------------------       ------- -------
create                          493     145
append                          1038    175
read                            1690    1462
readdir                         9256    11228
ls-l                            5823    3832
mkdir                           746     489
rmdir                           558     378
stat                            4335    4129
chmod                           2018    2040
setxattr                        2870    2859
getxattr                        11677   12670
rename                          182     268
delete-renamed                  1299    891


Volume info for RUN154
 
Volume Name: testvol
Type: Distributed-Replicate
Volume ID: 62c9d7aa-620f-42a6-b6ac-15d3931c0925
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: rhs-gp-srv2.lab.eng.blr.redhat.com:/data/datastore
Brick2: rhs-gp-srv5.lab.eng.blr.redhat.com:/data/datastore
Brick3: rhs-gp-srv6.lab.eng.blr.redhat.com:/data/datastore
Brick4: rhs-gp-srv16.lab.eng.blr.redhat.com:/data/datastore
Options Reconfigured:
nfs.mem-factor: 20
performance.quick-read: on
performance.open-behind: on


Volume info for RUN170
 
Volume Name: testvol
Type: Distributed-Replicate
Volume ID: a7a142e4-6e14-4e55-9fa4-a9b22ac92cbc
Status: Started
Snap Volume: no
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: rhs-gp-srv2.lab.eng.blr.redhat.com:/data/datastore
Brick2: rhs-gp-srv5.lab.eng.blr.redhat.com:/data/datastore
Brick3: rhs-gp-srv6.lab.eng.blr.redhat.com:/data/datastore
Brick4: rhs-gp-srv16.lab.eng.blr.redhat.com:/data/datastore
Options Reconfigured:
nfs.mem-factor: 20
performance.quick-read: on
performance.open-behind: on
performance.readdir-ahead: on
auto-delete: disable
snap-max-soft-limit: 90
snap-max-hard-limit: 256

Comment 2 santosh pradhan 2014-07-22 17:33:01 UTC
Anush,
You had filed one more bug BZ 1067284 which is same as this. Just the build is different. Wanted to mark it as duplicate of that, but ll check with you at office.

The major change:
Throttling in RPC layer i.e. outstanding-rpc-limit which would put limitation but with large I/O, that helps.

If you want the same performance as before, you can turn them OFF in NFS and server as I mentioned in BZ 1067284 which is still in NEEDINFO status.

-Santosh

Comment 3 santosh pradhan 2014-07-22 17:35:10 UTC
Why nfs.mem-factor is being tuned?

Comment 5 santosh pradhan 2014-07-26 11:31:34 UTC

Anush,
Did you have a chance to look at it?

If you agree with Ben's observation/test results, probably we can close the issue.

Thanks,
Santosh

Comment 6 santosh pradhan 2014-08-11 11:39:34 UTC
CLOSED it for now. Please feel free to reopen if you face it with latest build.


Note You need to log in before you can comment on or make changes to this bug.