Hide Forgot
Description of problem: We see performance regression in creates (-70.50%), append (-83.09%) and directory operations on NFS mount in comparison with RHS 2.1 build 3.4.0.59rhs-1.el6rhs Version-Release number of selected component (if applicable): 3.6.0-24.1.el6rhs How reproducible: Run as part of performance regression suite Steps to Reproduce: 1. Create a 2x2 Distributed-Replicate volume with 2 clients 2. Run smallfile test suite: https://github.com/bengland2/smallfile Actual results: ======= Throughput create ========= decision parameters: sample type = throughput confidence threshold = 95.00 % max. pct. deviation = 10.00 % regression threshold = 10.00 % sample stats for baseline: min = 493.648580 max = 494.691803 mean = 494.191710 sd = 0.522941 pct.dev. = 0.11 % sample stats for current: min = 145.281489 max = 146.432456 mean = 145.805070 sd = 0.582463 pct.dev. = 0.40 % CHANGE -70.50 percent magnitude of change is at least 70.10% /usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean. Please note that: - numpy.mean axis argument defaults to None, not 0 - numpy.mean has a ddof argument to replace bias in a more general manner. scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x, axis=0, ddof=1). axis=0, ddof=1).""", DeprecationWarning) t-test t-statistic = 770.881103 probability = 0.000000 t-test says that mean of two sample sets differs with probability 100.00% probability that sample sets have same mean = 0.0000 declaring a performance regression test FAILURE because of lower throughput RESULT:10 ======= Throughput append ========= decision parameters: sample type = throughput confidence threshold = 95.00 % max. pct. deviation = 10.00 % regression threshold = 10.00 % sample stats for baseline: min = 1029.065275 max = 1049.452872 mean = 1038.682042 sd = 10.242677 pct.dev. = 0.99 % sample stats for current: min = 175.168563 max = 176.346088 mean = 175.622468 sd = 0.633404 pct.dev. = 0.36 % CHANGE -83.09 percent magnitude of change is at least 82.11% /usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean. Please note that: - numpy.mean axis argument defaults to None, not 0 - numpy.mean has a ddof argument to replace bias in a more general manner. scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x, axis=0, ddof=1). axis=0, ddof=1).""", DeprecationWarning) t-test t-statistic = 145.666311 probability = 0.000000 t-test says that mean of two sample sets differs with probability 100.00% probability that sample sets have same mean = 0.0000 declaring a performance regression test FAILURE because of lower throughput RESULT:10 ======= Throughput read ========= decision parameters: sample type = throughput confidence threshold = 95.00 % max. pct. deviation = 10.00 % regression threshold = 10.00 % sample stats for baseline: min = 1652.487523 max = 1711.558338 mean = 1690.394255 sd = 32.902546 pct.dev. = 1.95 % sample stats for current: min = 1441.697021 max = 1492.139744 mean = 1462.348419 sd = 26.434273 pct.dev. = 1.81 % CHANGE -13.49 percent magnitude of change is at least 11.54% /usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean. Please note that: - numpy.mean axis argument defaults to None, not 0 - numpy.mean has a ddof argument to replace bias in a more general manner. scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x, axis=0, ddof=1). axis=0, ddof=1).""", DeprecationWarning) t-test t-statistic = 9.358545 probability = 0.000726 t-test says that mean of two sample sets differs with probability 99.93% probability that sample sets have same mean = 0.0007 declaring a performance regression test FAILURE because of lower throughput RESULT:10 ======= Throughput readdir ========= decision parameters: sample type = throughput confidence threshold = 95.00 % max. pct. deviation = 10.00 % regression threshold = 10.00 % sample stats for baseline: min = 9243.819577 max = 9267.905076 mean = 9257.024533 sd = 12.209832 pct.dev. = 0.13 % sample stats for current: min = 11028.228853 max = 11368.462301 mean = 11228.344092 sd = 177.874766 pct.dev. = 1.58 % CHANGE 21.30 percent magnitude of change is at least 19.71% /usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean. Please note that: - numpy.mean axis argument defaults to None, not 0 - numpy.mean has a ddof argument to replace bias in a more general manner. scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x, axis=0, ddof=1). axis=0, ddof=1).""", DeprecationWarning) t-test t-statistic = -19.150607 probability = 0.000044 t-test says that mean of two sample sets differs with probability 100.00% probability that sample sets have same mean = 0.0000 current sample set is statistically better than baseline sample set RESULT:0 ======= Throughput ls-l ========= decision parameters: sample type = throughput confidence threshold = 95.00 % max. pct. deviation = 10.00 % regression threshold = 10.00 % sample stats for baseline: min = 4518.678532 max = 6671.560516 mean = 5823.508967 sd = 1146.826239 pct.dev. = 19.69 % sample stats for current: min = 3764.305345 max = 3869.956399 mean = 3832.686769 sd = 59.300035 pct.dev. = 1.55 % ERROR: pct. deviation of 19.69 is too high for baseline samples RESULT:11 ======= Throughput mkdir ========= decision parameters: sample type = throughput confidence threshold = 95.00 % max. pct. deviation = 10.00 % regression threshold = 10.00 % sample stats for baseline: min = 733.029315 max = 762.575098 mean = 746.807376 sd = 14.873042 pct.dev. = 1.99 % sample stats for current: min = 477.255789 max = 503.437896 mean = 490.027308 sd = 13.102747 pct.dev. = 2.67 % CHANGE -34.38 percent magnitude of change is at least 31.71% /usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean. Please note that: - numpy.mean axis argument defaults to None, not 0 - numpy.mean has a ddof argument to replace bias in a more general manner. scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x, axis=0, ddof=1). axis=0, ddof=1).""", DeprecationWarning) t-test t-statistic = 22.438137 probability = 0.000023 t-test says that mean of two sample sets differs with probability 100.00% probability that sample sets have same mean = 0.0000 declaring a performance regression test FAILURE because of lower throughput RESULT:10 ======= Throughput rmdir ========= decision parameters: sample type = throughput confidence threshold = 95.00 % max. pct. deviation = 10.00 % regression threshold = 10.00 % sample stats for baseline: min = 554.047717 max = 566.721321 mean = 558.589879 sd = 7.058134 pct.dev. = 1.26 % sample stats for current: min = 360.613211 max = 403.665958 mean = 379.176939 sd = 22.129540 pct.dev. = 5.84 % CHANGE -32.12 percent magnitude of change is at least 26.28% /usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean. Please note that: - numpy.mean axis argument defaults to None, not 0 - numpy.mean has a ddof argument to replace bias in a more general manner. scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x, axis=0, ddof=1). axis=0, ddof=1).""", DeprecationWarning) t-test t-statistic = 13.378427 probability = 0.000181 t-test says that mean of two sample sets differs with probability 99.98% probability that sample sets have same mean = 0.0002 declaring a performance regression test FAILURE because of lower throughput RESULT:10 ======= Throughput stat ========= decision parameters: sample type = throughput confidence threshold = 95.00 % max. pct. deviation = 10.00 % regression threshold = 10.00 % sample stats for baseline: min = 4274.219088 max = 4434.828514 mean = 4335.970257 sd = 86.495947 pct.dev. = 1.99 % sample stats for current: min = 3908.265340 max = 4247.890344 mean = 4129.556218 sd = 191.797694 pct.dev. = 4.64 % CHANGE -4.76 percent magnitude of change is at least 0.12% /usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean. Please note that: - numpy.mean axis argument defaults to None, not 0 - numpy.mean has a ddof argument to replace bias in a more general manner. scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x, axis=0, ddof=1). axis=0, ddof=1).""", DeprecationWarning) t-test t-statistic = 1.699243 probability = 0.164501 t-test says that mean of two sample sets differs with probability 83.55% probability that sample sets have same mean = 0.1645 sample sets are statistically indistinguishable for specified confidence level RESULT:0 ======= Throughput chmod ========= decision parameters: sample type = throughput confidence threshold = 95.00 % max. pct. deviation = 10.00 % regression threshold = 10.00 % sample stats for baseline: min = 1976.240010 max = 2054.878358 mean = 2019.279094 sd = 39.843577 pct.dev. = 1.97 % sample stats for current: min = 1930.467085 max = 2232.464141 mean = 2041.222884 sd = 166.309913 pct.dev. = 8.15 % CHANGE 1.09 percent magnitude of change is less than standard deviation of samples /usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean. Please note that: - numpy.mean axis argument defaults to None, not 0 - numpy.mean has a ddof argument to replace bias in a more general manner. scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x, axis=0, ddof=1). axis=0, ddof=1).""", DeprecationWarning) t-test t-statistic = -0.222247 probability = 0.835008 t-test says that mean of two sample sets differs with probability 16.50% probability that sample sets have same mean = 0.8350 sample sets are statistically indistinguishable for specified confidence level RESULT:0 ======= Throughput setxattr ========= decision parameters: sample type = throughput confidence threshold = 95.00 % max. pct. deviation = 10.00 % regression threshold = 10.00 % sample stats for baseline: min = 2807.907130 max = 2933.084229 mean = 2870.635283 sd = 62.589017 pct.dev. = 2.18 % sample stats for current: min = 2823.884808 max = 2891.817321 mean = 2859.885453 sd = 34.148540 pct.dev. = 1.19 % CHANGE -0.37 percent magnitude of change is less than standard deviation of samples /usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean. Please note that: - numpy.mean axis argument defaults to None, not 0 - numpy.mean has a ddof argument to replace bias in a more general manner. scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x, axis=0, ddof=1). axis=0, ddof=1).""", DeprecationWarning) t-test t-statistic = 0.261144 probability = 0.806876 t-test says that mean of two sample sets differs with probability 19.31% probability that sample sets have same mean = 0.8069 sample sets are statistically indistinguishable for specified confidence level RESULT:0 ======= Throughput getxattr ========= decision parameters: sample type = throughput confidence threshold = 95.00 % max. pct. deviation = 10.00 % regression threshold = 10.00 % sample stats for baseline: min = 11627.131972 max = 11728.358597 mean = 11677.918053 sd = 50.614197 pct.dev. = 0.43 % sample stats for current: min = 12619.969594 max = 12755.719380 mean = 12671.562069 sd = 73.500697 pct.dev. = 0.58 % CHANGE 8.51 percent magnitude of change is at least 7.93% /usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean. Please note that: - numpy.mean axis argument defaults to None, not 0 - numpy.mean has a ddof argument to replace bias in a more general manner. scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x, axis=0, ddof=1). axis=0, ddof=1).""", DeprecationWarning) t-test t-statistic = -19.285094 probability = 0.000043 t-test says that mean of two sample sets differs with probability 100.00% probability that sample sets have same mean = 0.0000 current sample set is statistically better than baseline sample set RESULT:0 ======= Throughput rename ========= decision parameters: sample type = throughput confidence threshold = 95.00 % max. pct. deviation = 10.00 % regression threshold = 10.00 % sample stats for baseline: min = 182.264552 max = 183.033306 mean = 182.527871 sd = 0.437848 pct.dev. = 0.24 % sample stats for current: min = 264.937331 max = 271.017554 mean = 268.500543 sd = 3.172251 pct.dev. = 1.18 % CHANGE 47.10 percent magnitude of change is at least 45.92% /usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean. Please note that: - numpy.mean axis argument defaults to None, not 0 - numpy.mean has a ddof argument to replace bias in a more general manner. scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x, axis=0, ddof=1). axis=0, ddof=1).""", DeprecationWarning) t-test t-statistic = -46.500278 probability = 0.000001 t-test says that mean of two sample sets differs with probability 100.00% probability that sample sets have same mean = 0.0000 current sample set is statistically better than baseline sample set RESULT:0 ======= Throughput delete-renamed ========= decision parameters: sample type = throughput confidence threshold = 95.00 % max. pct. deviation = 10.00 % regression threshold = 10.00 % sample stats for baseline: min = 1284.636443 max = 1314.856695 mean = 1299.809573 sd = 15.110520 pct.dev. = 1.16 % sample stats for current: min = 811.306814 max = 948.673979 mean = 892.269959 sd = 71.901304 pct.dev. = 8.06 % CHANGE -31.35 percent magnitude of change is at least 23.30% /usr/lib64/python2.6/site-packages/scipy/stats/stats.py:420: DeprecationWarning: scipy.stats.mean is deprecated; please update your code to use numpy.mean. Please note that: - numpy.mean axis argument defaults to None, not 0 - numpy.mean has a ddof argument to replace bias in a more general manner. scipy.stats.mean(a, bias=True) can be replaced by numpy.mean(x, axis=0, ddof=1). axis=0, ddof=1).""", DeprecationWarning) t-test t-statistic = 9.607469 probability = 0.000656 t-test says that mean of two sample sets differs with probability 99.93% probability that sample sets have same mean = 0.0007 declaring a performance regression test FAILURE because of lower throughput RESULT:10 run154 - nfs - 3.4.0.59rhs-1.el6rhs - distrep - (quota no) - Threads:8, files:10000, file-size:64 run170 - nfs - 3.6.0-24.1.el6rhs - distrep - (quota no) - Threads:8, files:10000, file-size:64 Operations RUN154 RUN170 ------------------------- ------- ------- create 493 145 append 1038 175 read 1690 1462 readdir 9256 11228 ls-l 5823 3832 mkdir 746 489 rmdir 558 378 stat 4335 4129 chmod 2018 2040 setxattr 2870 2859 getxattr 11677 12670 rename 182 268 delete-renamed 1299 891 Volume info for RUN154 Volume Name: testvol Type: Distributed-Replicate Volume ID: 62c9d7aa-620f-42a6-b6ac-15d3931c0925 Status: Started Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: rhs-gp-srv2.lab.eng.blr.redhat.com:/data/datastore Brick2: rhs-gp-srv5.lab.eng.blr.redhat.com:/data/datastore Brick3: rhs-gp-srv6.lab.eng.blr.redhat.com:/data/datastore Brick4: rhs-gp-srv16.lab.eng.blr.redhat.com:/data/datastore Options Reconfigured: nfs.mem-factor: 20 performance.quick-read: on performance.open-behind: on Volume info for RUN170 Volume Name: testvol Type: Distributed-Replicate Volume ID: a7a142e4-6e14-4e55-9fa4-a9b22ac92cbc Status: Started Snap Volume: no Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: rhs-gp-srv2.lab.eng.blr.redhat.com:/data/datastore Brick2: rhs-gp-srv5.lab.eng.blr.redhat.com:/data/datastore Brick3: rhs-gp-srv6.lab.eng.blr.redhat.com:/data/datastore Brick4: rhs-gp-srv16.lab.eng.blr.redhat.com:/data/datastore Options Reconfigured: nfs.mem-factor: 20 performance.quick-read: on performance.open-behind: on performance.readdir-ahead: on auto-delete: disable snap-max-soft-limit: 90 snap-max-hard-limit: 256
Anush, You had filed one more bug BZ 1067284 which is same as this. Just the build is different. Wanted to mark it as duplicate of that, but ll check with you at office. The major change: Throttling in RPC layer i.e. outstanding-rpc-limit which would put limitation but with large I/O, that helps. If you want the same performance as before, you can turn them OFF in NFS and server as I mentioned in BZ 1067284 which is still in NEEDINFO status. -Santosh
Why nfs.mem-factor is being tuned?
Anush, Did you have a chance to look at it? If you agree with Ben's observation/test results, probably we can close the issue. Thanks, Santosh
CLOSED it for now. Please feel free to reopen if you face it with latest build.