Description of problem: ----------------------- Testbed : 12*(4+2),6 servers,6 workload generating clients. Benchmark : 3.1.3 with io-threads enabled. 3.2 testing was done with io-threads enabled and mdcache parameters set. It looks like we have regressed with 3.2 on large file writes and reads over SMB3 : ****************** Sequential Writes ****************** 3.1.3 : 1001869 kB/sec 3.2 : 773809.89 kB/sec Regression : 23% ****************** Sequential Reads ****************** 3.1.3 : 1499580.48 kB/sec 3.2 : 1106730.49 kB/sec Regression : 26% AIO was disabled at all times. Server Profiles will be attached soon. Version-Release number of selected component (if applicable): ------------------------------------------------------------- glusterfs-3.8.4-10.el7rhgs.x86_64 How reproducible: ----------------- Always Actual results: --------------- Sequential workloads on SMB 3 mounts have regressed by 25%. Expected results: ----------------- Regression Threshold is within 10% Additional info: ---------------- > Server Profile will be attached. > Client and Server OS : RHEL 7.3 > *Vol config* : Volume Name: butcher Type: Distributed-Disperse Volume ID: 4a377ad9-0c87-4553-b45f-95ab0590c055 Status: Started Snapshot Count: 0 Number of Bricks: 12 x (4 + 2) = 72 Transport-type: tcp Bricks: Brick1: gqas008.sbu.lab.eng.bos.redhat.com:/bricks1/brick Brick2: gqas009.sbu.lab.eng.bos.redhat.com:/bricks1/brick Brick3: gqas010.sbu.lab.eng.bos.redhat.com:/bricks1/brick Brick4: gqas014.sbu.lab.eng.bos.redhat.com:/bricks1/brick Brick5: gqas001.sbu.lab.eng.bos.redhat.com:/bricks1/brick Brick6: gqas016.sbu.lab.eng.bos.redhat.com:/bricks1/brick Brick7: gqas008.sbu.lab.eng.bos.redhat.com:/bricks2/brick Brick8: gqas009.sbu.lab.eng.bos.redhat.com:/bricks2/brick Brick9: gqas010.sbu.lab.eng.bos.redhat.com:/bricks2/brick Brick10: gqas014.sbu.lab.eng.bos.redhat.com:/bricks2/brick Brick11: gqas001.sbu.lab.eng.bos.redhat.com:/bricks2/brick Brick12: gqas016.sbu.lab.eng.bos.redhat.com:/bricks2/brick Brick13: gqas008.sbu.lab.eng.bos.redhat.com:/bricks3/brick Brick14: gqas009.sbu.lab.eng.bos.redhat.com:/bricks3/brick Brick15: gqas010.sbu.lab.eng.bos.redhat.com:/bricks3/brick Brick16: gqas014.sbu.lab.eng.bos.redhat.com:/bricks3/brick Brick17: gqas001.sbu.lab.eng.bos.redhat.com:/bricks3/brick Brick18: gqas016.sbu.lab.eng.bos.redhat.com:/bricks3/brick Brick19: gqas008.sbu.lab.eng.bos.redhat.com:/bricks4/brick Brick20: gqas009.sbu.lab.eng.bos.redhat.com:/bricks4/brick Brick21: gqas010.sbu.lab.eng.bos.redhat.com:/bricks4/brick Brick22: gqas014.sbu.lab.eng.bos.redhat.com:/bricks4/brick Brick23: gqas001.sbu.lab.eng.bos.redhat.com:/bricks4/brick Brick24: gqas016.sbu.lab.eng.bos.redhat.com:/bricks4/brick Brick25: gqas008.sbu.lab.eng.bos.redhat.com:/bricks5/brick Brick26: gqas009.sbu.lab.eng.bos.redhat.com:/bricks5/brick Brick27: gqas010.sbu.lab.eng.bos.redhat.com:/bricks5/brick Brick28: gqas014.sbu.lab.eng.bos.redhat.com:/bricks5/brick Brick29: gqas001.sbu.lab.eng.bos.redhat.com:/bricks5/brick Brick30: gqas016.sbu.lab.eng.bos.redhat.com:/bricks5/brick Brick31: gqas008.sbu.lab.eng.bos.redhat.com:/bricks6/brick Brick32: gqas009.sbu.lab.eng.bos.redhat.com:/bricks6/brick Brick33: gqas010.sbu.lab.eng.bos.redhat.com:/bricks6/brick Brick34: gqas014.sbu.lab.eng.bos.redhat.com:/bricks6/brick Brick35: gqas001.sbu.lab.eng.bos.redhat.com:/bricks6/brick Brick36: gqas016.sbu.lab.eng.bos.redhat.com:/bricks6/brick Brick37: gqas008.sbu.lab.eng.bos.redhat.com:/bricks7/brick Brick38: gqas009.sbu.lab.eng.bos.redhat.com:/bricks7/brick Brick39: gqas010.sbu.lab.eng.bos.redhat.com:/bricks7/brick Brick40: gqas014.sbu.lab.eng.bos.redhat.com:/bricks7/brick Brick41: gqas001.sbu.lab.eng.bos.redhat.com:/bricks7/brick Brick42: gqas016.sbu.lab.eng.bos.redhat.com:/bricks7/brick Brick43: gqas008.sbu.lab.eng.bos.redhat.com:/bricks8/brick Brick44: gqas009.sbu.lab.eng.bos.redhat.com:/bricks8/brick Brick45: gqas010.sbu.lab.eng.bos.redhat.com:/bricks8/brick Brick46: gqas014.sbu.lab.eng.bos.redhat.com:/bricks8/brick Brick47: gqas001.sbu.lab.eng.bos.redhat.com:/bricks8/brick Brick48: gqas016.sbu.lab.eng.bos.redhat.com:/bricks8/brick Brick49: gqas008.sbu.lab.eng.bos.redhat.com:/bricks9/brick Brick50: gqas009.sbu.lab.eng.bos.redhat.com:/bricks9/brick Brick51: gqas010.sbu.lab.eng.bos.redhat.com:/bricks9/brick Brick52: gqas014.sbu.lab.eng.bos.redhat.com:/bricks9/brick Brick53: gqas001.sbu.lab.eng.bos.redhat.com:/bricks9/brick Brick54: gqas016.sbu.lab.eng.bos.redhat.com:/bricks9/brick Brick55: gqas008.sbu.lab.eng.bos.redhat.com:/bricks10/brick Brick56: gqas009.sbu.lab.eng.bos.redhat.com:/bricks10/brick Brick57: gqas010.sbu.lab.eng.bos.redhat.com:/bricks10/brick Brick58: gqas014.sbu.lab.eng.bos.redhat.com:/bricks10/brick Brick59: gqas001.sbu.lab.eng.bos.redhat.com:/bricks10/brick Brick60: gqas016.sbu.lab.eng.bos.redhat.com:/bricks10/brick Brick61: gqas008.sbu.lab.eng.bos.redhat.com:/bricks11/brick Brick62: gqas009.sbu.lab.eng.bos.redhat.com:/bricks11/brick Brick63: gqas010.sbu.lab.eng.bos.redhat.com:/bricks11/brick Brick64: gqas014.sbu.lab.eng.bos.redhat.com:/bricks11/brick Brick65: gqas001.sbu.lab.eng.bos.redhat.com:/bricks11/brick Brick66: gqas016.sbu.lab.eng.bos.redhat.com:/bricks11/brick Brick67: gqas008.sbu.lab.eng.bos.redhat.com:/bricks12/brick Brick68: gqas009.sbu.lab.eng.bos.redhat.com:/bricks12/brick Brick69: gqas010.sbu.lab.eng.bos.redhat.com:/bricks12/brick Brick70: gqas014.sbu.lab.eng.bos.redhat.com:/bricks12/brick Brick71: gqas001.sbu.lab.eng.bos.redhat.com:/bricks12/brick Brick72: gqas016.sbu.lab.eng.bos.redhat.com:/bricks12/brick Options Reconfigured: diagnostics.count-fop-hits: on diagnostics.latency-measurement: on cluster.lookup-optimize: on performance.client-io-threads: on nfs.disable: off performance.readdir-ahead: on transport.address-family: inet server.event-threads: 4 client.event-threads: 4 features.cache-invalidation: on features.cache-invalidation-timeout: 600 performance.stat-prefetch: on performance.cache-samba-metadata: on performance.cache-invalidation: on performance.md-cache-timeout: 600 [root@gqas008 ~]#
After setting io-threads to off,I do not see any improvement(unlike on a *2 deployment). Sequential Writes(io-threads off) : 717642.56 kB/sec Sequential Reads(io-threads off) : 1111601.68 kB/sec
Created attachment 1235326 [details] 3.2_SMB3
upstream patch : http://review.gluster.org/16377
downstream patch : https://code.engineering.redhat.com/gerrit/#/c/95870/
Sequential Writes within 10% of the benchmark on 3.8.4-13,Verified. Sequential Read regression tracked via https://bugzilla.redhat.com/show_bug.cgi?id=1419807.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2017-0486.html