Red Hat Bugzilla – Bug 1419859
[Perf] : Renames are off target by 28% on EC FUSE mounts
Last modified: 2017-03-28 02:49:38 EDT
Description of problem: ----------------------- A regression on renames seems to have been introduced in 3.2 on EC over FUSE : 3.1.3 : 724 files/sec 3.8.4-13 : 521 filessec Regression : -28% Version-Release number of selected component (if applicable): -------------------------------------------------------------- 3.8.4-13 How reproducible: ----------------- 100% Actual results: --------------- 28% regression on renames. Expected results: ----------------- Regression Threshold : 10% Additional info: --------------- Volume Name: butcher Type: Distributed-Disperse Volume ID: b5a2fba9-ea65-4ddd-be3c-c9f05868d623 Status: Started Snapshot Count: 0 Number of Bricks: 12 x (4 + 2) = 72 Transport-type: tcp Bricks: Brick1: gqas013.sbu.lab.eng.bos.redhat.com:/bricks1/brick Brick2: gqas005.sbu.lab.eng.bos.redhat.com:/bricks1/brick Brick3: gqas006.sbu.lab.eng.bos.redhat.com:/bricks1/brick Brick4: gqas008.sbu.lab.eng.bos.redhat.com:/bricks1/brick Brick5: gqas009.sbu.lab.eng.bos.redhat.com:/bricks1/brick Brick6: gqas015.sbu.lab.eng.bos.redhat.com:/bricks1/brick Brick7: gqas013.sbu.lab.eng.bos.redhat.com:/bricks2/brick Brick8: gqas005.sbu.lab.eng.bos.redhat.com:/bricks2/brick Brick9: gqas006.sbu.lab.eng.bos.redhat.com:/bricks2/brick Brick10: gqas008.sbu.lab.eng.bos.redhat.com:/bricks2/brick Brick11: gqas009.sbu.lab.eng.bos.redhat.com:/bricks2/brick Brick12: gqas015.sbu.lab.eng.bos.redhat.com:/bricks2/brick Brick13: gqas013.sbu.lab.eng.bos.redhat.com:/bricks3/brick Brick14: gqas005.sbu.lab.eng.bos.redhat.com:/bricks3/brick Brick15: gqas006.sbu.lab.eng.bos.redhat.com:/bricks3/brick Brick16: gqas008.sbu.lab.eng.bos.redhat.com:/bricks3/brick Brick17: gqas009.sbu.lab.eng.bos.redhat.com:/bricks3/brick Brick18: gqas015.sbu.lab.eng.bos.redhat.com:/bricks3/brick Brick19: gqas013.sbu.lab.eng.bos.redhat.com:/bricks4/brick Brick20: gqas005.sbu.lab.eng.bos.redhat.com:/bricks4/brick Brick21: gqas006.sbu.lab.eng.bos.redhat.com:/bricks4/brick Brick22: gqas008.sbu.lab.eng.bos.redhat.com:/bricks4/brick Brick23: gqas009.sbu.lab.eng.bos.redhat.com:/bricks4/brick Brick24: gqas015.sbu.lab.eng.bos.redhat.com:/bricks4/brick Brick25: gqas013.sbu.lab.eng.bos.redhat.com:/bricks5/brick Brick26: gqas005.sbu.lab.eng.bos.redhat.com:/bricks5/brick Brick27: gqas006.sbu.lab.eng.bos.redhat.com:/bricks5/brick Brick28: gqas008.sbu.lab.eng.bos.redhat.com:/bricks5/brick Brick29: gqas009.sbu.lab.eng.bos.redhat.com:/bricks5/brick Brick30: gqas015.sbu.lab.eng.bos.redhat.com:/bricks5/brick Brick31: gqas013.sbu.lab.eng.bos.redhat.com:/bricks6/brick Brick32: gqas005.sbu.lab.eng.bos.redhat.com:/bricks6/brick Brick33: gqas006.sbu.lab.eng.bos.redhat.com:/bricks6/brick Brick34: gqas008.sbu.lab.eng.bos.redhat.com:/bricks6/brick Brick35: gqas009.sbu.lab.eng.bos.redhat.com:/bricks6/brick Brick36: gqas015.sbu.lab.eng.bos.redhat.com:/bricks6/brick Brick37: gqas013.sbu.lab.eng.bos.redhat.com:/bricks7/brick Brick38: gqas005.sbu.lab.eng.bos.redhat.com:/bricks7/brick Brick39: gqas006.sbu.lab.eng.bos.redhat.com:/bricks7/brick Brick40: gqas008.sbu.lab.eng.bos.redhat.com:/bricks7/brick Brick41: gqas009.sbu.lab.eng.bos.redhat.com:/bricks7/brick Brick42: gqas015.sbu.lab.eng.bos.redhat.com:/bricks7/brick Brick43: gqas013.sbu.lab.eng.bos.redhat.com:/bricks8/brick Brick44: gqas005.sbu.lab.eng.bos.redhat.com:/bricks8/brick Brick45: gqas006.sbu.lab.eng.bos.redhat.com:/bricks8/brick Brick46: gqas008.sbu.lab.eng.bos.redhat.com:/bricks8/brick Brick47: gqas009.sbu.lab.eng.bos.redhat.com:/bricks8/brick Brick48: gqas015.sbu.lab.eng.bos.redhat.com:/bricks8/brick Brick49: gqas013.sbu.lab.eng.bos.redhat.com:/bricks9/brick Brick50: gqas005.sbu.lab.eng.bos.redhat.com:/bricks9/brick Brick51: gqas006.sbu.lab.eng.bos.redhat.com:/bricks9/brick Brick52: gqas008.sbu.lab.eng.bos.redhat.com:/bricks9/brick Brick53: gqas009.sbu.lab.eng.bos.redhat.com:/bricks9/brick Brick54: gqas015.sbu.lab.eng.bos.redhat.com:/bricks9/brick Brick55: gqas013.sbu.lab.eng.bos.redhat.com:/bricks10/brick Brick56: gqas005.sbu.lab.eng.bos.redhat.com:/bricks10/brick Brick57: gqas006.sbu.lab.eng.bos.redhat.com:/bricks10/brick Brick58: gqas008.sbu.lab.eng.bos.redhat.com:/bricks10/brick Brick59: gqas009.sbu.lab.eng.bos.redhat.com:/bricks10/brick Brick60: gqas015.sbu.lab.eng.bos.redhat.com:/bricks10/brick Brick61: gqas013.sbu.lab.eng.bos.redhat.com:/bricks11/brick Brick62: gqas005.sbu.lab.eng.bos.redhat.com:/bricks11/brick Brick63: gqas006.sbu.lab.eng.bos.redhat.com:/bricks11/brick Brick64: gqas008.sbu.lab.eng.bos.redhat.com:/bricks11/brick Brick65: gqas009.sbu.lab.eng.bos.redhat.com:/bricks11/brick Brick66: gqas015.sbu.lab.eng.bos.redhat.com:/bricks11/brick Brick67: gqas013.sbu.lab.eng.bos.redhat.com:/bricks12/brick Brick68: gqas005.sbu.lab.eng.bos.redhat.com:/bricks12/brick Brick69: gqas006.sbu.lab.eng.bos.redhat.com:/bricks12/brick Brick70: gqas008.sbu.lab.eng.bos.redhat.com:/bricks12/brick Brick71: gqas009.sbu.lab.eng.bos.redhat.com:/bricks12/brick Brick72: gqas015.sbu.lab.eng.bos.redhat.com:/bricks12/brick Options Reconfigured: diagnostics.count-fop-hits: on diagnostics.latency-measurement: on cluster.lookup-optimize: on nfs.disable: off performance.readdir-ahead: on transport.address-family: inet features.cache-invalidation: on features.cache-invalidation-timeout: 600 performance.stat-prefetch: on performance.cache-samba-metadata: on performance.cache-invalidation: on performance.md-cache-timeout: 600 server.event-threads: 4 client.event-threads: 4 performance.client-io-threads: on [root@gqas013 ~]#
Regression from 3.1.3,adding Regression keyword.
RCA of this issue is same of BZ 1406723. Upstream patch : http://review.gluster.org/#/c/16298/
A new upstream patch https://review.gluster.org/#/c/16821/ is posted with a different alternative.
3.1.3 : 724 files/sec 3.8.4-17 : 549 files/sec regression : ~24% There is still a 24% regression. Failing QA and moving it back to Assigned.
(In reply to Ambarish from comment #9) > 3.1.3 : 724 files/sec > > 3.8.4-17 : 549 files/sec > > > regression : ~24% > > > There is still a 24% regression. > > Failing QA and moving it back to Assigned. Number of network operations is exactly same as 3.1.3 but the avg latencies increased in 3.2.0 Based on the testing I did, it seems to be because of l[g/s]etxattr for trusted.ec.dirty. I just provided a build to Ambarish to confirm if that brings the number closer.
upstream patch : https://review.gluster.org/16865
https://code.engineering.redhat.com/gerrit/99547
bug added to erratum RHSA-2016:24866 moving bz to ON_QA
3.1.3 : 724 files/sec 3.8.4-18 : 694 files/sec Within the regression threshold on 3.8.4-18,Verified.
(In reply to Ambarish from comment #14) > 3.1.3 : 724 files/sec > > 3.8.4-18 : 694 files/sec > > > Within the regression threshold on 3.8.4-18,Verified. Just want to know what could be the reason for this, if you don't mind could you attach strace/profile info of these runs, so that I can find what is the reason for this difference.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2017-0486.html