Bug 1419859 - [Perf] : Renames are off target by 28% on EC FUSE mounts
Summary: [Perf] : Renames are off target by 28% on EC FUSE mounts
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: disperse
Version: rhgs-3.2
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: RHGS 3.2.0
Assignee: Ashish Pandey
QA Contact: Ambarish
URL:
Whiteboard:
Depends On:
Blocks: 1351528
TreeView+ depends on / blocked
 
Reported: 2017-02-07 09:17 UTC by Ambarish
Modified: 2017-03-28 06:49 UTC (History)
9 users (show)

Fixed In Version: glusterfs-3.8.4-18
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-03-23 06:04:47 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2017:0486 normal SHIPPED_LIVE Moderate: Red Hat Gluster Storage 3.2.0 security, bug fix, and enhancement update 2017-03-23 09:18:45 UTC
Red Hat Bugzilla 1406723 None None None Never

Internal Links: 1406723

Description Ambarish 2017-02-07 09:17:22 UTC
Description of problem:
-----------------------

A regression on renames seems to have been introduced in 3.2 on EC over FUSE :

3.1.3 : 724 files/sec

3.8.4-13 :  521 filessec

Regression : -28%


Version-Release number of selected component (if applicable):
--------------------------------------------------------------

3.8.4-13

How reproducible:
-----------------

100%


Actual results:
---------------

28% regression on renames.

Expected results:
-----------------

Regression Threshold : 10%

Additional info:
---------------

Volume Name: butcher
Type: Distributed-Disperse
Volume ID: b5a2fba9-ea65-4ddd-be3c-c9f05868d623
Status: Started
Snapshot Count: 0
Number of Bricks: 12 x (4 + 2) = 72
Transport-type: tcp
Bricks:
Brick1: gqas013.sbu.lab.eng.bos.redhat.com:/bricks1/brick
Brick2: gqas005.sbu.lab.eng.bos.redhat.com:/bricks1/brick
Brick3: gqas006.sbu.lab.eng.bos.redhat.com:/bricks1/brick
Brick4: gqas008.sbu.lab.eng.bos.redhat.com:/bricks1/brick
Brick5: gqas009.sbu.lab.eng.bos.redhat.com:/bricks1/brick
Brick6: gqas015.sbu.lab.eng.bos.redhat.com:/bricks1/brick
Brick7: gqas013.sbu.lab.eng.bos.redhat.com:/bricks2/brick
Brick8: gqas005.sbu.lab.eng.bos.redhat.com:/bricks2/brick
Brick9: gqas006.sbu.lab.eng.bos.redhat.com:/bricks2/brick
Brick10: gqas008.sbu.lab.eng.bos.redhat.com:/bricks2/brick
Brick11: gqas009.sbu.lab.eng.bos.redhat.com:/bricks2/brick
Brick12: gqas015.sbu.lab.eng.bos.redhat.com:/bricks2/brick
Brick13: gqas013.sbu.lab.eng.bos.redhat.com:/bricks3/brick
Brick14: gqas005.sbu.lab.eng.bos.redhat.com:/bricks3/brick
Brick15: gqas006.sbu.lab.eng.bos.redhat.com:/bricks3/brick
Brick16: gqas008.sbu.lab.eng.bos.redhat.com:/bricks3/brick
Brick17: gqas009.sbu.lab.eng.bos.redhat.com:/bricks3/brick
Brick18: gqas015.sbu.lab.eng.bos.redhat.com:/bricks3/brick
Brick19: gqas013.sbu.lab.eng.bos.redhat.com:/bricks4/brick
Brick20: gqas005.sbu.lab.eng.bos.redhat.com:/bricks4/brick
Brick21: gqas006.sbu.lab.eng.bos.redhat.com:/bricks4/brick
Brick22: gqas008.sbu.lab.eng.bos.redhat.com:/bricks4/brick
Brick23: gqas009.sbu.lab.eng.bos.redhat.com:/bricks4/brick
Brick24: gqas015.sbu.lab.eng.bos.redhat.com:/bricks4/brick
Brick25: gqas013.sbu.lab.eng.bos.redhat.com:/bricks5/brick
Brick26: gqas005.sbu.lab.eng.bos.redhat.com:/bricks5/brick
Brick27: gqas006.sbu.lab.eng.bos.redhat.com:/bricks5/brick
Brick28: gqas008.sbu.lab.eng.bos.redhat.com:/bricks5/brick
Brick29: gqas009.sbu.lab.eng.bos.redhat.com:/bricks5/brick
Brick30: gqas015.sbu.lab.eng.bos.redhat.com:/bricks5/brick
Brick31: gqas013.sbu.lab.eng.bos.redhat.com:/bricks6/brick
Brick32: gqas005.sbu.lab.eng.bos.redhat.com:/bricks6/brick
Brick33: gqas006.sbu.lab.eng.bos.redhat.com:/bricks6/brick
Brick34: gqas008.sbu.lab.eng.bos.redhat.com:/bricks6/brick
Brick35: gqas009.sbu.lab.eng.bos.redhat.com:/bricks6/brick
Brick36: gqas015.sbu.lab.eng.bos.redhat.com:/bricks6/brick
Brick37: gqas013.sbu.lab.eng.bos.redhat.com:/bricks7/brick
Brick38: gqas005.sbu.lab.eng.bos.redhat.com:/bricks7/brick
Brick39: gqas006.sbu.lab.eng.bos.redhat.com:/bricks7/brick
Brick40: gqas008.sbu.lab.eng.bos.redhat.com:/bricks7/brick
Brick41: gqas009.sbu.lab.eng.bos.redhat.com:/bricks7/brick
Brick42: gqas015.sbu.lab.eng.bos.redhat.com:/bricks7/brick
Brick43: gqas013.sbu.lab.eng.bos.redhat.com:/bricks8/brick
Brick44: gqas005.sbu.lab.eng.bos.redhat.com:/bricks8/brick
Brick45: gqas006.sbu.lab.eng.bos.redhat.com:/bricks8/brick
Brick46: gqas008.sbu.lab.eng.bos.redhat.com:/bricks8/brick
Brick47: gqas009.sbu.lab.eng.bos.redhat.com:/bricks8/brick
Brick48: gqas015.sbu.lab.eng.bos.redhat.com:/bricks8/brick
Brick49: gqas013.sbu.lab.eng.bos.redhat.com:/bricks9/brick
Brick50: gqas005.sbu.lab.eng.bos.redhat.com:/bricks9/brick
Brick51: gqas006.sbu.lab.eng.bos.redhat.com:/bricks9/brick
Brick52: gqas008.sbu.lab.eng.bos.redhat.com:/bricks9/brick
Brick53: gqas009.sbu.lab.eng.bos.redhat.com:/bricks9/brick
Brick54: gqas015.sbu.lab.eng.bos.redhat.com:/bricks9/brick
Brick55: gqas013.sbu.lab.eng.bos.redhat.com:/bricks10/brick
Brick56: gqas005.sbu.lab.eng.bos.redhat.com:/bricks10/brick
Brick57: gqas006.sbu.lab.eng.bos.redhat.com:/bricks10/brick
Brick58: gqas008.sbu.lab.eng.bos.redhat.com:/bricks10/brick
Brick59: gqas009.sbu.lab.eng.bos.redhat.com:/bricks10/brick
Brick60: gqas015.sbu.lab.eng.bos.redhat.com:/bricks10/brick
Brick61: gqas013.sbu.lab.eng.bos.redhat.com:/bricks11/brick
Brick62: gqas005.sbu.lab.eng.bos.redhat.com:/bricks11/brick
Brick63: gqas006.sbu.lab.eng.bos.redhat.com:/bricks11/brick
Brick64: gqas008.sbu.lab.eng.bos.redhat.com:/bricks11/brick
Brick65: gqas009.sbu.lab.eng.bos.redhat.com:/bricks11/brick
Brick66: gqas015.sbu.lab.eng.bos.redhat.com:/bricks11/brick
Brick67: gqas013.sbu.lab.eng.bos.redhat.com:/bricks12/brick
Brick68: gqas005.sbu.lab.eng.bos.redhat.com:/bricks12/brick
Brick69: gqas006.sbu.lab.eng.bos.redhat.com:/bricks12/brick
Brick70: gqas008.sbu.lab.eng.bos.redhat.com:/bricks12/brick
Brick71: gqas009.sbu.lab.eng.bos.redhat.com:/bricks12/brick
Brick72: gqas015.sbu.lab.eng.bos.redhat.com:/bricks12/brick
Options Reconfigured:
diagnostics.count-fop-hits: on
diagnostics.latency-measurement: on
cluster.lookup-optimize: on
nfs.disable: off
performance.readdir-ahead: on
transport.address-family: inet
features.cache-invalidation: on
features.cache-invalidation-timeout: 600
performance.stat-prefetch: on
performance.cache-samba-metadata: on
performance.cache-invalidation: on
performance.md-cache-timeout: 600
server.event-threads: 4
client.event-threads: 4
performance.client-io-threads: on
[root@gqas013 ~]#

Comment 2 Ambarish 2017-02-07 09:18:46 UTC
Regression from 3.1.3,adding Regression keyword.

Comment 5 Atin Mukherjee 2017-02-08 06:30:08 UTC
RCA of this issue is same of BZ 1406723.

Upstream patch : http://review.gluster.org/#/c/16298/

Comment 6 Atin Mukherjee 2017-03-03 06:48:15 UTC
A new upstream patch https://review.gluster.org/#/c/16821/ is posted with a different alternative.

Comment 9 Ambarish 2017-03-07 11:30:54 UTC
3.1.3 : 724 files/sec

3.8.4-17 : 549 files/sec


regression : ~24%


There is still a 24% regression.

Failing QA and moving it back to Assigned.

Comment 10 Pranith Kumar K 2017-03-07 11:35:41 UTC
(In reply to Ambarish from comment #9)
> 3.1.3 : 724 files/sec
> 
> 3.8.4-17 : 549 files/sec
> 
> 
> regression : ~24%
> 
> 
> There is still a 24% regression.
> 
> Failing QA and moving it back to Assigned.

Number of network operations is exactly same as 3.1.3 but the avg latencies increased in 3.2.0 Based on the testing I did, it seems to be because of l[g/s]etxattr for trusted.ec.dirty. I just provided a build to Ambarish to confirm if that brings the number closer.

Comment 11 Atin Mukherjee 2017-03-07 13:03:56 UTC
upstream patch : https://review.gluster.org/16865

Comment 12 Pranith Kumar K 2017-03-07 14:12:45 UTC
https://code.engineering.redhat.com/gerrit/99547

Comment 13 Milind Changire 2017-03-08 12:20:21 UTC
bug added to erratum RHSA-2016:24866
moving bz to ON_QA

Comment 14 Ambarish 2017-03-09 15:44:56 UTC
3.1.3 : 724 files/sec

3.8.4-18 : 694 files/sec


Within the regression threshold on 3.8.4-18,Verified.

Comment 15 Pranith Kumar K 2017-03-15 18:55:07 UTC
(In reply to Ambarish from comment #14)
> 3.1.3 : 724 files/sec
> 
> 3.8.4-18 : 694 files/sec
> 
> 
> Within the regression threshold on 3.8.4-18,Verified.

Just want to know what could be the reason for this, if you don't mind could you attach strace/profile info of these runs, so that I can find what is the reason for this difference.

Comment 17 errata-xmlrpc 2017-03-23 06:04:47 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2017-0486.html


Note You need to log in before you can comment on or make changes to this bug.