Bug 1287531

Summary: Perf: Metadata operation(ls -l) performance regression.
Product: [Community] GlusterFS Reporter: Anuradha <atalur>
Component: replicateAssignee: Anuradha <atalur>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: high Docs Contact:
Priority: high    
Version: 3.7.6CC: atalur, bengland, bturner, bugs, jharriga, mpillai, pkarampu, rcyriac, rhs-bugs, riehecky, ryan.clough, sasundar, shberry, smohan
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: glusterfs-3.7.7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1250803 Environment:
Last Closed: 2016-02-17 07:24:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1250241, 1250803, 1315144    
Bug Blocks:    

Comment 1 Vijay Bellur 2015-12-02 10:27:08 UTC
REVIEW: http://review.gluster.org/12847 (mgmt/gluster: Handle tier brick volgen) posted (#1) for review on release-3.7 by Anuradha Talur (atalur)

Comment 2 Vijay Bellur 2015-12-02 10:30:09 UTC
REVIEW: http://review.gluster.org/12848 (glusterd/afr : Readdirp performance improvement) posted (#1) for review on release-3.7 by Anuradha Talur (atalur)

Comment 3 Vijay Bellur 2015-12-02 12:52:35 UTC
REVIEW: http://review.gluster.org/12852 (features/index : Readdirp performance improvement) posted (#1) for review on release-3.7 by Anuradha Talur (atalur)

Comment 4 Vijay Bellur 2015-12-02 12:53:19 UTC
REVIEW: http://review.gluster.org/12852 (features/index : Readdirp performance improvement) posted (#2) for review on release-3.7 by Anuradha Talur (atalur)

Comment 5 Vijay Bellur 2015-12-02 12:56:35 UTC
REVIEW: http://review.gluster.org/12853 (cluster/afr : Readdirp performance enhancement) posted (#1) for review on release-3.7 by Anuradha Talur (atalur)

Comment 6 Vijay Bellur 2015-12-02 13:00:25 UTC
REVIEW: http://review.gluster.org/12854 (heal : Changed heal info to process all indices directories) posted (#1) for review on release-3.7 by Anuradha Talur (atalur)

Comment 7 Vijay Bellur 2015-12-15 10:01:12 UTC
REVIEW: http://review.gluster.org/12971 (features/index : Prevent logging due to NULL dict) posted (#2) for review on release-3.7 by Anuradha Talur (atalur)

Comment 8 Vijay Bellur 2015-12-21 09:14:27 UTC
COMMIT: http://review.gluster.org/12848 committed in release-3.7 by Pranith Kumar Karampuri (pkarampu) 
------
commit 63b5f373c7bd7d78d49b06c4c03e43941ffbfbd9
Author: Anuradha Talur <atalur>
Date:   Wed Oct 21 16:49:49 2015 +0530

    glusterd/afr : Readdirp performance improvement
    
            Backport of http://review.gluster.org/#/c/12467/
    
    Add xlator options to index xlator with xattrs that
    it needs to keep track of.
    
    Change-Id: If818673be5e626f77e65cc3a340f8cdd624179c2
    BUG: 1287531
    Signed-off-by: Anuradha Talur <atalur>
    Reviewed-on: http://review.gluster.org/12467
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Krutika Dhananjay <kdhananj>
    Tested-by: NetBSD Build System <jenkins.org>
    Reviewed-by: Ravishankar N <ravishankar>
    Reviewed-by: Pranith Kumar Karampuri <pkarampu>
    Signed-off-by: Anuradha Talur <atalur>
    Reviewed-on: http://review.gluster.org/12848

Comment 9 Vijay Bellur 2015-12-21 09:16:24 UTC
COMMIT: http://review.gluster.org/12852 committed in release-3.7 by Pranith Kumar Karampuri (pkarampu) 
------
commit 95a576dec754af6609c5356e07003c54058eb675
Author: Anuradha Talur <atalur>
Date:   Thu Nov 12 19:41:33 2015 +0530

    features/index : Readdirp performance improvement
    
           Backport of: http://review.gluster.org/#/c/12468
    
    Things done :
    Index now maintains two different directories :
    1) for pending xattrs (trusted.afr.volname-client-n,
                           trusted.ec.dirty)
    
    2) for on going xattrs (trusted.afr.dirty)
    
    Based on the xattr, index decides the directory
    to add/del entry in/from.
    
    Change-Id: Ie04c02ea2d862cf80426a871a9a1e80b0773d9fd
    BUG: 1287531
    Signed-off-by: Anuradha Talur <atalur>
    Reviewed-on: http://review.gluster.org/12468
    Tested-by: NetBSD Build System <jenkins.org>
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Pranith Kumar Karampuri <pkarampu>
    Signed-off-by: Anuradha Talur <atalur>
    Reviewed-on: http://review.gluster.org/12852
    Reviewed-by: Krutika Dhananjay <kdhananj>

Comment 10 Vijay Bellur 2015-12-21 09:18:49 UTC
COMMIT: http://review.gluster.org/12853 committed in release-3.7 by Pranith Kumar Karampuri (pkarampu) 
------
commit cf1667cf2fdc9306ae23db1c171b6bdb225153d7
Author: Anuradha Talur <atalur>
Date:   Thu Nov 12 19:45:10 2015 +0530

    cluster/afr : Readdirp performance enhancement
    
            Backport of: http://review.gluster.org/#/c/12507/
    
    Things done :
    1) during lookup and inode_refresh as part of read_txn,
    request is sent to detect if heal is required or not.
    
    2) If heal is required, be conservative in setting the
    readdirp entry inodes to NULL, otherwise don't be.
    
    3) Self-heal-daemon now crawls both indices/xattrop
    and indices/dirty directory while healing.
    
    Change-Id: Ic4a4da63fb7e0726eab5f341a200859b29cf7eb7
    BUG: 1287531
    Signed-off-by: Anuradha Talur <atalur>
    Reviewed-on: http://review.gluster.org/12507
    Tested-by: Gluster Build System <jenkins.com>
    Tested-by: NetBSD Build System <jenkins.org>
    Reviewed-by: Pranith Kumar Karampuri <pkarampu>
    Signed-off-by: Anuradha Talur <atalur>
    Reviewed-on: http://review.gluster.org/12853

Comment 11 Vijay Bellur 2015-12-21 09:19:09 UTC
COMMIT: http://review.gluster.org/12854 committed in release-3.7 by Pranith Kumar Karampuri (pkarampu) 
------
commit fbcd691b593cba29ba9c54d3da2c685787f719be
Author: Anuradha Talur <atalur>
Date:   Mon Nov 23 15:56:28 2015 +0530

    heal : Changed heal info to process all indices directories
    
            Backport of http://review.gluster.org/#/c/12658/
    
    Change-Id: Ida863844e14309b6526c1b8434273fbf05c410d2
    BUG: 1287531
    Signed-off-by: Anuradha Talur <atalur>
    Reviewed-on: http://review.gluster.org/12854
    Tested-by: NetBSD Build System <jenkins.org>
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Pranith Kumar Karampuri <pkarampu>

Comment 12 Vijay Bellur 2015-12-21 10:35:15 UTC
REVIEW: http://review.gluster.org/13037 (heal : Do not print heal count on ENOTCONN) posted (#1) for review on release-3.7 by Anuradha Talur (atalur)

Comment 13 Vijay Bellur 2015-12-22 06:56:13 UTC
COMMIT: http://review.gluster.org/13037 committed in release-3.7 by Pranith Kumar Karampuri (pkarampu) 
------
commit 21368d565d94f4dea2ff119c86bf93cd90514848
Author: Anuradha Talur <atalur>
Date:   Wed Dec 9 14:50:41 2015 +0530

    heal : Do not print heal count on ENOTCONN
    
            Backport of http://review.gluster.org/12919
    
    When a brick is not reachable due to ENOTCONN,
    there are no entries gathered and this information
    should not be printed.
    It is a bug intorduced due to commit
    9c378026e9561595586a817fee0b439e2c863a22, fixing it.
    
            >Change-Id: I45559a9560c297854ea6b4177f86e0be30dc6b78
            >BUG: 1250803
            >Signed-off-by: Anuradha Talur <atalur>
            >Reviewed-on: http://review.gluster.org/12919
            >Tested-by: NetBSD Build System <jenkins.org>
            >Tested-by: Gluster Build System <jenkins.com>
            >Reviewed-by: Pranith Kumar Karampuri <pkarampu>
    
    Change-Id: I5b217bc246a863d75b9d993886a6abada768745e
    BUG: 1287531
    Signed-off-by: Anuradha Talur <atalur>
    Reviewed-on: http://review.gluster.org/13037
    Reviewed-by: Pranith Kumar Karampuri <pkarampu>
    Tested-by: Gluster Build System <jenkins.com>
    Tested-by: NetBSD Build System <jenkins.org>

Comment 14 Ben England 2016-01-04 15:06:25 UTC
cc'ing others in perf team, thanks for fixing this, look forward to seeing results.

Comment 15 Kaushal 2016-04-19 07:49:44 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.7, please open a new bug report.

glusterfs-3.7.7 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] https://www.gluster.org/pipermail/gluster-users/2016-February/025292.html
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user