Bug 1465854

Summary: Regression: Heal info takes longer time when a brick is down
Product: [Community] GlusterFS Reporter: Ashish Pandey <aspandey>
Component: disperseAssignee: Ashish Pandey <aspandey>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.11CC: amukherj, bugs, nchilaka, pkarampu, rhs-bugs, sheggodu, storage-qa-internal
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-3.11.2 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1464091 Environment:
Last Closed: 2017-08-12 13:07:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1463108, 1464091    
Bug Blocks:    

Comment 1 Ashish Pandey 2017-06-28 11:30:37 UTC
Description of problem:
=========================
(A possible regression)
When a brick is down the heal info of a disperse volume takes significantly longer time.
for eg: without any heal pending or on a fresh setup, with all bricks up, the heal info response time for  a 1x(4+2) volume is hardly 1 sec
where as with one brick down the same setup takes 11 Sec

and in one case where heal pending entries were there the heal info was hung(the re were hardly 10 entries on each brick)
If I hit the hang again, will raise a seperate bug

Version-Release number of selected component (if applicable):
===
3.8.4-28

How reproducible:
==
always

Steps to Reproduce:
1.create a 4+2 ecvolume and check heal info time (it will be hardly 1 sec)
2.kill one brick
3.check heal info time(takes about 11 sec)

Comment 2 Worker Ant 2017-06-28 11:32:58 UTC
REVIEW: https://review.gluster.org/17642 (ec: Increase notification in all the cases) posted (#1) for review on release-3.11 by Ashish Pandey (aspandey)

Comment 3 Worker Ant 2017-07-03 12:43:53 UTC
COMMIT: https://review.gluster.org/17642 committed in release-3.11 by Shyamsundar Ranganathan (srangana) 
------
commit 1764ba4654bcdac9155e81727e1467b024e4a01b
Author: Ashish Pandey <aspandey>
Date:   Thu Jun 22 17:06:40 2017 +0530

    ec: Increase notification in all the cases
    
    Problem:
    "gluster v heal <volname> info" is taking
    long time to respond when a brick is down.
    
    RCA:
    Heal info command does virtual mount.
    EC wait for 10 seconds, before sending UP call to upper xlator,
    to get notification (DOWN or UP) from all the bricks.
    
    Currently, we are increasing ec->xl_notify_count based on
    the current status of the brick. So, if a DOWN event notification
    has come and brick is already down, we are not increasing
    ec->xl_notify_count in ec_handle_down.
    
    Solution:
    Handle DOWN even as notification irrespective of what
    is the current status of brick.
    
    >Change-Id: I0acac0db7ec7622d4c0584692e88ad52f45a910f
    >BUG: 1464091
    >Signed-off-by: Ashish Pandey <aspandey>
    >Reviewed-on: https://review.gluster.org/17606
    >Tested-by: Pranith Kumar Karampuri <pkarampu>
    >Reviewed-by: Pranith Kumar Karampuri <pkarampu>
    >Smoke: Gluster Build System <jenkins.org>
    >CentOS-regression: Gluster Build System <jenkins.org>
    >Reviewed-by: Xavier Hernandez <xhernandez>
    >NetBSD-regression: NetBSD Build System <jenkins.org>
    >Signed-off-by: Ashish Pandey <aspandey>
    
    Change-Id: I0acac0db7ec7622d4c0584692e88ad52f45a910f
    BUG: 1465854
    Signed-off-by: Ashish Pandey <aspandey>
    Reviewed-on: https://review.gluster.org/17642
    Smoke: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    Reviewed-by: Xavier Hernandez <xhernandez>
    CentOS-regression: Gluster Build System <jenkins.org>

Comment 4 Shyamsundar 2017-08-12 13:07:04 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.11.2, please open a new bug report.

glusterfs-3.11.2 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/gluster-users/2017-July/031908.html
[2] https://www.gluster.org/pipermail/gluster-users/