Bug 1142612 - files with open fd's getting into split-brain when bricks goes offline and comes back online
Summary: files with open fd's getting into split-brain when bricks goes offline and co...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: replicate
Version: 3.6.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
Assignee: Pranith Kumar K
QA Contact:
URL:
Whiteboard: Regression
Depends On: 1131466 1142601
Blocks: 1142614
TreeView+ depends on / blocked
 
Reported: 2014-09-17 06:15 UTC by Pranith Kumar K
Modified: 2014-11-10 15:13 UTC (History)
3 users (show)

Fixed In Version: glusterfs-3.6.1
Clone Of: 1142601
: 1142614 (view as bug list)
Environment:
Last Closed: 2014-11-10 15:13:42 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:

Attachments (Terms of Use)

Comment 1 Anand Avati 2014-09-17 06:16:27 UTC 
REVIEW: http://review.gluster.org/8756 (cluster/afr: Launch self-heal only when all the brick status is known) posted (#1) for review on release-3.6 by Pranith Kumar Karampuri (pkarampu)
Comment 2 Anand Avati 2014-09-29 12:54:03 UTC 
REVIEW: http://review.gluster.org/8756 (cluster/afr: Launch self-heal only when all the brick status is known) posted (#2) for review on release-3.6 by Pranith Kumar Karampuri (pkarampu)
Comment 3 Anand Avati 2014-09-30 06:30:35 UTC 
COMMIT: http://review.gluster.org/8756 committed in release-3.6 by Vijay Bellur (vbellur) 
------
commit 86b4c0319d4275859575720eced3200583942cfb
Author: Pranith Kumar K <pkarampu>
Date:   Wed Sep 17 11:33:23 2014 +0530

    cluster/afr: Launch self-heal only when all the brick status is known
    
    Problem:
    File goes into split-brain because of wrong erasing of xattrs.
    
    RCA:
    The issue happens because index self-heal is triggered even before all the
    bricks are up. So what ends up happening while erasing the xattrs is, xattrs
    are erased only on the sink brick for the brick that it thinks is up leading to
    split-brain
    
    Example:
    lets say the xattrs before heal started are:
    brick 2:
    trusted.afr.vol1-client-2=0x000000020000000000000000
    trusted.afr.vol1-client-3=0x000000020000000000000000
    
    brick 3:
    trusted.afr.vol1-client-2=0x000010040000000000000000
    trusted.afr.vol1-client-3=0x000000000000000000000000
    
    if only brick-2 came up at the time of triggering the self-heal only
    'trusted.afr.vol1-client-2' is erased leading to the following xattrs:
    
    brick 2:
    trusted.afr.vol1-client-2=0x000000000000000000000000
    trusted.afr.vol1-client-3=0x000000020000000000000000
    
    brick 3:
    trusted.afr.vol1-client-2=0x000010040000000000000000
    trusted.afr.vol1-client-3=0x000000000000000000000000
    
    So the file goes into split-brain.
    
    BUG: 1142612
    Change-Id: I0c8b66e154f03b636db052c97745399a7cca265b
    Signed-off-by: Pranith Kumar K <pkarampu>
    Reviewed-on: http://review.gluster.org/8756
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Krutika Dhananjay <kdhananj>
    Reviewed-by: Vijay Bellur <vbellur>
Comment 4 Niels de Vos 2014-11-10 15:13:42 UTC 
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.6.1, please reopen this bug report.

glusterfs-3.6.1 has been announced [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://supercolony.gluster.org/pipermail/gluster-users/2014-November/019410.html
[2] http://supercolony.gluster.org/mailman/listinfo/gluster-users
Note You need to log in before you can comment on or make changes to this bug.