Bug 1320020 - add-brick on a replicate volume could lead to data-loss
Summary: add-brick on a replicate volume could lead to data-loss
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: replicate
Version: 3.7.9
Hardware: Unspecified
OS: Unspecified
high
unspecified
Target Milestone: ---
Assignee: Anuradha
QA Contact:
URL:
Whiteboard:
Depends On: 1276203
Blocks: 1248998 Gluster-HC-1 1365455 1366440 1366444
TreeView+ depends on / blocked
 
Reported: 2016-03-22 07:00 UTC by Anuradha
Modified: 2016-09-20 02:00 UTC (History)
5 users (show)

Fixed In Version: glusterfs-3.7.10
Clone Of: 1276203
Environment:
Last Closed: 2016-04-19 07:00:16 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:

Attachments (Terms of Use)

Description Anuradha 2016-03-22 07:00:31 UTC 
+++ This bug was initially created as a clone of Bug #1276203 +++

Description of problem:
On increasing the replica count of a replicate volume (by add-brick command),
self-heal upon failure of a fop on old-brick and success on newly added brick could lead to reverse heal and hence data loss.

Pending xattrs should be marked indicating the new brick doesn't have the latest copy of data yet.
Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

--- Additional comment from Vijay Bellur on 2015-10-29 01:18:04 EDT ---

REVIEW: http://review.gluster.org/12451 (glusterd / afr : Enable auto heal when replica count increases) posted (#1) for review on master by Anuradha Talur (atalur)

--- Additional comment from Vijay Bellur on 2015-10-29 03:40:55 EDT ---

REVIEW: http://review.gluster.org/12451 (glusterd / afr : Enable auto heal when replica count increases) posted (#2) for review on master by Anuradha Talur (atalur)

--- Additional comment from Vijay Bellur on 2016-01-13 01:18:27 EST ---

REVIEW: http://review.gluster.org/12451 (glusterd / afr : Enable auto heal when replica count increases) posted (#3) for review on master by Anuradha Talur (atalur)

--- Additional comment from Vijay Bellur on 2016-01-13 01:18:30 EST ---

REVIEW: http://review.gluster.org/12454 (afr :  Enable auto heal when replica count increases) posted (#3) for review on master by Anuradha Talur (atalur)

--- Additional comment from Vijay Bellur on 2016-02-03 02:35:54 EST ---

REVIEW: http://review.gluster.org/12454 (afr :  Enable auto heal when replica count increases) posted (#4) for review on master by Anuradha Talur (atalur)

--- Additional comment from Vijay Bellur on 2016-02-23 00:29:21 EST ---

REVIEW: http://review.gluster.org/12451 (glusterd / afr : Enable auto heal when replica count increases) posted (#4) for review on master by Anuradha Talur (atalur)

--- Additional comment from Vijay Bellur on 2016-02-23 00:29:24 EST ---

REVIEW: http://review.gluster.org/12454 (afr : Enable auto heal when replica count increases) posted (#5) for review on master by Anuradha Talur (atalur)

--- Additional comment from Vijay Bellur on 2016-02-23 01:43:38 EST ---

REVIEW: http://review.gluster.org/12451 (glusterd / afr : Enable auto heal when replica count increases) posted (#5) for review on master by Anuradha Talur (atalur)

--- Additional comment from Vijay Bellur on 2016-02-23 01:43:41 EST ---

REVIEW: http://review.gluster.org/12454 (afr : Enable auto heal when replica count increases) posted (#6) for review on master by Anuradha Talur (atalur)

--- Additional comment from Vijay Bellur on 2016-02-29 00:14:15 EST ---

REVIEW: http://review.gluster.org/12451 (glusterd / afr : Enable auto heal when replica count increases) posted (#6) for review on master by Anuradha Talur (atalur)

--- Additional comment from Vijay Bellur on 2016-03-02 02:32:46 EST ---

REVIEW: http://review.gluster.org/12451 (glusterd / afr : Enable auto heal when replica count increases) posted (#7) for review on master by Anuradha Talur (atalur)

--- Additional comment from Vijay Bellur on 2016-03-03 03:55:01 EST ---

REVIEW: http://review.gluster.org/12454 (afr : Enable auto heal when replica count increases) posted (#7) for review on master by Anuradha Talur (atalur)

--- Additional comment from Vijay Bellur on 2016-03-03 07:27:31 EST ---

REVIEW: http://review.gluster.org/12454 (afr : Enable auto heal when replica count increases) posted (#8) for review on master by Anuradha Talur (atalur)

--- Additional comment from Vijay Bellur on 2016-03-14 04:25:47 EDT ---

REVIEW: http://review.gluster.org/12451 (glusterd / afr : Enable auto heal when replica count increases) posted (#8) for review on master by Anuradha Talur (atalur)

--- Additional comment from Vijay Bellur on 2016-03-14 04:25:50 EDT ---

REVIEW: http://review.gluster.org/12454 (afr : Enable auto heal when replica count increases) posted (#9) for review on master by Anuradha Talur (atalur)

--- Additional comment from Vijay Bellur on 2016-03-16 01:25:46 EDT ---

REVIEW: http://review.gluster.org/12451 (glusterd / afr : Enable auto heal when replica count increases) posted (#9) for review on master by Anuradha Talur (atalur)

--- Additional comment from Vijay Bellur on 2016-03-17 02:18:59 EDT ---

REVIEW: http://review.gluster.org/12454 (afr : Enable auto heal when replica count increases) posted (#10) for review on master by Anuradha Talur (atalur)

--- Additional comment from Vijay Bellur on 2016-03-21 13:51:11 EDT ---

COMMIT: http://review.gluster.org/12451 committed in master by Atin Mukherjee (amukherj) 
------
commit 020bc022c342c4c015e29c63399757e36d653a49
Author: Anuradha Talur <atalur>
Date:   Wed Mar 16 10:55:09 2016 +0530

    glusterd / afr : Enable auto heal when replica count increases
    
    In replicate volumes, when a brick is added to a replicate
    group, heal to the new brick should be triggered.
    Also, the new brick should not be considered as source for
    healing till it is up to date.
    
    Previously, extended attributes had to be set manually on
    the bricks for this to happen. This patch is part 1 patch
    to automate this process.
    
    Change-Id: I29958448618372bfde23bf1dac5dd23dba1ad98f
    BUG: 1276203
    Signed-off-by: Anuradha Talur <atalur>
    Reviewed-on: http://review.gluster.org/12451
    Reviewed-by: Atin Mukherjee <amukherj>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.com>
    Reviewed-by: Ravishankar N <ravishankar>
    Smoke: Gluster Build System <jenkins.com>

--- Additional comment from Vijay Bellur on 2016-03-22 01:37:28 EDT ---

COMMIT: http://review.gluster.org/12454 committed in master by Pranith Kumar Karampuri (pkarampu) 
------
commit 8eaa3506ead4f11b81b146a9e56575c79f3aad7b
Author: Anuradha Talur <atalur>
Date:   Tue Feb 23 10:56:51 2016 +0530

    afr : Enable auto heal when replica count increases
    
    This patch is part two change to prevent data loss
    in a replicate volume on doing a add-brick operation.
    
    Problem: After doing add-brick, there is a chance
    that self heal might happen from the newly added
    brick rather than the source brick, leading to data loss.
    
    Solution: Mark pending changelogs on afr children for
    the new afr-child so that heal is performed in the
    correct direction.
    
    Change-Id: I11871e55eef3593aec874f92214a2d97da229b17
    BUG: 1276203
    Signed-off-by: Anuradha Talur <atalur>
    Reviewed-on: http://review.gluster.org/12454
    Smoke: Gluster Build System <jenkins.com>
    Reviewed-by: Pranith Kumar Karampuri <pkarampu>
    Tested-by: Pranith Kumar Karampuri <pkarampu>
    CentOS-regression: Gluster Build System <jenkins.com>
    NetBSD-regression: NetBSD Build System <jenkins.org>
Comment 1 Vijay Bellur 2016-03-22 08:43:22 UTC 
REVIEW: http://review.gluster.org/13806 (glusterd / afr : Enable auto heal when replica count increases) posted (#1) for review on release-3.7 by Anuradha Talur (atalur)
Comment 2 Vijay Bellur 2016-03-22 08:43:24 UTC 
REVIEW: http://review.gluster.org/13807 (afr : Enable auto heal when replica count increases) posted (#1) for review on release-3.7 by Anuradha Talur (atalur)
Comment 3 Vijay Bellur 2016-03-22 08:44:04 UTC 
REVIEW: http://review.gluster.org/13806 (glusterd / afr : Enable auto heal when replica count increases) posted (#2) for review on release-3.7 by Anuradha Talur (atalur)
Comment 4 Vijay Bellur 2016-03-23 10:21:50 UTC 
COMMIT: http://review.gluster.org/13806 committed in release-3.7 by Atin Mukherjee (amukherj) 
------
commit f5f43424994859f263165f0cdebedffd5f08fdcc
Author: Anuradha Talur <atalur>
Date:   Wed Mar 16 10:55:09 2016 +0530

    glusterd / afr : Enable auto heal when replica count increases
    
            Backport of http://review.gluster.org/12451
    
    In replicate volumes, when a brick is added to a replicate
    group, heal to the new brick should be triggered.
    Also, the new brick should not be considered as source for
    healing till it is up to date.
    
    Previously, extended attributes had to be set manually on
    the bricks for this to happen. This patch is part 1 patch
    to automate this process.
    
            >Change-Id: I29958448618372bfde23bf1dac5dd23dba1ad98f
            >BUG: 1276203
            >Signed-off-by: Anuradha Talur <atalur>
            >Reviewed-on: http://review.gluster.org/12451
            >Reviewed-by: Atin Mukherjee <amukherj>
            >NetBSD-regression: NetBSD Build System <jenkins.org>
            >CentOS-regression: Gluster Build System <jenkins.com>
            >Reviewed-by: Ravishankar N <ravishankar>
            >Smoke: Gluster Build System <jenkins.com>
    Signed-off-by: Anuradha Talur <atalur>
    
    Conflicts:
    	libglusterfs/src/globals.h
    	xlators/mgmt/glusterd/src/glusterd-replace-brick.c
    
    Change-Id: Ica83592aab8edbe49e2bb9d8d4824cf5c76324b7
    BUG: 1320020
    Reviewed-on: http://review.gluster.org/13806
    Smoke: Gluster Build System <jenkins.com>
    Tested-by: Anuradha Talur <atalur>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.com>
    Reviewed-by: Atin Mukherjee <amukherj>
Comment 5 Vijay Bellur 2016-03-23 11:49:47 UTC 
COMMIT: http://review.gluster.org/13807 committed in release-3.7 by Raghavendra G (rgowdapp) 
------
commit cf91abba2e6bf22794b04d143f226906e44bf5d5
Author: Anuradha Talur <atalur>
Date:   Tue Mar 22 14:03:38 2016 +0530

    afr : Enable auto heal when replica count increases
    
                Backport of: http://review.gluster.org/12454
    
        This patch is part two change to prevent data loss
        in a replicate volume on doing a add-brick operation.
    
        Problem: After doing add-brick, there is a chance that
        self heal might happen from the newly added brick rather
        than the source brick, leading to data loss.
    
        Solution: Mark pending changelogs on afr children for
        the new afr-child so that heal is performed in the
        correct direction.
    
                >Change-Id: I11871e55eef3593aec874f92214a2d97da229b17
                >BUG: 1276203
                >Signed-off-by: Anuradha Talur <atalur>
                >Reviewed-on: http://review.gluster.org/12454
                >Smoke: Gluster Build System <jenkins.com>
                >Reviewed-by: Pranith Kumar Karampuri <pkarampu>
                >Tested-by: Pranith Kumar Karampuri <pkarampu>
                >CentOS-regression: Gluster Build System <jenkins.com>
                >NetBSD-regression: NetBSD Build System <jenkins.org>
    
    Change-Id: Iae6af44f97e612cb3ee8c642254ec3d15ac063f5
    BUG: 1320020
    Signed-off-by: Anuradha Talur <atalur>
    Reviewed-on: http://review.gluster.org/13807
    Smoke: Gluster Build System <jenkins.com>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.com>
    Reviewed-by: Raghavendra G <rgowdapp>
Comment 6 Kaushal 2016-04-19 07:00:16 UTC 
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.10, please open a new bug report.

glusterfs-3.7.10 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] https://www.gluster.org/pipermail/gluster-users/2016-April/026164.html
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user
Note You need to log in before you can comment on or make changes to this bug.