1207829 – Incomplete self-heal and split-brain on directories found when self-healing files/dirs on a replaced disk

Bug 1207829 - Incomplete self-heal and split-brain on directories found when self-healing files/dirs on a replaced disk

Summary: Incomplete self-heal and split-brain on directories found when self-healing f...

Keywords:
Status:	CLOSED CURRENTRELEASE
Alias:	None
Product:	GlusterFS
Classification:	Community
Component:	replicate
Sub Component:
Version:	mainline
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	high
Target Milestone:	---
Assignee:	Anuradha
QA Contact:
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1140649 1232173 1255611
TreeView+	depends on / blocked

Reported:	2015-03-31 19:28 UTC by Anuradha
Modified:	2016-09-20 02:01 UTC (History)
CC List:	6 users (show)
Fixed In Version:	glusterfs-3.8rc2
Doc Type:	Bug Fix
Doc Text:
Clone Of:	1140649
Clones:	1232173 1255611 (view as bug list)
Environment:
Last Closed:	2016-06-16 12:46:47 UTC
Regression:	---
Mount Type:	---
Documentation:	---
CRM:
Verified Versions:
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Comment 1 Anand Avati 2015-03-31 19:37:45 UTC

REVIEW: http://review.gluster.org/10076 (glusterd : set afr pending xattrs on replace brick) posted (#1) for review on master by Anuradha Talur (atalur)

Comment 2 Anand Avati 2015-04-02 07:38:16 UTC

REVIEW: http://review.gluster.org/10076 (glusterd : set afr pending xattrs on replace brick) posted (#2) for review on master by Anuradha Talur (atalur)

Comment 3 Anand Avati 2015-04-02 08:50:11 UTC

REVIEW: http://review.gluster.org/10076 (glusterd : set afr pending xattrs on replace brick) posted (#3) for review on master by Anuradha Talur (atalur)

Comment 4 Anand Avati 2015-04-02 09:10:29 UTC

REVIEW: http://review.gluster.org/10076 (glusterd : set afr pending xattrs on replace brick) posted (#4) for review on master by Anuradha Talur (atalur)

Comment 5 Anand Avati 2015-04-29 10:32:39 UTC

REVIEW: http://review.gluster.org/10076 (glusterd : set afr pending xattrs on replace brick) posted (#5) for review on master by Anuradha Talur (atalur)

Comment 6 Anand Avati 2015-04-29 10:36:13 UTC

REVIEW: http://review.gluster.org/10076 (glusterd : set afr pending xattrs on replace brick) posted (#6) for review on master by Anuradha Talur (atalur)

Comment 7 Anand Avati 2015-04-29 10:46:16 UTC

REVIEW: http://review.gluster.org/10076 (glusterd : set afr pending xattrs on replace brick) posted (#7) for review on master by Anuradha Talur (atalur)

Comment 8 Anand Avati 2015-04-29 11:27:22 UTC

REVIEW: http://review.gluster.org/10448 (cluster/afr : set pending xattrs for replaced brick) posted (#1) for review on master by Anuradha Talur (atalur)

Comment 9 Anand Avati 2015-05-05 13:40:34 UTC

REVIEW: http://review.gluster.org/10076 (glusterd : set afr pending xattrs on replace brick) posted (#8) for review on master by Anuradha Talur (atalur)

Comment 10 Anand Avati 2015-05-06 11:43:55 UTC

REVIEW: http://review.gluster.org/10076 (glusterd : set afr pending xattrs on replace brick) posted (#9) for review on master by Anuradha Talur (atalur)

Comment 11 Anand Avati 2015-05-07 09:12:04 UTC

REVIEW: http://review.gluster.org/10076 (glusterd : set afr pending xattrs on replace brick) posted (#10) for review on master by Anuradha Talur (atalur)

Comment 12 Anand Avati 2015-05-11 13:11:05 UTC

REVIEW: http://review.gluster.org/10448 (cluster/afr : set pending xattrs for replaced brick) posted (#2) for review on master by Anuradha Talur (atalur)

Comment 13 Anand Avati 2015-05-11 13:14:52 UTC

REVIEW: http://review.gluster.org/10076 (glusterd : set afr pending xattrs on replace brick) posted (#11) for review on master by Anuradha Talur (atalur)

Comment 14 Anand Avati 2015-05-13 11:05:12 UTC

REVIEW: http://review.gluster.org/10076 (glusterd : set afr pending xattrs on replace brick) posted (#12) for review on master by Anuradha Talur (atalur)

Comment 15 Anand Avati 2015-05-13 11:42:45 UTC

REVIEW: http://review.gluster.org/10448 (cluster/afr : set pending xattrs for replaced brick) posted (#3) for review on master by Anuradha Talur (atalur)

Comment 16 Anand Avati 2015-05-18 08:06:47 UTC

REVIEW: http://review.gluster.org/10076 (glusterd : set afr pending xattrs on replace brick) posted (#13) for review on master by Anuradha Talur (atalur)

Comment 17 Anand Avati 2015-05-18 08:38:18 UTC

REVIEW: http://review.gluster.org/10448 (cluster/afr : set pending xattrs for replaced brick) posted (#4) for review on master by Anuradha Talur (atalur)

Comment 18 Anand Avati 2015-05-19 06:08:08 UTC

REVIEW: http://review.gluster.org/10448 (cluster/afr : set pending xattrs for replaced brick) posted (#5) for review on master by Anuradha Talur (atalur)

Comment 19 Anand Avati 2015-05-20 10:44:47 UTC

REVIEW: http://review.gluster.org/10076 (glusterd : set afr pending xattrs on replace brick) posted (#14) for review on master by Anuradha Talur (atalur)

Comment 20 Anand Avati 2015-05-27 07:32:13 UTC

REVIEW: http://review.gluster.org/10076 (glusterd : set afr pending xattrs on replace brick) posted (#15) for review on master by Anuradha Talur (atalur)

Comment 21 Anand Avati 2015-05-27 07:32:16 UTC

REVIEW: http://review.gluster.org/10448 (cluster/afr : set pending xattrs for replaced brick) posted (#6) for review on master by Anuradha Talur (atalur)

Comment 22 Anand Avati 2015-05-27 08:30:04 UTC

REVIEW: http://review.gluster.org/10448 (cluster/afr : set pending xattrs for replaced brick) posted (#7) for review on master by Anuradha Talur (atalur)

Comment 23 Anand Avati 2015-05-28 04:56:57 UTC

REVIEW: http://review.gluster.org/10076 (glusterd : set afr pending xattrs on replace brick) posted (#16) for review on master by Atin Mukherjee (amukherj)

Comment 24 Anand Avati 2015-05-29 12:00:37 UTC

REVIEW: http://review.gluster.org/10076 (glusterd/ afr : set afr pending xattrs on replace brick) posted (#17) for review on master by Anuradha Talur (atalur)

Comment 25 Anand Avati 2015-05-30 10:42:19 UTC

REVIEW: http://review.gluster.org/10076 (glusterd/ afr : set afr pending xattrs on replace brick) posted (#18) for review on master by Niels de Vos (ndevos)

Comment 26 Niels de Vos 2015-05-30 19:06:53 UTC

Dropped this bug from the glusterfs-3.7.1 tracker, this bug has been filed against "mainline". Please clone this bug, and have the new close block "glusterfs-3.7.1" and that new bug should depend on this one (1207829).

Comment 27 Anand Avati 2015-06-01 08:33:11 UTC

REVIEW: http://review.gluster.org/10448 (cluster/afr : set pending xattrs for replaced brick) posted (#8) for review on master by Anuradha Talur (atalur)

Comment 28 Anand Avati 2015-06-01 09:29:43 UTC

REVIEW: http://review.gluster.org/10076 (glusterd/ afr : set afr pending xattrs on replace brick) posted (#19) for review on master by Anuradha Talur (atalur)

Comment 29 Anand Avati 2015-06-02 11:03:20 UTC

REVIEW: http://review.gluster.org/10076 (glusterd/ afr : set afr pending xattrs on replace brick) posted (#20) for review on master by Anuradha Talur (atalur)

Comment 30 Anand Avati 2015-06-11 07:43:40 UTC

REVIEW: http://review.gluster.org/10076 (glusterd/ afr : set afr pending xattrs on replace brick) posted (#22) for review on master by Anuradha Talur (atalur)

Comment 31 Anand Avati 2015-06-11 09:33:55 UTC

REVIEW: http://review.gluster.org/10448 (cluster/afr : set pending xattrs for replaced brick) posted (#10) for review on master by Anuradha Talur (atalur)

Comment 32 Anand Avati 2015-06-11 12:51:10 UTC

REVIEW: http://review.gluster.org/10448 (cluster/afr : set pending xattrs for replaced brick) posted (#11) for review on master by Anuradha Talur (atalur)

Comment 33 Anand Avati 2015-06-15 06:39:53 UTC

REVIEW: http://review.gluster.org/10076 (glusterd/ afr : set afr pending xattrs on replace brick) posted (#23) for review on master by Anuradha Talur (atalur)

Comment 34 Anand Avati 2015-06-24 11:26:41 UTC

REVIEW: http://review.gluster.org/10076 (glusterd/ afr : set afr pending xattrs on replace brick) posted (#24) for review on master by Anuradha Talur (atalur)

Comment 35 Anand Avati 2015-06-24 11:26:43 UTC

REVIEW: http://review.gluster.org/10448 (cluster/afr : set pending xattrs for replaced brick) posted (#12) for review on master by Anuradha Talur (atalur)

Comment 36 Anand Avati 2015-06-25 09:23:55 UTC

REVIEW: http://review.gluster.org/10448 (cluster/afr : set pending xattrs for replaced brick) posted (#13) for review on master by Anuradha Talur (atalur)

Comment 37 Anand Avati 2015-06-25 16:12:30 UTC

REVIEW: http://review.gluster.org/10448 (cluster/afr : set pending xattrs for replaced brick) posted (#14) for review on master by Anuradha Talur (atalur)

Comment 38 Anand Avati 2015-06-26 02:34:37 UTC

COMMIT: http://review.gluster.org/10448 committed in master by Pranith Kumar Karampuri (pkarampu) 
------
commit cd05e036117a27377af8ff4a1cdba09f1de0a8eb
Author: Anuradha <atalur>
Date:   Thu Jun 11 14:58:05 2015 +0530

    cluster/afr : set pending xattrs for replaced brick
    
    This patch is part two change to prevent data loss
    in a replicate volume on doing a replace-brick commit
    force operation.
    
    Problem: After doing replace-brick commit force, there is a
    chance that self heal might happen from the replaced (sink) brick
    rather than the source brick leading to data loss.
    
    Solution: Mark pending changelogs on afr children for
    the replaced afr-child so that heal is performed in the
    correct direction.
    
    Change-Id: Icb9807e49b4c1c4f1dcab115318d9a58ccf95675
    BUG: 1207829
    Signed-off-by: Anuradha Talur <atalur>
    Reviewed-on: http://review.gluster.org/10448
    Reviewed-by: Pranith Kumar Karampuri <pkarampu>
    Tested-by: NetBSD Build System <jenkins.org>
    Reviewed-by: Krutika Dhananjay <kdhananj>

Comment 39 Anand Avati 2015-06-26 03:38:27 UTC

REVIEW: http://review.gluster.org/11413 (libglusterfs: Fix build failure) posted (#1) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 40 Anand Avati 2015-06-26 03:39:20 UTC

COMMIT: http://review.gluster.org/11413 committed in master by Pranith Kumar Karampuri (pkarampu) 
------
commit c9cbddf3d936f2bf1ac46084a74051096a5ac9a3
Author: Pranith Kumar K <pkarampu>
Date:   Fri Jun 26 09:05:26 2015 +0530

    libglusterfs: Fix build failure
    
    Change-Id: Ie589c866a53c0cfc167e31d1d14a11bf58c8dabf
    BUG: 1207829
    Signed-off-by: Pranith Kumar K <pkarampu>
    Reviewed-on: http://review.gluster.org/11413

Comment 41 Anand Avati 2015-06-26 05:56:02 UTC

COMMIT: http://review.gluster.org/10076 committed in master by Pranith Kumar Karampuri (pkarampu) 
------
commit 4f76b8d11a93cfb74667f6b3051186b86c1ec55b
Author: Anuradha <atalur>
Date:   Fri Jun 5 16:46:39 2015 +0530

    glusterd/ afr : set afr pending xattrs on replace brick
    
    This patch is part one change to prevent data loss
    in a replicate volume on doing a replace-brick commit
    force operation.
    
    Problem: After doing replace-brick commit force, there is a
    chance that self heal happens from the replaced (sink) brick
    rather than the source brick leading to data loss.
    
    Solution: During the commit phase of replace brick, after old
    brick is brought down, create a temporary mount and perform
    setfattr operation (on virtual xattr) indicating AFR to mark
    the replaced brick as sink.
    
    As a part of this change replace-brick command is being changed
    to use mgmt_v3 framework rather than op-state-machine framework.
    
    Many thanks to Krishnan Parthasarathi for helping me out on this.
    
    Change-Id: If0d51b5b3cef5b34d5672d46ea12eaa9d35fd894
    BUG: 1207829
    Signed-off-by: Anuradha <atalur>
    Reviewed-on: http://review.gluster.org/10076
    Tested-by: NetBSD Build System <jenkins.org>
    Reviewed-by: Ravishankar N <ravishankar>
    Reviewed-by: Pranith Kumar Karampuri <pkarampu>

Comment 42 Anand Avati 2015-06-26 06:19:41 UTC

REVIEW: http://review.gluster.org/11416 (tests: fix basic/afr/replace-brick-self-heal.t failure) posted (#1) for review on master by Ravishankar N (ravishankar)

Comment 43 Anand Avati 2015-06-26 09:57:43 UTC

COMMIT: http://review.gluster.org/11416 committed in master by Pranith Kumar Karampuri (pkarampu) 
------
commit 264776f7af6dffabc2216cf46fd1885a309f6e62
Author: Ravishankar N <ravishankar>
Date:   Fri Jun 26 11:46:45 2015 +0530

    tests: fix basic/afr/replace-brick-self-heal.t failure
    
    Test fails with:
    not ok 28 Got "Binary file (standard input) matches" instead of "qwerty"
    FAILED COMMAND: qwerty get_text_xattr user.test
    /d/backends/patchy1_new/file5.txt
    not ok 29 Got "Binary file (standard input) matches" instead of "qwerty"
    FAILED COMMAND: qwerty get_text_xattr user.test
    /d/backends/patchy0/file5.txt
    Failed 2/29 subtests
    
    Fix:
    Pass -a flag to grep
    
    Change-Id: I69626fbf95a9ff756046363c5627cf98ea3f1df8
    BUG: 1207829
    Signed-off-by: Ravishankar N <ravishankar>
    Reviewed-on: http://review.gluster.org/11416
    Tested-by: NetBSD Build System <jenkins.org>
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Pranith Kumar Karampuri <pkarampu>

Comment 44 Nagaprasad Sathyanarayana 2015-10-25 15:02:12 UTC

Fix for this BZ is already present in a GlusterFS release. You can find clone of this BZ, fixed in a GlusterFS release and closed. Hence closing this mainline BZ as well.

Comment 45 Niels de Vos 2016-06-16 12:46:47 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.0, please open a new bug report.

glusterfs-3.8.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://blog.gluster.org/2016/06/glusterfs-3-8-released/
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Note You need to log in before you can comment on or make changes to this bug.