Red Hat Bugzilla – Bug 1276203
add-brick on a replicate volume could lead to data-loss
Last modified: 2016-09-19 22:00:51 EDT
Description of problem: On increasing the replica count of a replicate volume (by add-brick command), self-heal upon failure of a fop on old-brick and success on newly added brick could lead to reverse heal and hence data loss. Pending xattrs should be marked indicating the new brick doesn't have the latest copy of data yet. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
REVIEW: http://review.gluster.org/12451 (glusterd / afr : Enable auto heal when replica count increases) posted (#1) for review on master by Anuradha Talur (atalur@redhat.com)
REVIEW: http://review.gluster.org/12451 (glusterd / afr : Enable auto heal when replica count increases) posted (#2) for review on master by Anuradha Talur (atalur@redhat.com)
REVIEW: http://review.gluster.org/12451 (glusterd / afr : Enable auto heal when replica count increases) posted (#3) for review on master by Anuradha Talur (atalur@redhat.com)
REVIEW: http://review.gluster.org/12454 (afr : Enable auto heal when replica count increases) posted (#3) for review on master by Anuradha Talur (atalur@redhat.com)
REVIEW: http://review.gluster.org/12454 (afr : Enable auto heal when replica count increases) posted (#4) for review on master by Anuradha Talur (atalur@redhat.com)
REVIEW: http://review.gluster.org/12451 (glusterd / afr : Enable auto heal when replica count increases) posted (#4) for review on master by Anuradha Talur (atalur@redhat.com)
REVIEW: http://review.gluster.org/12454 (afr : Enable auto heal when replica count increases) posted (#5) for review on master by Anuradha Talur (atalur@redhat.com)
REVIEW: http://review.gluster.org/12451 (glusterd / afr : Enable auto heal when replica count increases) posted (#5) for review on master by Anuradha Talur (atalur@redhat.com)
REVIEW: http://review.gluster.org/12454 (afr : Enable auto heal when replica count increases) posted (#6) for review on master by Anuradha Talur (atalur@redhat.com)
REVIEW: http://review.gluster.org/12451 (glusterd / afr : Enable auto heal when replica count increases) posted (#6) for review on master by Anuradha Talur (atalur@redhat.com)
REVIEW: http://review.gluster.org/12451 (glusterd / afr : Enable auto heal when replica count increases) posted (#7) for review on master by Anuradha Talur (atalur@redhat.com)
REVIEW: http://review.gluster.org/12454 (afr : Enable auto heal when replica count increases) posted (#7) for review on master by Anuradha Talur (atalur@redhat.com)
REVIEW: http://review.gluster.org/12454 (afr : Enable auto heal when replica count increases) posted (#8) for review on master by Anuradha Talur (atalur@redhat.com)
REVIEW: http://review.gluster.org/12451 (glusterd / afr : Enable auto heal when replica count increases) posted (#8) for review on master by Anuradha Talur (atalur@redhat.com)
REVIEW: http://review.gluster.org/12454 (afr : Enable auto heal when replica count increases) posted (#9) for review on master by Anuradha Talur (atalur@redhat.com)
REVIEW: http://review.gluster.org/12451 (glusterd / afr : Enable auto heal when replica count increases) posted (#9) for review on master by Anuradha Talur (atalur@redhat.com)
REVIEW: http://review.gluster.org/12454 (afr : Enable auto heal when replica count increases) posted (#10) for review on master by Anuradha Talur (atalur@redhat.com)
COMMIT: http://review.gluster.org/12451 committed in master by Atin Mukherjee (amukherj@redhat.com) ------ commit 020bc022c342c4c015e29c63399757e36d653a49 Author: Anuradha Talur <atalur@redhat.com> Date: Wed Mar 16 10:55:09 2016 +0530 glusterd / afr : Enable auto heal when replica count increases In replicate volumes, when a brick is added to a replicate group, heal to the new brick should be triggered. Also, the new brick should not be considered as source for healing till it is up to date. Previously, extended attributes had to be set manually on the bricks for this to happen. This patch is part 1 patch to automate this process. Change-Id: I29958448618372bfde23bf1dac5dd23dba1ad98f BUG: 1276203 Signed-off-by: Anuradha Talur <atalur@redhat.com> Reviewed-on: http://review.gluster.org/12451 Reviewed-by: Atin Mukherjee <amukherj@redhat.com> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Ravishankar N <ravishankar@redhat.com> Smoke: Gluster Build System <jenkins@build.gluster.com>
COMMIT: http://review.gluster.org/12454 committed in master by Pranith Kumar Karampuri (pkarampu@redhat.com) ------ commit 8eaa3506ead4f11b81b146a9e56575c79f3aad7b Author: Anuradha Talur <atalur@redhat.com> Date: Tue Feb 23 10:56:51 2016 +0530 afr : Enable auto heal when replica count increases This patch is part two change to prevent data loss in a replicate volume on doing a add-brick operation. Problem: After doing add-brick, there is a chance that self heal might happen from the newly added brick rather than the source brick, leading to data loss. Solution: Mark pending changelogs on afr children for the new afr-child so that heal is performed in the correct direction. Change-Id: I11871e55eef3593aec874f92214a2d97da229b17 BUG: 1276203 Signed-off-by: Anuradha Talur <atalur@redhat.com> Reviewed-on: http://review.gluster.org/12454 Smoke: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Pranith Kumar Karampuri <pkarampu@redhat.com> Tested-by: Pranith Kumar Karampuri <pkarampu@redhat.com> CentOS-regression: Gluster Build System <jenkins@build.gluster.com> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.0, please open a new bug report. glusterfs-3.8.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://blog.gluster.org/2016/06/glusterfs-3-8-released/ [2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user