Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1136769

Summary:	AFR: Provide a gluster CLI for automated resolution of split-brains.
Product:	[Community] GlusterFS	Reporter:	Ravishankar N <ravishankar>
Component:	replicate	Assignee:	Ravishankar N <ravishankar>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	mainline	CC:	bugs, gluster-bugs
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	glusterfs-3.7.0	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:
Clones:	1163588 (view as bug list)		Environment:
Last Closed:	2015-05-14 17:27:29 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1163588

Description Ravishankar N 2014-09-03 09:30:39 UTC

Extend the gluster heal command to provide a new CLI to automatically resolve split-brain based on various policies.

gluster-devel thread http://supercolony.gluster.org/pipermail/gluster-devel/2014-August/041868.html

Comment 1 Anand Avati 2014-12-29 07:41:30 UTC

REVIEW: http://review.gluster.org/9357 (cluster/afr: split-brain resolution CLI) posted (#1) for review on master by Ravishankar N (ravishankar)

Comment 2 Anand Avati 2015-01-02 06:51:54 UTC

REVIEW: http://review.gluster.org/9375 (libglusterfs: change signature of syncop_(f)getxattr) posted (#1) for review on master by Ravishankar N (ravishankar)

Comment 3 Anand Avati 2015-01-02 06:51:58 UTC

REVIEW: http://review.gluster.org/9376 (api: export glfs_resolve()) posted (#1) for review on master by Ravishankar N (ravishankar)

Comment 4 Anand Avati 2015-01-02 06:52:01 UTC

REVIEW: http://review.gluster.org/9377 (cluster/afr: split-brain resolution CLI) posted (#1) for review on master by Ravishankar N (ravishankar)

Comment 5 Anand Avati 2015-01-04 06:54:58 UTC

REVIEW: http://review.gluster.org/9377 (cluster/afr: split-brain resolution CLI) posted (#2) for review on master by Ravishankar N (ravishankar)

Comment 6 Anand Avati 2015-01-04 06:55:06 UTC

REVIEW: http://review.gluster.org/9376 (api: export glfs_resolve()) posted (#2) for review on master by Ravishankar N (ravishankar)

Comment 7 Anand Avati 2015-01-04 06:55:09 UTC

REVIEW: http://review.gluster.org/9375 (libglusterfs: change signature of syncop_(f)getxattr) posted (#2) for review on master by Ravishankar N (ravishankar)

Comment 8 Anand Avati 2015-01-06 06:00:14 UTC

COMMIT: http://review.gluster.org/9375 committed in master by Vijay Bellur (vbellur) 
------
commit c4ab37c02e9edc23d0637e23d6f2b42d0827dad2
Author: Ravishankar N <ravishankar>
Date:   Thu Jan 1 15:16:41 2015 +0000

    libglusterfs: change signature of syncop_(f)getxattr
    
    Pass xdata dict to syncop_(f)getxattr calls.
    
    This patch [1/3] is required as a part of afr automated split-brain resolution
    implementation.
    
    Change-Id: I3970b3dd6daf64681a031e37f8e9afb14fb3d668
    BUG: 1136769
    Signed-off-by: Ravishankar N <ravishankar>
    Reviewed-on: http://review.gluster.org/9375
    Reviewed-by: Pranith Kumar Karampuri <pkarampu>
    Reviewed-by: Niels de Vos <ndevos>
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Shyamsundar Ranganathan <srangana>
    Reviewed-by: Vijay Bellur <vbellur>

Comment 9 Anand Avati 2015-01-06 16:19:04 UTC

REVIEW: http://review.gluster.org/9376 (api: export glfs_resolve()) posted (#3) for review on master by Shyamsundar Ranganathan (srangana)

Comment 10 Anand Avati 2015-01-06 17:14:21 UTC

COMMIT: http://review.gluster.org/9376 committed in master by Shyamsundar Ranganathan (srangana) 
------
commit cd338784d39b9a4a55cc3c6a1613b90a2a63a8e3
Author: Ravishankar N <ravishankar>
Date:   Thu Jan 1 15:55:22 2015 +0000

    api: export glfs_resolve()
    
    glfs_resolve() is an internal function in libgfapi. Export it so that it can be
    used by glfs-heal.c. The change is done in line with the approach followed in
    commit 7e497871d11a3a527e2ce192e4274322631f27d0 (http://review.gluster.org/9036)
    
    This patch [2/3] is required as a part of afr automated split-brain resolution
    implementation.
    
    Change-Id: If9057f2a037fd25a0dfa231683bfbaa72d3cb1b2
    BUG: 1136769
    Signed-off-by: Ravishankar N <ravishankar>
    Reviewed-on: http://review.gluster.org/9376
    Reviewed-by: Niels de Vos <ndevos>
    Reviewed-by: Shyamsundar Ranganathan <srangana>
    Tested-by: Shyamsundar Ranganathan <srangana>

Comment 11 Anand Avati 2015-01-08 12:15:39 UTC

REVIEW: http://review.gluster.org/9377 (cluster/afr: split-brain resolution CLI) posted (#3) for review on master by Ravishankar N (ravishankar)

Comment 12 Anand Avati 2015-01-09 07:08:46 UTC

REVIEW: http://review.gluster.org/9377 (cluster/afr: split-brain resolution CLI) posted (#4) for review on master by Ravishankar N (ravishankar)

Comment 13 Anand Avati 2015-01-09 11:51:47 UTC

REVIEW: http://review.gluster.org/9377 (cluster/afr: split-brain resolution CLI) posted (#5) for review on master by Ravishankar N (ravishankar)

Comment 14 Anand Avati 2015-01-13 07:26:00 UTC

REVIEW: http://review.gluster.org/9377 (cluster/afr: split-brain resolution CLI) posted (#6) for review on master by Ravishankar N (ravishankar)

Comment 15 Anand Avati 2015-01-15 09:28:42 UTC

COMMIT: http://review.gluster.org/9377 committed in master by Pranith Kumar Karampuri (pkarampu) 
------
commit 8beaf169e39b262416e2274a028292379d39b310
Author: Ravishankar N <ravishankar>
Date:   Fri Jan 9 14:43:22 2015 +0000

    cluster/afr: split-brain resolution CLI
    
    Extend the AFR heal command to include automated split-brain resolution.
    
    This patch [3/3] is the final patch for afr automated split-brain resolution
    implementation.
    
    "gluster volume heal <VOLNAME> [full | statistics [heal-count [replica
    <HOSTNAME:BRICKNAME>]] |info [healed | heal-failed | split-brain]| split-brain
    {bigger-file <FILE> |source-brick <HOSTNAME:BRICKNAME> [<FILE>]}]"
    
    The new additions being:
    1.gluster volume heal <VOLNAME> split-brain bigger-file <FILE>
    Locates the replica containing the FILE, selects bigger-file as source and
    completes heal.
    
    2.gluster volume heal <VOLNAME> split-brain source-brick <HOSTNAME:BRICKNAME>
    <FILE>
    Selects <FILE> present in <HOSTNAME:BRICKNAME> as source and completes heal.
    
    3.gluster volume heal <VOLNAME> split-brain <HOSTNAME:BRICKNAME>
    Selects all split-brained files in <HOSTNAME:BRICKNAME> as source and completes
    heal.
    
    Note: <FILE> can be either the full file name as seen from the root of the
    volume (or) the gfid-string representation of the file, which sometimes gets
    displayed in the heal info command's output.
    
    Entry/gfid split-brain resolution is not supported.
    
    Example can be found in the test case.
    
    Change-Id: I4649733922d406f14f28ee9033a5cb627b9538b3
    BUG: 1136769
    Signed-off-by: Ravishankar N <ravishankar>
    Reviewed-on: http://review.gluster.org/9377
    Reviewed-by: Pranith Kumar Karampuri <pkarampu>
    Tested-by: Pranith Kumar Karampuri <pkarampu>
    Tested-by: Gluster Build System <jenkins.com>

Comment 16 Niels de Vos 2015-05-14 17:27:29 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 17 Niels de Vos 2015-05-14 17:35:34 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 18 Niels de Vos 2015-05-14 17:37:56 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 19 Niels de Vos 2015-05-14 17:43:41 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user