Bug 1225318

Summary:	glusterd could crash in remove-brick-status when local remove-brick process has just completed
Product:	[Community] GlusterFS	Reporter:	krishnan parthasarathi <kparthas>
Component:	glusterd	Assignee:	krishnan parthasarathi <kparthas>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	3.7.0	CC:	amukherj, bugs, gluster-bugs, nsathyan
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	glusterfs-3.7.1	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:	1223338
Clones:	1227168 (view as bug list)		Environment:
Last Closed:	2015-06-02 08:03:26 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1223338
Bug Blocks:	1219955, 1223634, 1227168, 1233025

Description krishnan parthasarathi 2015-05-27 06:28:33 UTC

+++ This bug was initially created as a clone of Bug #1223338 +++

Description of problem:
Glusterd process could crash while executing remove-brick-status command around the time when the local remove-brick process (i.e, rebalance process) has completed migrating data. 

Version-Release number of selected component (if applicable):
glusterfs-3.7.0

How reproducible:
Intermittent

Steps to Reproduce:
1. Create and start a volume.
2. Add files/directories as required.
3. Remove one or more bricks using remove-brick-start command.
4. Issue remove-brick-status command around the time when the local remove-brick
   process is completed.

Actual results:
glusterd process crashes.

Expected results:
glusterd shouldn't crash. It would be helpful if the remove-brick-status command failed saying that the rebalance process may have just completed with
migration of data from the bricks being removed.

Additional info:
The above steps are representative of when the issue can be seen but not really helpful if you wish to automate this. The following link leads to the regression test, as part of GlusterFS regression test suite, that has hit this problem more often. This could help those interested in automation.

https://github.com/gluster/glusterfs/blob/master/tests/bugs/glusterd/bug-974007.t

--- Additional comment from Anand Avati on 2015-05-21 03:11:46 EDT ---

REVIEW: http://review.gluster.org/10872 (glusterd: fix double-free of rebalance process' rpc object) posted (#1) for review on master by Krishnan Parthasarathi (kparthas)

--- Additional comment from Anand Avati on 2015-05-21 23:13:13 EDT ---

REVIEW: http://review.gluster.org/10872 (glusterd: fix double-free of rebalance process' rpc object) posted (#2) for review on master by Krishnan Parthasarathi (kparthas)

--- Additional comment from Anand Avati on 2015-05-26 05:58:45 EDT ---

REVIEW: http://review.gluster.org/10872 (glusterd: fix double-free of rebalance process' rpc object) posted (#3) for review on master by Krishnan Parthasarathi (kparthas)

--- Additional comment from Anand Avati on 2015-05-26 09:42:02 EDT ---

COMMIT: http://review.gluster.org/10872 committed in master by Kaushal M (kaushal) 
------
commit f59a1431e38ad644b6bdd4adbdf94ff028c9c60d
Author: Krishnan Parthasarathi <kparthas>
Date:   Wed May 20 18:23:58 2015 +0530

    glusterd: fix double-free of rebalance process' rpc object
    
    Change-Id: I0c79c4de47a160b1ecf3a8994eedc02e3f5002a9
    BUG: 1223338
    Signed-off-by: Krishnan Parthasarathi <kparthas>
    Reviewed-on: http://review.gluster.org/10872
    Tested-by: NetBSD Build System
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Atin Mukherjee <amukherj>
    Reviewed-by: Kaushal M <kaushal>

Comment 1 Anand Avati 2015-05-28 03:51:42 UTC

REVIEW: http://review.gluster.org/10932 (glusterd: fix double-free of rebalance process' rpc object) posted (#3) for review on release-3.7 by Krishnan Parthasarathi (kparthas)

Comment 2 Anand Avati 2015-05-30 06:34:17 UTC

REVIEW: http://review.gluster.org/10932 (glusterd: fix double-free of rebalance process' rpc object) posted (#4) for review on release-3.7 by Vijay Bellur (vbellur)

Comment 3 Anand Avati 2015-05-30 14:05:32 UTC

COMMIT: http://review.gluster.org/10932 committed in release-3.7 by Vijay Bellur (vbellur) 
------
commit 0308c7a2800f52810093af780c42b6c6616c2aed
Author: Krishnan Parthasarathi <kparthas>
Date:   Wed May 20 18:23:58 2015 +0530

    glusterd: fix double-free of rebalance process' rpc object
    
    Change-Id: I0c79c4de47a160b1ecf3a8994eedc02e3f5002a9
    BUG: 1225318
    Signed-off-by: Krishnan Parthasarathi <kparthas>
    Reviewed-on: http://review.gluster.org/10872
    Reviewed-by: Atin Mukherjee <amukherj>
    Reviewed-by: Kaushal M <kaushal>
    Reviewed-on: http://review.gluster.org/10932
    Tested-by: NetBSD Build System
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Vijay Bellur <vbellur>

Comment 4 Niels de Vos 2015-06-02 08:03:26 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.1, please reopen this bug report.

glusterfs-3.7.1 has been announced on the Gluster Packaging mailinglist [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.packaging/1
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 5 Niels de Vos 2015-07-06 14:00:32 UTC

*** Bug 1223634 has been marked as a duplicate of this bug. ***