Bug 1465559

Summary:	multiple brick processes seen on gluster(fs)d restart in brick multiplexing
Product:	[Community] GlusterFS	Reporter:	Atin Mukherjee <amukherj>
Component:	glusterd	Assignee:	Atin Mukherjee <amukherj>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	mainline	CC:	bugs
Target Milestone:	---	Keywords:	Triaged
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	glusterfs-3.12.0	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1466608 (view as bug list)		Environment:
Last Closed:	2017-09-05 17:35:13 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1466608, 1473327

Description Atin Mukherjee 2017-06-27 15:45:39 UTC

Description of problem:

On a brick multiplexing test setup if all the gluster processes are killed and glusterd is started there is a race window where one of the brick will fail to attach to one of the running brick instances and while doing so, it will try to bring up a new process resulting in deleting the socket file which has been used by one of the other brick processes resulting into brick disconnect too.

Version-Release number of selected component (if applicable):
mainline

How reproducible:
Often, not always

Comment 1 Worker Ant 2017-06-27 15:54:26 UTC

REVIEW: https://review.gluster.org/17640 (glusterd: mark brickinfo to started on successful attach) posted (#1) for review on master by Atin Mukherjee (amukherj)

Comment 2 Worker Ant 2017-06-27 15:55:46 UTC

REVIEW: https://review.gluster.org/17640 (glusterd: mark brickinfo to started on successful attach) posted (#2) for review on master by Atin Mukherjee (amukherj)

Comment 3 Worker Ant 2017-06-28 06:19:05 UTC

COMMIT: https://review.gluster.org/17640 committed in master by Atin Mukherjee (amukherj) 
------
commit 24d09edf4b13d72a8707c801939921de0d32d4dd
Author: Atin Mukherjee <amukherj>
Date:   Tue Jun 27 21:09:49 2017 +0530

    glusterd: mark brickinfo to started on successful attach
    
    brickinfo's port & status should be filled up only when attach brick is
    successful.
    
    Change-Id: I68b181be37cb94d176f0f4692e8d9dac5493181c
    BUG: 1465559
    Signed-off-by: Atin Mukherjee <amukherj>
    Reviewed-on: https://review.gluster.org/17640
    Reviewed-by: Jeff Darcy <jeff.us>
    Smoke: Gluster Build System <jenkins.org>
    Reviewed-by: Samikshan Bairagya <samikshan>
    CentOS-regression: Gluster Build System <jenkins.org>

Comment 4 Atin Mukherjee 2017-07-20 12:27:23 UTC

Found another race where glusterd was restarted glusterd_brick_start () is called multiple times due to friend handshaking and in one instance when one of the brick was attempted to be attached to the existing brick process, send_attach_req failed as the first brick itself was still not up and then we did a synlock_unlock ( )followed by a sleep of 1 sec, before the same thread woke up, another thread tried to start the same brick process and then it assumed that it has to start a fresh brick process.

Comment 5 Worker Ant 2017-07-20 12:51:56 UTC

REVIEW: https://review.gluster.org/17840 (glusterd: fix brick start race) posted (#1) for review on master by Atin Mukherjee (amukherj)

Comment 6 Worker Ant 2017-07-20 21:50:23 UTC

COMMIT: https://review.gluster.org/17840 committed in master by Jeff Darcy (jeff.us) 
------
commit d095c02eb9796ca2ec2a24931c28f057c403f834
Author: Atin Mukherjee <amukherj>
Date:   Thu Jul 20 18:11:14 2017 +0530

    glusterd: fix brick start race
    
    Problem:
    
    Another race where glusterd was restarted glusterd_brick_start () is called
    multiple times due to friend handshaking and in one instance when one of the
    brick was attempted to be attached to the existing brick process,
    send_attach_req failed as the first brick itself was still not up and then we
    did a synlock_unlock () followed by a sleep of 1 sec, before the same thread
    woke up, another thread tried to start the same brick process and then it
    assumed that it has to start a fresh brick process.
    
    Solution:
    
    1. If brick is in starting phase (brickinfo->status ==
    GF_BRICK_STARTING), no need for a reattempt to
    start the brick.
    2. While initiating attach_req set brickinfo->status to
    GF_BRICK_STARTING
    
    Change-Id: Ib007b6199ec36fdab4214a1d37f99d7f65ef64da
    BUG: 1465559
    Signed-off-by: Atin Mukherjee <amukherj>
    Reviewed-on: https://review.gluster.org/17840
    Reviewed-by: Amar Tumballi <amarts>
    Smoke: Gluster Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: Jeff Darcy <jeff.us>

Comment 7 Shyamsundar 2017-09-05 17:35:13 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.12.0, please open a new bug report.

glusterfs-3.12.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/announce/2017-September/000082.html
[2] https://www.gluster.org/pipermail/gluster-users/