Description of problem: On a brick multiplexing test setup if all the gluster processes are killed and glusterd is started there is a race window where one of the brick will fail to attach to one of the running brick instances and while doing so, it will try to bring up a new process resulting in deleting the socket file which has been used by one of the other brick processes resulting into brick disconnect too. Version-Release number of selected component (if applicable): mainline How reproducible: Often, not always
REVIEW: https://review.gluster.org/17640 (glusterd: mark brickinfo to started on successful attach) posted (#1) for review on master by Atin Mukherjee (amukherj)
REVIEW: https://review.gluster.org/17640 (glusterd: mark brickinfo to started on successful attach) posted (#2) for review on master by Atin Mukherjee (amukherj)
COMMIT: https://review.gluster.org/17640 committed in master by Atin Mukherjee (amukherj) ------ commit 24d09edf4b13d72a8707c801939921de0d32d4dd Author: Atin Mukherjee <amukherj> Date: Tue Jun 27 21:09:49 2017 +0530 glusterd: mark brickinfo to started on successful attach brickinfo's port & status should be filled up only when attach brick is successful. Change-Id: I68b181be37cb94d176f0f4692e8d9dac5493181c BUG: 1465559 Signed-off-by: Atin Mukherjee <amukherj> Reviewed-on: https://review.gluster.org/17640 Reviewed-by: Jeff Darcy <jeff.us> Smoke: Gluster Build System <jenkins.org> Reviewed-by: Samikshan Bairagya <samikshan> CentOS-regression: Gluster Build System <jenkins.org>
Found another race where glusterd was restarted glusterd_brick_start () is called multiple times due to friend handshaking and in one instance when one of the brick was attempted to be attached to the existing brick process, send_attach_req failed as the first brick itself was still not up and then we did a synlock_unlock ( )followed by a sleep of 1 sec, before the same thread woke up, another thread tried to start the same brick process and then it assumed that it has to start a fresh brick process.
REVIEW: https://review.gluster.org/17840 (glusterd: fix brick start race) posted (#1) for review on master by Atin Mukherjee (amukherj)
COMMIT: https://review.gluster.org/17840 committed in master by Jeff Darcy (jeff.us) ------ commit d095c02eb9796ca2ec2a24931c28f057c403f834 Author: Atin Mukherjee <amukherj> Date: Thu Jul 20 18:11:14 2017 +0530 glusterd: fix brick start race Problem: Another race where glusterd was restarted glusterd_brick_start () is called multiple times due to friend handshaking and in one instance when one of the brick was attempted to be attached to the existing brick process, send_attach_req failed as the first brick itself was still not up and then we did a synlock_unlock () followed by a sleep of 1 sec, before the same thread woke up, another thread tried to start the same brick process and then it assumed that it has to start a fresh brick process. Solution: 1. If brick is in starting phase (brickinfo->status == GF_BRICK_STARTING), no need for a reattempt to start the brick. 2. While initiating attach_req set brickinfo->status to GF_BRICK_STARTING Change-Id: Ib007b6199ec36fdab4214a1d37f99d7f65ef64da BUG: 1465559 Signed-off-by: Atin Mukherjee <amukherj> Reviewed-on: https://review.gluster.org/17840 Reviewed-by: Amar Tumballi <amarts> Smoke: Gluster Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.org> Reviewed-by: Jeff Darcy <jeff.us>
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.12.0, please open a new bug report. glusterfs-3.12.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://lists.gluster.org/pipermail/announce/2017-September/000082.html [2] https://www.gluster.org/pipermail/gluster-users/