Bug 1455912

+++ This bug was initially created as a clone of Bug #1454865 +++

+++ This bug was initially created as a clone of Bug #1448833 +++

Description of problem:
=======================
heal info command output shows the status of the bricks as  "Transport endpoint is not connected" though bricks are up and running.

Version-Release number of selected component (if applicable):
mainline

How reproducible:
=================
always

Steps to Reproduce:
===================
1) Create a Distributed-Replicate volume and enable brick mux on it.
2) Start the volume and FUSE mount it on a client.
3) Set cluster.self-heal-daemon to off.
4) Create a 10 directory on the mount point.
5) Kill one brick of one of the replica sets in the volume and modify the permissions of all directories.
6) Start volume with force option.
7) Kill the other brick in the same replica set and modify permissions of the directory again.
8) Start volume with force option. Examine the output of `gluster volume heal <vol-name> info' command on the server.

Actual results:
===============
heal info command output shows the status of the bricks as  "Transport endpoint is not connected" though bricks are up and running.


RCA:

When we stop the volume GlusterD actually sends two terminate request to brick process one during brick op phase and another during commit phase. Without multiplexing, it wasn't causing any problem, because the process was supposed to stop. But with multiplexing, it is just a detach which will be executed twice. Now those two requests can be executed at the same time, if that happens we may delete the graph entry twice as we are not taking any locks during the link modification of graph in glusterfs_handle_detach.

So the linked list will me moved twice which is results a deletion of an independent brick.

--- Additional comment from Worker Ant on 2017-05-23 11:31:50 EDT ---

REVIEW: https://review.gluster.org/17374 (glusterfsd: process attach and detach request inside lock) posted (#1) for review on master by Atin Mukherjee (amukherj)

--- Additional comment from Worker Ant on 2017-05-23 11:32:29 EDT ---

REVIEW: https://review.gluster.org/17374 (glusterfsd: process attach and detach request inside lock) posted (#2) for review on master by Atin Mukherjee (amukherj)

--- Additional comment from Worker Ant on 2017-05-24 03:48:04 EDT ---

REVIEW: https://review.gluster.org/17374 (glusterfsd: process attach and detach request inside lock) posted (#3) for review on master by Atin Mukherjee (amukherj)

--- Additional comment from Worker Ant on 2017-05-24 10:50:37 EDT ---

REVIEW: https://review.gluster.org/17374 (glusterfsd: process attach and detach request inside lock) posted (#4) for review on master by Atin Mukherjee (amukherj)

--- Additional comment from Worker Ant on 2017-05-25 08:32:46 EDT ---

REVIEW: https://review.gluster.org/17374 (glusterfsd: process attach and detach request inside lock) posted (#5) for review on master by Atin Mukherjee (amukherj)

--- Additional comment from Worker Ant on 2017-05-26 08:11:32 EDT ---

COMMIT: https://review.gluster.org/17374 committed in master by Jeff Darcy (jeff.us) 
------
commit 3ca5ae2f3bff2371042b607b8e8a218bf316b48c
Author: Atin Mukherjee <amukherj>
Date:   Fri May 19 21:04:53 2017 +0530

    glusterfsd: process attach and detach request inside lock
    
    With brick multiplexing, there is a high possibility that attach and
    detach requests might be parallely processed and to avoid a concurrent
    update to the same graph list, a mutex lock is required.
    
    Credits : Rafi (rkavunga) for the RCA of this issue
    
    Change-Id: Ic8e6d1708655c8a143c5a3690968dfa572a32a9c
    BUG: 1454865
    Signed-off-by: Atin Mukherjee <amukherj>
    Reviewed-on: https://review.gluster.org/17374
    Smoke: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: Jeff Darcy <jeff.us>