Bug 1453087 - Brick Multiplexing: On reboot of a node Brick multiplexing feature lost on that node as multiple brick processes get spawned
Summary: Brick Multiplexing: On reboot of a node Brick multiplexing feature lost on th...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: glusterd
Version: 3.10
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
Assignee: Samikshan Bairagya
QA Contact:
URL:
Whiteboard: brick-multiplexing
Depends On: 1451248
Blocks: 1450889 1453086 glusterfs-3.10.3
TreeView+ depends on / blocked
 
Reported: 2017-05-22 06:18 UTC by Samikshan Bairagya
Modified: 2017-06-06 06:08 UTC (History)
7 users (show)

Fixed In Version: glusterfs-3.10.3
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1451248
Environment:
Last Closed: 2017-06-06 06:08:56 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:


Attachments (Terms of Use)

Description Samikshan Bairagya 2017-05-22 06:18:01 UTC
+++ This bug was initially created as a clone of Bug #1451248 +++

+++ This bug was initially created as a clone of Bug #1450889 +++

Description of problem:
========================
When you reboot a node with brick mux enabled and multi volume setup, I see that many glusterfsd are spawned and hence we lose the brick mux feature/


Version-Release number of selected component (if applicable):
========
3.8.4-25

How reproducible:
========
always

Steps to Reproduce:
1.have a 3 node setup with brick mux enabled, and vols say from v1..v10 with each volume being a 1x3 and one brick per node (all independent LVs)
2.we can see that only one glusterfsd per node exists
3.now reboot node1
4. on successful reboot, following is the status 



Last login: Mon May 15 15:56:55 2017 from dhcp35-77.lab.eng.blr.redhat.com
[root@dhcp35-45 ~]# ps -ef|grep glusterfsd
root      4693     1 42 15:56 ?        00:02:07 /usr/sbin/glusterfsd -s 10.70.35.45 --volfile-id 1.10.70.35.45.rhs-brick1-1 -p /var/lib/glusterd/vols/1/run/10.70.35.45-rhs-brick1-1.pid -S /var/run/gluster/a19832cf9844ad10112aba39eba569a6.socket --brick-name /rhs/brick1/1 -l /var/log/glusterfs/bricks/rhs-brick1-1.log --xlator-option *-posix.glusterd-uuid=e4f737cd-59a2-4392-aa3d-4230f698f128 --brick-port 49152 --xlator-option 1-server.listen-port=49152
root      4701     1  0 15:56 ?        00:00:00 /usr/sbin/glusterfsd -s 10.70.35.45 --volfile-id 10.10.70.35.45.rhs-brick10-10 -p /var/lib/glusterd/vols/10/run/10.70.35.45-rhs-brick10-10.pid -S /var/run/gluster/fd40f022ab677d36e57793a60cc16166.socket --brick-name /rhs/brick10/10 -l /var/log/glusterfs/bricks/rhs-brick10-10.log --xlator-option *-posix.glusterd-uuid=e4f737cd-59a2-4392-aa3d-4230f698f128 --brick-port 49153 --xlator-option 10-server.listen-port=49153
root      4709     1  0 15:56 ?        00:00:00 /usr/sbin/glusterfsd -s 10.70.35.45 --volfile-id 2.10.70.35.45.rhs-brick2-2 -p /var/lib/glusterd/vols/2/run/10.70.35.45-rhs-brick2-2.pid -S /var/run/gluster/898f4e556d871cfb1613d6ff121bd5e6.socket --brick-name /rhs/brick2/2 -l /var/log/glusterfs/bricks/rhs-brick2-2.log --xlator-option *-posix.glusterd-uuid=e4f737cd-59a2-4392-aa3d-4230f698f128 --brick-port 49154 --xlator-option 2-server.listen-port=49154
root      4719     1  0 15:56 ?        00:00:00 /usr/sbin/glusterfsd -s 10.70.35.45 --volfile-id 3.10.70.35.45.rhs-brick3-3 -p /var/lib/glusterd/vols/3/run/10.70.35.45-rhs-brick3-3.pid -S /var/run/gluster/af3354d92921146c0e8d3bebdcbec907.socket --brick-name /rhs/brick3/3 -l /var/log/glusterfs/bricks/rhs-brick3-3.log --xlator-option *-posix.glusterd-uuid=e4f737cd-59a2-4392-aa3d-4230f698f128 --brick-port 49155 --xlator-option 3-server.listen-port=49155
root      4728     1 44 15:56 ?        00:02:13 /usr/sbin/glusterfsd -s 10.70.35.45 --volfile-id 4.10.70.35.45.rhs-brick4-4 -p /var/lib/glusterd/vols/4/run/10.70.35.45-rhs-brick4-4.pid -S /var/run/gluster/cafb15e7ed1d462ddf513e7cf80ca718.socket --brick-name /rhs/brick4/4 -l /var/log/glusterfs/bricks/rhs-brick4-4.log --xlator-option *-posix.glusterd-uuid=e4f737cd-59a2-4392-aa3d-4230f698f128 --brick-port 49156 --xlator-option 4-server.listen-port=49156
root      4734     1  0 15:56 ?        00:00:00 /usr/sbin/glusterfsd -s 10.70.35.45 --volfile-id 5.10.70.35.45.rhs-brick5-5 -p /var/lib/glusterd/vols/5/run/10.70.35.45-rhs-brick5-5.pid -S /var/run/gluster/5a92ed518f554fe96a3c3f4a1ecf5cb3.socket --brick-name /rhs/brick5/5 -l /var/log/glusterfs/bricks/rhs-brick5-5.log --xlator-option *-posix.glusterd-uuid=e4f737cd-59a2-4392-aa3d-4230f698f128 --brick-port 49157 --xlator-option 5-server.listen-port=49157

--- Additional comment from Worker Ant on 2017-05-16 07:05:13 EDT ---

REVIEW: https://review.gluster.org/17307 (glusterd: Don't spawn new glusterfsds on node reboot with brick-mux) posted (#1) for review on master by Samikshan Bairagya (samikshan)

--- Additional comment from Worker Ant on 2017-05-17 16:37:34 EDT ---

REVIEW: https://review.gluster.org/17307 (glusterd: Don't spawn new glusterfsds on node reboot with brick-mux) posted (#2) for review on master by Samikshan Bairagya (samikshan)

--- Additional comment from Worker Ant on 2017-05-18 07:56:46 EDT ---

REVIEW: https://review.gluster.org/17307 (glusterd: Don't spawn new glusterfsds on node reboot with brick-mux) posted (#3) for review on master by Samikshan Bairagya (samikshan)

--- Additional comment from Worker Ant on 2017-05-18 12:45:32 EDT ---

COMMIT: https://review.gluster.org/17307 committed in master by Jeff Darcy (jeff.us) 
------
commit 13e7b3b354a252ad4065f7b2f0f805c40a3c5d18
Author: Samikshan Bairagya <samikshan>
Date:   Tue May 16 15:07:21 2017 +0530

    glusterd: Don't spawn new glusterfsds on node reboot with brick-mux
    
    With brick multiplexing enabled, upon a node reboot new bricks were
    not being attached to the first spawned brick process even though
    there wasn't any compatibility issues.
    
    The reason for this is that upon glusterd restart after a node
    reboot, since brick services aren't running, glusterd starts the
    bricks in a "no-wait" mode. So after a brick process is spawned for
    the first brick, there isn't enough time for the corresponding pid
    file to get populated with a value before the compatibilty check is
    made for the next brick.
    
    This commit solves this by iteratively waiting for the pidfile to be
    populated in the brick compatibility comparison stage before checking
    if the brick process is alive.
    
    Change-Id: Ibd1f8e54c63e4bb04162143c9d70f09918a44aa4
    BUG: 1451248
    Signed-off-by: Samikshan Bairagya <samikshan>
    Reviewed-on: https://review.gluster.org/17307
    Reviewed-by: Atin Mukherjee <amukherj>
    Smoke: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>

Comment 1 Worker Ant 2017-05-22 06:20:49 UTC
REVIEW: https://review.gluster.org/17352 (glusterd: Don't spawn new glusterfsds on node reboot with brick-mux) posted (#1) for review on release-3.10 by Samikshan Bairagya (samikshan)

Comment 2 Worker Ant 2017-05-30 13:36:44 UTC
COMMIT: https://review.gluster.org/17352 committed in release-3.10 by Raghavendra Talur (rtalur) 
------
commit 1a90d86296f6529423a4450bc1e0b3bb12e4f0a2
Author: Samikshan Bairagya <samikshan>
Date:   Tue May 16 15:07:21 2017 +0530

    glusterd: Don't spawn new glusterfsds on node reboot with brick-mux
    
    With brick multiplexing enabled, upon a node reboot new bricks were
    not being attached to the first spawned brick process even though
    there wasn't any compatibility issues.
    
    The reason for this is that upon glusterd restart after a node
    reboot, since brick services aren't running, glusterd starts the
    bricks in a "no-wait" mode. So after a brick process is spawned for
    the first brick, there isn't enough time for the corresponding pid
    file to get populated with a value before the compatibilty check is
    made for the next brick.
    
    This commit solves this by iteratively waiting for the pidfile to be
    populated in the brick compatibility comparison stage before checking
    if the brick process is alive.
    
    > Reviewed-on: https://review.gluster.org/17307
    > Reviewed-by: Atin Mukherjee <amukherj>
    > Smoke: Gluster Build System <jenkins.org>
    > NetBSD-regression: NetBSD Build System <jenkins.org>
    > CentOS-regression: Gluster Build System <jenkins.org>
    
    (cherry picked from commit 13e7b3b354a252ad4065f7b2f0f805c40a3c5d18)
    
    Change-Id: Ibd1f8e54c63e4bb04162143c9d70f09918a44aa4
    BUG: 1453087
    Signed-off-by: Samikshan Bairagya <samikshan>
    Reviewed-on: https://review.gluster.org/17352
    Smoke: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: Atin Mukherjee <amukherj>
    Reviewed-by: Raghavendra Talur <rtalur>

Comment 3 Worker Ant 2017-05-31 04:38:04 UTC
REVIEW: https://review.gluster.org/17425 (glusterd: Eliminate race in brick compatibility checking stage) posted (#1) for review on release-3.10 by Samikshan Bairagya (samikshan)

Comment 4 Worker Ant 2017-05-31 15:07:02 UTC
COMMIT: https://review.gluster.org/17425 committed in release-3.10 by Raghavendra Talur (rtalur) 
------
commit e91bfdec983ce41a395fe0b89d2a31fb5a6564a3
Author: Samikshan Bairagya <samikshan>
Date:   Tue May 23 19:32:24 2017 +0530

    glusterd: Eliminate race in brick compatibility checking stage
    
    In https://review.gluster.org/17307/, while looking for compatible
    bricks for multiplexing, it is checked if the brick pidfile exists
    before checking if the corresponding brick process is running.
    
    However checking if the brick process is running just after
    checking if the pidfile exists isn't enough since there might be
    race conditions where the pidfile has been created but hasn't
    been updated with a pid value yet. This commit solves that by
    making sure that we wait iteratively till the pid value is updated
    as well.
    
    > Reviewed-on: https://review.gluster.org/17375
    > Smoke: Gluster Build System <jenkins.org>
    > Reviewed-by: Atin Mukherjee <amukherj>
    > NetBSD-regression: NetBSD Build System <jenkins.org>
    > CentOS-regression: Gluster Build System <jenkins.org>
    
    (cherry picked from commit a8624b8b13a1f4222e4d3e33fa5836d7b45369bc)
    
    Change-Id: Ib7a158f95566486f7c1f84b6357c9b89e4c797ae
    BUG: 1453087
    Signed-off-by: Samikshan Bairagya <samikshan>
    Reviewed-on: https://review.gluster.org/17425
    NetBSD-regression: NetBSD Build System <jenkins.org>
    Tested-by: Raghavendra Talur <rtalur>
    CentOS-regression: Gluster Build System <jenkins.org>
    Smoke: Gluster Build System <jenkins.org>
    Reviewed-by: Raghavendra Talur <rtalur>

Comment 5 Raghavendra Talur 2017-06-06 06:08:56 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.10.3, please open a new bug report.

glusterfs-3.10.3 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/gluster-users/2017-June/031399.html
[2] https://www.gluster.org/pipermail/gluster-users/


Note You need to log in before you can comment on or make changes to this bug.