Bug 924726 - A series of graph switches when there is no activity on mountpoint can result in socket leaks
Summary: A series of graph switches when there is no activity on mountpoint can result...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: fuse
Version: mainline
Hardware: Unspecified
OS: Unspecified
high
unspecified
Target Milestone: ---
Assignee: Raghavendra G
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-03-22 11:34 UTC by Raghavendra G
Modified: 2014-11-11 08:23 UTC (History)
2 users (show)

Fixed In Version: glusterfs-3.6.0beta1
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-11-11 08:23:37 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:


Attachments (Terms of Use)

Description Raghavendra G 2013-03-22 11:34:35 UTC
Description of problem:
After a graph switch, a PARENT_DOWN event indicates protocol/client to shutdown all its sockets. However, this event will be sent only when the first fop is received. Also, this event is sent only on previously active graph. So, if there are multiple graph switches without any activity on mountpoint, we'll be left with a list of graphs with their sockets still open.

Version-Release number of selected component (if applicable):
master

How reproducible:
always

Steps to Reproduce:
1. note down the number of sockets opened by glusterfs client using netstat -nap.
2.when there is no activity on mountpoint, do a series of volume set operations (like setting a perf translator on/off). 
3. netstat -nap outputs show a constantly increasing list of opened sockets.
4. ls on mounpoint only cleans up sockets consumed by previously active graph. The number of sockets opened by glusterfs won't return to the value at the start of the test.
  
Actual results:
The number of sockets opened by glusterfs won't return to the value at the start of the test.

Expected results:
The number of sockets opened should return to the value in step 1. after we complete step 4.


Additional info:

Comment 1 Vijay Bellur 2013-03-27 12:16:57 UTC
REVIEW: http://review.gluster.org/4713 (mount/fuse: cleanup old graphs.) posted (#3) for review on master by Raghavendra G (raghavendra)

Comment 2 Anand Avati 2013-12-09 11:19:07 UTC
REVIEW: http://review.gluster.org/4713 (mount/fuse: cleanup old graphs.) posted (#5) for review on master by Raghavendra G (rgowdapp)

Comment 3 Anand Avati 2013-12-09 18:19:09 UTC
REVIEW: http://review.gluster.org/4713 (mount/fuse: cleanup old graphs.) posted (#6) for review on master by Raghavendra G (rgowdapp)

Comment 4 Anand Avati 2013-12-10 02:52:39 UTC
REVIEW: http://review.gluster.org/4713 (mount/fuse: cleanup old graphs.) posted (#7) for review on master by Raghavendra G (rgowdapp)

Comment 5 Anand Avati 2013-12-10 17:32:06 UTC
COMMIT: http://review.gluster.org/4713 committed in master by Anand Avati (avati) 
------
commit c492b2cf8a18f09975da24e60330d0454cbd1e4e
Author: Raghavendra G <rgowdapp>
Date:   Mon Dec 9 16:10:56 2013 +0530

    mount/fuse: cleanup old graphs.
    
    After a graph switch, a PARENT_DOWN event from fuse indicates
    protocol/client to shutdown all its sockets. However, this event will
    be sent only when the first fop is received from fuse mount after
    graph switch. Also, this event is sent only on previously active graph.
    So, if there are multiple graph switches when there is no activity on
    mountpoint, we'll be left with a list of graphs with their sockets
    still open.
    
    This patch fixes the issue by sending PARENT_DOWN to previously
    non-active graph when a new graph is available irrespective of whether
    there is an activity on mount-point.
    
    Change-Id: I9e676658d797c0b2dd3ed5ca5a56271bd94b4bb6
    BUG: 924726
    Signed-off-by: Raghavendra G <rgowdapp>
    Reviewed-on: http://review.gluster.org/4713
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Anand Avati <avati>

Comment 6 Anand Avati 2014-01-27 10:32:12 UTC
REVIEW: http://review.gluster.org/6813 (mount/fuse: cleanup old graphs.) posted (#1) for review on release-3.5 by Raghavendra G (rgowdapp)

Comment 7 Anand Avati 2014-01-28 02:18:15 UTC
REVIEW: http://review.gluster.org/6813 (mount/fuse: cleanup old graphs.) posted (#2) for review on release-3.5 by Raghavendra G (rgowdapp)

Comment 8 Anand Avati 2014-01-28 16:44:03 UTC
COMMIT: http://review.gluster.org/6813 committed in release-3.5 by Vijay Bellur (vbellur) 
------
commit c7ba4f37b048399481e14167a9a649f13a365cf0
Author: Raghavendra G <rgowdapp>
Date:   Mon Dec 9 16:10:56 2013 +0530

    mount/fuse: cleanup old graphs.
    
    After a graph switch, a PARENT_DOWN event from fuse indicates
    protocol/client to shutdown all its sockets. However, this event will
    be sent only when the first fop is received from fuse mount after
    graph switch. Also, this event is sent only on previously active graph.
    So, if there are multiple graph switches when there is no activity on
    mountpoint, we'll be left with a list of graphs with their sockets
    still open.
    
    This patch fixes the issue by sending PARENT_DOWN to previously
    non-active graph when a new graph is available irrespective of whether
    there is an activity on mount-point.
    
    Change-Id: I9e676658d797c0b2dd3ed5ca5a56271bd94b4bb6
    BUG: 924726
    Signed-off-by: Raghavendra G <rgowdapp>
    Reviewed-on: http://review.gluster.org/4713
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Anand Avati <avati>
    Reviewed-on: http://review.gluster.org/6813
    Reviewed-by: Vijay Bellur <vbellur>

Comment 9 Niels de Vos 2014-09-22 12:31:36 UTC
A beta release for GlusterFS 3.6.0 has been released. Please verify if the release solves this bug report for you. In case the glusterfs-3.6.0beta1 release does not have a resolution for this issue, leave a comment in this bug and move the status to ASSIGNED. If this release fixes the problem for you, leave a note and change the status to VERIFIED.

Packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update (possibly an "updates-testing" repository) infrastructure for your distribution.

[1] http://supercolony.gluster.org/pipermail/gluster-users/2014-September/018836.html
[2] http://supercolony.gluster.org/pipermail/gluster-users/

Comment 10 Niels de Vos 2014-11-11 08:23:37 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.6.1, please reopen this bug report.

glusterfs-3.6.1 has been announced [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://supercolony.gluster.org/pipermail/gluster-users/2014-November/019410.html
[2] http://supercolony.gluster.org/mailman/listinfo/gluster-users


Note You need to log in before you can comment on or make changes to this bug.