Bug 1450267

Summary: nl-cache xlator leaks timer wheel and other memory
Product: [Community] GlusterFS Reporter: Poornima G <pgurusid>
Component: gluster-smbAssignee: Poornima G <pgurusid>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 3.11CC: bugs
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-3.11.0 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-05-30 18:52:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Poornima G 2017-05-12 05:09:58 UTC
Description of problem:
1. nl-cache in the init, inits timer-wheel thread but in the glfs_fini its not cleaned up
2. nl-cache fini() is not implemented

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Worker Ant 2017-05-12 05:10:18 UTC
REVIEW: https://review.gluster.org/17262 (core: make the per glusterfs_ctx_t timer-wheel refcounted) posted (#1) for review on release-3.11 by Poornima G (pgurusid)

Comment 2 Worker Ant 2017-05-12 11:14:06 UTC
REVIEW: https://review.gluster.org/17263 (nl-cache: free nlc_conf_t in fini()) posted (#1) for review on release-3.11 by Niels de Vos (ndevos)

Comment 3 Worker Ant 2017-05-12 13:32:35 UTC
COMMIT: https://review.gluster.org/17262 committed in release-3.11 by Shyamsundar Ranganathan (srangana) 
------
commit 45a5cea1ad028bdff5f33770df8ecdd9ac69b6f1
Author: Niels de Vos <ndevos>
Date:   Mon Apr 17 15:50:07 2017 +0530

    core: make the per glusterfs_ctx_t timer-wheel refcounted
    
    xlators can use a 'global' timer-wheel for scheduling events. This
    timer-wheel is managed per glusterfs_ctx_t, but does not need to be
    allocated for every graph. When an xlator wants to use the timer-wheel,
    it will be instanciated on demand, and provided to xlators that request
    it later on.
    
    By adding a reference counter to the glusterfs_ctx_t for the
    timer-wheel, the threads and structures can be cleaned up when the last
    xlator does not have a need for it anymore. In general, the xlators
    request the timer-wheel in init(), and they should return it in fini().
    
    Because the timer-wheel is managed per glusterfs_ctx_t, the functions
    can be added to ctx.c and do not need to live in their very minimal
    tw.[ch] files.
    
    
    >Reported-by: Poornima G <pgurusid>
    >Signed-off-by: Niels de Vos <ndevos>
    >Reviewed-on: https://review.gluster.org/17068
    >NetBSD-regression: NetBSD Build System <jenkins.org>
    >CentOS-regression: Gluster Build System <jenkins.org>
    >Smoke: Gluster Build System <jenkins.org>
    >Reviewed-by: Amar Tumballi <amarts>
    >Reviewed-by: Zhou Zhengping <johnzzpcrystal>
    >Reviewed-by: Kaleb KEITHLEY <kkeithle>
    >(cherry picked from commit 73fcf3a874b2049da31d01b8363d1ac85c9488c2)
    
    Change-Id: I19d225b39aaa272d9005ba7adc3104c3764f1572
    BUG: 1450267
    Reviewed-on: https://review.gluster.org/17262
    Tested-by: Poornima G <pgurusid>
    Smoke: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: Niels de Vos <ndevos>

Comment 4 Worker Ant 2017-05-16 00:29:21 UTC
COMMIT: https://review.gluster.org/17263 committed in release-3.11 by Shyamsundar Ranganathan (srangana) 
------
commit 9a20120790a462608a6121504bab27e4e910b471
Author: Niels de Vos <ndevos>
Date:   Fri May 12 13:12:39 2017 +0200

    nl-cache: free nlc_conf_t in fini()
    
    The (xlator_t*)->private structure in negative-lookup-cache is allocated
    in the init() function of the xlator, but never free'd. Valgrind
    detected this as:
    
        656 bytes in 1 blocks are definitely lost in loss record X of Y
           at 0x..+ calloc (/builddir/build/BUILD/valgrind-3.11.0/coregrind/m_replacemalloc/vg_replace_malloc.c:711)
           by 0x.. __gf_calloc (/usr/src/debug/glusterfs-3.11dev/libglusterfs/src/mem-pool.c:117)
           by 0x.. init (/usr/src/debug/glusterfs-3.11dev/xlators/performance/nl-cache/src/nl-cache.c:669)
           by 0x.. __xlator_init (/usr/src/debug/glusterfs-3.11dev/libglusterfs/src/xlator.c:472)
           by 0x.. xlator_init (/usr/src/debug/glusterfs-3.11dev/libglusterfs/src/xlator.c:498)
           by 0x.. glusterfs_graph_init (/usr/src/debug/glusterfs-3.11dev/libglusterfs/src/graph.c:321)
           by 0x.. glusterfs_graph_activate (/usr/src/debug/glusterfs-3.11dev/libglusterfs/src/graph.c:693)
           by 0x.. glfs_process_volfp (/usr/src/debug/glusterfs-3.11dev/api/src/glfs-mgmt.c:79)
           by 0x.. glfs_volumes_init (/usr/src/debug/glusterfs-3.11dev/api/src/glfs.c:160)
           by 0x.. glfs_init_common (/usr/src/debug/glusterfs-3.11dev/api/src/glfs.c:868)
           by 0x.. glfs_init@@GFAPI_3.4.0 (/usr/src/debug/glusterfs-3.11dev/api/src/glfs.c:913)
           by 0x.. main (/root/gluster-debug/gfapi-load-volfile/gfapi-load-volfile.c:54)
    
    When the xlators is unloaded, it should free the resources it allocated.
    This can easily be done in the fini() function.
    
    Cherry picked from commit d7e9dcfad228f385ad64526b1f06b55e98b06964:
    > Change-Id: I079e78cc207145bc542e2282fc4cf2bb4dadc28a
    > BUG: 1442569
    > Signed-off-by: Niels de Vos <ndevos>
    > Reviewed-on: https://review.gluster.org/17143
    > Smoke: Gluster Build System <jenkins.org>
    > NetBSD-regression: NetBSD Build System <jenkins.org>
    > CentOS-regression: Gluster Build System <jenkins.org>
    > Reviewed-by: Jeff Darcy <jeff.us>
    
    Change-Id: I079e78cc207145bc542e2282fc4cf2bb4dadc28a
    BUG: 1450267
    Signed-off-by: Niels de Vos <ndevos>
    Reviewed-on: https://review.gluster.org/17263
    Smoke: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: Zhou Zhengping <johnzzpcrystal>
    Reviewed-by: Poornima G <pgurusid>

Comment 5 Shyamsundar 2017-05-30 18:52:18 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.11.0, please open a new bug report.

glusterfs-3.11.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/announce/2017-May/000073.html
[2] https://www.gluster.org/pipermail/gluster-users/