Bug 1471028 - glusterfs process leaking memory when error occurs
Summary: glusterfs process leaking memory when error occurs
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: fuse
Version: 3.10
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Niels de Vos
QA Contact:
URL:
Whiteboard:
: 1470329 (view as bug list)
Depends On: 1470220
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-07-14 09:38 UTC by Niels de Vos
Modified: 2017-08-21 13:40 UTC (History)
2 users (show)

Fixed In Version: glusterfs-3.10.5
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1470220
Environment:
Last Closed: 2017-08-21 13:40:58 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:


Attachments (Terms of Use)

Description Niels de Vos 2017-07-14 09:38:41 UTC
+++ This bug was initially created as a clone of Bug #1470220 +++

Description of problem:

When an error occurs on nodes, it is possible that an error unwinding function is called instead of the fuse_release function. In this case, the current state of the code will leak a 88 bytes ctx structure.

Version-Release number of selected component (if applicable):
3.10.3

How reproducible:
100%

Steps to Reproduce:
1.Setup a 3 nodes replicaset (2 nodes might be enough)
2.Write the same file over and over again on one node
3.Try to read that same file over and over again on a second node.

Actual results:
==19736== Thread 1:
==19736== 88 bytes in 1 blocks are definitely lost in loss record 257 of 591
==19736==    at 0x4C277BB: calloc (vg_replace_malloc.c:593)
==19736==    by 0x4E90C31: __gf_calloc (mem-pool.c:117)
==19736==    by 0xD56373D: __fuse_fd_ctx_check_n_create (fuse-bridge.c:79)
==19736==    by 0xD56381E: fuse_fd_ctx_check_n_create (fuse-bridge.c:108)
==19736==    by 0xD56F473: fuse_open_resume (fuse-bridge.c:2148)
==19736==    by 0xD564CE9: fuse_fop_resume (fuse-bridge.c:556)
==19736==    by 0xD562803: fuse_resolve_done (fuse-resolve.c:663)
==19736==    by 0xD5628D9: fuse_resolve_all (fuse-resolve.c:690)
==19736==    by 0xD5627E4: fuse_resolve (fuse-resolve.c:654)
==19736==    by 0xD5628B0: fuse_resolve_all (fuse-resolve.c:686)
==19736==    by 0xD562937: fuse_resolve_continue (fuse-resolve.c:706)
==19736==    by 0xD561CDE: fuse_resolve_inode (fuse-resolve.c:364)
==19736==    by 0xD5627D6: fuse_resolve (fuse-resolve.c:651)
==19736==    by 0xD56285B: fuse_resolve_all (fuse-resolve.c:679)
==19736==    by 0xD562975: fuse_resolve_and_resume (fuse-resolve.c:718)
==19736==    by 0xD56FAF8: fuse_open (fuse-bridge.c:2185)
==19736==    by 0xD57DA2A: fuse_thread_proc (fuse-bridge.c:5068)
==19736==    by 0x5C32AA0: start_thread (pthread_create.c:301)
==19736==    by 0x1633C6FF: ???

Expected results:
no leaks

Additional info:
on our production environment, this happens often enough that we must restart the gluster process every 2-3 months to avoid OOM

--- Additional comment from Worker Ant on 2017-07-14 11:35:28 CEST ---

COMMIT: https://review.gluster.org/17759 committed in master by Niels de Vos (ndevos) 
------
commit 5ee383fed9f6408d303aa539dda071275021f8e4
Author: Danny Couture <couture.danny>
Date:   Wed Jul 5 09:55:17 2017 -0400

    fuse: memory leak fixes
    
    Fix fuse ctx memory leak in case an error occurs and the cleanup path
    is different than usual. Also fix a memory leak in logging if
    eh_save_history() fails.
    
    Change-Id: I7ec967c807b0ed91184e5b958be70702215c46c9
    BUG: 1470220
    Signed-off-by: Danny Couture <couture.danny>
    Reviewed-on: https://review.gluster.org/17759
    Reviewed-by: Niels de Vos <ndevos>
    Smoke: Gluster Build System <jenkins.org>
    Reviewed-by: N Balachandran <nbalacha>
    Reviewed-by: Prashanth Pai <ppai>
    Reviewed-by: Amar Tumballi <amarts>
    Tested-by: Amar Tumballi <amarts>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: Raghavendra G <rgowdapp>

Comment 1 Worker Ant 2017-07-14 09:44:17 UTC
REVIEW: https://review.gluster.org/17758 (fuse: memory leak fixes) posted (#2) for review on release-3.10 by Niels de Vos (ndevos)

Comment 2 Niels de Vos 2017-07-14 09:47:33 UTC
*** Bug 1470329 has been marked as a duplicate of this bug. ***

Comment 3 Worker Ant 2017-07-19 21:33:38 UTC
COMMIT: https://review.gluster.org/17758 committed in release-3.10 by Raghavendra Talur (rtalur) 
------
commit 904c211802e1d834ec05c21831aa5b51a416341c
Author: Danny Couture <couture.danny>
Date:   Wed Jul 5 09:55:17 2017 -0400

    fuse: memory leak fixes
    
    Fix fuse ctx memory leak in case an error occurs and the cleanup path
    is different than usual. Also fix a memory leak in logging if
    eh_save_history() fails.
    
    Cherry picked from commit 5ee383fed9f6408d303aa539dda071275021f8e4:
    > Change-Id: I7ec967c807b0ed91184e5b958be70702215c46c9
    > BUG: 1470220
    > Signed-off-by: Danny Couture <couture.danny>
    > Reviewed-on: https://review.gluster.org/17759
    > Reviewed-by: Niels de Vos <ndevos>
    > Smoke: Gluster Build System <jenkins.org>
    > Reviewed-by: N Balachandran <nbalacha>
    > Reviewed-by: Prashanth Pai <ppai>
    > Reviewed-by: Amar Tumballi <amarts>
    > Tested-by: Amar Tumballi <amarts>
    > CentOS-regression: Gluster Build System <jenkins.org>
    > Reviewed-by: Raghavendra G <rgowdapp>
    
    Change-Id: I7ec967c807b0ed91184e5b958be70702215c46c9
    BUG: 1471028
    Signed-off-by: Niels de Vos <ndevos>
    Reviewed-on: https://review.gluster.org/17758
    Smoke: Gluster Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>

Comment 4 Shyamsundar 2017-08-21 13:40:58 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.10.5, please open a new bug report.

glusterfs-3.10.5 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/announce/2017-August/000079.html
[2] https://www.gluster.org/pipermail/gluster-users/


Note You need to log in before you can comment on or make changes to this bug.