Description of problem: When an error occurs on nodes, it is possible that an error unwinding function is called instead of the fuse_release function. In this case, the current state of the code will leak a 88 bytes ctx structure. Version-Release number of selected component (if applicable): 3.10.3 How reproducible: 100% Steps to Reproduce: 1.Setup a 3 nodes replicaset (2 nodes might be enough) 2.Write the same file over and over again on one node 3.Try to read that same file over and over again on a second node. Actual results: ==19736== Thread 1: ==19736== 88 bytes in 1 blocks are definitely lost in loss record 257 of 591 ==19736== at 0x4C277BB: calloc (vg_replace_malloc.c:593) ==19736== by 0x4E90C31: __gf_calloc (mem-pool.c:117) ==19736== by 0xD56373D: __fuse_fd_ctx_check_n_create (fuse-bridge.c:79) ==19736== by 0xD56381E: fuse_fd_ctx_check_n_create (fuse-bridge.c:108) ==19736== by 0xD56F473: fuse_open_resume (fuse-bridge.c:2148) ==19736== by 0xD564CE9: fuse_fop_resume (fuse-bridge.c:556) ==19736== by 0xD562803: fuse_resolve_done (fuse-resolve.c:663) ==19736== by 0xD5628D9: fuse_resolve_all (fuse-resolve.c:690) ==19736== by 0xD5627E4: fuse_resolve (fuse-resolve.c:654) ==19736== by 0xD5628B0: fuse_resolve_all (fuse-resolve.c:686) ==19736== by 0xD562937: fuse_resolve_continue (fuse-resolve.c:706) ==19736== by 0xD561CDE: fuse_resolve_inode (fuse-resolve.c:364) ==19736== by 0xD5627D6: fuse_resolve (fuse-resolve.c:651) ==19736== by 0xD56285B: fuse_resolve_all (fuse-resolve.c:679) ==19736== by 0xD562975: fuse_resolve_and_resume (fuse-resolve.c:718) ==19736== by 0xD56FAF8: fuse_open (fuse-bridge.c:2185) ==19736== by 0xD57DA2A: fuse_thread_proc (fuse-bridge.c:5068) ==19736== by 0x5C32AA0: start_thread (pthread_create.c:301) ==19736== by 0x1633C6FF: ??? Expected results: no leaks Additional info: on our production environment, this happens often enough that we must restart the gluster process every 2-3 months to avoid OOM
REVIEW: https://review.gluster.org/17759 (memory leak fixes) posted (#1) for review on master by Danny Couture (couture.danny)
I just confirmed the bug for mainline @ a4a417e29c5b2d63e6bf5efae4f0ccf30a39647f
REVIEW: https://review.gluster.org/17759 (memory leak fixes) posted (#2) for review on master by Danny Couture (couture.danny)
REVIEW: https://review.gluster.org/17759 (memory leak fixes) posted (#3) for review on master by Danny Couture (couture.danny)
REVIEW: https://review.gluster.org/17759 (fuse: memory leak fixes) posted (#4) for review on master by Niels de Vos (ndevos)
COMMIT: https://review.gluster.org/17759 committed in master by Niels de Vos (ndevos) ------ commit 5ee383fed9f6408d303aa539dda071275021f8e4 Author: Danny Couture <couture.danny> Date: Wed Jul 5 09:55:17 2017 -0400 fuse: memory leak fixes Fix fuse ctx memory leak in case an error occurs and the cleanup path is different than usual. Also fix a memory leak in logging if eh_save_history() fails. Change-Id: I7ec967c807b0ed91184e5b958be70702215c46c9 BUG: 1470220 Signed-off-by: Danny Couture <couture.danny> Reviewed-on: https://review.gluster.org/17759 Reviewed-by: Niels de Vos <ndevos> Smoke: Gluster Build System <jenkins.org> Reviewed-by: N Balachandran <nbalacha> Reviewed-by: Prashanth Pai <ppai> Reviewed-by: Amar Tumballi <amarts> Tested-by: Amar Tumballi <amarts> CentOS-regression: Gluster Build System <jenkins.org> Reviewed-by: Raghavendra G <rgowdapp>
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.12.0, please open a new bug report. glusterfs-3.12.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://lists.gluster.org/pipermail/announce/2017-September/000082.html [2] https://www.gluster.org/pipermail/gluster-users/