Description of problem: Glfs_fini, doesn't free the resources allocated in glfs_new Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. Write a sample libgfapi application, the threads, memory and other resources are not freed after glfs_fini. 2. 3. Actual results: Expected results: The resources should be freed. Additional info:
REVIEW: http://review.gluster.org/7642 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#1) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/7642 (libgfapi: In glfs_fini() free all the resources allocated in glfs_new.) posted (#2) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/7642 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#3) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/7642 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#4) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/7642 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#5) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/7642 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#6) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/7642 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#7) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/7884 (tests: Add gfapi test case) posted (#1) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/7884 (tests: Add gfapi test case) posted (#2) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/7884 (tests: Add gfapi test case) posted (#3) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/7642 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#8) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/7642 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#9) for review on master by Poornima G (pgurusid)
*** Bug 1005164 has been marked as a duplicate of this bug. ***
*** Bug 1061229 has been marked as a duplicate of this bug. ***
I tried a naive backport to 3.5.x of the patch on review (that is: apply a96350fa2b68626b8592d5cbd67405e4d8416cca first and do not try to cleanup ctx->logbuf_pool), but when trying to take a snapshot using libvirt, I get the following: [2014-09-27 17:34:05.886392] E [afr-common.c:4168:afr_notify] 0-virtualization-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2014-09-27 17:34:06.848988] E [client.c:2162:client_rpc_notify] 0-client: private structure of the xlator is NULL [2014-09-27 17:34:06.849018] E [client.c:2162:client_rpc_notify] 0-client: private structure of the xlator is NULL [2014-09-27 17:34:06.849169] E [io-cache.c:2080:fini] (-->/usr/lib64/libglusterfs.so.0(+0x1b485) [0x311ca247485] (-->/usr/lib64/libglusterfs.so.0(+0x1b485) [0x311ca247485] (-->/usr/lib64/libglusterfs.so.0(+0x1b4f1) [0x311ca2474f1]))) 0-: Assertion failed: list_empty (&table->inode_lru[i]) [2014-09-27 17:34:06.849196] E [io-cache.c:2083:fini] (-->/usr/lib64/libglusterfs.so.0(+0x1b485) [0x311ca247485] (-->/usr/lib64/libglusterfs.so.0(+0x1b485) [0x311ca247485] (-->/usr/lib64/libglusterfs.so.0(+0x1b4f1) [0x311ca2474f1]))) 0-: Assertion failed: list_empty (&table->inodes) [2014-09-27 17:34:06.849654] E [iobuf.c:127:__iobuf_arena_destroy_iobufs] (-->/usr/lib64/libgfapi.so.0(glfs_fini+0x1e9) [0x311ca4cfa79] (-->/usr/lib64/libglusterfs.so.0(iobuf_pool_destroy+0x6f) [0x311ca27627f] (-->/usr/lib64/libglusterfs.so.0(__iobuf_arena_destroy+0x22) [0x311ca275c92]))) 0-: Assertion failed: iobuf->ref == 0 [2014-09-27 17:34:06.849690] E [iobuf.c:127:__iobuf_arena_destroy_iobufs] (-->/usr/lib64/libgfapi.so.0(glfs_fini+0x1e9) [0x311ca4cfa79] (-->/usr/lib64/libglusterfs.so.0(iobuf_pool_destroy+0x6f) [0x311ca27627f] (-->/usr/lib64/libglusterfs.so.0(__iobuf_arena_destroy+0x22) [0x311ca275c92]))) 0-: Assertion failed: iobuf->ref == 0 Formatting 'gluster://10.1.120.11/virtualization/vm-persistent/0f83f084-8080-413e-b558-b678e504836e/04cb1ecd-eeb7-46e1-88d5-c6c890f44b95.snap01.qcow2', fmt=qcow2 size=32212254720 backing_file='gluster://10.1.120.11:24007/virtualization/vm-persistent/0f83f084-8080-413e-b558-b678e504836e/04cb1ecd-eeb7-46e1-88d5-c6c890f44b95.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off [2014-09-27 17:34:06.870307] E [afr-common.c:4168:afr_notify] 0-virtualization-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. [2014-09-27 17:34:07.851604] E [client.c:2162:client_rpc_notify] 0-client: private structure of the xlator is NULL [2014-09-27 17:34:07.851634] E [client.c:2162:client_rpc_notify] 0-client: private structure of the xlator is NULL [2014-09-27 17:34:07.851695] E [io-cache.c:2080:fini] (-->/usr/lib64/libglusterfs.so.0(+0x1b485) [0x311ca247485] (-->/usr/lib64/libglusterfs.so.0(+0x1b485) [0x311ca247485] (-->/usr/lib64/libglusterfs.so.0(+0x1b4f1) [0x311ca2474f1]))) 0-: Assertion failed: list_empty (&table->inode_lru[i]) [2014-09-27 17:34:07.851722] E [io-cache.c:2083:fini] (-->/usr/lib64/libglusterfs.so.0(+0x1b485) [0x311ca247485] (-->/usr/lib64/libglusterfs.so.0(+0x1b485) [0x311ca247485] (-->/usr/lib64/libglusterfs.so.0(+0x1b4f1) [0x311ca2474f1]))) 0-: Assertion failed: list_empty (&table->inodes) [2014-09-27 17:34:07.851782] E [quick-read.c:1056:qr_inode_table_destroy] (-->/usr/lib64/libglusterfs.so.0(+0x1b485) [0x311ca247485] (-->/usr/lib64/libglusterfs.so.0(+0x1b4f1) [0x311ca2474f1] (-->/usr/lib64/glusterfs/3.5.2/xlator/performance/quick-read.so(fini+0x3a) [0x311bb7ce9ea]))) 0-: Assertion failed: list_empty (&priv->table.lru[i]) After which qemu gets unresponsive and gets terminated by libvirt and libvirt must be restarted since it is unable to acquire a state change lock. Sorry, didn't have time to test with glusterfs-3.6 yet
Looks like, glfs_fini should wait for CHILD_DOWN event from all the xlators and only then proceed with cleanup. Trying to reproduce with the latest code to validate the analysis. Thank you for trying it out.
REVIEW: http://review.gluster.org/7642 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#10) for review on master by Poornima G (pgurusid)
Tiziano, when you can test this patch successfully (or not), please go to http://review.gluster.org/7642, login, click 'review' and mark the change as 'Verified' (or Failed). This will give you credit on the patch when it gets merged (a Tested-by line gets added then). More verifications and reviews make it easier for the maintainers to accept the change. Thanks!
REVIEW: http://review.gluster.org/7642 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#11) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/7642 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#12) for review on master by Poornima G (pgurusid)
Created attachment 985112 [details] Valgrind report without this patch
Created attachment 985113 [details] Valgrind report with this patch
Current state of the patch: Now with this patch, atleast 80-85% of the resource leaks are handled. Below are the numbers of memory and threads usage for 20 loops of glfs_init() and glfs_fini() (without any file/data operations): Without this patch: - the virtual size of the process 3015mb - ~80 threads leak. With this patch: - the virtual size of the process reduces to 314mb - All the threads are joined. For reference, attaching the valgrind report for both the cases. There is quite a bit of memory that is not freed, two major places remaining for cleanup are: - memory allocated by individual xlators(inode_ctx, fd_ctx, local pools) to be freed in each xlator. - rpc_clnt object is not being destroyed even after PARENT_DOWN, something to be fixed.
REVIEW: http://review.gluster.org/7642 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#13) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/7642 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#14) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9580 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#1) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/7642 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#15) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9680 (protocol/client: defer cleanup of private until RPC notifications are handled.) posted (#1) for review on master by Krishnan Parthasarathi (kparthas)
REVIEW: http://review.gluster.org/7642 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#16) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9693 (syncop: Add the code to destroy the synenv processors and syncenv structures) posted (#1) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9696 (libglusterfs: Add functions for xlator and graph cleanup.) posted (#1) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9699 (libglusterfs: Change iobuf_pool_destroy to destroy all the arenas.) posted (#1) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9700 (libglusterfs: Add inode table cleanup APIs.) posted (#1) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9701 (libglusterfs: Add timer thread destroy code.) posted (#1) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9699 (libglusterfs: Change iobuf_pool_destroy to destroy all the arenas.) posted (#2) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9691 (event_pool: Add the code to destroy the poller threads and event pool gracefully.) posted (#3) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9701 (libglusterfs: Add timer thread destroy code.) posted (#2) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9700 (libglusterfs: Add inode table cleanup APIs.) posted (#2) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9680 (protocol/client: defer cleanup of private until RPC notifications are handled.) posted (#2) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9699 (libglusterfs: Change iobuf_pool_destroy to destroy all the arenas.) posted (#3) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9696 (libglusterfs: Add functions for xlator and graph cleanup.) posted (#2) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9693 (syncop: Add the code to destroy the synenv processors and syncenv structures) posted (#2) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/7642 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#17) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9700 (libglusterfs: Add inode table cleanup APIs.) posted (#3) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/7642 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#18) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9693 (syncop: Add the code to destroy the synenv processors and syncenv structures) posted (#3) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9691 (event_pool: Add the code to destroy the poller threads and event pool gracefully.) posted (#4) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9725 (epoll: Fix a leak in slot allocation.) posted (#1) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9680 (protocol/client: defer cleanup of private until RPC notifications are handled.) posted (#3) for review on master by Krishnan Parthasarathi (kparthas)
REVIEW: http://review.gluster.org/9701 (libglusterfs: Add timer thread destroy code.) posted (#3) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9700 (libglusterfs: Add inode table cleanup APIs.) posted (#4) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9680 (protocol/client: defer cleanup of private until RPC notifications are handled.) posted (#4) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9699 (libglusterfs: Change iobuf_pool_destroy to destroy all the arenas.) posted (#4) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9696 (libglusterfs: Add functions for xlator and graph cleanup.) posted (#3) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9693 (syncop: Add the code to destroy the synenv processors and syncenv structures) posted (#4) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/7642 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#19) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9691 (event_pool: Add the code to destroy the poller threads and event pool gracefully.) posted (#5) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9701 (libglusterfs: Add timer thread destroy code.) posted (#4) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9700 (libglusterfs: Add inode table cleanup APIs.) posted (#5) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9680 (protocol/client: defer cleanup of private until RPC notifications are handled.) posted (#5) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9699 (libglusterfs: Change iobuf_pool_destroy to destroy all the arenas.) posted (#5) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9696 (libglusterfs: Add functions for xlator and graph cleanup.) posted (#4) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9693 (syncop: Add the code to destroy the synenv processors and syncenv structures) posted (#5) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/7642 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#20) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9691 (event_pool: Add the code to destroy the poller threads and event pool gracefully.) posted (#6) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9701 (libglusterfs: Add timer thread destroy code.) posted (#5) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9700 (libglusterfs: Add inode table cleanup APIs.) posted (#6) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9680 (protocol/client: defer cleanup of private until RPC notifications are handled.) posted (#6) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9699 (libglusterfs: Change iobuf_pool_destroy to destroy all the arenas.) posted (#6) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9696 (libglusterfs: Add functions for xlator and graph cleanup.) posted (#5) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9693 (syncop: Add the code to destroy the synenv processors and syncenv structures) posted (#6) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/7642 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#21) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9691 (event_pool: Add the code to destroy the poller threads and event pool gracefully.) posted (#7) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9701 (libglusterfs: Add timer thread destroy code.) posted (#6) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9700 (libglusterfs: Add inode table cleanup APIs.) posted (#7) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9699 (libglusterfs: Change iobuf_pool_destroy to destroy all the arenas.) posted (#7) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9680 (protocol/client: defer cleanup of private until RPC notifications are handled.) posted (#7) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9696 (libglusterfs: Add functions for xlator and graph cleanup.) posted (#6) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9693 (syncop: Add the code to destroy the synenv processors and syncenv structures) posted (#7) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/7642 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#22) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9691 (event_pool: Add the code to destroy the poller threads and event pool gracefully.) posted (#8) for review on master by Poornima G (pgurusid)
COMMIT: http://review.gluster.org/9725 committed in master by Vijay Bellur (vbellur) ------ commit 60cff8ab4f5fa2ee7bea028937ba23674e1ee0cc Author: Poornima G <pgurusid> Date: Mon Feb 23 16:10:33 2015 +0530 epoll: Fix a leak in slot allocation. When the slots_used in a table becomes 0, the table will not get reused, leading to a leak. This patch fixes the leak. Change-Id: Ib86826d287368174ea7ebe0d0d64b2dec574634e BUG: 1093594 Signed-off-by: Poornima G <pgurusid> Reviewed-on: http://review.gluster.org/9725 Reviewed-by: Krishnan Parthasarathi <kparthas> Reviewed-by: Prashanth Pai <ppai> Reviewed-by: Shyamsundar Ranganathan <srangana> Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Vijay Bellur <vbellur>
COMMIT: http://review.gluster.org/9680 committed in master by Vijay Bellur (vbellur) ------ commit efe09fff57dc01088168e60bff355b769c1bc7f7 Author: Krishnan Parthasarathi <kparthas> Date: Wed Feb 18 15:16:17 2015 +0530 protocol/client: defer cleanup of private until RPC notifications are handled. This fix is required for glfs_fini to be able to perform fini on client xlators in a graph. We are deferring freeing of client xlator's private until all RPC related resources are destroyed. This guarantees that client xlator would free RPC related resources provided its private structures are still accessible via its this pointer. 'Weak' property: If there are no epoll threads executing after calling fini() on a client xlator, then all its RPC related resources are guaranteed to be freed. We can now free the corresponding 'this' pointer. Change-Id: Ie00b14dda096ac128e1c37e0032f07d17fd701ce BUG: 1093594 Signed-off-by: Krishnan Parthasarathi <kparthas> Reviewed-on: http://review.gluster.org/9680 Reviewed-by: Rajesh Joseph <rjoseph> Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Vijay Bellur <vbellur>
REVIEW: http://review.gluster.org/9699 (libglusterfs: Change iobuf_pool_destroy to destroy all the arenas.) posted (#8) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9691 (event_pool: Add the code to destroy the poller threads and event pool gracefully.) posted (#9) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9700 (libglusterfs: Add inode table cleanup APIs.) posted (#8) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9693 (syncop: Add the code to destroy the synenv processors and syncenv structures) posted (#8) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9693 (syncop: Add the code to destroy the synenv processors and syncenv structures) posted (#9) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9699 (libglusterfs: Change iobuf_pool_destroy to destroy all the arenas.) posted (#9) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9699 (libglusterfs: Change iobuf_pool_destroy to destroy all the arenas.) posted (#10) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9700 (libglusterfs: Add inode table cleanup APIs.) posted (#9) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9699 (libglusterfs: Change iobuf_pool_destroy to destroy all the arenas.) posted (#11) for review on master by Poornima G (pgurusid)
COMMIT: http://review.gluster.org/9700 committed in master by Vijay Bellur (vbellur) ------ commit 5e12c658d6c2dc92fd336b745036641f054b3a40 Author: Poornima G <pgurusid> Date: Thu Feb 19 04:49:35 2015 +0530 libglusterfs: Add inode table cleanup APIs. Took the inode context free code from the patch http://review.gluster.org/#/c/4775/18/libglusterfs/src/inode.c Change-Id: I05fc025763fe4ce61dc61503de27ec1d3a203e50 BUG: 1093594 Signed-off-by: Poornima G <pgurusid> Reviewed-on: http://review.gluster.org/9700 Reviewed-by: Raghavendra G <rgowdapp> Reviewed-by: Vijay Bellur <vbellur> Tested-by: Gluster Build System <jenkins.com>
COMMIT: http://review.gluster.org/9693 committed in master by Vijay Bellur (vbellur) ------ commit 84e6eac265c87be5e436f3e7a41f40dc2d9c65c5 Author: Poornima G <pgurusid> Date: Thu Feb 19 01:24:29 2015 +0530 syncop: Add the code to destroy the synenv processors and syncenv structures Change-Id: I28020eb2fc08d886cd7c05ff96daf7ebb4264ffe BUG: 1093594 Signed-off-by: Poornima G <pgurusid> Reviewed-on: http://review.gluster.org/9693 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Vijay Bellur <vbellur>
COMMIT: http://review.gluster.org/9701 committed in master by Vijay Bellur (vbellur) ------ commit 7e416b6d00d626219c8d2067be720a915c4f85b1 Author: Poornima G <pgurusid> Date: Thu Feb 19 05:29:02 2015 +0530 libglusterfs: Add timer thread destroy code. Change-Id: Iafbbbfd9319751742b3c79419e1dd8e2958fee07 BUG: 1093594 Signed-off-by: Poornima G <pgurusid> Reviewed-on: http://review.gluster.org/9701 Reviewed-by: Rajesh Joseph <rjoseph> Reviewed-by: Krishnan Parthasarathi <kparthas> Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Vijay Bellur <vbellur>
COMMIT: http://review.gluster.org/9696 committed in master by Vijay Bellur (vbellur) ------ commit fc54f75ea49605e7fb5808e3fc01dfaa6b7c4649 Author: Poornima G <pgurusid> Date: Thu Feb 19 03:45:34 2015 +0530 libglusterfs: Add functions for xlator and graph cleanup. Change-Id: If341e3c0a559aa5bbca9c1263a241c6592c59706 BUG: 1093594 Signed-off-by: Poornima G <pgurusid> Reviewed-on: http://review.gluster.org/9696 Reviewed-by: Rajesh Joseph <rjoseph> Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Vijay Bellur <vbellur>
REVIEW: http://review.gluster.org/9691 (event_pool: Add the code to destroy the poller threads and event pool gracefully.) posted (#10) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9691 (event_pool: Add the code to destroy the poller threads and event pool gracefully.) posted (#11) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/7642 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#23) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/7642 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#24) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9691 (event_pool: Add the code to destroy the poller threads and event pool gracefully.) posted (#12) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9699 (libglusterfs: Change iobuf_pool_destroy to destroy all the arenas) posted (#12) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/9699 (libglusterfs: Change iobuf_pool_destroy to destroy all the arenas.) posted (#13) for review on master by Poornima G (pgurusid)
COMMIT: http://review.gluster.org/9691 committed in master by Shyamsundar Ranganathan (srangana) ------ commit 2acfbcf34ed77985905b2d122adbfd541eb01db1 Author: Poornima G <pgurusid> Date: Wed Feb 18 21:40:46 2015 +0530 event_pool: Add the code to destroy the poller threads and event pool gracefully. Change-Id: I49b6ceebb45773620c318fb5d20b81623db75ab6 BUG: 1093594 Signed-off-by: Poornima G <pgurusid> Reviewed-on: http://review.gluster.org/9691 Reviewed-by: Krishnan Parthasarathi <kparthas> Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Shyamsundar Ranganathan <srangana>
COMMIT: http://review.gluster.org/7642 committed in master by Shyamsundar Ranganathan (srangana) ------ commit 9c5011b8e49e34d736ba7cfadc0802e8b80682a7 Author: Poornima G <pgurusid> Date: Thu Feb 19 21:29:02 2015 +0530 glfs_fini: Clean up all the resources allocated in glfs_new. Initially even after calling glfs_fini(), all the threads created during init and many other resources like memory pool, iobuf pool, event pool and other memory allocs were not being freed. With this patch these resources are freed in glfs_fini(). The two thumb rules followed in this patch are: - The threads are not killed, they are made to exit voluntarily, once the queued tasks are completed. The main thread waits for the other threads to exit. - Free the memory pools and destroy the graphs only after all the other threads are stopped, so that there are less chances of hitting access after free. Resources freed and its order: 1. Destroy the inode table of all the graphs - Call forget on all the inodes. This will not be required when the cleanup during graph switch is implemented to perform inode table destroy. 2. Deactivate the current graph, call fini of all the xlators. 3. Syncenv destroy - Join the synctask threads and cleanup syncenv resources Sets the destroy mode, complete the existing synctasks, then join the synctask threads. After entering the destroy mode, -if a new synctask is submitted, it fails. -if syncenv_new() is called, it will end up creating new threads, but this is called only during init. 4. Poller thread destroy Register an event handler which sets the destroy mode for the poller. Once the poller is done processing all the events, it exits. 5. Tear down the logging framework The log file is closed and the log level is set to none, after this point no log messages appear either in log file or in stderr. 6. Destroy the timer thread Set the destroy bit, once the pending timer events are processed the timer thread exits. Note: Log infrastructure should be shutdown before destroying the timer thread as gf_log uses timers. 7. Destroy the glusterfs_ctx_t For all the graphs(active and passive), free graph, xlator structs and few other lists. Free the memory pools - iobuf pool, event pool, dict, logbuf pool, stub mem pool, stack mem pool, frame mem pool. Few things not addressed in this patch: 1. rpc_transport object not destroyed, the PARENT_DOWN should have destroyed this object but has not, needs to be addressed as a part of different patch 2. Each xlator fini should clean up the local pool allocated by its xlator. Needs to be addresses as a part of different patch. 3. Each xlator should implement forget to free its inode_ctx. Needs to be addresses as a part of different patch. 3. Few other leaks reported by valgrind. 4. fd and fd contexts The numbers: The resource usage by the test case in this patch: Without the fix, Memory: ~3GB; Threads: ~81 With this fix, Memory: 300MB; Threads: 1(main thread) Change-Id: I96b9277541737aa8372b4e6c9eed380cb871e7c2 BUG: 1093594 Signed-off-by: Poornima G <pgurusid> Reviewed-on: http://review.gluster.org/7642 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Rajesh Joseph <rjoseph> Reviewed-by: Raghavendra Talur <rtalur> Reviewed-by: Krishnan Parthasarathi <kparthas> Reviewed-by: Pranith Kumar Karampuri <pkarampu> Reviewed-by: Shyamsundar Ranganathan <srangana>
COMMIT: http://review.gluster.org/9699 committed in master by Shyamsundar Ranganathan (srangana) ------ commit a42db19a5dd796f5387b0765f5fc05cef462df22 Author: Poornima G <pgurusid> Date: Thu Feb 19 04:09:26 2015 +0530 libglusterfs: Change iobuf_pool_destroy to destroy all the arenas. Change-Id: If5e9d4ce98f845d3b52565ac62970959e663497f BUG: 1093594 Signed-off-by: Poornima G <pgurusid> Reviewed-on: http://review.gluster.org/9699 Reviewed-by: Raghavendra Talur <rtalur> Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Rajesh Joseph <rjoseph> Reviewed-by: Shyamsundar Ranganathan <srangana>
REVIEW: http://review.gluster.org/9871 (libglusterfs: Fix compile warning in inode.c) posted (#1) for review on master by Ravishankar N (ravishankar)
REVIEW: http://review.gluster.org/9961 (tests: move libgfapi tests to their own gfapi subdir) posted (#1) for review on master by Niels de Vos (ndevos)
COMMIT: http://review.gluster.org/9961 committed in master by Niels de Vos (ndevos) ------ commit 8992b549df536eaf5ff6315221f66191f2eff7ef Author: Niels de Vos <ndevos> Date: Sun Mar 22 12:13:42 2015 +0100 tests: move libgfapi tests to their own gfapi subdir Also, glfs_fini() is currently not stable yet, this test case causes many regression failures. The .t file has been renamed to .sh so that the test does not get run automatically, but can be run easily by hand. BUG: 1093594 Change-Id: I63fa4ddf798a505bc94d13d32dd02f22a9b7ab73 Signed-off-by: Niels de Vos <ndevos> Reviewed-on: http://review.gluster.org/9961 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Shyamsundar Ranganathan <srangana>
REVIEW: http://review.gluster.org/8092 (tests: Added a script to test the glfs_fini hang fix) posted (#4) for review on master by soumya k (skoduri)
REVIEW: http://review.gluster.org/8092 (tests: Added a script to test the glfs_fini hang fix) posted (#5) for review on master by soumya k (skoduri)
REVIEW: http://review.gluster.org/8092 (tests: Added a script to test the glfs_fini hang fix) posted (#6) for review on master by soumya k (skoduri)
REVIEW: http://review.gluster.org/8092 (tests: Added a script to test the glfs_fini hang fix) posted (#7) for review on master by soumya k (skoduri)
REVIEW: http://review.gluster.org/8092 (tests: Added a script to test the glfs_fini hang fix) posted (#8) for review on master by soumya k (skoduri)
COMMIT: http://review.gluster.org/8092 committed in master by Kaleb KEITHLEY (kkeithle) ------ commit bf8a5c9f21381690dc8ba25bcfba44619a6e2434 Author: Soumya Koduri <skoduri> Date: Tue Jun 17 17:44:11 2014 +0530 tests: Added a script to test the glfs_fini hang fix The script has been added as '.sh' instead of '.t' to avoid regression tests failure which may happen due to instability of glfs_fini. In addition, there is still some support needed to compile libgfapi* tests in our current regression test framework. BUG: 1093594 Change-Id: If66817f330320e8c092340008b36616e1d954127 Signed-off-by: Soumya Koduri <skoduri> Reviewed-on: http://review.gluster.org/8092 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Humble Devassy Chirammal <humble.devassy> Reviewed-by: Kaleb KEITHLEY <kkeithle>
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report. glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939 [2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user