Bug 1093594 - Glfs_fini() not freeing the resources
Summary: Glfs_fini() not freeing the resources
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: libgfapi
Version: pre-release
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Poornima G
QA Contact: Sudhir D
URL:
Whiteboard:
: 1005164 1061229 (view as bug list)
Depends On:
Blocks: 1104039 1134050 1142647 1143800
TreeView+ depends on / blocked
 
Reported: 2014-05-02 07:34 UTC by Poornima G
Modified: 2016-05-03 17:01 UTC (History)
12 users (show)

Fixed In Version: glusterfs-3.7.0
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1104039 (view as bug list)
Environment:
Last Closed: 2015-05-14 17:25:43 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:


Attachments (Terms of Use)
Valgrind report without this patch (170.81 KB, text/plain)
2015-01-28 12:18 UTC, Poornima G
no flags Details
Valgrind report with this patch (271.70 KB, text/plain)
2015-01-28 12:20 UTC, Poornima G
no flags Details

Description Poornima G 2014-05-02 07:34:39 UTC
Description of problem:
Glfs_fini, doesn't free the resources allocated in glfs_new

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Write a sample libgfapi application, the threads, memory and other resources are not freed after glfs_fini.
2.
3.

Actual results:


Expected results:
The resources should be freed.

Additional info:

Comment 1 Anand Avati 2014-05-02 07:35:22 UTC
REVIEW: http://review.gluster.org/7642 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#1) for review on master by Poornima G (pgurusid@redhat.com)

Comment 2 Anand Avati 2014-05-02 11:05:33 UTC
REVIEW: http://review.gluster.org/7642 (libgfapi: In glfs_fini() free all the resources allocated in glfs_new.) posted (#2) for review on master by Poornima G (pgurusid@redhat.com)

Comment 3 Anand Avati 2014-05-07 11:09:18 UTC
REVIEW: http://review.gluster.org/7642 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#3) for review on master by Poornima G (pgurusid@redhat.com)

Comment 4 Anand Avati 2014-05-13 13:03:29 UTC
REVIEW: http://review.gluster.org/7642 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#4) for review on master by Poornima G (pgurusid@redhat.com)

Comment 5 Anand Avati 2014-05-16 11:37:03 UTC
REVIEW: http://review.gluster.org/7642 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#5) for review on master by Poornima G (pgurusid@redhat.com)

Comment 6 Anand Avati 2014-05-19 12:30:55 UTC
REVIEW: http://review.gluster.org/7642 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#6) for review on master by Poornima G (pgurusid@redhat.com)

Comment 7 Anand Avati 2014-05-23 09:23:25 UTC
REVIEW: http://review.gluster.org/7642 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#7) for review on master by Poornima G (pgurusid@redhat.com)

Comment 8 Anand Avati 2014-05-27 07:09:23 UTC
REVIEW: http://review.gluster.org/7884 (tests: Add gfapi test case) posted (#1) for review on master by Poornima G (pgurusid@redhat.com)

Comment 9 Anand Avati 2014-05-27 09:42:21 UTC
REVIEW: http://review.gluster.org/7884 (tests: Add gfapi test case) posted (#2) for review on master by Poornima G (pgurusid@redhat.com)

Comment 10 Anand Avati 2014-06-03 04:09:28 UTC
REVIEW: http://review.gluster.org/7884 (tests: Add gfapi test case) posted (#3) for review on master by Poornima G (pgurusid@redhat.com)

Comment 11 Anand Avati 2014-06-06 07:07:38 UTC
REVIEW: http://review.gluster.org/7642 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#8) for review on master by Poornima G (pgurusid@redhat.com)

Comment 12 Anand Avati 2014-06-10 04:19:22 UTC
REVIEW: http://review.gluster.org/7642 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#9) for review on master by Poornima G (pgurusid@redhat.com)

Comment 13 santosh pradhan 2014-06-10 10:37:51 UTC
*** Bug 1005164 has been marked as a duplicate of this bug. ***

Comment 14 Niels de Vos 2014-06-24 16:14:09 UTC
*** Bug 1061229 has been marked as a duplicate of this bug. ***

Comment 15 Tiziano Müller 2014-09-27 17:47:40 UTC
I tried a naive backport to 3.5.x of the patch on review (that is: apply a96350fa2b68626b8592d5cbd67405e4d8416cca first and do not try to cleanup ctx->logbuf_pool), but when trying to take a snapshot using libvirt, I get the following:

[2014-09-27 17:34:05.886392] E [afr-common.c:4168:afr_notify] 0-virtualization-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
[2014-09-27 17:34:06.848988] E [client.c:2162:client_rpc_notify] 0-client: private structure of the xlator is NULL
[2014-09-27 17:34:06.849018] E [client.c:2162:client_rpc_notify] 0-client: private structure of the xlator is NULL
[2014-09-27 17:34:06.849169] E [io-cache.c:2080:fini] (-->/usr/lib64/libglusterfs.so.0(+0x1b485) [0x311ca247485] (-->/usr/lib64/libglusterfs.so.0(+0x1b485) [0x311ca247485] (-->/usr/lib64/libglusterfs.so.0(+0x1b4f1) [0x311ca2474f1]))) 0-: Assertion failed: list_empty (&table->inode_lru[i])
[2014-09-27 17:34:06.849196] E [io-cache.c:2083:fini] (-->/usr/lib64/libglusterfs.so.0(+0x1b485) [0x311ca247485] (-->/usr/lib64/libglusterfs.so.0(+0x1b485) [0x311ca247485] (-->/usr/lib64/libglusterfs.so.0(+0x1b4f1) [0x311ca2474f1]))) 0-: Assertion failed: list_empty (&table->inodes)
[2014-09-27 17:34:06.849654] E [iobuf.c:127:__iobuf_arena_destroy_iobufs] (-->/usr/lib64/libgfapi.so.0(glfs_fini+0x1e9) [0x311ca4cfa79] (-->/usr/lib64/libglusterfs.so.0(iobuf_pool_destroy+0x6f) [0x311ca27627f] (-->/usr/lib64/libglusterfs.so.0(__iobuf_arena_destroy+0x22) [0x311ca275c92]))) 0-: Assertion failed: iobuf->ref == 0
[2014-09-27 17:34:06.849690] E [iobuf.c:127:__iobuf_arena_destroy_iobufs] (-->/usr/lib64/libgfapi.so.0(glfs_fini+0x1e9) [0x311ca4cfa79] (-->/usr/lib64/libglusterfs.so.0(iobuf_pool_destroy+0x6f) [0x311ca27627f] (-->/usr/lib64/libglusterfs.so.0(__iobuf_arena_destroy+0x22) [0x311ca275c92]))) 0-: Assertion failed: iobuf->ref == 0
Formatting 'gluster://10.1.120.11/virtualization/vm-persistent/0f83f084-8080-413e-b558-b678e504836e/04cb1ecd-eeb7-46e1-88d5-c6c890f44b95.snap01.qcow2', fmt=qcow2 size=32212254720 backing_file='gluster://10.1.120.11:24007/virtualization/vm-persistent/0f83f084-8080-413e-b558-b678e504836e/04cb1ecd-eeb7-46e1-88d5-c6c890f44b95.qcow2' backing_fmt='qcow2' encryption=off cluster_size=65536 lazy_refcounts=off 
[2014-09-27 17:34:06.870307] E [afr-common.c:4168:afr_notify] 0-virtualization-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
[2014-09-27 17:34:07.851604] E [client.c:2162:client_rpc_notify] 0-client: private structure of the xlator is NULL
[2014-09-27 17:34:07.851634] E [client.c:2162:client_rpc_notify] 0-client: private structure of the xlator is NULL
[2014-09-27 17:34:07.851695] E [io-cache.c:2080:fini] (-->/usr/lib64/libglusterfs.so.0(+0x1b485) [0x311ca247485] (-->/usr/lib64/libglusterfs.so.0(+0x1b485) [0x311ca247485] (-->/usr/lib64/libglusterfs.so.0(+0x1b4f1) [0x311ca2474f1]))) 0-: Assertion failed: list_empty (&table->inode_lru[i])
[2014-09-27 17:34:07.851722] E [io-cache.c:2083:fini] (-->/usr/lib64/libglusterfs.so.0(+0x1b485) [0x311ca247485] (-->/usr/lib64/libglusterfs.so.0(+0x1b485) [0x311ca247485] (-->/usr/lib64/libglusterfs.so.0(+0x1b4f1) [0x311ca2474f1]))) 0-: Assertion failed: list_empty (&table->inodes)
[2014-09-27 17:34:07.851782] E [quick-read.c:1056:qr_inode_table_destroy] (-->/usr/lib64/libglusterfs.so.0(+0x1b485) [0x311ca247485] (-->/usr/lib64/libglusterfs.so.0(+0x1b4f1) [0x311ca2474f1] (-->/usr/lib64/glusterfs/3.5.2/xlator/performance/quick-read.so(fini+0x3a) [0x311bb7ce9ea]))) 0-: Assertion failed: list_empty (&priv->table.lru[i])

After which qemu gets unresponsive and gets terminated by libvirt and libvirt must be restarted since it is unable to acquire a state change lock.

Sorry, didn't have time to test with glusterfs-3.6 yet

Comment 16 Poornima G 2014-10-01 08:43:03 UTC
Looks like, glfs_fini should wait for CHILD_DOWN event from all the xlators and only then proceed with cleanup. Trying to reproduce with the latest code to validate the analysis.

Thank you for trying it out.

Comment 17 Anand Avati 2014-10-01 09:23:31 UTC
REVIEW: http://review.gluster.org/7642 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#10) for review on master by Poornima G (pgurusid@redhat.com)

Comment 18 Niels de Vos 2014-10-04 10:26:07 UTC
Tiziano, when you can test this patch successfully (or not), please go to http://review.gluster.org/7642, login, click 'review' and mark the change as 'Verified' (or Failed). This will give you credit on the patch when it gets merged (a Tested-by line gets added then). More verifications and reviews make it easier for the maintainers to accept the change.

Thanks!

Comment 19 Anand Avati 2015-01-13 11:15:32 UTC
REVIEW: http://review.gluster.org/7642 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#11) for review on master by Poornima G (pgurusid@redhat.com)

Comment 20 Anand Avati 2015-01-15 05:28:45 UTC
REVIEW: http://review.gluster.org/7642 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#12) for review on master by Poornima G (pgurusid@redhat.com)

Comment 21 Poornima G 2015-01-28 12:18:34 UTC
Created attachment 985112 [details]
Valgrind report without this patch

Comment 22 Poornima G 2015-01-28 12:20:13 UTC
Created attachment 985113 [details]
Valgrind report with this patch

Comment 23 Poornima G 2015-01-28 12:21:11 UTC
Current state of the patch:

Now with this patch, atleast 80-85% of the resource leaks are handled.
Below are the numbers of memory and threads usage for 20 loops of glfs_init() and glfs_fini() (without any file/data operations):

Without this patch:
- the virtual size of the process 3015mb
- ~80 threads leak.

With this patch:
- the virtual size of the process reduces to 314mb
- All the threads are joined.

For reference, attaching the valgrind report for both the cases.

There is quite a bit of memory that is not freed, two major places remaining for cleanup are:
- memory allocated by individual xlators(inode_ctx, fd_ctx, local pools) to be freed in each xlator.
- rpc_clnt object is not being destroyed even after PARENT_DOWN, something to be fixed.

Comment 24 Anand Avati 2015-01-29 12:12:49 UTC
REVIEW: http://review.gluster.org/7642 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#13) for review on master by Poornima G (pgurusid@redhat.com)

Comment 25 Anand Avati 2015-01-30 05:39:05 UTC
REVIEW: http://review.gluster.org/7642 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#14) for review on master by Poornima G (pgurusid@redhat.com)

Comment 26 Anand Avati 2015-02-04 11:32:59 UTC
REVIEW: http://review.gluster.org/9580 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#1) for review on master by Poornima G (pgurusid@redhat.com)

Comment 27 Anand Avati 2015-02-04 11:37:13 UTC
REVIEW: http://review.gluster.org/7642 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#15) for review on master by Poornima G (pgurusid@redhat.com)

Comment 28 Anand Avati 2015-02-18 11:35:00 UTC
REVIEW: http://review.gluster.org/9680 (protocol/client: defer cleanup of private until RPC notifications are handled.) posted (#1) for review on master by Krishnan Parthasarathi (kparthas@redhat.com)

Comment 29 Anand Avati 2015-02-18 15:00:29 UTC
REVIEW: http://review.gluster.org/7642 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#16) for review on master by Poornima G (pgurusid@redhat.com)

Comment 30 Anand Avati 2015-02-19 07:58:24 UTC
REVIEW: http://review.gluster.org/9693 (syncop: Add the code to destroy the synenv processors and         syncenv structures) posted (#1) for review on master by Poornima G (pgurusid@redhat.com)

Comment 31 Anand Avati 2015-02-19 10:22:02 UTC
REVIEW: http://review.gluster.org/9696 (libglusterfs: Add functions for xlator and graph cleanup.) posted (#1) for review on master by Poornima G (pgurusid@redhat.com)

Comment 32 Anand Avati 2015-02-19 10:55:16 UTC
REVIEW: http://review.gluster.org/9699 (libglusterfs: Change iobuf_pool_destroy to destroy all the arenas.) posted (#1) for review on master by Poornima G (pgurusid@redhat.com)

Comment 33 Anand Avati 2015-02-19 11:43:29 UTC
REVIEW: http://review.gluster.org/9700 (libglusterfs: Add inode table cleanup APIs.) posted (#1) for review on master by Poornima G (pgurusid@redhat.com)

Comment 34 Anand Avati 2015-02-19 12:02:39 UTC
REVIEW: http://review.gluster.org/9701 (libglusterfs: Add timer thread destroy code.) posted (#1) for review on master by Poornima G (pgurusid@redhat.com)

Comment 35 Anand Avati 2015-02-19 12:14:26 UTC
REVIEW: http://review.gluster.org/9699 (libglusterfs: Change iobuf_pool_destroy to destroy all the arenas.) posted (#2) for review on master by Poornima G (pgurusid@redhat.com)

Comment 36 Anand Avati 2015-02-19 13:23:26 UTC
REVIEW: http://review.gluster.org/9691 (event_pool: Add the code to destroy the poller threads and event pool gracefully.) posted (#3) for review on master by Poornima G (pgurusid@redhat.com)

Comment 37 Anand Avati 2015-02-20 04:01:53 UTC
REVIEW: http://review.gluster.org/9701 (libglusterfs: Add timer thread destroy code.) posted (#2) for review on master by Poornima G (pgurusid@redhat.com)

Comment 38 Anand Avati 2015-02-20 04:01:56 UTC
REVIEW: http://review.gluster.org/9700 (libglusterfs: Add inode table cleanup APIs.) posted (#2) for review on master by Poornima G (pgurusid@redhat.com)

Comment 39 Anand Avati 2015-02-20 04:01:59 UTC
REVIEW: http://review.gluster.org/9680 (protocol/client: defer cleanup of private until RPC notifications are handled.) posted (#2) for review on master by Poornima G (pgurusid@redhat.com)

Comment 40 Anand Avati 2015-02-20 04:02:02 UTC
REVIEW: http://review.gluster.org/9699 (libglusterfs: Change iobuf_pool_destroy to destroy all the arenas.) posted (#3) for review on master by Poornima G (pgurusid@redhat.com)

Comment 41 Anand Avati 2015-02-20 04:02:08 UTC
REVIEW: http://review.gluster.org/9696 (libglusterfs: Add functions for xlator and graph cleanup.) posted (#2) for review on master by Poornima G (pgurusid@redhat.com)

Comment 42 Anand Avati 2015-02-20 04:02:11 UTC
REVIEW: http://review.gluster.org/9693 (syncop: Add the code to destroy the synenv processors and         syncenv structures) posted (#2) for review on master by Poornima G (pgurusid@redhat.com)

Comment 43 Anand Avati 2015-02-20 04:02:14 UTC
REVIEW: http://review.gluster.org/7642 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#17) for review on master by Poornima G (pgurusid@redhat.com)

Comment 44 Anand Avati 2015-02-20 04:33:59 UTC
REVIEW: http://review.gluster.org/9700 (libglusterfs: Add inode table cleanup APIs.) posted (#3) for review on master by Poornima G (pgurusid@redhat.com)

Comment 45 Anand Avati 2015-02-20 04:34:53 UTC
REVIEW: http://review.gluster.org/7642 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#18) for review on master by Poornima G (pgurusid@redhat.com)

Comment 46 Anand Avati 2015-02-23 08:24:20 UTC
REVIEW: http://review.gluster.org/9693 (syncop: Add the code to destroy the synenv processors and         syncenv structures) posted (#3) for review on master by Poornima G (pgurusid@redhat.com)

Comment 47 Anand Avati 2015-02-23 10:37:41 UTC
REVIEW: http://review.gluster.org/9691 (event_pool: Add the code to destroy the poller threads and event pool gracefully.) posted (#4) for review on master by Poornima G (pgurusid@redhat.com)

Comment 48 Anand Avati 2015-02-23 10:44:30 UTC
REVIEW: http://review.gluster.org/9725 (epoll: Fix a leak in slot allocation.) posted (#1) for review on master by Poornima G (pgurusid@redhat.com)

Comment 49 Anand Avati 2015-02-24 05:28:08 UTC
REVIEW: http://review.gluster.org/9680 (protocol/client: defer cleanup of private until RPC notifications are handled.) posted (#3) for review on master by Krishnan Parthasarathi (kparthas@redhat.com)

Comment 50 Anand Avati 2015-02-24 12:43:12 UTC
REVIEW: http://review.gluster.org/9701 (libglusterfs: Add timer thread destroy code.) posted (#3) for review on master by Poornima G (pgurusid@redhat.com)

Comment 51 Anand Avati 2015-02-24 12:43:16 UTC
REVIEW: http://review.gluster.org/9700 (libglusterfs: Add inode table cleanup APIs.) posted (#4) for review on master by Poornima G (pgurusid@redhat.com)

Comment 52 Anand Avati 2015-02-24 12:43:24 UTC
REVIEW: http://review.gluster.org/9680 (protocol/client: defer cleanup of private until RPC notifications are handled.) posted (#4) for review on master by Poornima G (pgurusid@redhat.com)

Comment 53 Anand Avati 2015-02-24 12:43:26 UTC
REVIEW: http://review.gluster.org/9699 (libglusterfs: Change iobuf_pool_destroy to destroy all the arenas.) posted (#4) for review on master by Poornima G (pgurusid@redhat.com)

Comment 54 Anand Avati 2015-02-24 12:43:33 UTC
REVIEW: http://review.gluster.org/9696 (libglusterfs: Add functions for xlator and graph cleanup.) posted (#3) for review on master by Poornima G (pgurusid@redhat.com)

Comment 55 Anand Avati 2015-02-24 12:43:36 UTC
REVIEW: http://review.gluster.org/9693 (syncop: Add the code to destroy the synenv processors and         syncenv structures) posted (#4) for review on master by Poornima G (pgurusid@redhat.com)

Comment 56 Anand Avati 2015-02-24 12:43:39 UTC
REVIEW: http://review.gluster.org/7642 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#19) for review on master by Poornima G (pgurusid@redhat.com)

Comment 57 Anand Avati 2015-02-24 12:43:43 UTC
REVIEW: http://review.gluster.org/9691 (event_pool: Add the code to destroy the poller threads and event pool gracefully.) posted (#5) for review on master by Poornima G (pgurusid@redhat.com)

Comment 58 Anand Avati 2015-02-25 11:19:55 UTC
REVIEW: http://review.gluster.org/9701 (libglusterfs: Add timer thread destroy code.) posted (#4) for review on master by Poornima G (pgurusid@redhat.com)

Comment 59 Anand Avati 2015-02-25 11:19:59 UTC
REVIEW: http://review.gluster.org/9700 (libglusterfs: Add inode table cleanup APIs.) posted (#5) for review on master by Poornima G (pgurusid@redhat.com)

Comment 60 Anand Avati 2015-02-25 11:20:02 UTC
REVIEW: http://review.gluster.org/9680 (protocol/client: defer cleanup of private until RPC notifications are handled.) posted (#5) for review on master by Poornima G (pgurusid@redhat.com)

Comment 61 Anand Avati 2015-02-25 11:20:05 UTC
REVIEW: http://review.gluster.org/9699 (libglusterfs: Change iobuf_pool_destroy to destroy all the arenas.) posted (#5) for review on master by Poornima G (pgurusid@redhat.com)

Comment 62 Anand Avati 2015-02-25 11:20:16 UTC
REVIEW: http://review.gluster.org/9696 (libglusterfs: Add functions for xlator and graph cleanup.) posted (#4) for review on master by Poornima G (pgurusid@redhat.com)

Comment 63 Anand Avati 2015-02-25 11:20:19 UTC
REVIEW: http://review.gluster.org/9693 (syncop: Add the code to destroy the synenv processors and         syncenv structures) posted (#5) for review on master by Poornima G (pgurusid@redhat.com)

Comment 64 Anand Avati 2015-02-25 11:20:22 UTC
REVIEW: http://review.gluster.org/7642 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#20) for review on master by Poornima G (pgurusid@redhat.com)

Comment 65 Anand Avati 2015-02-25 11:20:25 UTC
REVIEW: http://review.gluster.org/9691 (event_pool: Add the code to destroy the poller threads and event pool gracefully.) posted (#6) for review on master by Poornima G (pgurusid@redhat.com)

Comment 66 Anand Avati 2015-02-27 05:37:56 UTC
REVIEW: http://review.gluster.org/9701 (libglusterfs: Add timer thread destroy code.) posted (#5) for review on master by Poornima G (pgurusid@redhat.com)

Comment 67 Anand Avati 2015-02-27 05:38:01 UTC
REVIEW: http://review.gluster.org/9700 (libglusterfs: Add inode table cleanup APIs.) posted (#6) for review on master by Poornima G (pgurusid@redhat.com)

Comment 68 Anand Avati 2015-02-27 05:38:06 UTC
REVIEW: http://review.gluster.org/9680 (protocol/client: defer cleanup of private until RPC notifications are handled.) posted (#6) for review on master by Poornima G (pgurusid@redhat.com)

Comment 69 Anand Avati 2015-02-27 05:38:15 UTC
REVIEW: http://review.gluster.org/9699 (libglusterfs: Change iobuf_pool_destroy to destroy all the arenas.) posted (#6) for review on master by Poornima G (pgurusid@redhat.com)

Comment 70 Anand Avati 2015-02-27 05:38:21 UTC
REVIEW: http://review.gluster.org/9696 (libglusterfs: Add functions for xlator and graph cleanup.) posted (#5) for review on master by Poornima G (pgurusid@redhat.com)

Comment 71 Anand Avati 2015-02-27 05:38:25 UTC
REVIEW: http://review.gluster.org/9693 (syncop: Add the code to destroy the synenv processors and         syncenv structures) posted (#6) for review on master by Poornima G (pgurusid@redhat.com)

Comment 72 Anand Avati 2015-02-27 05:38:28 UTC
REVIEW: http://review.gluster.org/7642 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#21) for review on master by Poornima G (pgurusid@redhat.com)

Comment 73 Anand Avati 2015-02-27 05:38:35 UTC
REVIEW: http://review.gluster.org/9691 (event_pool: Add the code to destroy the poller threads and event pool gracefully.) posted (#7) for review on master by Poornima G (pgurusid@redhat.com)

Comment 74 Anand Avati 2015-02-27 11:01:30 UTC
REVIEW: http://review.gluster.org/9701 (libglusterfs: Add timer thread destroy code.) posted (#6) for review on master by Poornima G (pgurusid@redhat.com)

Comment 75 Anand Avati 2015-02-27 11:01:34 UTC
REVIEW: http://review.gluster.org/9700 (libglusterfs: Add inode table cleanup APIs.) posted (#7) for review on master by Poornima G (pgurusid@redhat.com)

Comment 76 Anand Avati 2015-02-27 11:01:37 UTC
REVIEW: http://review.gluster.org/9699 (libglusterfs: Change iobuf_pool_destroy to destroy all the arenas.) posted (#7) for review on master by Poornima G (pgurusid@redhat.com)

Comment 77 Anand Avati 2015-02-27 11:01:40 UTC
REVIEW: http://review.gluster.org/9680 (protocol/client: defer cleanup of private until RPC notifications are handled.) posted (#7) for review on master by Poornima G (pgurusid@redhat.com)

Comment 78 Anand Avati 2015-02-27 11:01:44 UTC
REVIEW: http://review.gluster.org/9696 (libglusterfs: Add functions for xlator and graph cleanup.) posted (#6) for review on master by Poornima G (pgurusid@redhat.com)

Comment 79 Anand Avati 2015-02-27 11:01:47 UTC
REVIEW: http://review.gluster.org/9693 (syncop: Add the code to destroy the synenv processors and         syncenv structures) posted (#7) for review on master by Poornima G (pgurusid@redhat.com)

Comment 80 Anand Avati 2015-02-27 11:01:50 UTC
REVIEW: http://review.gluster.org/7642 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#22) for review on master by Poornima G (pgurusid@redhat.com)

Comment 81 Anand Avati 2015-02-27 11:01:54 UTC
REVIEW: http://review.gluster.org/9691 (event_pool: Add the code to destroy the poller threads and event pool gracefully.) posted (#8) for review on master by Poornima G (pgurusid@redhat.com)

Comment 82 Anand Avati 2015-03-02 08:56:10 UTC
COMMIT: http://review.gluster.org/9725 committed in master by Vijay Bellur (vbellur@redhat.com) 
------
commit 60cff8ab4f5fa2ee7bea028937ba23674e1ee0cc
Author: Poornima G <pgurusid@redhat.com>
Date:   Mon Feb 23 16:10:33 2015 +0530

    epoll: Fix a leak in slot allocation.
    
    When the slots_used in a table becomes 0, the table will not
    get reused, leading to a leak.
    
    This patch fixes the leak.
    
    Change-Id: Ib86826d287368174ea7ebe0d0d64b2dec574634e
    BUG: 1093594
    Signed-off-by: Poornima G <pgurusid@redhat.com>
    Reviewed-on: http://review.gluster.org/9725
    Reviewed-by: Krishnan Parthasarathi <kparthas@redhat.com>
    Reviewed-by: Prashanth Pai <ppai@redhat.com>
    Reviewed-by: Shyamsundar Ranganathan <srangana@redhat.com>
    Tested-by: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Vijay Bellur <vbellur@redhat.com>

Comment 83 Anand Avati 2015-03-02 09:16:55 UTC
COMMIT: http://review.gluster.org/9680 committed in master by Vijay Bellur (vbellur@redhat.com) 
------
commit efe09fff57dc01088168e60bff355b769c1bc7f7
Author: Krishnan Parthasarathi <kparthas@redhat.com>
Date:   Wed Feb 18 15:16:17 2015 +0530

    protocol/client: defer cleanup of private until RPC notifications are handled.
    
    This fix is required for glfs_fini to be able to perform fini on client
    xlators in a graph. We are deferring freeing of client xlator's private
    until all RPC related resources are destroyed. This guarantees that
    client xlator would free RPC related resources provided its private
    structures are still accessible via its this pointer.
    
    'Weak' property: If there are no epoll threads executing after calling
    fini() on a client xlator, then all its RPC related resources are
    guaranteed to be freed. We can now free the corresponding 'this'
    pointer.
    
    Change-Id: Ie00b14dda096ac128e1c37e0032f07d17fd701ce
    BUG: 1093594
    Signed-off-by: Krishnan Parthasarathi <kparthas@redhat.com>
    Reviewed-on: http://review.gluster.org/9680
    Reviewed-by: Rajesh Joseph <rjoseph@redhat.com>
    Tested-by: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Vijay Bellur <vbellur@redhat.com>

Comment 84 Anand Avati 2015-03-02 09:38:26 UTC
REVIEW: http://review.gluster.org/9699 (libglusterfs: Change iobuf_pool_destroy to destroy all the arenas.) posted (#8) for review on master by Poornima G (pgurusid@redhat.com)

Comment 85 Anand Avati 2015-03-02 11:52:16 UTC
REVIEW: http://review.gluster.org/9691 (event_pool: Add the code to destroy the poller threads and event pool gracefully.) posted (#9) for review on master by Poornima G (pgurusid@redhat.com)

Comment 86 Anand Avati 2015-03-02 13:29:07 UTC
REVIEW: http://review.gluster.org/9700 (libglusterfs: Add inode table cleanup APIs.) posted (#8) for review on master by Poornima G (pgurusid@redhat.com)

Comment 87 Anand Avati 2015-03-02 13:46:39 UTC
REVIEW: http://review.gluster.org/9693 (syncop: Add the code to destroy the synenv processors and         syncenv structures) posted (#8) for review on master by Poornima G (pgurusid@redhat.com)

Comment 88 Anand Avati 2015-03-02 13:50:25 UTC
REVIEW: http://review.gluster.org/9693 (syncop: Add the code to destroy the synenv processors and         syncenv structures) posted (#9) for review on master by Poornima G (pgurusid@redhat.com)

Comment 89 Anand Avati 2015-03-02 14:02:57 UTC
REVIEW: http://review.gluster.org/9699 (libglusterfs: Change iobuf_pool_destroy to destroy all the arenas.) posted (#9) for review on master by Poornima G (pgurusid@redhat.com)

Comment 90 Anand Avati 2015-03-02 14:20:26 UTC
REVIEW: http://review.gluster.org/9699 (libglusterfs: Change iobuf_pool_destroy to destroy all the arenas.) posted (#10) for review on master by Poornima G (pgurusid@redhat.com)

Comment 91 Anand Avati 2015-03-02 14:40:39 UTC
REVIEW: http://review.gluster.org/9700 (libglusterfs: Add inode table cleanup APIs.) posted (#9) for review on master by Poornima G (pgurusid@redhat.com)

Comment 92 Anand Avati 2015-03-03 04:17:27 UTC
REVIEW: http://review.gluster.org/9699 (libglusterfs: Change iobuf_pool_destroy to destroy all the arenas.) posted (#11) for review on master by Poornima G (pgurusid@redhat.com)

Comment 93 Anand Avati 2015-03-03 04:28:53 UTC
COMMIT: http://review.gluster.org/9700 committed in master by Vijay Bellur (vbellur@redhat.com) 
------
commit 5e12c658d6c2dc92fd336b745036641f054b3a40
Author: Poornima G <pgurusid@redhat.com>
Date:   Thu Feb 19 04:49:35 2015 +0530

    libglusterfs: Add inode table cleanup APIs.
    
    Took the inode context free code from the patch
    http://review.gluster.org/#/c/4775/18/libglusterfs/src/inode.c
    
    Change-Id: I05fc025763fe4ce61dc61503de27ec1d3a203e50
    BUG: 1093594
    Signed-off-by: Poornima G <pgurusid@redhat.com>
    Reviewed-on: http://review.gluster.org/9700
    Reviewed-by: Raghavendra G <rgowdapp@redhat.com>
    Reviewed-by: Vijay Bellur <vbellur@redhat.com>
    Tested-by: Gluster Build System <jenkins@build.gluster.com>

Comment 94 Anand Avati 2015-03-03 04:30:58 UTC
COMMIT: http://review.gluster.org/9693 committed in master by Vijay Bellur (vbellur@redhat.com) 
------
commit 84e6eac265c87be5e436f3e7a41f40dc2d9c65c5
Author: Poornima G <pgurusid@redhat.com>
Date:   Thu Feb 19 01:24:29 2015 +0530

    syncop: Add the code to destroy the synenv processors and
            syncenv structures
    
    Change-Id: I28020eb2fc08d886cd7c05ff96daf7ebb4264ffe
    BUG: 1093594
    Signed-off-by: Poornima G <pgurusid@redhat.com>
    Reviewed-on: http://review.gluster.org/9693
    Tested-by: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Vijay Bellur <vbellur@redhat.com>

Comment 95 Anand Avati 2015-03-03 04:32:03 UTC
COMMIT: http://review.gluster.org/9701 committed in master by Vijay Bellur (vbellur@redhat.com) 
------
commit 7e416b6d00d626219c8d2067be720a915c4f85b1
Author: Poornima G <pgurusid@redhat.com>
Date:   Thu Feb 19 05:29:02 2015 +0530

    libglusterfs: Add timer thread destroy code.
    
    Change-Id: Iafbbbfd9319751742b3c79419e1dd8e2958fee07
    BUG: 1093594
    Signed-off-by: Poornima G <pgurusid@redhat.com>
    Reviewed-on: http://review.gluster.org/9701
    Reviewed-by: Rajesh Joseph <rjoseph@redhat.com>
    Reviewed-by: Krishnan Parthasarathi <kparthas@redhat.com>
    Tested-by: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Vijay Bellur <vbellur@redhat.com>

Comment 96 Anand Avati 2015-03-03 04:34:43 UTC
COMMIT: http://review.gluster.org/9696 committed in master by Vijay Bellur (vbellur@redhat.com) 
------
commit fc54f75ea49605e7fb5808e3fc01dfaa6b7c4649
Author: Poornima G <pgurusid@redhat.com>
Date:   Thu Feb 19 03:45:34 2015 +0530

    libglusterfs: Add functions for xlator and graph cleanup.
    
    Change-Id: If341e3c0a559aa5bbca9c1263a241c6592c59706
    BUG: 1093594
    Signed-off-by: Poornima G <pgurusid@redhat.com>
    Reviewed-on: http://review.gluster.org/9696
    Reviewed-by: Rajesh Joseph <rjoseph@redhat.com>
    Tested-by: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Vijay Bellur <vbellur@redhat.com>

Comment 97 Anand Avati 2015-03-03 06:40:09 UTC
REVIEW: http://review.gluster.org/9691 (event_pool: Add the code to destroy the poller threads and event pool gracefully.) posted (#10) for review on master by Poornima G (pgurusid@redhat.com)

Comment 98 Anand Avati 2015-03-03 06:45:48 UTC
REVIEW: http://review.gluster.org/9691 (event_pool: Add the code to destroy the poller threads and event pool gracefully.) posted (#11) for review on master by Poornima G (pgurusid@redhat.com)

Comment 99 Anand Avati 2015-03-03 07:26:48 UTC
REVIEW: http://review.gluster.org/7642 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#23) for review on master by Poornima G (pgurusid@redhat.com)

Comment 100 Anand Avati 2015-03-03 07:59:24 UTC
REVIEW: http://review.gluster.org/7642 (glfs_fini: Clean up all the resources allocated in glfs_new.) posted (#24) for review on master by Poornima G (pgurusid@redhat.com)

Comment 101 Anand Avati 2015-03-03 08:04:21 UTC
REVIEW: http://review.gluster.org/9691 (event_pool: Add the code to destroy the poller threads and event pool gracefully.) posted (#12) for review on master by Poornima G (pgurusid@redhat.com)

Comment 102 Anand Avati 2015-03-03 09:59:33 UTC
REVIEW: http://review.gluster.org/9699 (libglusterfs: Change iobuf_pool_destroy to destroy all the arenas) posted (#12) for review on master by Poornima G (pgurusid@redhat.com)

Comment 103 Anand Avati 2015-03-03 10:06:20 UTC
REVIEW: http://review.gluster.org/9699 (libglusterfs: Change iobuf_pool_destroy to destroy all the arenas.) posted (#13) for review on master by Poornima G (pgurusid@redhat.com)

Comment 104 Anand Avati 2015-03-03 14:45:25 UTC
COMMIT: http://review.gluster.org/9691 committed in master by Shyamsundar Ranganathan (srangana@redhat.com) 
------
commit 2acfbcf34ed77985905b2d122adbfd541eb01db1
Author: Poornima G <pgurusid@redhat.com>
Date:   Wed Feb 18 21:40:46 2015 +0530

    event_pool: Add the code to destroy the poller threads and event pool gracefully.
    
    Change-Id: I49b6ceebb45773620c318fb5d20b81623db75ab6
    BUG: 1093594
    Signed-off-by: Poornima G <pgurusid@redhat.com>
    Reviewed-on: http://review.gluster.org/9691
    Reviewed-by: Krishnan Parthasarathi <kparthas@redhat.com>
    Tested-by: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Shyamsundar Ranganathan <srangana@redhat.com>

Comment 105 Anand Avati 2015-03-04 15:15:17 UTC
COMMIT: http://review.gluster.org/7642 committed in master by Shyamsundar Ranganathan (srangana@redhat.com) 
------
commit 9c5011b8e49e34d736ba7cfadc0802e8b80682a7
Author: Poornima G <pgurusid@redhat.com>
Date:   Thu Feb 19 21:29:02 2015 +0530

    glfs_fini: Clean up all the resources allocated in glfs_new.
    
    Initially even after calling glfs_fini(), all the threads created
    during init and many other resources like memory pool, iobuf pool,
    event pool and other memory allocs were not being freed.
    With this patch these resources are freed in glfs_fini().
    
    The two thumb rules followed in this patch are:
    - The threads are not killed, they are made to exit voluntarily,
      once the queued tasks are completed. The main thread waits for
      the other threads to exit.
    - Free the memory pools and destroy the graphs only after all the
      other threads are stopped, so that there are less chances of
      hitting access after free.
    
    Resources freed and its order:
    1. Destroy the inode table of all the graphs - Call forget on all the inodes.
       This will not be required when the cleanup during graph switch is
       implemented to perform inode table destroy.
    2. Deactivate the current graph, call fini of all the xlators.
    3. Syncenv destroy - Join the synctask threads and cleanup syncenv resources
       Sets the destroy mode, complete the existing synctasks, then join the
       synctask threads.
       After entering the destroy mode,
            -if a new synctask is submitted, it fails.
            -if syncenv_new() is called, it will end up creating new threads,
            but this is called only during init.
    4. Poller thread destroy
       Register an event handler which sets the destroy mode for the poller.
       Once the poller is done processing all the events, it exits.
    5. Tear down the logging framework
       The log file is closed and the log level is set to none, after this
       point no log messages appear either in log file or in stderr.
    6. Destroy the timer thread
       Set the destroy bit, once the pending timer events are processed
       the timer thread exits.
       Note: Log infrastructure should be shutdown before destroying the timer
       thread as gf_log uses timers.
    7. Destroy the glusterfs_ctx_t
       For all the graphs(active and passive), free graph, xlator structs and few other lists.
       Free the memory pools - iobuf pool, event pool, dict, logbuf pool,
       stub mem pool, stack mem pool, frame mem pool.
    
    Few things not addressed in this patch:
    1. rpc_transport object not destroyed, the PARENT_DOWN should have
       destroyed this object but has not, needs to be addressed as a part
       of different patch
    2. Each xlator fini should clean up the local pool allocated by its xlator.
       Needs to be addresses as a part of different patch.
    3. Each xlator should implement forget to free its inode_ctx.
       Needs to be addresses as a part of different patch.
    3. Few other leaks reported by valgrind.
    4. fd and fd contexts
    
    The numbers:
    The resource usage by the test case in this patch:
    Without the fix, Memory: ~3GB; Threads: ~81
    With this fix, Memory: 300MB; Threads: 1(main thread)
    
    Change-Id: I96b9277541737aa8372b4e6c9eed380cb871e7c2
    BUG: 1093594
    Signed-off-by: Poornima G <pgurusid@redhat.com>
    Reviewed-on: http://review.gluster.org/7642
    Tested-by: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Rajesh Joseph <rjoseph@redhat.com>
    Reviewed-by: Raghavendra Talur <rtalur@redhat.com>
    Reviewed-by: Krishnan Parthasarathi <kparthas@redhat.com>
    Reviewed-by: Pranith Kumar Karampuri <pkarampu@redhat.com>
    Reviewed-by: Shyamsundar Ranganathan <srangana@redhat.com>

Comment 106 Anand Avati 2015-03-04 17:15:15 UTC
COMMIT: http://review.gluster.org/9699 committed in master by Shyamsundar Ranganathan (srangana@redhat.com) 
------
commit a42db19a5dd796f5387b0765f5fc05cef462df22
Author: Poornima G <pgurusid@redhat.com>
Date:   Thu Feb 19 04:09:26 2015 +0530

    libglusterfs: Change iobuf_pool_destroy to destroy all the arenas.
    
    Change-Id: If5e9d4ce98f845d3b52565ac62970959e663497f
    BUG: 1093594
    Signed-off-by: Poornima G <pgurusid@redhat.com>
    Reviewed-on: http://review.gluster.org/9699
    Reviewed-by: Raghavendra Talur <rtalur@redhat.com>
    Tested-by: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Rajesh Joseph <rjoseph@redhat.com>
    Reviewed-by: Shyamsundar Ranganathan <srangana@redhat.com>

Comment 107 Anand Avati 2015-03-13 02:57:19 UTC
REVIEW: http://review.gluster.org/9871 (libglusterfs: Fix compile warning in inode.c) posted (#1) for review on master by Ravishankar N (ravishankar@redhat.com)

Comment 108 Anand Avati 2015-03-22 11:19:20 UTC
REVIEW: http://review.gluster.org/9961 (tests: move libgfapi tests to their own gfapi subdir) posted (#1) for review on master by Niels de Vos (ndevos@redhat.com)

Comment 109 Anand Avati 2015-03-23 15:24:35 UTC
COMMIT: http://review.gluster.org/9961 committed in master by Niels de Vos (ndevos@redhat.com) 
------
commit 8992b549df536eaf5ff6315221f66191f2eff7ef
Author: Niels de Vos <ndevos@redhat.com>
Date:   Sun Mar 22 12:13:42 2015 +0100

    tests: move libgfapi tests to their own gfapi subdir
    
    Also, glfs_fini() is currently not stable yet, this test case causes
    many regression failures. The .t file has been renamed to .sh so that
    the test does not get run automatically, but can be run easily by hand.
    
    BUG: 1093594
    Change-Id: I63fa4ddf798a505bc94d13d32dd02f22a9b7ab73
    Signed-off-by: Niels de Vos <ndevos@redhat.com>
    Reviewed-on: http://review.gluster.org/9961
    Tested-by: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Shyamsundar Ranganathan <srangana@redhat.com>

Comment 110 Anand Avati 2015-03-26 07:59:31 UTC
REVIEW: http://review.gluster.org/8092 (tests: Added a script to test the glfs_fini hang fix) posted (#4) for review on master by soumya k (skoduri@redhat.com)

Comment 111 Anand Avati 2015-03-26 09:55:46 UTC
REVIEW: http://review.gluster.org/8092 (tests: Added a script to test the glfs_fini hang fix) posted (#5) for review on master by soumya k (skoduri@redhat.com)

Comment 112 Anand Avati 2015-03-26 12:15:40 UTC
REVIEW: http://review.gluster.org/8092 (tests: Added a script to test the glfs_fini hang fix) posted (#6) for review on master by soumya k (skoduri@redhat.com)

Comment 113 Anand Avati 2015-03-26 12:15:57 UTC
REVIEW: http://review.gluster.org/8092 (tests: Added a script to test the glfs_fini hang fix) posted (#7) for review on master by soumya k (skoduri@redhat.com)

Comment 114 Anand Avati 2015-03-26 12:19:29 UTC
REVIEW: http://review.gluster.org/8092 (tests: Added a script to test the glfs_fini hang fix) posted (#8) for review on master by soumya k (skoduri@redhat.com)

Comment 115 Anand Avati 2015-03-26 19:26:42 UTC
COMMIT: http://review.gluster.org/8092 committed in master by Kaleb KEITHLEY (kkeithle@redhat.com) 
------
commit bf8a5c9f21381690dc8ba25bcfba44619a6e2434
Author: Soumya Koduri <skoduri@redhat.com>
Date:   Tue Jun 17 17:44:11 2014 +0530

    tests: Added a script to test the glfs_fini hang fix
    
    The script has been added as '.sh' instead of '.t' to avoid regression
    tests failure which may happen due to instability of glfs_fini.
    
    In addition, there is still some support needed to compile libgfapi*
    tests in our current regression test framework.
    
    BUG: 1093594
    Change-Id: If66817f330320e8c092340008b36616e1d954127
    Signed-off-by: Soumya Koduri <skoduri@redhat.com>
    Reviewed-on: http://review.gluster.org/8092
    Tested-by: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Humble Devassy Chirammal <humble.devassy@gmail.com>
    Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>

Comment 116 Niels de Vos 2015-05-14 17:25:43 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 117 Niels de Vos 2015-05-14 17:35:26 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 118 Niels de Vos 2015-05-14 17:37:48 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 119 Niels de Vos 2015-05-14 17:42:18 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user


Note You need to log in before you can comment on or make changes to this bug.