Created attachment 574469 [details] glusterfs process statedump. Description of problem: statedump output - [mallinfo] mallinfo_arena=2080006144 mallinfo_ordblks=98988 mallinfo_smblks=4 mallinfo_hblks=13 mallinfo_hblkhd=18313216 mallinfo_usmblks=0 mallinfo_fsmblks=400 mallinfo_uordblks=2076200960 mallinfo_fordblks=3805184 mallinfo_keepcost=29040 pool-name=fuse:fd_t hot-count=0 cold-count=1024 padded_sizeof=100 alloc-count=6911 max-alloc=1 pool-misses=0 max-stdalloc=0 -----=----- pool-name=fuse:dentry_t hot-count=1672 cold-count=31096 padded_sizeof=84 alloc-count=120694 max-alloc=32768 pool-misses=2072 max-stdalloc=2072 -----=----- pool-name=fuse:inode_t hot-count=1674 cold-count=31094 padded_sizeof=148 alloc-count=249800 max-alloc=32768 pool-misses=4273 max-stdalloc=2074 -----=----- pool-name=master-client-0:struct saved_frame hot-count=1 cold-count=511 padded_sizeof=124 alloc-count=186227 max-alloc=4 pool-misses=0 max-stdalloc=0 -----=----- pool-name=master-client-0:struct rpc_req hot-count=1 cold-count=511 padded_sizeof=2236 alloc-count=186227 max-alloc=4 pool-misses=0 max-stdalloc=0 pool-name=master-client-0:clnt_local_t hot-count=1 cold-count=63 padded_sizeof=1284 alloc-count=179163 max-alloc=3 pool-misses=0 max-stdalloc=0 -----=----- pool-name=master-client-1:struct saved_frame hot-count=0 cold-count=512 padded_sizeof=124 alloc-count=187084 max-alloc=4 pool-misses=0 max-stdalloc=0 -----=----- pool-name=master-client-1:struct rpc_req hot-count=0 cold-count=512 padded_sizeof=2236 alloc-count=187084 max-alloc=4 pool-misses=0 max-stdalloc=0 -----=----- pool-name=master-client-1:clnt_local_t hot-count=0 cold-count=64 padded_sizeof=1284 alloc-count=180021 max-alloc=3 pool-misses=0 max-stdalloc=0
mostly looks like some fragmentation looking at uord_blks and ord_blks
Next time you run these set of tests, please run it through Valgrind, so we can capture the leaks well. One of the possibility is quick-read's dictionary getting cached in md-cache, which can lead to a huge leak (Thanks to Brian Foster/Avati on the md-cache/quick-read causing memory consumption)
> One of the possibility is quick-read's dictionary getting cached in md-cache, > which can lead to a huge leak (Thanks to Brian Foster/Avati on the > md-cache/quick-read causing memory consumption) Correction: Thanks to Brian Foster and Avati on *finding* the memory consumption issue when md-cache and quick-read are used together.
Taking this out of Beta Blocker considering multiple patches which have gone into fix the obvious memory leaks. Only seriously pending tasks would be to handle md-cache/quick-read memory consumption behavior, for which Brian Foster already sent a patch.
the patch to handle quick-read/md-cache dict's memory consumption has gone in. http://review.gluster.com/3268