Description of problem: Glusterd mem leak when run command gluster volume status Version-Release number of selected component (if applicable): v3.6.9 How reproducible: Steps to Reproduce: 1. Build a gluster cluster 2. Create a volume with multiple bricks, and start it 3. Run the command gluster volume status several time 4. Check the memory of the glusterd process all the time Actual results: the memory of glusterd process grow all the time. Expected results: The memory is kept a small range Additional info: The dump file is below: DUMP-START-TIME: 2016-08-23 09:11:42.543685 [mallinfo] mallinfo_arena=197533696 mallinfo_ordblks=85725 mallinfo_smblks=337 mallinfo_hblks=9 mallinfo_hblkhd=11153408 mallinfo_usmblks=0 mallinfo_fsmblks=31568 mallinfo_uordblks=193679680 mallinfo_fordblks=3854016 mallinfo_keepcost=134512 [global.glusterfs - Memory usage] num_types=122 [global.glusterfs - usage-type gf_common_mt_event_pool memusage] size=144 num_allocs=1 max_size=144 max_num_allocs=1 total_allocs=1 [global.glusterfs - usage-type gf_common_mt_reg memusage] size=393216 num_allocs=1 max_size=393216 max_num_allocs=1 total_allocs=1 [global.glusterfs - usage-type gf_common_mt_epoll_event memusage] size=3132 num_allocs=1 max_size=3132 max_num_allocs=1 total_allocs=1 [global.glusterfs - usage-type gf_common_mt_xlator_t memusage] size=2600 num_allocs=1 max_size=2600 max_num_allocs=1 total_allocs=1 [global.glusterfs - usage-type gf_common_mt_volume_opt_list_t memusage] size=24 num_allocs=1 max_size=24 max_num_allocs=1 total_allocs=1 [global.glusterfs - usage-type gf_common_mt_gf_timer_t memusage] size=56 num_allocs=1 max_size=56 max_num_allocs=1 total_allocs=32 [global.glusterfs - usage-type gf_common_mt_gf_timer_registry_t memusage] size=168 num_allocs=1 max_size=168 max_num_allocs=1 total_allocs=1 [global.glusterfs - usage-type gf_common_mt_iobuf memusage] size=109536 num_allocs=8 max_size=109536 max_num_allocs=8 total_allocs=8 [global.glusterfs - usage-type gf_common_mt_iobuf_arena memusage] size=1872 num_allocs=9 max_size=1872 max_num_allocs=9 total_allocs=9 [global.glusterfs - usage-type gf_common_mt_iobuf_pool memusage] size=1776 num_allocs=1 max_size=1776 max_num_allocs=1 total_allocs=1 [global.glusterfs - usage-type gf_common_mt_asprintf memusage] size=225 num_allocs=9 max_size=320 max_num_allocs=10 total_allocs=11 [global.glusterfs - usage-type gf_common_mt_strdup memusage] size=275 num_allocs=13 max_size=457 max_num_allocs=20 total_allocs=33 [global.glusterfs - usage-type gf_common_mt_char memusage] size=145 num_allocs=6 max_size=24576 max_num_allocs=6 total_allocs=9 [global.glusterfs - usage-type gf_common_mt_mem_pool memusage] size=840 num_allocs=7 max_size=840 max_num_allocs=7 total_allocs=7 [global.glusterfs - usage-type gf_common_mt_long memusage] size=9006080 num_allocs=7 max_size=9006080 max_num_allocs=7 total_allocs=7 [global.glusterfs - usage-type gf_common_mt_glusterfs_graph_t memusage] size=192 num_allocs=1 max_size=192 max_num_allocs=1 total_allocs=1 [global.glusterfs - usage-type gf_common_mt_cliententry_t memusage] size=2048 num_allocs=1 max_size=2048 max_num_allocs=1 total_allocs=1 [global.glusterfs - usage-type gf_common_mt_clienttable_t memusage] size=32 num_allocs=1 max_size=32 max_num_allocs=1 total_allocs=1 [global.glusterfs - usage-type gfd_mt_call_pool_t memusage] size=48 num_allocs=1 max_size=48 max_num_allocs=1 total_allocs=1 [mempool] -----=----- pool-name=management:struct saved_frame hot-count=0 cold-count=16 padded_sizeof=124 alloc-count=0 max-alloc=0 pool-misses=0 cur-stdalloc=0 max-stdalloc=0 -----=----- pool-name=management:struct rpc_req hot-count=0 cold-count=16 padded_sizeof=588 alloc-count=0 max-alloc=0 pool-misses=0 cur-stdalloc=0 max-stdalloc=0 -----=----- pool-name=management:struct saved_frame hot-count=0 cold-count=16 padded_sizeof=124 alloc-count=0 max-alloc=0 pool-misses=0 cur-stdalloc=0 max-stdalloc=0 -----=----- pool-name=management:struct rpc_req hot-count=0 cold-count=16 padded_sizeof=588 alloc-count=0 max-alloc=0 pool-misses=0 cur-stdalloc=0 max-stdalloc=0 -----=----- pool-name=management:struct saved_frame hot-count=0 cold-count=16 padded_sizeof=124 alloc-count=10 max-alloc=3 pool-misses=0 cur-stdalloc=0 max-stdalloc=0 -----=----- pool-name=management:struct rpc_req hot-count=0 cold-count=16 padded_sizeof=588 alloc-count=10 max-alloc=3 pool-misses=0 cur-stdalloc=0 max-stdalloc=0 -----=----- pool-name=management:struct saved_frame hot-count=0 cold-count=16 padded_sizeof=124 alloc-count=12 max-alloc=3 pool-misses=0 cur-stdalloc=0 max-stdalloc=0 -----=----- pool-name=management:struct rpc_req hot-count=0 cold-count=16 padded_sizeof=588 alloc-count=12 max-alloc=3 pool-misses=0 cur-stdalloc=0 max-stdalloc=0 -----=----- pool-name=management:struct saved_frame hot-count=0 cold-count=16 padded_sizeof=124 alloc-count=9 max-alloc=3 pool-misses=0 cur-stdalloc=0 max-stdalloc=0 -----=----- pool-name=management:struct rpc_req hot-count=0 cold-count=16 padded_sizeof=588 alloc-count=9 max-alloc=3 pool-misses=0 cur-stdalloc=0 max-stdalloc=0 -----=----- pool-name=management:rpcsvc_request_t hot-count=0 cold-count=8 padded_sizeof=2828 alloc-count=0 max-alloc=0 pool-misses=0 cur-stdalloc=0 max-stdalloc=0 -----=----- pool-name=management:rpcsvc_request_t hot-count=1 cold-count=63 padded_sizeof=2828 alloc-count=485477 max-alloc=2 pool-misses=0 cur-stdalloc=0 max-stdalloc=0 -----=----- pool-name=glusterfs:log_buf_t hot-count=0 cold-count=256 padded_sizeof=140 alloc-count=3 max-alloc=2 pool-misses=0 cur-stdalloc=0 max-stdalloc=0 -----=----- pool-name=glusterfs:data_t hot-count=16384 cold-count=0 padded_sizeof=52 alloc-count=9191727 max-alloc=16384 pool-misses=8881844 cur-stdalloc=469415 max-stdalloc=469491 -----=----- pool-name=glusterfs:data_pair_t hot-count=16384 cold-count=0 padded_sizeof=68 alloc-count=8059283 max-alloc=16384 pool-misses=7732988 cur-stdalloc=388449 max-stdalloc=388524 -----=----- pool-name=glusterfs:dict_t hot-count=4096 cold-count=0 padded_sizeof=140 alloc-count=647167 max-alloc=4096 pool-misses=614499 cur-stdalloc=76877 max-stdalloc=76879 -----=----- pool-name=glusterfs:call_stub_t hot-count=0 cold-count=1024 padded_sizeof=3756 alloc-count=0 max-alloc=0 pool-misses=0 cur-stdalloc=0 max-stdalloc=0 -----=----- pool-name=glusterfs:call_stack_t hot-count=1 cold-count=1023 padded_sizeof=1836 alloc-count=485321 max-alloc=7 pool-misses=0 cur-stdalloc=0 max-stdalloc=0 -----=----- pool-name=glusterfs:call_frame_t hot-count=0 cold-count=4096 padded_sizeof=172 alloc-count=0 max-alloc=0 pool-misses=0 cur-stdalloc=0 max-stdalloc=0 [iobuf.global] iobuf_pool=0x7f1fb00a5a40 iobuf_pool.default_page_size=131072 iobuf_pool.arena_size=12976128 iobuf_pool.arena_cnt=8 iobuf_pool.request_misses=0 [purge.1] purge.1.mem_base=0x7f1fae7ce000 purge.1.active_cnt=0 purge.1.passive_cnt=1024 purge.1.alloc_cnt=647542 purge.1.max_active=3 purge.1.page_size=128 [arena.2] arena.2.mem_base=0x7f1fae78e000 arena.2.active_cnt=1 arena.2.passive_cnt=511 arena.2.alloc_cnt=647097 arena.2.max_active=3 arena.2.page_size=512 [arena.2.active_iobuf.1] arena.2.active_iobuf.1.ref=1 arena.2.active_iobuf.1.ptr=0x7f1fae7cda00 [purge.3] purge.3.mem_base=0x7f1fae68e000 purge.3.active_cnt=0 purge.3.passive_cnt=512 purge.3.alloc_cnt=83144 purge.3.max_active=2 purge.3.page_size=2048 [purge.4] purge.4.mem_base=0x7f1fad026000 purge.4.active_cnt=0 purge.4.passive_cnt=128 purge.4.alloc_cnt=78594 purge.4.max_active=2 purge.4.page_size=8192 [arena.5] arena.5.mem_base=0x7f1face26000 arena.5.active_cnt=0 arena.5.passive_cnt=64 arena.5.alloc_cnt=0 arena.5.max_active=0 arena.5.page_size=32768 [arena.6] arena.6.mem_base=0x7f1faca26000 arena.6.active_cnt=0 arena.6.passive_cnt=32 arena.6.alloc_cnt=0 arena.6.max_active=0 arena.6.page_size=131072 [arena.7] arena.7.mem_base=0x7f1fac826000 arena.7.active_cnt=0 arena.7.passive_cnt=8 arena.7.alloc_cnt=0 arena.7.max_active=0 arena.7.page_size=262144 [arena.8] arena.8.mem_base=0x7f1fac626000 arena.8.active_cnt=0 arena.8.passive_cnt=2 arena.8.alloc_cnt=0 arena.8.max_active=0 arena.8.page_size=1048576 [global.callpool] callpool_address=0x7f1fb00a58e0 callpool.cnt=1 [global.callpool.stack.1] stack=0x7f1fac3ae5d0 uid=0 gid=0 pid=0 unique=0 lk-owner= op=stack type=0 cnt=1 [global.callpool.stack.1.frame.1] frame=0x7f1fac3aec38 ref_count=0 translator=management complete=0 [active graph - 1] [mgmt/glusterd.management - Memory usage] num_types=174 [mgmt/glusterd.management - usage-type 0 memusage] size=0 num_allocs=0 max_size=256 max_num_allocs=8 total_allocs=168 [mgmt/glusterd.management - usage-type gf_common_mt_dnscache6 memusage] size=48 num_allocs=3 max_size=48 max_num_allocs=3 total_allocs=3 [mgmt/glusterd.management - usage-type gf_common_mt_xlator_t memusage] size=0 num_allocs=0 max_size=20800 max_num_allocs=8 total_allocs=8 [mgmt/glusterd.management - usage-type gf_common_mt_xlator_list_t memusage] size=0 num_allocs=0 max_size=224 max_num_allocs=14 total_allocs=14 [mgmt/glusterd.management - usage-type gf_common_mt_volume_opt_list_t memusage] size=168 num_allocs=7 max_size=168 max_num_allocs=7 total_allocs=7 [mgmt/glusterd.management - usage-type gf_common_mt_gf_timer_t memusage] size=280 num_allocs=5 max_size=616 max_num_allocs=11 total_allocs=3713 [mgmt/glusterd.management - usage-type gf_common_mt_iobref memusage] size=24 num_allocs=1 max_size=96 max_num_allocs=4 total_allocs=971015 [mgmt/glusterd.management - usage-type gf_common_mt_memdup memusage] size=2671795 num_allocs=242895 max_size=2672452 max_num_allocs=242953 total_allocs=7169696 [mgmt/glusterd.management - usage-type gf_common_mt_asprintf memusage] size=324249 num_allocs=80978 max_size=324332 max_num_allocs=80989 total_allocs=1051637 [mgmt/glusterd.management - usage-type gf_common_mt_strdup memusage] size=2149 num_allocs=68 max_size=2796 max_num_allocs=102 total_allocs=243165 [mgmt/glusterd.management - usage-type gf_common_mt_socket_private_t memusage] size=5376 num_allocs=12 max_size=5824 max_num_allocs=13 total_allocs=30 [mgmt/glusterd.management - usage-type gf_common_mt_ioq memusage] size=0 num_allocs=0 max_size=312 max_num_allocs=1 total_allocs=485507 [mgmt/glusterd.management - usage-type gf_common_mt_char memusage] size=8096761 num_allocs=485795 max_size=8097749 max_num_allocs=485872 total_allocs=9192049 [mgmt/glusterd.management - usage-type gf_common_mt_mem_pool memusage] size=61588332 num_allocs=934753 max_size=61597524 max_num_allocs=934905 total_allocs=17229343 [mgmt/glusterd.management - usage-type gf_common_mt_long memusage] size=260576 num_allocs=12 max_size=260576 max_num_allocs=12 total_allocs=12 [mgmt/glusterd.management - usage-type gf_common_mt_rpcsvc_auth_list memusage] size=576 num_allocs=8 max_size=576 max_num_allocs=8 total_allocs=8 [mgmt/glusterd.management - usage-type gf_common_mt_rpcsvc_t memusage] size=400 num_allocs=2 max_size=400 max_num_allocs=2 total_allocs=2 [mgmt/glusterd.management - usage-type gf_common_mt_rpcsvc_program_t memusage] size=1320 num_allocs=11 max_size=1320 max_num_allocs=11 total_allocs=11 [mgmt/glusterd.management - usage-type gf_common_mt_rpcsvc_listener_t memusage] size=320 num_allocs=2 max_size=320 max_num_allocs=2 total_allocs=2 [mgmt/glusterd.management - usage-type gf_common_mt_rpcsvc_wrapper_t memusage] size=64 num_allocs=2 max_size=96 max_num_allocs=3 total_allocs=20 [mgmt/glusterd.management - usage-type gf_common_mt_rpcclnt_t memusage] size=1480 num_allocs=5 max_size=1480 max_num_allocs=5 total_allocs=5 [mgmt/glusterd.management - usage-type gf_common_mt_rpcclnt_savedframe_t memusage] size=1000 num_allocs=5 max_size=1200 max_num_allocs=6 total_allocs=7 [mgmt/glusterd.management - usage-type gf_common_mt_rpc_trans_t memusage] size=33408 num_allocs=12 max_size=36192 max_num_allocs=13 total_allocs=31 [mgmt/glusterd.management - usage-type gf_common_mt_rpc_trans_pollin_t memusage] size=0 num_allocs=0 max_size=296 max_num_allocs=1 total_allocs=485508 [mgmt/glusterd.management - usage-type gf_common_mt_rpc_trans_reqinfo_t memusage] size=0 num_allocs=0 max_size=64 max_num_allocs=1 total_allocs=31 [mgmt/glusterd.management - usage-type gf_common_mt_run_argv memusage] size=0 num_allocs=0 max_size=512 max_num_allocs=1 total_allocs=2 [mgmt/glusterd.management - usage-type gf_common_mt_run_logbuf memusage] size=0 num_allocs=0 max_size=178 max_num_allocs=1 total_allocs=1 [mgmt/glusterd.management - usage-type gf_common_mt_store_handle_t memusage] size=416 num_allocs=13 max_size=416 max_num_allocs=13 total_allocs=16 [mgmt/glusterd.management - usage-type gf_common_mt_store_iter_t memusage] size=0 num_allocs=0 max_size=8208 max_num_allocs=2 total_allocs=12 [mgmt/glusterd.management - usage-type gf_common_mt_iobrefs memusage] size=128 num_allocs=1 max_size=512 max_num_allocs=4 total_allocs=971015 [mgmt/glusterd.management - usage-type gf_common_mt_uuid_t memusage] size=1295200 num_allocs=80950 max_size=1295216 max_num_allocs=80951 total_allocs=323597 [mgmt/glusterd.management - usage-type gf_common_mt_mgmt_v3_lock_obj_t memusage] size=0 num_allocs=0 max_size=16 max_num_allocs=1 total_allocs=80846 [mgmt/glusterd.management - usage-type gf_common_mt_txn_opinfo_obj_t memusage] size=9067296 num_allocs=80958 max_size=9067296 max_num_allocs=80958 total_allocs=161804 [mgmt/glusterd.management - usage-type gf_sock_mt_lock_array memusage] size=1640 num_allocs=1 max_size=1640 max_num_allocs=1 total_allocs=1 [mgmt/glusterd.management - usage-type gf_gld_mt_glusterd_conf_t memusage] size=4688 num_allocs=1 max_size=4688 max_num_allocs=1 total_allocs=1 [mgmt/glusterd.management - usage-type gf_gld_mt_peerinfo_t memusage] size=816 num_allocs=3 max_size=816 max_num_allocs=3 total_allocs=3 [mgmt/glusterd.management - usage-type gf_gld_mt_friend_sm_event_t memusage] size=0 num_allocs=0 max_size=80 max_num_allocs=2 total_allocs=16 [mgmt/glusterd.management - usage-type gf_gld_mt_friend_req_ctx_t memusage] size=0 num_allocs=0 max_size=48 max_num_allocs=1 total_allocs=4 [mgmt/glusterd.management - usage-type gf_gld_mt_friend_update_ctx_t memusage] size=0 num_allocs=0 max_size=32 max_num_allocs=1 total_allocs=8 [mgmt/glusterd.management - usage-type gf_gld_mt_op_sm_event_t memusage] size=0 num_allocs=0 max_size=96 max_num_allocs=2 total_allocs=647084 [mgmt/glusterd.management - usage-type gf_gld_mt_op_lock_ctx_t memusage] size=0 num_allocs=0 max_size=32 max_num_allocs=1 total_allocs=161692 [mgmt/glusterd.management - usage-type mem_type memusage] size=40 num_allocs=1 max_size=40 max_num_allocs=1 total_allocs=161803 [mgmt/glusterd.management - usage-type mem_type memusage] size=0 num_allocs=0 max_size=40 max_num_allocs=1 total_allocs=161795 [mgmt/glusterd.management - usage-type gf_gld_mt_probe_ctx_t memusage] size=0 num_allocs=0 max_size=32 max_num_allocs=1 total_allocs=4 [mgmt/glusterd.management - usage-type gf_gld_mt_glusterd_volinfo_t memusage] size=1040 num_allocs=1 max_size=1040 max_num_allocs=1 total_allocs=1 [mgmt/glusterd.management - usage-type gf_gld_mt_glusterd_brickinfo_t memusage] size=79232 num_allocs=4 max_size=99040 max_num_allocs=5 total_allocs=5 [mgmt/glusterd.management - usage-type gf_gld_mt_peer_hostname_t memusage] size=72 num_allocs=3 max_size=72 max_num_allocs=3 total_allocs=3 [mgmt/glusterd.management - usage-type gf_gld_mt_peerctx_t memusage] size=120 num_allocs=3 max_size=120 max_num_allocs=3 total_allocs=3 [mgmt/glusterd.management - usage-type gf_gld_mt_sm_tr_log_t memusage] size=4800 num_allocs=4 max_size=4800 max_num_allocs=4 total_allocs=4 [mgmt/glusterd.management - usage-type gf_gld_mt_nodesrv_t memusage] size=48 num_allocs=3 max_size=48 max_num_allocs=3 total_allocs=3 [mgmt/glusterd.management - usage-type gf_gld_mt_charptr memusage] size=448 num_allocs=3 max_size=632 max_num_allocs=4 total_allocs=4 [mgmt/glusterd.management - usage-type gf_gld_mt_hooks_priv_t memusage] size=120 num_allocs=1 max_size=120 max_num_allocs=1 total_allocs=1 DUMP-END-TIME: 2016-08-23 09:11:42.553990
GlusterFS-3.6 is nearing its End-Of-Life, only important security bugs still make a chance on getting fixed. Moving this to the mainline 'version'. If this needs to get fixed in 3.7 or 3.8 this bug should get cloned.
I try to reproduce the issue, but after multiple trials issue is not being reproduced.. I have used mainline build for my testing. Reproduction steps. 1. Create a 2 node cluster 2. Create volume with 3 bricks on both the nodes. 3. Start the volume. 4. Execute the command gluster volume status in a script for 1 hr, 5. Monitor memory of glusterd process. I didn't see memory is increasing for glusterd process when gluster v status command ie being executed. As issue is not getting reproduced in mainline, closing the bug.