Bug 1369382

Summary: Glusterd mem leak when run command gluster volume status
Product: [Community] GlusterFS Reporter: Yu.Changcai <einstcrazy>
Component: glusterdAssignee: Gaurav Yadav <gyadav>
Status: CLOSED WORKSFORME QA Contact:
Severity: high Docs Contact:
Priority: medium    
Version: mainlineCC: amukherj, bugs, kaushal
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-02-07 10:43:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Yu.Changcai 2016-08-23 09:14:27 UTC 
Description of problem:

Glusterd mem leak when run command gluster volume status


Version-Release number of selected component (if applicable):
v3.6.9

How reproducible:


Steps to Reproduce:
1. Build a gluster cluster
2. Create a volume with multiple bricks, and start it
3. Run the command gluster volume status several time
4. Check the memory of the glusterd process all the time

Actual results:
the memory of glusterd process grow all the time.

Expected results:
The memory is kept a small range

Additional info:
The dump file is below:

DUMP-START-TIME: 2016-08-23 09:11:42.543685

[mallinfo]
mallinfo_arena=197533696
mallinfo_ordblks=85725
mallinfo_smblks=337
mallinfo_hblks=9
mallinfo_hblkhd=11153408
mallinfo_usmblks=0
mallinfo_fsmblks=31568
mallinfo_uordblks=193679680
mallinfo_fordblks=3854016
mallinfo_keepcost=134512

[global.glusterfs - Memory usage]
num_types=122

[global.glusterfs - usage-type gf_common_mt_event_pool memusage]
size=144
num_allocs=1
max_size=144
max_num_allocs=1
total_allocs=1

[global.glusterfs - usage-type gf_common_mt_reg memusage]
size=393216
num_allocs=1
max_size=393216
max_num_allocs=1
total_allocs=1

[global.glusterfs - usage-type gf_common_mt_epoll_event memusage]
size=3132
num_allocs=1
max_size=3132
max_num_allocs=1
total_allocs=1

[global.glusterfs - usage-type gf_common_mt_xlator_t memusage]
size=2600
num_allocs=1
max_size=2600
max_num_allocs=1
total_allocs=1

[global.glusterfs - usage-type gf_common_mt_volume_opt_list_t memusage]
size=24
num_allocs=1
max_size=24
max_num_allocs=1
total_allocs=1

[global.glusterfs - usage-type gf_common_mt_gf_timer_t memusage]
size=56
num_allocs=1
max_size=56
max_num_allocs=1
total_allocs=32

[global.glusterfs - usage-type gf_common_mt_gf_timer_registry_t memusage]
size=168
num_allocs=1
max_size=168
max_num_allocs=1
total_allocs=1

[global.glusterfs - usage-type gf_common_mt_iobuf memusage]
size=109536
num_allocs=8
max_size=109536
max_num_allocs=8
total_allocs=8

[global.glusterfs - usage-type gf_common_mt_iobuf_arena memusage]
size=1872
num_allocs=9
max_size=1872
max_num_allocs=9
total_allocs=9

[global.glusterfs - usage-type gf_common_mt_iobuf_pool memusage]
size=1776
num_allocs=1
max_size=1776
max_num_allocs=1
total_allocs=1

[global.glusterfs - usage-type gf_common_mt_asprintf memusage]
size=225
num_allocs=9
max_size=320
max_num_allocs=10
total_allocs=11

[global.glusterfs - usage-type gf_common_mt_strdup memusage]
size=275
num_allocs=13
max_size=457
max_num_allocs=20
total_allocs=33

[global.glusterfs - usage-type gf_common_mt_char memusage]
size=145
num_allocs=6
max_size=24576
max_num_allocs=6
total_allocs=9

[global.glusterfs - usage-type gf_common_mt_mem_pool memusage]
size=840
num_allocs=7
max_size=840
max_num_allocs=7
total_allocs=7

[global.glusterfs - usage-type gf_common_mt_long memusage]
size=9006080
num_allocs=7
max_size=9006080
max_num_allocs=7
total_allocs=7

[global.glusterfs - usage-type gf_common_mt_glusterfs_graph_t memusage]
size=192
num_allocs=1
max_size=192
max_num_allocs=1
total_allocs=1

[global.glusterfs - usage-type gf_common_mt_cliententry_t memusage]
size=2048
num_allocs=1
max_size=2048
max_num_allocs=1
total_allocs=1

[global.glusterfs - usage-type gf_common_mt_clienttable_t memusage]
size=32
num_allocs=1
max_size=32
max_num_allocs=1
total_allocs=1

[global.glusterfs - usage-type gfd_mt_call_pool_t memusage]
size=48
num_allocs=1
max_size=48
max_num_allocs=1
total_allocs=1

[mempool]
-----=-----
pool-name=management:struct saved_frame
hot-count=0
cold-count=16
padded_sizeof=124
alloc-count=0
max-alloc=0
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=management:struct rpc_req
hot-count=0
cold-count=16
padded_sizeof=588
alloc-count=0
max-alloc=0
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=management:struct saved_frame
hot-count=0
cold-count=16
padded_sizeof=124
alloc-count=0
max-alloc=0
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=management:struct rpc_req
hot-count=0
cold-count=16
padded_sizeof=588
alloc-count=0
max-alloc=0
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=management:struct saved_frame
hot-count=0
cold-count=16
padded_sizeof=124
alloc-count=10
max-alloc=3
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=management:struct rpc_req
hot-count=0
cold-count=16
padded_sizeof=588
alloc-count=10
max-alloc=3
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=management:struct saved_frame
hot-count=0
cold-count=16
padded_sizeof=124
alloc-count=12
max-alloc=3
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=management:struct rpc_req
hot-count=0
cold-count=16
padded_sizeof=588
alloc-count=12
max-alloc=3
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=management:struct saved_frame
hot-count=0
cold-count=16
padded_sizeof=124
alloc-count=9
max-alloc=3
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=management:struct rpc_req
hot-count=0
cold-count=16
padded_sizeof=588
alloc-count=9
max-alloc=3
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=management:rpcsvc_request_t
hot-count=0
cold-count=8
padded_sizeof=2828
alloc-count=0
max-alloc=0
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=management:rpcsvc_request_t
hot-count=1
cold-count=63
padded_sizeof=2828
alloc-count=485477
max-alloc=2
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=glusterfs:log_buf_t
hot-count=0
cold-count=256
padded_sizeof=140
alloc-count=3
max-alloc=2
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=glusterfs:data_t
hot-count=16384
cold-count=0
padded_sizeof=52
alloc-count=9191727
max-alloc=16384
pool-misses=8881844
cur-stdalloc=469415
max-stdalloc=469491
-----=-----
pool-name=glusterfs:data_pair_t
hot-count=16384
cold-count=0
padded_sizeof=68
alloc-count=8059283
max-alloc=16384
pool-misses=7732988
cur-stdalloc=388449
max-stdalloc=388524
-----=-----
pool-name=glusterfs:dict_t
hot-count=4096
cold-count=0
padded_sizeof=140
alloc-count=647167
max-alloc=4096
pool-misses=614499
cur-stdalloc=76877
max-stdalloc=76879
-----=-----
pool-name=glusterfs:call_stub_t
hot-count=0
cold-count=1024
padded_sizeof=3756
alloc-count=0
max-alloc=0
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=glusterfs:call_stack_t
hot-count=1
cold-count=1023
padded_sizeof=1836
alloc-count=485321
max-alloc=7
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=glusterfs:call_frame_t
hot-count=0
cold-count=4096
padded_sizeof=172
alloc-count=0
max-alloc=0
pool-misses=0
cur-stdalloc=0
max-stdalloc=0

[iobuf.global]
iobuf_pool=0x7f1fb00a5a40
iobuf_pool.default_page_size=131072
iobuf_pool.arena_size=12976128
iobuf_pool.arena_cnt=8
iobuf_pool.request_misses=0

[purge.1]
purge.1.mem_base=0x7f1fae7ce000
purge.1.active_cnt=0
purge.1.passive_cnt=1024
purge.1.alloc_cnt=647542
purge.1.max_active=3
purge.1.page_size=128

[arena.2]
arena.2.mem_base=0x7f1fae78e000
arena.2.active_cnt=1
arena.2.passive_cnt=511
arena.2.alloc_cnt=647097
arena.2.max_active=3
arena.2.page_size=512

[arena.2.active_iobuf.1]
arena.2.active_iobuf.1.ref=1
arena.2.active_iobuf.1.ptr=0x7f1fae7cda00

[purge.3]
purge.3.mem_base=0x7f1fae68e000
purge.3.active_cnt=0
purge.3.passive_cnt=512
purge.3.alloc_cnt=83144
purge.3.max_active=2
purge.3.page_size=2048

[purge.4]
purge.4.mem_base=0x7f1fad026000
purge.4.active_cnt=0
purge.4.passive_cnt=128
purge.4.alloc_cnt=78594
purge.4.max_active=2
purge.4.page_size=8192

[arena.5]
arena.5.mem_base=0x7f1face26000
arena.5.active_cnt=0
arena.5.passive_cnt=64
arena.5.alloc_cnt=0
arena.5.max_active=0
arena.5.page_size=32768

[arena.6]
arena.6.mem_base=0x7f1faca26000
arena.6.active_cnt=0
arena.6.passive_cnt=32
arena.6.alloc_cnt=0
arena.6.max_active=0
arena.6.page_size=131072

[arena.7]
arena.7.mem_base=0x7f1fac826000
arena.7.active_cnt=0
arena.7.passive_cnt=8
arena.7.alloc_cnt=0
arena.7.max_active=0
arena.7.page_size=262144

[arena.8]
arena.8.mem_base=0x7f1fac626000
arena.8.active_cnt=0
arena.8.passive_cnt=2
arena.8.alloc_cnt=0
arena.8.max_active=0
arena.8.page_size=1048576

[global.callpool]
callpool_address=0x7f1fb00a58e0
callpool.cnt=1

[global.callpool.stack.1]
stack=0x7f1fac3ae5d0
uid=0
gid=0
pid=0
unique=0
lk-owner=
op=stack
type=0
cnt=1

[global.callpool.stack.1.frame.1]
frame=0x7f1fac3aec38
ref_count=0
translator=management
complete=0

[active graph - 1]

[mgmt/glusterd.management - Memory usage]
num_types=174

[mgmt/glusterd.management - usage-type 0 memusage]
size=0
num_allocs=0
max_size=256
max_num_allocs=8
total_allocs=168

[mgmt/glusterd.management - usage-type gf_common_mt_dnscache6 memusage]
size=48
num_allocs=3
max_size=48
max_num_allocs=3
total_allocs=3

[mgmt/glusterd.management - usage-type gf_common_mt_xlator_t memusage]
size=0
num_allocs=0
max_size=20800
max_num_allocs=8
total_allocs=8

[mgmt/glusterd.management - usage-type gf_common_mt_xlator_list_t memusage]
size=0
num_allocs=0
max_size=224
max_num_allocs=14
total_allocs=14

[mgmt/glusterd.management - usage-type gf_common_mt_volume_opt_list_t memusage]
size=168
num_allocs=7
max_size=168
max_num_allocs=7
total_allocs=7

[mgmt/glusterd.management - usage-type gf_common_mt_gf_timer_t memusage]
size=280
num_allocs=5
max_size=616
max_num_allocs=11
total_allocs=3713

[mgmt/glusterd.management - usage-type gf_common_mt_iobref memusage]
size=24
num_allocs=1
max_size=96
max_num_allocs=4
total_allocs=971015

[mgmt/glusterd.management - usage-type gf_common_mt_memdup memusage]
size=2671795
num_allocs=242895
max_size=2672452
max_num_allocs=242953
total_allocs=7169696

[mgmt/glusterd.management - usage-type gf_common_mt_asprintf memusage]
size=324249
num_allocs=80978
max_size=324332
max_num_allocs=80989
total_allocs=1051637

[mgmt/glusterd.management - usage-type gf_common_mt_strdup memusage]
size=2149
num_allocs=68
max_size=2796
max_num_allocs=102
total_allocs=243165

[mgmt/glusterd.management - usage-type gf_common_mt_socket_private_t memusage]
size=5376
num_allocs=12
max_size=5824
max_num_allocs=13
total_allocs=30

[mgmt/glusterd.management - usage-type gf_common_mt_ioq memusage]
size=0
num_allocs=0
max_size=312
max_num_allocs=1
total_allocs=485507

[mgmt/glusterd.management - usage-type gf_common_mt_char memusage]
size=8096761
num_allocs=485795
max_size=8097749
max_num_allocs=485872
total_allocs=9192049

[mgmt/glusterd.management - usage-type gf_common_mt_mem_pool memusage]
size=61588332
num_allocs=934753
max_size=61597524
max_num_allocs=934905
total_allocs=17229343

[mgmt/glusterd.management - usage-type gf_common_mt_long memusage]
size=260576
num_allocs=12
max_size=260576
max_num_allocs=12
total_allocs=12

[mgmt/glusterd.management - usage-type gf_common_mt_rpcsvc_auth_list memusage]
size=576
num_allocs=8
max_size=576
max_num_allocs=8
total_allocs=8

[mgmt/glusterd.management - usage-type gf_common_mt_rpcsvc_t memusage]
size=400
num_allocs=2
max_size=400
max_num_allocs=2
total_allocs=2

[mgmt/glusterd.management - usage-type gf_common_mt_rpcsvc_program_t memusage]
size=1320
num_allocs=11
max_size=1320
max_num_allocs=11
total_allocs=11

[mgmt/glusterd.management - usage-type gf_common_mt_rpcsvc_listener_t memusage]
size=320
num_allocs=2
max_size=320
max_num_allocs=2
total_allocs=2

[mgmt/glusterd.management - usage-type gf_common_mt_rpcsvc_wrapper_t memusage]
size=64
num_allocs=2
max_size=96
max_num_allocs=3
total_allocs=20

[mgmt/glusterd.management - usage-type gf_common_mt_rpcclnt_t memusage]
size=1480
num_allocs=5
max_size=1480
max_num_allocs=5
total_allocs=5

[mgmt/glusterd.management - usage-type gf_common_mt_rpcclnt_savedframe_t memusage]
size=1000
num_allocs=5
max_size=1200
max_num_allocs=6
total_allocs=7

[mgmt/glusterd.management - usage-type gf_common_mt_rpc_trans_t memusage]
size=33408
num_allocs=12
max_size=36192
max_num_allocs=13
total_allocs=31

[mgmt/glusterd.management - usage-type gf_common_mt_rpc_trans_pollin_t memusage]
size=0
num_allocs=0
max_size=296
max_num_allocs=1
total_allocs=485508

[mgmt/glusterd.management - usage-type gf_common_mt_rpc_trans_reqinfo_t memusage]
size=0
num_allocs=0
max_size=64
max_num_allocs=1
total_allocs=31

[mgmt/glusterd.management - usage-type gf_common_mt_run_argv memusage]
size=0
num_allocs=0
max_size=512
max_num_allocs=1
total_allocs=2

[mgmt/glusterd.management - usage-type gf_common_mt_run_logbuf memusage]
size=0
num_allocs=0
max_size=178
max_num_allocs=1
total_allocs=1

[mgmt/glusterd.management - usage-type gf_common_mt_store_handle_t memusage]
size=416
num_allocs=13
max_size=416
max_num_allocs=13
total_allocs=16

[mgmt/glusterd.management - usage-type gf_common_mt_store_iter_t memusage]
size=0
num_allocs=0
max_size=8208
max_num_allocs=2
total_allocs=12

[mgmt/glusterd.management - usage-type gf_common_mt_iobrefs memusage]
size=128
num_allocs=1
max_size=512
max_num_allocs=4
total_allocs=971015

[mgmt/glusterd.management - usage-type gf_common_mt_uuid_t memusage]
size=1295200
num_allocs=80950
max_size=1295216
max_num_allocs=80951
total_allocs=323597

[mgmt/glusterd.management - usage-type gf_common_mt_mgmt_v3_lock_obj_t memusage]
size=0
num_allocs=0
max_size=16
max_num_allocs=1
total_allocs=80846

[mgmt/glusterd.management - usage-type gf_common_mt_txn_opinfo_obj_t memusage]
size=9067296
num_allocs=80958
max_size=9067296
max_num_allocs=80958
total_allocs=161804

[mgmt/glusterd.management - usage-type gf_sock_mt_lock_array memusage]
size=1640
num_allocs=1
max_size=1640
max_num_allocs=1
total_allocs=1

[mgmt/glusterd.management - usage-type gf_gld_mt_glusterd_conf_t memusage]
size=4688
num_allocs=1
max_size=4688
max_num_allocs=1
total_allocs=1

[mgmt/glusterd.management - usage-type gf_gld_mt_peerinfo_t memusage]
size=816
num_allocs=3
max_size=816
max_num_allocs=3
total_allocs=3

[mgmt/glusterd.management - usage-type gf_gld_mt_friend_sm_event_t memusage]
size=0
num_allocs=0
max_size=80
max_num_allocs=2
total_allocs=16

[mgmt/glusterd.management - usage-type gf_gld_mt_friend_req_ctx_t memusage]
size=0
num_allocs=0
max_size=48
max_num_allocs=1
total_allocs=4

[mgmt/glusterd.management - usage-type gf_gld_mt_friend_update_ctx_t memusage]
size=0
num_allocs=0
max_size=32
max_num_allocs=1
total_allocs=8

[mgmt/glusterd.management - usage-type gf_gld_mt_op_sm_event_t memusage]
size=0
num_allocs=0
max_size=96
max_num_allocs=2
total_allocs=647084

[mgmt/glusterd.management - usage-type gf_gld_mt_op_lock_ctx_t memusage]
size=0
num_allocs=0
max_size=32
max_num_allocs=1
total_allocs=161692

[mgmt/glusterd.management - usage-type mem_type memusage]
size=40
num_allocs=1
max_size=40
max_num_allocs=1
total_allocs=161803

[mgmt/glusterd.management - usage-type mem_type memusage]
size=0
num_allocs=0
max_size=40
max_num_allocs=1
total_allocs=161795

[mgmt/glusterd.management - usage-type gf_gld_mt_probe_ctx_t memusage]
size=0
num_allocs=0
max_size=32
max_num_allocs=1
total_allocs=4

[mgmt/glusterd.management - usage-type gf_gld_mt_glusterd_volinfo_t memusage]
size=1040
num_allocs=1
max_size=1040
max_num_allocs=1
total_allocs=1

[mgmt/glusterd.management - usage-type gf_gld_mt_glusterd_brickinfo_t memusage]
size=79232
num_allocs=4
max_size=99040
max_num_allocs=5
total_allocs=5

[mgmt/glusterd.management - usage-type gf_gld_mt_peer_hostname_t memusage]
size=72
num_allocs=3
max_size=72
max_num_allocs=3
total_allocs=3

[mgmt/glusterd.management - usage-type gf_gld_mt_peerctx_t memusage]
size=120
num_allocs=3
max_size=120
max_num_allocs=3
total_allocs=3

[mgmt/glusterd.management - usage-type gf_gld_mt_sm_tr_log_t memusage]
size=4800
num_allocs=4
max_size=4800
max_num_allocs=4
total_allocs=4

[mgmt/glusterd.management - usage-type gf_gld_mt_nodesrv_t memusage]
size=48
num_allocs=3
max_size=48
max_num_allocs=3
total_allocs=3

[mgmt/glusterd.management - usage-type gf_gld_mt_charptr memusage]
size=448
num_allocs=3
max_size=632
max_num_allocs=4
total_allocs=4

[mgmt/glusterd.management - usage-type gf_gld_mt_hooks_priv_t memusage]
size=120
num_allocs=1
max_size=120
max_num_allocs=1
total_allocs=1

DUMP-END-TIME: 2016-08-23 09:11:42.553990
Comment 2 Kaushal 2016-08-30 12:17:12 UTC 
GlusterFS-3.6 is nearing its End-Of-Life, only important security bugs still make a chance on getting fixed. Moving this to the mainline 'version'. If this needs to get fixed in 3.7 or 3.8 this bug should get cloned.
Comment 3 Gaurav Yadav 2017-02-07 10:43:37 UTC 
I try to reproduce the issue, but after multiple trials issue is not being reproduced..
I have used mainline build for my testing.

Reproduction steps.
1. Create a 2 node cluster
2. Create volume with 3 bricks on both the nodes.
3. Start the volume.
4. Execute the command gluster volume status in a script for 1 hr, 
5. Monitor memory of glusterd process.

I didn't see memory is increasing for glusterd process when gluster v status command ie being executed.

As issue is not getting reproduced in mainline, closing the bug.