Bug 1455223

[detached from 12245.smallfile]
[root@dhcp37-162 glusterfs]# free -h
              total        used        free      shared  buff/cache   available
Mem:           3.7G        3.0G        213M        3.2M        487M        413M
Swap:          3.9G        406M        3.5G



[root@dhcp37-162 ~]# free -h
              total        used        free      shared  buff/cache   available
Mem:           3.7G        1.7G        1.5G        3.6M        487M        1.7G
Swap:          3.9G         66M        3.8G
[root@dhcp37-162 ~]# 


[root@dhcp37-162 ~]# top -n 1 -b|grep gluster
11063 root      20   0 3401188 1.561g   2164 S   0.0 42.2   1019:48 glusterfs
[root@dhcp37-162 ~]# 
[root@dhcp37-162 ~]# 
[root@dhcp37-162 ~]# cd /mnt/cross3-29/
[root@dhcp37-162 cross3-29]# ls
dhcp37-162.lab.eng.blr.redhat.com  dir1  kernel  miketest  smallfile
[root@dhcp37-162 cross3-29]# #rm -rf *
[root@dhcp37-162 cross3-29]# kill -USR1 11063


10.70.35.130:cross3-29             10G  102M  9.9G   1% /mnt/cross3-29
[root@dhcp37-162 ~]# 
[root@dhcp37-162 ~]# 
[root@dhcp37-162 ~]# 
[root@dhcp37-162 ~]# 
[root@dhcp37-162 ~]# 
[root@dhcp37-162 ~]# free -h
              total        used        free      shared  buff/cache   available
Mem:           3.7G        1.7G        1.5G        4.1M        488M        1.7G
Swap:          3.9G         66M        3.8G
[root@dhcp37-162 ~]# 
[root@dhcp37-162 ~]# top -n 1 -b|egrep "gluster|RES"
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
11063 root      20   0 3401188 1.561g   2164 S   0.0 42.2   1019:49 glusterfs

Brick multiplexing is a server side feature and has nothing to do with clients.

sosreports available @ http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/nchilaka/bug.1455223/

Kindly refer to following files for more related info:
Under each client:
1)statedumps are available taken at different times, check the most recent ones
2)info.log====> has the memory consumption from top command and df -h and mount details taken at the time of raising bug
3)refer to <hostname>.top.log for the complete free -h and top o/p since IOs were triggered
eg:dhcp37-107.lab.eng.blr.redhat.com.top.log


Refer to dhcp37-162.lab.eng.blr.redhat.com logs for the latest one, on this node's the bug was raised

> Also, note that from statedumps found one datastructure mentioning about 4GB
> size in all the clients
> 
> gf_fuse_mt_iov_base
> [mount/fuse.fuse - usage-type gf_fuse_mt_iov_base memusage]
> size=4279449616
> num_allocs=1
> max_size=4294964384
> max_num_allocs=20
> total_allocs=80757263

This is a false positive (for leaks). There is a bug in memory accounting and due to this we tend to see huge values for this data types. We've seen this in other setups too. We can ignore this.

From one of the statedumps:

[mallinfo]
mallinfo_arena=1699667968
mallinfo_ordblks=3409
mallinfo_smblks=66
mallinfo_hblks=5
mallinfo_hblkhd=11993088
mallinfo_usmblks=0
mallinfo_fsmblks=6144
mallinfo_uordblks=1748048
mallinfo_fordblks=1697919920
mallinfo_keepcost=16176

look at values mallinfo_fordblks (freed bytes) and mallinfo_arena (total allocated bytes), they are _almost_ equal. And mallinfo_arena ~= 1.5 GB (shown as the memory consumed by glusterfs process). So, all the memory shown is held in freed chunks of malloc subsystem. Note that malloc might not return memory to system immediately after free and keep it in its cache for future allocations. So, I think there is no leak.

@Nag,

Can we close this as NOT A BUG?

For reference on what individual members of mallinfo mean, please check "man 3 mallinfo"

Development Management has reviewed and declined this request.
You may appeal this decision by reopening this request.