Bug 1399584

Summary: Memory leaks after glfs_new+set_volfile_server+init+fini
Product: [Community] GlusterFS Reporter: Mateusz Slupny <mateusz.slupny>
Component: libgfapiAssignee: Mateusz Slupny <mateusz.slupny>
Status: CLOSED CURRENTRELEASE QA Contact: Sudhir D <sdharane>
Severity: medium Docs Contact:
Priority: medium    
Version: mainlineCC: bugs, c.affolter
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: glusterfs-4.1.3 (or later) Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-08-29 03:35:33 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1397177    

Description Mateusz Slupny 2016-11-29 10:43:36 UTC 
Description of problem:

Extends the test from https://bugzilla.redhat.com/show_bug.cgi?id=1397177.

After running a simple application valgrind reports several memory leaks. Loop is added to highlight actual repeating leaks and not occurrences one-time initialization.

Application code:

#include <glusterfs/api/glfs.h>

void glusterCatastrophe()
{
  glfs_t* gluster = glfs_new("hot");
  glfs_set_volfile_server(gluster, "tcp", "10.10.27.20", 24007);
  glfs_init(gluster);
  glfs_fini(gluster);
}

int main()
{
  for (int i = 0; i < 10; ++i)
  {
    glusterCatastrophe();
  }

  return 0;
}

valgrind output:

==31753== Memcheck, a memory error detector
==31753== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==31753== Using Valgrind-3.12.0.SVN and LibVEX; rerun with -h for copyright info
==31753== Command: ./simple
==31753== 
==31753== 
==31753== HEAP SUMMARY:
==31753==     in use at exit: 137,819,294 bytes in 11,711 blocks
==31753==   total heap usage: 47,402 allocs, 35,691 frees, 164,418,897 bytes allocated
==31753== 
==31753== 780 bytes in 10 blocks are definitely lost in loss record 253 of 459
==31753==    at 0x4C2CB3F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBBCF: __gf_malloc (mem-pool.c:142)
==31753==    by 0x68BBEBB: gf_vasprintf (mem-pool.c:221)
==31753==    by 0x68BBFD0: gf_asprintf (mem-pool.c:240)
==31753==    by 0x687F3F8: data_from_int32 (dict.c:721)
==31753==    by 0x6881A21: dict_set_int32 (dict.c:1789)
==31753==    by 0x6D9E594: rpc_transport_inet_options_build (rpc-transport.c:681)
==31753==    by 0x6278FBD: glfs_mgmt_init (glfs-mgmt.c:917)
==31753==    by 0x6274863: glfs_volumes_init (glfs.c:266)
==31753==    by 0x6275D6B: glfs_init_common (glfs.c:989)
==31753==    by 0x6275F21: glfs_init@@GFAPI_3.4.0 (glfs.c:1034)
==31753==    by 0x1088D2: glusterCatastrophe() (in /home/matt/dev/glusterfs_memory_leak/simple)
==31753== 
==31753== 820 bytes in 10 blocks are definitely lost in loss record 259 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x6887817: gf_strdup (mem-pool.h:187)
==31753==    by 0x6888B6B: gf_log_init (logging.c:734)
==31753==    by 0x6275B4D: glfs_set_logging@@GFAPI_3.4.0 (glfs.c:903)
==31753==    by 0x62757A7: glfs_new@@GFAPI_3.4.0 (glfs.c:778)
==31753==    by 0x1088A3: glusterCatastrophe() (in /home/matt/dev/glusterfs_memory_leak/simple)
==31753==    by 0x1088FB: main (in /home/matt/dev/glusterfs_memory_leak/simple)
==31753== 
==31753== 840 bytes in 10 blocks are definitely lost in loss record 262 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x6D9C89D: gf_strdup (mem-pool.h:187)
==31753==    by 0x6D9E4F3: rpc_transport_inet_options_build (rpc-transport.c:667)
==31753==    by 0x6278FBD: glfs_mgmt_init (glfs-mgmt.c:917)
==31753==    by 0x6274863: glfs_volumes_init (glfs.c:266)
==31753==    by 0x6275D6B: glfs_init_common (glfs.c:989)
==31753==    by 0x6275F21: glfs_init@@GFAPI_3.4.0 (glfs.c:1034)
==31753==    by 0x1088D2: glusterCatastrophe() (in /home/matt/dev/glusterfs_memory_leak/simple)
==31753==    by 0x1088FB: main (in /home/matt/dev/glusterfs_memory_leak/simple)
==31753== 
==31753== 840 bytes in 10 blocks are definitely lost in loss record 263 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x687E5ED: dict_set_lk (dict.c:376)
==31753==    by 0x687E7C8: dict_set (dict.c:422)
==31753==    by 0x6882538: dict_set_dynstr (dict.c:2223)
==31753==    by 0x6D9E521: rpc_transport_inet_options_build (rpc-transport.c:673)
==31753==    by 0x6278FBD: glfs_mgmt_init (glfs-mgmt.c:917)
==31753==    by 0x6274863: glfs_volumes_init (glfs.c:266)
==31753==    by 0x6275D6B: glfs_init_common (glfs.c:989)
==31753==    by 0x6275F21: glfs_init@@GFAPI_3.4.0 (glfs.c:1034)
==31753==    by 0x1088D2: glusterCatastrophe() (in /home/matt/dev/glusterfs_memory_leak/simple)
==31753==    by 0x1088FB: main (in /home/matt/dev/glusterfs_memory_leak/simple)
==31753== 
==31753== 840 bytes in 10 blocks are definitely lost in loss record 264 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x687E5ED: dict_set_lk (dict.c:376)
==31753==    by 0x687E7C8: dict_set (dict.c:422)
==31753==    by 0x6881A4C: dict_set_int32 (dict.c:1795)
==31753==    by 0x6D9E594: rpc_transport_inet_options_build (rpc-transport.c:681)
==31753==    by 0x6278FBD: glfs_mgmt_init (glfs-mgmt.c:917)
==31753==    by 0x6274863: glfs_volumes_init (glfs.c:266)
==31753==    by 0x6275D6B: glfs_init_common (glfs.c:989)
==31753==    by 0x6275F21: glfs_init@@GFAPI_3.4.0 (glfs.c:1034)
==31753==    by 0x1088D2: glusterCatastrophe() (in /home/matt/dev/glusterfs_memory_leak/simple)
==31753==    by 0x1088FB: main (in /home/matt/dev/glusterfs_memory_leak/simple)
==31753== 
==31753== 840 bytes in 10 blocks are definitely lost in loss record 265 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x54ED71E: dht_init_subvolumes (dht-helper.c:1042)
==31753==    by 0x557573A: dht_init (dht-shared.c:757)
==31753==    by 0x68858C0: __xlator_init (xlator.c:403)
==31753==    by 0x68859E1: xlator_init (xlator.c:428)
==31753==    by 0x68CE7BA: glusterfs_graph_init (graph.c:320)
==31753==    by 0x68CF39F: glusterfs_graph_activate (graph.c:670)
==31753==    by 0x6276EBF: glfs_process_volfp (glfs-mgmt.c:78)
==31753==    by 0x62783E3: glfs_mgmt_getspec_cbk (glfs-mgmt.c:604)
==31753==    by 0x6DA1DFD: rpc_clnt_handle_reply (rpc-clnt.c:790)
==31753==    by 0x6DA2355: rpc_clnt_notify (rpc-clnt.c:961)
==31753== 
==31753== 870 bytes in 10 blocks are definitely lost in loss record 268 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x687E5ED: dict_set_lk (dict.c:376)
==31753==    by 0x687E7C8: dict_set (dict.c:422)
==31753==    by 0x6882452: dict_set_str (dict.c:2186)
==31753==    by 0x6D9E5FD: rpc_transport_inet_options_build (rpc-transport.c:688)
==31753==    by 0x6278FBD: glfs_mgmt_init (glfs-mgmt.c:917)
==31753==    by 0x6274863: glfs_volumes_init (glfs.c:266)
==31753==    by 0x6275D6B: glfs_init_common (glfs.c:989)
==31753==    by 0x6275F21: glfs_init@@GFAPI_3.4.0 (glfs.c:1034)
==31753==    by 0x1088D2: glusterCatastrophe() (in /home/matt/dev/glusterfs_memory_leak/simple)
==31753==    by 0x1088FB: main (in /home/matt/dev/glusterfs_memory_leak/simple)
==31753== 
==31753== 880 bytes in 10 blocks are definitely lost in loss record 276 of 459
==31753==    at 0x4C2CB3F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBBCF: __gf_malloc (mem-pool.c:142)
==31753==    by 0x688F1BC: gf_dnscache_init (common-utils.c:379)
==31753==    by 0x4E59760: init (io-stats.c:3786)
==31753==    by 0x68858C0: __xlator_init (xlator.c:403)
==31753==    by 0x68859E1: xlator_init (xlator.c:428)
==31753==    by 0x68CE7BA: glusterfs_graph_init (graph.c:320)
==31753==    by 0x68CF39F: glusterfs_graph_activate (graph.c:670)
==31753==    by 0x6276EBF: glfs_process_volfp (glfs-mgmt.c:78)
==31753==    by 0x62783E3: glfs_mgmt_getspec_cbk (glfs-mgmt.c:604)
==31753==    by 0x6DA1DFD: rpc_clnt_handle_reply (rpc-clnt.c:790)
==31753==    by 0x6DA2355: rpc_clnt_notify (rpc-clnt.c:961)
==31753== 
==31753== 880 bytes in 10 blocks are definitely lost in loss record 277 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x5C5231B: init (quick-read.c:1034)
==31753==    by 0x68858C0: __xlator_init (xlator.c:403)
==31753==    by 0x68859E1: xlator_init (xlator.c:428)
==31753==    by 0x68CE7BA: glusterfs_graph_init (graph.c:320)
==31753==    by 0x68CF39F: glusterfs_graph_activate (graph.c:670)
==31753==    by 0x6276EBF: glfs_process_volfp (glfs-mgmt.c:78)
==31753==    by 0x62783E3: glfs_mgmt_getspec_cbk (glfs-mgmt.c:604)
==31753==    by 0x6DA1DFD: rpc_clnt_handle_reply (rpc-clnt.c:790)
==31753==    by 0x6DA2355: rpc_clnt_notify (rpc-clnt.c:961)
==31753==    by 0x6D9E0D9: rpc_transport_notify (rpc-transport.c:537)
==31753== 
==31753== 950 bytes in 10 blocks are definitely lost in loss record 282 of 459
==31753==    at 0x4C2CB3F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBBCF: __gf_malloc (mem-pool.c:142)
==31753==    by 0x68BBEBB: gf_vasprintf (mem-pool.c:221)
==31753==    by 0x68BBFD0: gf_asprintf (mem-pool.c:240)
==31753==    by 0x5575C76: dht_init (dht-shared.c:822)
==31753==    by 0x68858C0: __xlator_init (xlator.c:403)
==31753==    by 0x68859E1: xlator_init (xlator.c:428)
==31753==    by 0x68CE7BA: glusterfs_graph_init (graph.c:320)
==31753==    by 0x68CF39F: glusterfs_graph_activate (graph.c:670)
==31753==    by 0x6276EBF: glfs_process_volfp (glfs-mgmt.c:78)
==31753==    by 0x62783E3: glfs_mgmt_getspec_cbk (glfs-mgmt.c:604)
==31753==    by 0x6DA1DFD: rpc_clnt_handle_reply (rpc-clnt.c:790)
==31753== 
==31753== 960 bytes in 10 blocks are definitely lost in loss record 286 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x54ED75E: dht_init_subvolumes (dht-helper.c:1048)
==31753==    by 0x557573A: dht_init (dht-shared.c:757)
==31753==    by 0x68858C0: __xlator_init (xlator.c:403)
==31753==    by 0x68859E1: xlator_init (xlator.c:428)
==31753==    by 0x68CE7BA: glusterfs_graph_init (graph.c:320)
==31753==    by 0x68CF39F: glusterfs_graph_activate (graph.c:670)
==31753==    by 0x6276EBF: glfs_process_volfp (glfs-mgmt.c:78)
==31753==    by 0x62783E3: glfs_mgmt_getspec_cbk (glfs-mgmt.c:604)
==31753==    by 0x6DA1DFD: rpc_clnt_handle_reply (rpc-clnt.c:790)
==31753==    by 0x6DA2355: rpc_clnt_notify (rpc-clnt.c:961)
==31753== 
==31753== 960 bytes in 10 blocks are definitely lost in loss record 287 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x54ED7E5: dht_init_subvolumes (dht-helper.c:1060)
==31753==    by 0x557573A: dht_init (dht-shared.c:757)
==31753==    by 0x68858C0: __xlator_init (xlator.c:403)
==31753==    by 0x68859E1: xlator_init (xlator.c:428)
==31753==    by 0x68CE7BA: glusterfs_graph_init (graph.c:320)
==31753==    by 0x68CF39F: glusterfs_graph_activate (graph.c:670)
==31753==    by 0x6276EBF: glfs_process_volfp (glfs-mgmt.c:78)
==31753==    by 0x62783E3: glfs_mgmt_getspec_cbk (glfs-mgmt.c:604)
==31753==    by 0x6DA1DFD: rpc_clnt_handle_reply (rpc-clnt.c:790)
==31753==    by 0x6DA2355: rpc_clnt_notify (rpc-clnt.c:961)
==31753== 
==31753== 1,010 bytes in 10 blocks are definitely lost in loss record 291 of 459
==31753==    at 0x4C2CB3F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBBCF: __gf_malloc (mem-pool.c:142)
==31753==    by 0x68BBEBB: gf_vasprintf (mem-pool.c:221)
==31753==    by 0x68BBFD0: gf_asprintf (mem-pool.c:240)
==31753==    by 0x5575C1C: dht_init (dht-shared.c:818)
==31753==    by 0x68858C0: __xlator_init (xlator.c:403)
==31753==    by 0x68859E1: xlator_init (xlator.c:428)
==31753==    by 0x68CE7BA: glusterfs_graph_init (graph.c:320)
==31753==    by 0x68CF39F: glusterfs_graph_activate (graph.c:670)
==31753==    by 0x6276EBF: glfs_process_volfp (glfs-mgmt.c:78)
==31753==    by 0x62783E3: glfs_mgmt_getspec_cbk (glfs-mgmt.c:604)
==31753==    by 0x6DA1DFD: rpc_clnt_handle_reply (rpc-clnt.c:790)
==31753== 
==31753== 1,050 bytes in 10 blocks are definitely lost in loss record 293 of 459
==31753==    at 0x4C2CB3F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBBCF: __gf_malloc (mem-pool.c:142)
==31753==    by 0x68BBEBB: gf_vasprintf (mem-pool.c:221)
==31753==    by 0x68BBFD0: gf_asprintf (mem-pool.c:240)
==31753==    by 0x5575C49: dht_init (dht-shared.c:820)
==31753==    by 0x68858C0: __xlator_init (xlator.c:403)
==31753==    by 0x68859E1: xlator_init (xlator.c:428)
==31753==    by 0x68CE7BA: glusterfs_graph_init (graph.c:320)
==31753==    by 0x68CF39F: glusterfs_graph_activate (graph.c:670)
==31753==    by 0x6276EBF: glfs_process_volfp (glfs-mgmt.c:78)
==31753==    by 0x62783E3: glfs_mgmt_getspec_cbk (glfs-mgmt.c:604)
==31753==    by 0x6DA1DFD: rpc_clnt_handle_reply (rpc-clnt.c:790)
==31753== 
==31753== 1,300 bytes in 10 blocks are definitely lost in loss record 304 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x68836EB: dict_allocate_and_serialize (dict.c:2797)
==31753==    by 0x627873F: glfs_volfile_fetch (glfs-mgmt.c:690)
==31753==    by 0x6278D6B: mgmt_rpc_notify (glfs-mgmt.c:846)
==31753==    by 0x6DA2410: rpc_clnt_notify (rpc-clnt.c:993)
==31753==    by 0x6D9E0D9: rpc_transport_notify (rpc-transport.c:537)
==31753==    by 0xD9B7677: socket_connect_finish (socket.c:2341)
==31753==    by 0xD9B7873: socket_event_handler (socket.c:2384)
==31753==    by 0x68F92AC: event_dispatch_epoll_handler (event-epoll.c:571)
==31753==    by 0x68F969A: event_dispatch_epoll_worker (event-epoll.c:674)
==31753==    by 0x6B78709: start_thread (pthread_create.c:333)
==31753== 
==31753== 1,400 bytes in 10 blocks are definitely lost in loss record 306 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x68BC9BB: mem_get (mem-pool.c:513)
==31753==    by 0x68BC7DE: mem_get0 (mem-pool.c:446)
==31753==    by 0x687E568: dict_set_lk (dict.c:358)
==31753==    by 0x687E7C8: dict_set (dict.c:422)
==31753==    by 0x6881A4C: dict_set_int32 (dict.c:1795)
==31753==    by 0x6D9E594: rpc_transport_inet_options_build (rpc-transport.c:681)
==31753==    by 0x6278FBD: glfs_mgmt_init (glfs-mgmt.c:917)
==31753==    by 0x6274863: glfs_volumes_init (glfs.c:266)
==31753==    by 0x6275D6B: glfs_init_common (glfs.c:989)
==31753==    by 0x6275F21: glfs_init@@GFAPI_3.4.0 (glfs.c:1034)
==31753== 
==31753== 1,400 bytes in 10 blocks are definitely lost in loss record 307 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x68BC9BB: mem_get (mem-pool.c:513)
==31753==    by 0x68BC7DE: mem_get0 (mem-pool.c:446)
==31753==    by 0x687E568: dict_set_lk (dict.c:358)
==31753==    by 0x687E7C8: dict_set (dict.c:422)
==31753==    by 0x6882452: dict_set_str (dict.c:2186)
==31753==    by 0x6D9E5FD: rpc_transport_inet_options_build (rpc-transport.c:688)
==31753==    by 0x6278FBD: glfs_mgmt_init (glfs-mgmt.c:917)
==31753==    by 0x6274863: glfs_volumes_init (glfs.c:266)
==31753==    by 0x6275D6B: glfs_init_common (glfs.c:989)
==31753==    by 0x6275F21: glfs_init@@GFAPI_3.4.0 (glfs.c:1034)
==31753== 
==31753== 1,640 bytes in 10 blocks are definitely lost in loss record 313 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x68BC9BB: mem_get (mem-pool.c:513)
==31753==    by 0x68BC7DE: mem_get0 (mem-pool.c:446)
==31753==    by 0x687DABD: get_new_data (dict.c:40)
==31753==    by 0x687FA26: data_from_dynstr (dict.c:900)
==31753==    by 0x688250D: dict_set_dynstr (dict.c:2217)
==31753==    by 0x6D9E521: rpc_transport_inet_options_build (rpc-transport.c:673)
==31753==    by 0x6278FBD: glfs_mgmt_init (glfs-mgmt.c:917)
==31753==    by 0x6274863: glfs_volumes_init (glfs.c:266)
==31753==    by 0x6275D6B: glfs_init_common (glfs.c:989)
==31753==    by 0x6275F21: glfs_init@@GFAPI_3.4.0 (glfs.c:1034)
==31753== 
==31753== 1,640 bytes in 10 blocks are definitely lost in loss record 314 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x68BC9BB: mem_get (mem-pool.c:513)
==31753==    by 0x68BC7DE: mem_get0 (mem-pool.c:446)
==31753==    by 0x687DABD: get_new_data (dict.c:40)
==31753==    by 0x687F3C2: data_from_int32 (dict.c:716)
==31753==    by 0x6881A21: dict_set_int32 (dict.c:1789)
==31753==    by 0x6D9E594: rpc_transport_inet_options_build (rpc-transport.c:681)
==31753==    by 0x6278FBD: glfs_mgmt_init (glfs-mgmt.c:917)
==31753==    by 0x6274863: glfs_volumes_init (glfs.c:266)
==31753==    by 0x6275D6B: glfs_init_common (glfs.c:989)
==31753==    by 0x6275F21: glfs_init@@GFAPI_3.4.0 (glfs.c:1034)
==31753== 
==31753== 1,640 bytes in 10 blocks are definitely lost in loss record 315 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x68BC9BB: mem_get (mem-pool.c:513)
==31753==    by 0x68BC7DE: mem_get0 (mem-pool.c:446)
==31753==    by 0x687DABD: get_new_data (dict.c:40)
==31753==    by 0x687F973: str_to_data (dict.c:878)
==31753==    by 0x6882427: dict_set_str (dict.c:2180)
==31753==    by 0x6D9E5FD: rpc_transport_inet_options_build (rpc-transport.c:688)
==31753==    by 0x6278FBD: glfs_mgmt_init (glfs-mgmt.c:917)
==31753==    by 0x6274863: glfs_volumes_init (glfs.c:266)
==31753==    by 0x6275D6B: glfs_init_common (glfs.c:989)
==31753==    by 0x6275F21: glfs_init@@GFAPI_3.4.0 (glfs.c:1034)
==31753== 
==31753== 1,680 bytes in 10 blocks are definitely lost in loss record 317 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x54ED7A8: dht_init_subvolumes (dht-helper.c:1054)
==31753==    by 0x557573A: dht_init (dht-shared.c:757)
==31753==    by 0x68858C0: __xlator_init (xlator.c:403)
==31753==    by 0x68859E1: xlator_init (xlator.c:428)
==31753==    by 0x68CE7BA: glusterfs_graph_init (graph.c:320)
==31753==    by 0x68CF39F: glusterfs_graph_activate (graph.c:670)
==31753==    by 0x6276EBF: glfs_process_volfp (glfs-mgmt.c:78)
==31753==    by 0x62783E3: glfs_mgmt_getspec_cbk (glfs-mgmt.c:604)
==31753==    by 0x6DA1DFD: rpc_clnt_handle_reply (rpc-clnt.c:790)
==31753==    by 0x6DA2355: rpc_clnt_notify (rpc-clnt.c:961)
==31753== 
==31753== 2,080 bytes in 20 blocks are definitely lost in loss record 330 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x4E583A6: ios_init_top_stats (io-stats.c:3438)
==31753==    by 0x4E594F0: init (io-stats.c:3758)
==31753==    by 0x68858C0: __xlator_init (xlator.c:403)
==31753==    by 0x68859E1: xlator_init (xlator.c:428)
==31753==    by 0x68CE7BA: glusterfs_graph_init (graph.c:320)
==31753==    by 0x68CF39F: glusterfs_graph_activate (graph.c:670)
==31753==    by 0x6276EBF: glfs_process_volfp (glfs-mgmt.c:78)
==31753==    by 0x62783E3: glfs_mgmt_getspec_cbk (glfs-mgmt.c:604)
==31753==    by 0x6DA1DFD: rpc_clnt_handle_reply (rpc-clnt.c:790)
==31753==    by 0x6DA2355: rpc_clnt_notify (rpc-clnt.c:961)
==31753== 
==31753== 2,250 bytes in 30 blocks are definitely lost in loss record 333 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x58188EB: init (afr.c:515)
==31753==    by 0x68858C0: __xlator_init (xlator.c:403)
==31753==    by 0x68859E1: xlator_init (xlator.c:428)
==31753==    by 0x68CE7BA: glusterfs_graph_init (graph.c:320)
==31753==    by 0x68CF39F: glusterfs_graph_activate (graph.c:670)
==31753==    by 0x6276EBF: glfs_process_volfp (glfs-mgmt.c:78)
==31753==    by 0x62783E3: glfs_mgmt_getspec_cbk (glfs-mgmt.c:604)
==31753==    by 0x6DA1DFD: rpc_clnt_handle_reply (rpc-clnt.c:790)
==31753==    by 0x6DA2355: rpc_clnt_notify (rpc-clnt.c:961)
==31753==    by 0x6D9E0D9: rpc_transport_notify (rpc-transport.c:537)
==31753== 
==31753== 2,560 bytes in 10 blocks are definitely lost in loss record 340 of 459
==31753==    at 0x4C2CB3F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x657DA8A: regcomp (regcomp.c:479)
==31753==    by 0x5574101: dht_init_regex (dht-shared.c:359)
==31753==    by 0x5575848: dht_init (dht-shared.c:778)
==31753==    by 0x68858C0: __xlator_init (xlator.c:403)
==31753==    by 0x68859E1: xlator_init (xlator.c:428)
==31753==    by 0x68CE7BA: glusterfs_graph_init (graph.c:320)
==31753==    by 0x68CF39F: glusterfs_graph_activate (graph.c:670)
==31753==    by 0x6276EBF: glfs_process_volfp (glfs-mgmt.c:78)
==31753==    by 0x62783E3: glfs_mgmt_getspec_cbk (glfs-mgmt.c:604)
==31753==    by 0x6DA1DFD: rpc_clnt_handle_reply (rpc-clnt.c:790)
==31753==    by 0x6DA2355: rpc_clnt_notify (rpc-clnt.c:961)
==31753== 
==31753== 2,940 bytes in 30 blocks are definitely lost in loss record 349 of 459
==31753==    at 0x4C2CB3F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBBCF: __gf_malloc (mem-pool.c:142)
==31753==    by 0x68BBEBB: gf_vasprintf (mem-pool.c:221)
==31753==    by 0x68BBFD0: gf_asprintf (mem-pool.c:240)
==31753==    by 0x5818A8C: init (afr.c:555)
==31753==    by 0x68858C0: __xlator_init (xlator.c:403)
==31753==    by 0x68859E1: xlator_init (xlator.c:428)
==31753==    by 0x68CE7BA: glusterfs_graph_init (graph.c:320)
==31753==    by 0x68CF39F: glusterfs_graph_activate (graph.c:670)
==31753==    by 0x6276EBF: glfs_process_volfp (glfs-mgmt.c:78)
==31753==    by 0x62783E3: glfs_mgmt_getspec_cbk (glfs-mgmt.c:604)
==31753==    by 0x6DA1DFD: rpc_clnt_handle_reply (rpc-clnt.c:790)
==31753== 
==31753== 6,660 bytes in 90 blocks are definitely lost in loss record 368 of 459
==31753==    at 0x4C2CB3F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBBCF: __gf_malloc (mem-pool.c:142)
==31753==    by 0x68BBEBB: gf_vasprintf (mem-pool.c:221)
==31753==    by 0x68BBFD0: gf_asprintf (mem-pool.c:240)
==31753==    by 0x687F7B5: data_from_uint32 (dict.c:823)
==31753==    by 0x6881DA0: dict_set_uint32 (dict.c:1925)
==31753==    by 0x52C53E8: client_setvolume (client-handshake.c:1373)
==31753==    by 0x52C6139: client_dump_version_cbk (client-handshake.c:1662)
==31753==    by 0x6DA1DFD: rpc_clnt_handle_reply (rpc-clnt.c:790)
==31753==    by 0x6DA2355: rpc_clnt_notify (rpc-clnt.c:961)
==31753==    by 0x6D9E0D9: rpc_transport_notify (rpc-transport.c:537)
==31753==    by 0xD9B734D: socket_event_poll_in (socket.c:2265)
==31753== 
==31753== 6,660 bytes in 90 blocks are definitely lost in loss record 369 of 459
==31753==    at 0x4C2CB3F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBBCF: __gf_malloc (mem-pool.c:142)
==31753==    by 0x68BBEBB: gf_vasprintf (mem-pool.c:221)
==31753==    by 0x68BBFD0: gf_asprintf (mem-pool.c:240)
==31753==    by 0x687F4C4: data_from_int16 (dict.c:741)
==31753==    by 0x68818F9: dict_set_int16 (dict.c:1744)
==31753==    by 0x52C545D: client_setvolume (client-handshake.c:1381)
==31753==    by 0x52C6139: client_dump_version_cbk (client-handshake.c:1662)
==31753==    by 0x6DA1DFD: rpc_clnt_handle_reply (rpc-clnt.c:790)
==31753==    by 0x6DA2355: rpc_clnt_notify (rpc-clnt.c:961)
==31753==    by 0x6D9E0D9: rpc_transport_notify (rpc-transport.c:537)
==31753==    by 0xD9B734D: socket_event_poll_in (socket.c:2265)
==31753== 
==31753== 7,200 bytes in 90 blocks are definitely lost in loss record 371 of 459
==31753==    at 0x4C2CB3F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBBCF: __gf_malloc (mem-pool.c:142)
==31753==    by 0x68BBEBB: gf_vasprintf (mem-pool.c:221)
==31753==    by 0x68BBFD0: gf_asprintf (mem-pool.c:240)
==31753==    by 0x687F3F8: data_from_int32 (dict.c:721)
==31753==    by 0x6881A21: dict_set_int32 (dict.c:1789)
==31753==    by 0x52C505B: client_setvolume (client-handshake.c:1302)
==31753==    by 0x52C6139: client_dump_version_cbk (client-handshake.c:1662)
==31753==    by 0x6DA1DFD: rpc_clnt_handle_reply (rpc-clnt.c:790)
==31753==    by 0x6DA2355: rpc_clnt_notify (rpc-clnt.c:961)
==31753==    by 0x6D9E0D9: rpc_transport_notify (rpc-transport.c:537)
==31753==    by 0xD9B734D: socket_event_poll_in (socket.c:2265)
==31753== 
==31753== 7,560 bytes in 90 blocks are definitely lost in loss record 372 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x687E5ED: dict_set_lk (dict.c:376)
==31753==    by 0x687E7C8: dict_set (dict.c:422)
==31753==    by 0x6882452: dict_set_str (dict.c:2186)
==31753==    by 0x52C5371: client_setvolume (client-handshake.c:1366)
==31753==    by 0x52C6139: client_dump_version_cbk (client-handshake.c:1662)
==31753==    by 0x6DA1DFD: rpc_clnt_handle_reply (rpc-clnt.c:790)
==31753==    by 0x6DA2355: rpc_clnt_notify (rpc-clnt.c:961)
==31753==    by 0x6D9E0D9: rpc_transport_notify (rpc-transport.c:537)
==31753==    by 0xD9B734D: socket_event_poll_in (socket.c:2265)
==31753==    by 0xD9B78B2: socket_event_handler (socket.c:2395)
==31753== 
==31753== 7,650 bytes in 90 blocks are definitely lost in loss record 375 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x687E5ED: dict_set_lk (dict.c:376)
==31753==    by 0x687E7C8: dict_set (dict.c:422)
==31753==    by 0x6881A4C: dict_set_int32 (dict.c:1795)
==31753==    by 0x52C505B: client_setvolume (client-handshake.c:1302)
==31753==    by 0x52C6139: client_dump_version_cbk (client-handshake.c:1662)
==31753==    by 0x6DA1DFD: rpc_clnt_handle_reply (rpc-clnt.c:790)
==31753==    by 0x6DA2355: rpc_clnt_notify (rpc-clnt.c:961)
==31753==    by 0x6D9E0D9: rpc_transport_notify (rpc-transport.c:537)
==31753==    by 0xD9B734D: socket_event_poll_in (socket.c:2265)
==31753==    by 0xD9B78B2: socket_event_handler (socket.c:2395)
==31753== 
==31753== 7,650 bytes in 90 blocks are definitely lost in loss record 376 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x687E5ED: dict_set_lk (dict.c:376)
==31753==    by 0x687E7C8: dict_set (dict.c:422)
==31753==    by 0x6882538: dict_set_dynstr (dict.c:2223)
==31753==    by 0x52C5254: client_setvolume (client-handshake.c:1349)
==31753==    by 0x52C6139: client_dump_version_cbk (client-handshake.c:1662)
==31753==    by 0x6DA1DFD: rpc_clnt_handle_reply (rpc-clnt.c:790)
==31753==    by 0x6DA2355: rpc_clnt_notify (rpc-clnt.c:961)
==31753==    by 0x6D9E0D9: rpc_transport_notify (rpc-transport.c:537)
==31753==    by 0xD9B734D: socket_event_poll_in (socket.c:2265)
==31753==    by 0xD9B78B2: socket_event_handler (socket.c:2395)
==31753== 
==31753== 7,830 bytes in 90 blocks are definitely lost in loss record 378 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x687E5ED: dict_set_lk (dict.c:376)
==31753==    by 0x687E7C8: dict_set (dict.c:422)
==31753==    by 0x6882452: dict_set_str (dict.c:2186)
==31753==    by 0x52C52C7: client_setvolume (client-handshake.c:1357)
==31753==    by 0x52C6139: client_dump_version_cbk (client-handshake.c:1662)
==31753==    by 0x6DA1DFD: rpc_clnt_handle_reply (rpc-clnt.c:790)
==31753==    by 0x6DA2355: rpc_clnt_notify (rpc-clnt.c:961)
==31753==    by 0x6D9E0D9: rpc_transport_notify (rpc-transport.c:537)
==31753==    by 0xD9B734D: socket_event_poll_in (socket.c:2265)
==31753==    by 0xD9B78B2: socket_event_handler (socket.c:2395)
==31753== 
==31753== 7,920 bytes in 90 blocks are definitely lost in loss record 380 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x687E5ED: dict_set_lk (dict.c:376)
==31753==    by 0x687E7C8: dict_set (dict.c:422)
==31753==    by 0x6881924: dict_set_int16 (dict.c:1750)
==31753==    by 0x52C545D: client_setvolume (client-handshake.c:1381)
==31753==    by 0x52C6139: client_dump_version_cbk (client-handshake.c:1662)
==31753==    by 0x6DA1DFD: rpc_clnt_handle_reply (rpc-clnt.c:790)
==31753==    by 0x6DA2355: rpc_clnt_notify (rpc-clnt.c:961)
==31753==    by 0x6D9E0D9: rpc_transport_notify (rpc-transport.c:537)
==31753==    by 0xD9B734D: socket_event_poll_in (socket.c:2265)
==31753==    by 0xD9B78B2: socket_event_handler (socket.c:2395)
==31753== 
==31753== 8,010 bytes in 90 blocks are definitely lost in loss record 381 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x687E5ED: dict_set_lk (dict.c:376)
==31753==    by 0x687E7C8: dict_set (dict.c:422)
==31753==    by 0x6881DCB: dict_set_uint32 (dict.c:1931)
==31753==    by 0x52C53E8: client_setvolume (client-handshake.c:1373)
==31753==    by 0x52C6139: client_dump_version_cbk (client-handshake.c:1662)
==31753==    by 0x6DA1DFD: rpc_clnt_handle_reply (rpc-clnt.c:790)
==31753==    by 0x6DA2355: rpc_clnt_notify (rpc-clnt.c:961)
==31753==    by 0x6D9E0D9: rpc_transport_notify (rpc-transport.c:537)
==31753==    by 0xD9B734D: socket_event_poll_in (socket.c:2265)
==31753==    by 0xD9B78B2: socket_event_handler (socket.c:2395)
==31753== 
==31753== 8,320 bytes in 80 blocks are definitely lost in loss record 384 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x4E5827A: ios_init_top_stats (io-stats.c:3426)
==31753==    by 0x4E594F0: init (io-stats.c:3758)
==31753==    by 0x68858C0: __xlator_init (xlator.c:403)
==31753==    by 0x68859E1: xlator_init (xlator.c:428)
==31753==    by 0x68CE7BA: glusterfs_graph_init (graph.c:320)
==31753==    by 0x68CF39F: glusterfs_graph_activate (graph.c:670)
==31753==    by 0x6276EBF: glfs_process_volfp (glfs-mgmt.c:78)
==31753==    by 0x62783E3: glfs_mgmt_getspec_cbk (glfs-mgmt.c:604)
==31753==    by 0x6DA1DFD: rpc_clnt_handle_reply (rpc-clnt.c:790)
==31753==    by 0x6DA2355: rpc_clnt_notify (rpc-clnt.c:961)
==31753== 
==31753== 10,136 (320 direct, 9,816 indirect) bytes in 2 blocks are definitely lost in loss record 388 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x6DA36A1: rpcclnt_cbk_program_register (rpc-clnt.c:1506)
==31753==    by 0x627910E: glfs_mgmt_init (glfs-mgmt.c:944)
==31753==    by 0x6274863: glfs_volumes_init (glfs.c:266)
==31753==    by 0x6275D6B: glfs_init_common (glfs.c:989)
==31753==    by 0x6275F21: glfs_init@@GFAPI_3.4.0 (glfs.c:1034)
==31753==    by 0x1088D2: glusterCatastrophe() (in /home/matt/dev/glusterfs_memory_leak/simple)
==31753==    by 0x1088FB: main (in /home/matt/dev/glusterfs_memory_leak/simple)
==31753== 
==31753== 11,430 bytes in 90 blocks are definitely lost in loss record 390 of 459
==31753==    at 0x4C2CB3F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBBCF: __gf_malloc (mem-pool.c:142)
==31753==    by 0x68BBEBB: gf_vasprintf (mem-pool.c:221)
==31753==    by 0x68BBFD0: gf_asprintf (mem-pool.c:240)
==31753==    by 0x52C51E5: client_setvolume (client-handshake.c:1339)
==31753==    by 0x52C6139: client_dump_version_cbk (client-handshake.c:1662)
==31753==    by 0x6DA1DFD: rpc_clnt_handle_reply (rpc-clnt.c:790)
==31753==    by 0x6DA2355: rpc_clnt_notify (rpc-clnt.c:961)
==31753==    by 0x6D9E0D9: rpc_transport_notify (rpc-transport.c:537)
==31753==    by 0xD9B734D: socket_event_poll_in (socket.c:2265)
==31753==    by 0xD9B78B2: socket_event_handler (socket.c:2395)
==31753==    by 0x68F92AC: event_dispatch_epoll_handler (event-epoll.c:571)
==31753== 
==31753== 12,600 bytes in 90 blocks are definitely lost in loss record 391 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x68BC9BB: mem_get (mem-pool.c:513)
==31753==    by 0x68BC7DE: mem_get0 (mem-pool.c:446)
==31753==    by 0x687E568: dict_set_lk (dict.c:358)
==31753==    by 0x687E7C8: dict_set (dict.c:422)
==31753==    by 0x6881A4C: dict_set_int32 (dict.c:1795)
==31753==    by 0x52C505B: client_setvolume (client-handshake.c:1302)
==31753==    by 0x52C6139: client_dump_version_cbk (client-handshake.c:1662)
==31753==    by 0x6DA1DFD: rpc_clnt_handle_reply (rpc-clnt.c:790)
==31753==    by 0x6DA2355: rpc_clnt_notify (rpc-clnt.c:961)
==31753==    by 0x6D9E0D9: rpc_transport_notify (rpc-transport.c:537)
==31753== 
==31753== 12,600 bytes in 90 blocks are definitely lost in loss record 392 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x68BC9BB: mem_get (mem-pool.c:513)
==31753==    by 0x68BC7DE: mem_get0 (mem-pool.c:446)
==31753==    by 0x687E568: dict_set_lk (dict.c:358)
==31753==    by 0x687E7C8: dict_set (dict.c:422)
==31753==    by 0x6882538: dict_set_dynstr (dict.c:2223)
==31753==    by 0x52C5254: client_setvolume (client-handshake.c:1349)
==31753==    by 0x52C6139: client_dump_version_cbk (client-handshake.c:1662)
==31753==    by 0x6DA1DFD: rpc_clnt_handle_reply (rpc-clnt.c:790)
==31753==    by 0x6DA2355: rpc_clnt_notify (rpc-clnt.c:961)
==31753==    by 0x6D9E0D9: rpc_transport_notify (rpc-transport.c:537)
==31753== 
==31753== 12,600 bytes in 90 blocks are definitely lost in loss record 393 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x68BC9BB: mem_get (mem-pool.c:513)
==31753==    by 0x68BC7DE: mem_get0 (mem-pool.c:446)
==31753==    by 0x687E568: dict_set_lk (dict.c:358)
==31753==    by 0x687E7C8: dict_set (dict.c:422)
==31753==    by 0x6882452: dict_set_str (dict.c:2186)
==31753==    by 0x52C52C7: client_setvolume (client-handshake.c:1357)
==31753==    by 0x52C6139: client_dump_version_cbk (client-handshake.c:1662)
==31753==    by 0x6DA1DFD: rpc_clnt_handle_reply (rpc-clnt.c:790)
==31753==    by 0x6DA2355: rpc_clnt_notify (rpc-clnt.c:961)
==31753==    by 0x6D9E0D9: rpc_transport_notify (rpc-transport.c:537)
==31753== 
==31753== 12,600 bytes in 90 blocks are definitely lost in loss record 394 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x68BC9BB: mem_get (mem-pool.c:513)
==31753==    by 0x68BC7DE: mem_get0 (mem-pool.c:446)
==31753==    by 0x687E568: dict_set_lk (dict.c:358)
==31753==    by 0x687E7C8: dict_set (dict.c:422)
==31753==    by 0x6882452: dict_set_str (dict.c:2186)
==31753==    by 0x52C5371: client_setvolume (client-handshake.c:1366)
==31753==    by 0x52C6139: client_dump_version_cbk (client-handshake.c:1662)
==31753==    by 0x6DA1DFD: rpc_clnt_handle_reply (rpc-clnt.c:790)
==31753==    by 0x6DA2355: rpc_clnt_notify (rpc-clnt.c:961)
==31753==    by 0x6D9E0D9: rpc_transport_notify (rpc-transport.c:537)
==31753== 
==31753== 12,600 bytes in 90 blocks are definitely lost in loss record 395 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x68BC9BB: mem_get (mem-pool.c:513)
==31753==    by 0x68BC7DE: mem_get0 (mem-pool.c:446)
==31753==    by 0x687E568: dict_set_lk (dict.c:358)
==31753==    by 0x687E7C8: dict_set (dict.c:422)
==31753==    by 0x6881DCB: dict_set_uint32 (dict.c:1931)
==31753==    by 0x52C53E8: client_setvolume (client-handshake.c:1373)
==31753==    by 0x52C6139: client_dump_version_cbk (client-handshake.c:1662)
==31753==    by 0x6DA1DFD: rpc_clnt_handle_reply (rpc-clnt.c:790)
==31753==    by 0x6DA2355: rpc_clnt_notify (rpc-clnt.c:961)
==31753==    by 0x6D9E0D9: rpc_transport_notify (rpc-transport.c:537)
==31753== 
==31753== 12,600 bytes in 90 blocks are definitely lost in loss record 396 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x68BC9BB: mem_get (mem-pool.c:513)
==31753==    by 0x68BC7DE: mem_get0 (mem-pool.c:446)
==31753==    by 0x687E568: dict_set_lk (dict.c:358)
==31753==    by 0x687E7C8: dict_set (dict.c:422)
==31753==    by 0x6881924: dict_set_int16 (dict.c:1750)
==31753==    by 0x52C545D: client_setvolume (client-handshake.c:1381)
==31753==    by 0x52C6139: client_dump_version_cbk (client-handshake.c:1662)
==31753==    by 0x6DA1DFD: rpc_clnt_handle_reply (rpc-clnt.c:790)
==31753==    by 0x6DA2355: rpc_clnt_notify (rpc-clnt.c:961)
==31753==    by 0x6D9E0D9: rpc_transport_notify (rpc-transport.c:537)
==31753== 
==31753== 14,760 bytes in 90 blocks are definitely lost in loss record 404 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x68BC9BB: mem_get (mem-pool.c:513)
==31753==    by 0x68BC7DE: mem_get0 (mem-pool.c:446)
==31753==    by 0x687DABD: get_new_data (dict.c:40)
==31753==    by 0x687F973: str_to_data (dict.c:878)
==31753==    by 0x6882427: dict_set_str (dict.c:2180)
==31753==    by 0x6D9D1F3: rpc_transport_load (rpc-transport.c:234)
==31753==    by 0x6DA2640: rpc_clnt_connection_init (rpc-clnt.c:1053)
==31753==    by 0x6DA294C: rpc_clnt_new (rpc-clnt.c:1142)
==31753==    by 0x5291029: client_init_rpc (client.c:2523)
==31753==    by 0x5291B50: init (client.c:2746)
==31753== 
==31753== 14,760 bytes in 90 blocks are definitely lost in loss record 405 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x68BC9BB: mem_get (mem-pool.c:513)
==31753==    by 0x68BC7DE: mem_get0 (mem-pool.c:446)
==31753==    by 0x687DABD: get_new_data (dict.c:40)
==31753==    by 0x687F3C2: data_from_int32 (dict.c:716)
==31753==    by 0x6881A21: dict_set_int32 (dict.c:1789)
==31753==    by 0x52C505B: client_setvolume (client-handshake.c:1302)
==31753==    by 0x52C6139: client_dump_version_cbk (client-handshake.c:1662)
==31753==    by 0x6DA1DFD: rpc_clnt_handle_reply (rpc-clnt.c:790)
==31753==    by 0x6DA2355: rpc_clnt_notify (rpc-clnt.c:961)
==31753==    by 0x6D9E0D9: rpc_transport_notify (rpc-transport.c:537)
==31753== 
==31753== 14,760 bytes in 90 blocks are definitely lost in loss record 406 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x68BC9BB: mem_get (mem-pool.c:513)
==31753==    by 0x68BC7DE: mem_get0 (mem-pool.c:446)
==31753==    by 0x687DABD: get_new_data (dict.c:40)
==31753==    by 0x687FA26: data_from_dynstr (dict.c:900)
==31753==    by 0x688250D: dict_set_dynstr (dict.c:2217)
==31753==    by 0x52C5254: client_setvolume (client-handshake.c:1349)
==31753==    by 0x52C6139: client_dump_version_cbk (client-handshake.c:1662)
==31753==    by 0x6DA1DFD: rpc_clnt_handle_reply (rpc-clnt.c:790)
==31753==    by 0x6DA2355: rpc_clnt_notify (rpc-clnt.c:961)
==31753==    by 0x6D9E0D9: rpc_transport_notify (rpc-transport.c:537)
==31753== 
==31753== 14,760 bytes in 90 blocks are definitely lost in loss record 407 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x68BC9BB: mem_get (mem-pool.c:513)
==31753==    by 0x68BC7DE: mem_get0 (mem-pool.c:446)
==31753==    by 0x687DABD: get_new_data (dict.c:40)
==31753==    by 0x687F973: str_to_data (dict.c:878)
==31753==    by 0x6882427: dict_set_str (dict.c:2180)
==31753==    by 0x52C52C7: client_setvolume (client-handshake.c:1357)
==31753==    by 0x52C6139: client_dump_version_cbk (client-handshake.c:1662)
==31753==    by 0x6DA1DFD: rpc_clnt_handle_reply (rpc-clnt.c:790)
==31753==    by 0x6DA2355: rpc_clnt_notify (rpc-clnt.c:961)
==31753==    by 0x6D9E0D9: rpc_transport_notify (rpc-transport.c:537)
==31753== 
==31753== 14,760 bytes in 90 blocks are definitely lost in loss record 408 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x68BC9BB: mem_get (mem-pool.c:513)
==31753==    by 0x68BC7DE: mem_get0 (mem-pool.c:446)
==31753==    by 0x687DABD: get_new_data (dict.c:40)
==31753==    by 0x687F973: str_to_data (dict.c:878)
==31753==    by 0x6882427: dict_set_str (dict.c:2180)
==31753==    by 0x52C5371: client_setvolume (client-handshake.c:1366)
==31753==    by 0x52C6139: client_dump_version_cbk (client-handshake.c:1662)
==31753==    by 0x6DA1DFD: rpc_clnt_handle_reply (rpc-clnt.c:790)
==31753==    by 0x6DA2355: rpc_clnt_notify (rpc-clnt.c:961)
==31753==    by 0x6D9E0D9: rpc_transport_notify (rpc-transport.c:537)
==31753== 
==31753== 14,760 bytes in 90 blocks are definitely lost in loss record 409 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x68BC9BB: mem_get (mem-pool.c:513)
==31753==    by 0x68BC7DE: mem_get0 (mem-pool.c:446)
==31753==    by 0x687DABD: get_new_data (dict.c:40)
==31753==    by 0x687F77F: data_from_uint32 (dict.c:818)
==31753==    by 0x6881DA0: dict_set_uint32 (dict.c:1925)
==31753==    by 0x52C53E8: client_setvolume (client-handshake.c:1373)
==31753==    by 0x52C6139: client_dump_version_cbk (client-handshake.c:1662)
==31753==    by 0x6DA1DFD: rpc_clnt_handle_reply (rpc-clnt.c:790)
==31753==    by 0x6DA2355: rpc_clnt_notify (rpc-clnt.c:961)
==31753==    by 0x6D9E0D9: rpc_transport_notify (rpc-transport.c:537)
==31753== 
==31753== 14,760 bytes in 90 blocks are definitely lost in loss record 410 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x68BC9BB: mem_get (mem-pool.c:513)
==31753==    by 0x68BC7DE: mem_get0 (mem-pool.c:446)
==31753==    by 0x687DABD: get_new_data (dict.c:40)
==31753==    by 0x687F48D: data_from_int16 (dict.c:736)
==31753==    by 0x68818F9: dict_set_int16 (dict.c:1744)
==31753==    by 0x52C545D: client_setvolume (client-handshake.c:1381)
==31753==    by 0x52C6139: client_dump_version_cbk (client-handshake.c:1662)
==31753==    by 0x6DA1DFD: rpc_clnt_handle_reply (rpc-clnt.c:790)
==31753==    by 0x6DA2355: rpc_clnt_notify (rpc-clnt.c:961)
==31753==    by 0x6D9E0D9: rpc_transport_notify (rpc-transport.c:537)
==31753== 
==31753== 16,560 bytes in 30 blocks are definitely lost in loss record 414 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x689AB7F: __inode_create (inode.c:614)
==31753==    by 0x689CA43: __inode_table_init_root (inode.c:1557)
==31753==    by 0x689CDE3: inode_table_new (inode.c:1638)
==31753==    by 0x57F6BAE: afr_selfheal_daemon_init (afr-self-heald.c:976)
==31753==    by 0x5818B07: init (afr.c:569)
==31753==    by 0x68858C0: __xlator_init (xlator.c:403)
==31753==    by 0x68859E1: xlator_init (xlator.c:428)
==31753==    by 0x68CE7BA: glusterfs_graph_init (graph.c:320)
==31753==    by 0x68CF39F: glusterfs_graph_activate (graph.c:670)
==31753==    by 0x6276EBF: glfs_process_volfp (glfs-mgmt.c:78)
==31753== 
==31753== 18,000 bytes in 30 blocks are definitely lost in loss record 415 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x57F6BF2: afr_selfheal_daemon_init (afr-self-heald.c:980)
==31753==    by 0x5818B07: init (afr.c:569)
==31753==    by 0x68858C0: __xlator_init (xlator.c:403)
==31753==    by 0x68859E1: xlator_init (xlator.c:428)
==31753==    by 0x68CE7BA: glusterfs_graph_init (graph.c:320)
==31753==    by 0x68CF39F: glusterfs_graph_activate (graph.c:670)
==31753==    by 0x6276EBF: glfs_process_volfp (glfs-mgmt.c:78)
==31753==    by 0x62783E3: glfs_mgmt_getspec_cbk (glfs-mgmt.c:604)
==31753==    by 0x6DA1DFD: rpc_clnt_handle_reply (rpc-clnt.c:790)
==31753==    by 0x6DA2355: rpc_clnt_notify (rpc-clnt.c:961)
==31753== 
==31753== 18,000 bytes in 30 blocks are definitely lost in loss record 416 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x57F6CB9: afr_selfheal_daemon_init (afr-self-heald.c:993)
==31753==    by 0x5818B07: init (afr.c:569)
==31753==    by 0x68858C0: __xlator_init (xlator.c:403)
==31753==    by 0x68859E1: xlator_init (xlator.c:428)
==31753==    by 0x68CE7BA: glusterfs_graph_init (graph.c:320)
==31753==    by 0x68CF39F: glusterfs_graph_activate (graph.c:670)
==31753==    by 0x6276EBF: glfs_process_volfp (glfs-mgmt.c:78)
==31753==    by 0x62783E3: glfs_mgmt_getspec_cbk (glfs-mgmt.c:604)
==31753==    by 0x6DA1DFD: rpc_clnt_handle_reply (rpc-clnt.c:790)
==31753==    by 0x6DA2355: rpc_clnt_notify (rpc-clnt.c:961)
==31753== 
==31753== 35,640 (2,240 direct, 33,400 indirect) bytes in 10 blocks are definitely lost in loss record 422 of 459
==31753==    at 0x4C2CA6F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x4C2EDEF: realloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x657C4FD: re_compile_internal (regcomp.c:747)
==31753==    by 0x657DAE9: regcomp (regcomp.c:498)
==31753==    by 0x5574101: dht_init_regex (dht-shared.c:359)
==31753==    by 0x5575848: dht_init (dht-shared.c:778)
==31753==    by 0x68858C0: __xlator_init (xlator.c:403)
==31753==    by 0x68859E1: xlator_init (xlator.c:428)
==31753==    by 0x68CE7BA: glusterfs_graph_init (graph.c:320)
==31753==    by 0x68CF39F: glusterfs_graph_activate (graph.c:670)
==31753==    by 0x6276EBF: glfs_process_volfp (glfs-mgmt.c:78)
==31753==    by 0x62783E3: glfs_mgmt_getspec_cbk (glfs-mgmt.c:604)
==31753== 
==31753== 37,850 bytes in 490 blocks are definitely lost in loss record 424 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x691B241: gf_strdup (mem-pool.h:187)
==31753==    by 0x691C324: volume_option (graph.y:240)
==31753==    by 0x691B7F1: graphyyparse (graph.y:69)
==31753==    by 0x691D18C: glusterfs_graph_construct (graph.y:590)
==31753==    by 0x6276D9F: glfs_process_volfp (glfs-mgmt.c:54)
==31753==    by 0x62783E3: glfs_mgmt_getspec_cbk (glfs-mgmt.c:604)
==31753==    by 0x6DA1DFD: rpc_clnt_handle_reply (rpc-clnt.c:790)
==31753==    by 0x6DA2355: rpc_clnt_notify (rpc-clnt.c:961)
==31753==    by 0x6D9E0D9: rpc_transport_notify (rpc-transport.c:537)
==31753==    by 0xD9B734D: socket_event_poll_in (socket.c:2265)
==31753== 
==31753== 37,890 bytes in 10 blocks are definitely lost in loss record 425 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x6277F22: glusterfs_oldvolfile_update (glfs-mgmt.c:490)
==31753==    by 0x627840B: glfs_mgmt_getspec_cbk (glfs-mgmt.c:610)
==31753==    by 0x6DA1DFD: rpc_clnt_handle_reply (rpc-clnt.c:790)
==31753==    by 0x6DA2355: rpc_clnt_notify (rpc-clnt.c:961)
==31753==    by 0x6D9E0D9: rpc_transport_notify (rpc-transport.c:537)
==31753==    by 0xD9B734D: socket_event_poll_in (socket.c:2265)
==31753==    by 0xD9B78B2: socket_event_handler (socket.c:2395)
==31753==    by 0x68F92AC: event_dispatch_epoll_handler (event-epoll.c:571)
==31753==    by 0x68F969A: event_dispatch_epoll_worker (event-epoll.c:674)
==31753==    by 0x6B78709: start_thread (pthread_create.c:333)
==31753== 
==31753== 59,040 bytes in 670 blocks are definitely lost in loss record 430 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x687E5ED: dict_set_lk (dict.c:376)
==31753==    by 0x687E7C8: dict_set (dict.c:422)
==31753==    by 0x6882538: dict_set_dynstr (dict.c:2223)
==31753==    by 0x691C349: volume_option (graph.y:241)
==31753==    by 0x691B7F1: graphyyparse (graph.y:69)
==31753==    by 0x691D18C: glusterfs_graph_construct (graph.y:590)
==31753==    by 0x6276D9F: glfs_process_volfp (glfs-mgmt.c:54)
==31753==    by 0x62783E3: glfs_mgmt_getspec_cbk (glfs-mgmt.c:604)
==31753==    by 0x6DA1DFD: rpc_clnt_handle_reply (rpc-clnt.c:790)
==31753==    by 0x6DA2355: rpc_clnt_notify (rpc-clnt.c:961)
==31753== 
==31753== 69,840 (2,880 direct, 66,960 indirect) bytes in 30 blocks are definitely lost in loss record 433 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x57F6DB5: afr_selfheal_daemon_init (afr-self-heald.c:1010)
==31753==    by 0x5818B07: init (afr.c:569)
==31753==    by 0x68858C0: __xlator_init (xlator.c:403)
==31753==    by 0x68859E1: xlator_init (xlator.c:428)
==31753==    by 0x68CE7BA: glusterfs_graph_init (graph.c:320)
==31753==    by 0x68CF39F: glusterfs_graph_activate (graph.c:670)
==31753==    by 0x6276EBF: glfs_process_volfp (glfs-mgmt.c:78)
==31753==    by 0x62783E3: glfs_mgmt_getspec_cbk (glfs-mgmt.c:604)
==31753==    by 0x6DA1DFD: rpc_clnt_handle_reply (rpc-clnt.c:790)
==31753==    by 0x6DA2355: rpc_clnt_notify (rpc-clnt.c:961)
==31753== 
==31753== 74,200 bytes in 530 blocks are definitely lost in loss record 434 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x68BC9BB: mem_get (mem-pool.c:513)
==31753==    by 0x68BC7DE: mem_get0 (mem-pool.c:446)
==31753==    by 0x687E568: dict_set_lk (dict.c:358)
==31753==    by 0x687E7C8: dict_set (dict.c:422)
==31753==    by 0x6882538: dict_set_dynstr (dict.c:2223)
==31753==    by 0x691C349: volume_option (graph.y:241)
==31753==    by 0x691B7F1: graphyyparse (graph.y:69)
==31753==    by 0x691D18C: glusterfs_graph_construct (graph.y:590)
==31753==    by 0x6276D9F: glfs_process_volfp (glfs-mgmt.c:54)
==31753==    by 0x62783E3: glfs_mgmt_getspec_cbk (glfs-mgmt.c:604)
==31753== 
==31753== 95,120 bytes in 580 blocks are definitely lost in loss record 436 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x68BC9BB: mem_get (mem-pool.c:513)
==31753==    by 0x68BC7DE: mem_get0 (mem-pool.c:446)
==31753==    by 0x687DABD: get_new_data (dict.c:40)
==31753==    by 0x687FA26: data_from_dynstr (dict.c:900)
==31753==    by 0x688250D: dict_set_dynstr (dict.c:2217)
==31753==    by 0x691C349: volume_option (graph.y:241)
==31753==    by 0x691B7F1: graphyyparse (graph.y:69)
==31753==    by 0x691D18C: glusterfs_graph_construct (graph.y:590)
==31753==    by 0x6276D9F: glfs_process_volfp (glfs-mgmt.c:54)
==31753==    by 0x62783E3: glfs_mgmt_getspec_cbk (glfs-mgmt.c:604)
==31753== 
==31753== 125,064 (3,600 direct, 121,464 indirect) bytes in 10 blocks are definitely lost in loss record 438 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x68BF3BC: iobuf_create_stdalloc_arena (iobuf.c:367)
==31753==    by 0x68BF738: iobuf_pool_new (iobuf.c:431)
==31753==    by 0x627410B: glusterfs_ctx_defaults_init (glfs.c:89)
==31753==    by 0x627577A: glfs_new@@GFAPI_3.4.0 (glfs.c:772)
==31753==    by 0x1088A3: glusterCatastrophe() (in /home/matt/dev/glusterfs_memory_leak/simple)
==31753==    by 0x1088FB: main (in /home/matt/dev/glusterfs_memory_leak/simple)
==31753== 
==31753== 256,080 (3,600 direct, 252,480 indirect) bytes in 30 blocks are definitely lost in loss record 446 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x68F3E39: eh_new (event-history.c:21)
==31753==    by 0x57F6D77: afr_selfheal_daemon_init (afr-self-heald.c:1005)
==31753==    by 0x5818B07: init (afr.c:569)
==31753==    by 0x68858C0: __xlator_init (xlator.c:403)
==31753==    by 0x68859E1: xlator_init (xlator.c:428)
==31753==    by 0x68CE7BA: glusterfs_graph_init (graph.c:320)
==31753==    by 0x68CF39F: glusterfs_graph_activate (graph.c:670)
==31753==    by 0x6276EBF: glfs_process_volfp (glfs-mgmt.c:78)
==31753==    by 0x62783E3: glfs_mgmt_getspec_cbk (glfs-mgmt.c:604)
==31753==    by 0x6DA1DFD: rpc_clnt_handle_reply (rpc-clnt.c:790)
==31753== 
==31753== 8,360,238 (192 direct, 8,360,046 indirect) bytes in 2 blocks are definitely lost in loss record 454 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x6885399: xlator_dynload (xlator.c:263)
==31753==    by 0x68854D2: xlator_set_type (xlator.c:295)
==31753==    by 0x62745B1: create_master (glfs.c:194)
==31753==    by 0x6275D23: glfs_init_common (glfs.c:981)
==31753==    by 0x6275F21: glfs_init@@GFAPI_3.4.0 (glfs.c:1034)
==31753==    by 0x1088D2: glusterCatastrophe() (in /home/matt/dev/glusterfs_memory_leak/simple)
==31753==    by 0x1088FB: main (in /home/matt/dev/glusterfs_memory_leak/simple)
==31753== 
==31753== 28,729,220 (3,200 direct, 28,726,020 indirect) bytes in 8 blocks are definitely lost in loss record 456 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x6DA2822: rpc_clnt_new (rpc-clnt.c:1112)
==31753==    by 0x6279014: glfs_mgmt_init (glfs-mgmt.c:927)
==31753==    by 0x6274863: glfs_volumes_init (glfs.c:266)
==31753==    by 0x6275D6B: glfs_init_common (glfs.c:989)
==31753==    by 0x6275F21: glfs_init@@GFAPI_3.4.0 (glfs.c:1034)
==31753==    by 0x1088D2: glusterCatastrophe() (in /home/matt/dev/glusterfs_memory_leak/simple)
==31753==    by 0x1088FB: main (in /home/matt/dev/glusterfs_memory_leak/simple)
==31753== 
==31753== 47,186,680 (1,120 direct, 47,185,560 indirect) bytes in 10 blocks are definitely lost in loss record 459 of 459
==31753==    at 0x4C2EB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31753==    by 0x68BBACF: __gf_calloc (mem-pool.c:117)
==31753==    by 0x4E3CCEB: ios_create_sample_buf (io-stats.c:508)
==31753==    by 0x4E3CE15: ios_init_sample_buf (io-stats.c:547)
==31753==    by 0x4E596BF: init (io-stats.c:3778)
==31753==    by 0x68858C0: __xlator_init (xlator.c:403)
==31753==    by 0x68859E1: xlator_init (xlator.c:428)
==31753==    by 0x68CE7BA: glusterfs_graph_init (graph.c:320)
==31753==    by 0x68CF39F: glusterfs_graph_activate (graph.c:670)
==31753==    by 0x6276EBF: glfs_process_volfp (glfs-mgmt.c:78)
==31753==    by 0x62783E3: glfs_mgmt_getspec_cbk (glfs-mgmt.c:604)
==31753==    by 0x6DA1DFD: rpc_clnt_handle_reply (rpc-clnt.c:790)
==31753== 
==31753== LEAK SUMMARY:
==31753==    definitely lost: 672,672 bytes in 4,922 blocks
==31753==    indirectly lost: 84,755,746 bytes in 3,646 blocks
==31753==      possibly lost: 52,263,754 bytes in 301 blocks
==31753==    still reachable: 127,122 bytes in 2,842 blocks
==31753==         suppressed: 0 bytes in 0 blocks
==31753== Reachable blocks (those to which a pointer was found) are not shown.
==31753== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==31753== 
==31753== For counts of detected and suppressed errors, rerun with: -v
==31753== ERROR SUMMARY: 116 errors from 116 contexts (suppressed: 0 from 0)

Actual results:

Several memory leaks are reported by valgrind.

Expected results:

No memory leaks should be reported.

Additional info:

Volume "hot" consists only of a single local brick.
Comment 1 Worker Ant 2016-11-29 11:19:23 UTC 
REVIEW: http://review.gluster.org/15967 (libgfapi: fix major memory leaks after cleanup) posted (#1) for review on master by Anonymous Coward (mateusz.slupny)
Comment 2 Worker Ant 2016-11-30 11:51:12 UTC 
REVIEW: http://review.gluster.org/15967 (libgfapi: fix major memory leaks after cleanup) posted (#2) for review on master by Mateusz Slupny (mateusz.slupny)
Comment 3 Worker Ant 2016-12-01 08:48:19 UTC 
REVIEW: http://review.gluster.org/15967 (libgfapi: fix major memory leaks after cleanup) posted (#3) for review on master by Mateusz Slupny (mateusz.slupny)
Comment 4 Amar Tumballi 2018-08-29 03:35:33 UTC 
This update is done in bulk based on the state of the patch and the time since last activity. If the issue is still seen, please reopen the bug.