Attachment 883770 Details for Bug 1083709 – Valgrind output

Valgrind output

valgrind.txt (text/plain), 59.46 KB, created by Ben Turner on 2014-04-07 21:38:52 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Ben Turner

Created: 2014-04-07 21:38:52 UTC

Size: 59.46 KB

patch

obsolete

>Valgrind process:
>
>** I am not 100% I am doing this properly but here is my process **
>1.  Mount all 3 clients with:
>    valgrind --leak-check=full /usr/sbin/glusterfs --volfile-server=gqas005.sbu.lab.eng.bos.redhat.com --volfile-id=/testvol /gluster-mount
>2.  Run iozone with:
>    iozone -+m /mnt/tests/rhs-tests/beaker/rhs/auto-tests/setup/rhs-setup/clients.ioz -+h gqas016.sbu.lab.eng.bos.redhat.com -C -w -c -e -i 0 -+n -r 64k -s 100m -t 12
>3.  When iozone finsihes unmount client:
>    # umount /gluster-mount
>
>IOZone command output:
>
>** Note - running with valgrind was so terribly slow I had to lower my file size to 100 MB to redue run time **
>
>[root@gqas016 ~]# iozone -+m /mnt/tests/rhs-tests/beaker/rhs/auto-tests/setup/rhs-setup/clients.ioz -+h gqas016.sbu.lab.eng.bos.redhat.com -C -w -c -e -i 0 -+n -r 64k -s 100m -t 12
>	Iozone: Performance Test of File I/O
>	        Version $Revision: 3.394 $
>		Compiled for 64 bit mode.
>		Build: linux 
>
>	Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
>	             Al Slater, Scott Rhine, Mike Wisner, Ken Goss
>	             Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
>	             Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
>	             Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
>	             Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
>	             Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer.
>	             Ben England.
>
>	Run began: Mon Apr  7 17:29:15 2014
>
>	Network distribution mode enabled.
>	Hostname = gqas016.sbu.lab.eng.bos.redhat.com
>	Setting no_unlink
>	Include close in write timing
>	Include fsync in write timing
>	No retest option selected
>	Record Size 64 KB
>	File size set to 102400 KB
>	Command line used: iozone -+m /mnt/tests/rhs-tests/beaker/rhs/auto-tests/setup/rhs-setup/clients.ioz -+h gqas016.sbu.lab.eng.bos.redhat.com -C -w -c -e -i 0 -+n -r 64k -s 100m -t 12
>	Output is in Kbytes/sec
>	Time Resolution = 0.000001 seconds.
>	Processor cache size set to 1024 Kbytes.
>	Processor cache line size set to 32 bytes.
>	File stride size set to 17 * record size.
>	Throughput test with 12 processes
>	Each process writes a 102400 Kbyte file in 64 Kbyte records
>
>	Test running:
>	Children see throughput for 12 initial writers 	=   34881.64 KB/sec
>	Min throughput per process 			=    1897.20 KB/sec 
>	Max throughput per process 			=    5922.38 KB/sec
>	Avg throughput per process 			=    2906.80 KB/sec
>	Min xfer 					=   32832.00 KB
>	Child[0] xfer count =   33984.00 KB, Throughput =    1960.56 KB/sec
>	Child[1] xfer count =   33472.00 KB, Throughput =    1934.46 KB/sec
>	Child[2] xfer count =   33472.00 KB, Throughput =    1938.83 KB/sec
>	Child[3] xfer count =   32832.00 KB, Throughput =    1897.20 KB/sec
>	Child[4] xfer count =   58496.00 KB, Throughput =    3354.88 KB/sec
>	Child[5] xfer count =   58496.00 KB, Throughput =    3368.61 KB/sec
>	Child[6] xfer count =   58432.00 KB, Throughput =    3393.85 KB/sec
>	Child[7] xfer count =   58496.00 KB, Throughput =    3377.13 KB/sec
>	Child[8] xfer count =  102400.00 KB, Throughput =    5922.38 KB/sec
>	Child[9] xfer count =   44544.00 KB, Throughput =    2586.74 KB/sec
>	Child[10] xfer count =   44672.00 KB, Throughput =    2577.53 KB/sec
>	Child[11] xfer count =   44480.00 KB, Throughput =    2569.47 KB/sec
>
>
>
>iozone test complete.
>
>
>Valgrind output:
>
>[root@gqas009 ~]# valgrind --leak-check=yes /usr/sbin/glusterfs --volfile-server=gqas005.sbu.lab.eng.bos.redhat.com --volfile-id=/testvol /gluster-mount/
>==2698== Memcheck, a memory error detector
>==2698== Copyright (C) 2002-2012, and GNU GPL'd, by Julian Seward et al.
>==2698== Using Valgrind-3.8.1 and LibVEX; rerun with -h for copyright info
>==2698== Command: /usr/sbin/glusterfs --volfile-server=gqas005.sbu.lab.eng.bos.redhat.com --volfile-id=/testvol /gluster-mount/
>==2698== 
>==2701== 
>==2701== HEAP SUMMARY:
>==2701==     in use at exit: 9,537,218 bytes in 75 blocks
>==2701==   total heap usage: 169 allocs, 94 frees, 9,554,955 bytes allocated
>==2701== 
>==2701== 11 bytes in 1 blocks are definitely lost in loss record 6 of 61
>==2701==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2701==    by 0x374D2463F2: __gf_calloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2701==    by 0x374D21995C: ??? (in /usr/lib64/libglusterfs.so.0.0.0)
>==2701==    by 0x374D219B29: dict_set (in /usr/lib64/libglusterfs.so.0.0.0)
>==2701==    by 0x405583: ??? (in /usr/sbin/glusterfsd)
>==2701==    by 0x40611A: create_fuse_mount (in /usr/sbin/glusterfsd)
>==2701==    by 0x40792E: main (in /usr/sbin/glusterfsd)
>==2701== 
>==2701== 14 bytes in 1 blocks are definitely lost in loss record 7 of 61
>==2701==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2701==    by 0x374D2463F2: __gf_calloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2701==    by 0x374D21995C: ??? (in /usr/lib64/libglusterfs.so.0.0.0)
>==2701==    by 0x374D219B29: dict_set (in /usr/lib64/libglusterfs.so.0.0.0)
>==2701==    by 0x405E99: ??? (in /usr/sbin/glusterfsd)
>==2701==    by 0x40611A: create_fuse_mount (in /usr/sbin/glusterfsd)
>==2701==    by 0x40792E: main (in /usr/sbin/glusterfsd)
>==2701== 
>==2701== 16 bytes in 1 blocks are definitely lost in loss record 12 of 61
>==2701==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2701==    by 0x374D2463F2: __gf_calloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2701==    by 0x4058B5: ??? (in /usr/sbin/glusterfsd)
>==2701==    by 0x40611A: create_fuse_mount (in /usr/sbin/glusterfsd)
>==2701==    by 0x40792E: main (in /usr/sbin/glusterfsd)
>==2701== 
>==2701== 17 bytes in 1 blocks are definitely lost in loss record 15 of 61
>==2701==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2701==    by 0x374D2463F2: __gf_calloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2701==    by 0x374D21995C: ??? (in /usr/lib64/libglusterfs.so.0.0.0)
>==2701==    by 0x374D219B29: dict_set (in /usr/lib64/libglusterfs.so.0.0.0)
>==2701==    by 0x4059AC: ??? (in /usr/sbin/glusterfsd)
>==2701==    by 0x40611A: create_fuse_mount (in /usr/sbin/glusterfsd)
>==2701==    by 0x40792E: main (in /usr/sbin/glusterfsd)
>==2701== 
>==2701== LEAK SUMMARY:
>==2701==    definitely lost: 58 bytes in 4 blocks
>==2701==    indirectly lost: 0 bytes in 0 blocks
>==2701==      possibly lost: 0 bytes in 0 blocks
>==2701==    still reachable: 9,537,160 bytes in 71 blocks
>==2701==         suppressed: 0 bytes in 0 blocks
>==2701== Reachable blocks (those to which a pointer was found) are not shown.
>==2701== To see them, rerun with: --leak-check=full --show-reachable=yes
>==2701== 
>==2701== For counts of detected and suppressed errors, rerun with: -v
>==2701== ERROR SUMMARY: 4 errors from 4 contexts (suppressed: 9 from 9)
>==2698== 
>==2698== HEAP SUMMARY:
>==2698==     in use at exit: 9,545,446 bytes in 76 blocks
>==2698==   total heap usage: 166 allocs, 90 frees, 9,562,906 bytes allocated
>==2698== 
>==2698== 11 bytes in 1 blocks are definitely lost in loss record 6 of 62
>==2698==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2698==    by 0x374D2463F2: __gf_calloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2698==    by 0x374D21995C: ??? (in /usr/lib64/libglusterfs.so.0.0.0)
>==2698==    by 0x374D219B29: dict_set (in /usr/lib64/libglusterfs.so.0.0.0)
>==2698==    by 0x405583: ??? (in /usr/sbin/glusterfsd)
>==2698==    by 0x40611A: create_fuse_mount (in /usr/sbin/glusterfsd)
>==2698==    by 0x40792E: main (in /usr/sbin/glusterfsd)
>==2698== 
>==2698== 14 bytes in 1 blocks are definitely lost in loss record 7 of 62
>==2698==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2698==    by 0x374D2463F2: __gf_calloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2698==    by 0x374D21995C: ??? (in /usr/lib64/libglusterfs.so.0.0.0)
>==2698==    by 0x374D219B29: dict_set (in /usr/lib64/libglusterfs.so.0.0.0)
>==2698==    by 0x405E99: ??? (in /usr/sbin/glusterfsd)
>==2698==    by 0x40611A: create_fuse_mount (in /usr/sbin/glusterfsd)
>==2698==    by 0x40792E: main (in /usr/sbin/glusterfsd)
>==2698== 
>==2698== 16 bytes in 1 blocks are definitely lost in loss record 12 of 62
>==2698==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2698==    by 0x374D2463F2: __gf_calloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2698==    by 0x4058B5: ??? (in /usr/sbin/glusterfsd)
>==2698==    by 0x40611A: create_fuse_mount (in /usr/sbin/glusterfsd)
>==2698==    by 0x40792E: main (in /usr/sbin/glusterfsd)
>==2698== 
>==2698== 17 bytes in 1 blocks are definitely lost in loss record 15 of 62
>==2698==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2698==    by 0x374D2463F2: __gf_calloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2698==    by 0x374D21995C: ??? (in /usr/lib64/libglusterfs.so.0.0.0)
>==2698==    by 0x374D219B29: dict_set (in /usr/lib64/libglusterfs.so.0.0.0)
>==2698==    by 0x4059AC: ??? (in /usr/sbin/glusterfsd)
>==2698==    by 0x40611A: create_fuse_mount (in /usr/sbin/glusterfsd)
>==2698==    by 0x40792E: main (in /usr/sbin/glusterfsd)
>==2698== 
>==2698== LEAK SUMMARY:
>==2698==    definitely lost: 58 bytes in 4 blocks
>==2698==    indirectly lost: 0 bytes in 0 blocks
>==2698==      possibly lost: 0 bytes in 0 blocks
>==2698==    still reachable: 9,545,388 bytes in 72 blocks
>==2698==         suppressed: 0 bytes in 0 blocks
>==2698== Reachable blocks (those to which a pointer was found) are not shown.
>==2698== To see them, rerun with: --leak-check=full --show-reachable=yes
>==2698== 
>==2698== For counts of detected and suppressed errors, rerun with: -v
>==2698== ERROR SUMMARY: 4 errors from 4 contexts (suppressed: 9 from 9)
>[root@gqas009 ~]# ==2702== Syscall param writev(vector[...]) points to uninitialised byte(s)
>==2702==    at 0x374BAE0C57: writev (in /lib64/libc-2.12.so)
>==2702==    by 0xC3195FF: ??? (in /usr/lib64/glusterfs/3.5qa2/xlator/mount/fuse.so)
>==2702==    by 0xC319775: ??? (in /usr/lib64/glusterfs/3.5qa2/xlator/mount/fuse.so)
>==2702==    by 0xC32C655: ??? (in /usr/lib64/glusterfs/3.5qa2/xlator/mount/fuse.so)
>==2702==    by 0xC32C886: ??? (in /usr/lib64/glusterfs/3.5qa2/xlator/mount/fuse.so)
>==2702==    by 0x106CA292: io_stats_lookup_cbk (in /usr/lib64/glusterfs/3.5qa2/xlator/debug/io-stats.so)
>==2702==    by 0x104B37AB: mdc_lookup_cbk (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/md-cache.so)
>==2702==    by 0x1009F563: qr_lookup_cbk (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/quick-read.so)
>==2702==    by 0xFE927A0: ioc_lookup_cbk (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/io-cache.so)
>==2702==    by 0xF84D300: dht_revalidate_cbk (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/dht.so)
>==2702==    by 0xF60F2E5: afr_discover_cbk (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/afr.so)
>==2702==    by 0xF3B10C6: client3_3_lookup_cbk (in /usr/lib64/glusterfs/3.5qa2/xlator/protocol/client.so)
>==2702==  Address 0x7feffe2cc is on thread 1's stack
>==2702== 
>
>** IOZone completed here **
>
>[root@gqas009 ~]# 
>[root@gqas009 ~]# 
>[root@gqas009 ~]# 
>[root@gqas009 ~]# 
>[root@gqas009 ~]# umount /gluster-mount
>[root@gqas009 ~]# ==2702== Warning: invalid file descriptor -1 in syscall close()
>==2702== 
>==2702== HEAP SUMMARY:
>==2702==     in use at exit: 39,565,031 bytes in 15,872 blocks
>==2702==   total heap usage: 697,763 allocs, 681,891 frees, 129,152,538 bytes allocated
>==2702== 
>==2702== 2 bytes in 1 blocks are possibly lost in loss record 5 of 400
>==2702==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2702==    by 0x374D2463F2: __gf_calloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0xF607847: afr_transaction_local_init (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/afr.so)
>==2702==    by 0xF5F4969: afr_transaction (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/afr.so)
>==2702==    by 0xF5ECF23: afr_do_writev (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/afr.so)
>==2702==    by 0xF5ED64A: afr_writev (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/afr.so)
>==2702==    by 0xF858C48: dht_writev (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/dht.so)
>==2702==    by 0xFA76EF3: wb_fulfill_head (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/write-behind.so)
>==2702==    by 0xFA7708A: wb_fulfill (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/write-behind.so)
>==2702==    by 0xFA78A91: wb_process_queue (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/write-behind.so)
>==2702==    by 0xFA79397: wb_writev (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/write-behind.so)
>==2702==    by 0xFC8257B: ra_writev (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/read-ahead.so)
>==2702== 
>==2702== 2 bytes in 1 blocks are definitely lost in loss record 6 of 400
>==2702==    at 0x4A069EE: malloc (vg_replace_malloc.c:270)
>==2702==    by 0x374D246179: __gf_malloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D2462DC: gf_vasprintf (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D246397: gf_asprintf (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D21A8EC: data_from_int8 (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D21A96B: dict_set_int8 (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0xF84AF0C: dht_check_and_set_acl_xattr_req (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/dht.so)
>==2702==    by 0xF84B5BF: dht_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/dht.so)
>==2702==    by 0x374D22147C: default_lookup (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D22147C: default_lookup (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0xFE8F26B: ioc_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/io-cache.so)
>==2702==    by 0x1009E916: qr_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/quick-read.so)
>==2702== 
>==2702== 2 bytes in 1 blocks are definitely lost in loss record 7 of 400
>==2702==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2702==    by 0x374D2463F2: __gf_calloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0xF607847: afr_transaction_local_init (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/afr.so)
>==2702==    by 0xF5F4969: afr_transaction (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/afr.so)
>==2702==    by 0xF5ECF23: afr_do_writev (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/afr.so)
>==2702==    by 0xF5ED64A: afr_writev (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/afr.so)
>==2702==    by 0xF858C48: dht_writev (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/dht.so)
>==2702==    by 0xFA76EF3: wb_fulfill_head (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/write-behind.so)
>==2702==    by 0xFA7708A: wb_fulfill (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/write-behind.so)
>==2702==    by 0xFA78A91: wb_process_queue (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/write-behind.so)
>==2702==    by 0xFA78C2E: wb_fulfill_cbk (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/write-behind.so)
>==2702==    by 0xF8587B7: dht_writev_cbk (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/dht.so)
>==2702== 
>==2702== 2 bytes in 1 blocks are definitely lost in loss record 8 of 400
>==2702==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2702==    by 0x374D2463F2: __gf_calloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0xF607865: afr_transaction_local_init (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/afr.so)
>==2702==    by 0xF5F4969: afr_transaction (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/afr.so)
>==2702==    by 0xF5ECF23: afr_do_writev (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/afr.so)
>==2702==    by 0xF5ED64A: afr_writev (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/afr.so)
>==2702==    by 0xF858C48: dht_writev (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/dht.so)
>==2702==    by 0xFA76EF3: wb_fulfill_head (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/write-behind.so)
>==2702==    by 0xFA7708A: wb_fulfill (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/write-behind.so)
>==2702==    by 0xFA78A91: wb_process_queue (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/write-behind.so)
>==2702==    by 0xFA78C2E: wb_fulfill_cbk (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/write-behind.so)
>==2702==    by 0xF8587B7: dht_writev_cbk (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/dht.so)
>==2702== 
>==2702== 6 bytes in 1 blocks are definitely lost in loss record 14 of 400
>==2702==    at 0x4A069EE: malloc (vg_replace_malloc.c:270)
>==2702==    by 0x374D246179: __gf_malloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D2462DC: gf_vasprintf (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D246397: gf_asprintf (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D21A7F4: data_from_uint64 (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x1009E977: qr_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/quick-read.so)
>==2702==    by 0x374D22147C: default_lookup (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x104B518D: mdc_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/md-cache.so)
>==2702==    by 0x106C1BC6: io_stats_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/debug/io-stats.so)
>==2702==    by 0xC318071: fuse_resolve_entry (in /usr/lib64/glusterfs/3.5qa2/xlator/mount/fuse.so)
>==2702==    by 0xC3182C7: fuse_resolve_parent (in /usr/lib64/glusterfs/3.5qa2/xlator/mount/fuse.so)
>==2702==    by 0xC3185A7: ??? (in /usr/lib64/glusterfs/3.5qa2/xlator/mount/fuse.so)
>==2702== 
>==2702== 8 bytes in 4 blocks are definitely lost in loss record 34 of 400
>==2702==    at 0x4A069EE: malloc (vg_replace_malloc.c:270)
>==2702==    by 0x374D246179: __gf_malloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D2462DC: gf_vasprintf (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D246397: gf_asprintf (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D21A6F2: data_from_uint32 (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D21A77A: dict_set_uint32 (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0xF3BD947: client_setvolume (in /usr/lib64/glusterfs/3.5qa2/xlator/protocol/client.so)
>==2702==    by 0xF3BF8CB: client_dump_version_cbk (in /usr/lib64/glusterfs/3.5qa2/xlator/protocol/client.so)
>==2702==    by 0x374C60DF74: rpc_clnt_handle_reply (in /usr/lib64/libgfrpc.so.0.0.0)
>==2702==    by 0x374C60F58E: rpc_clnt_notify (in /usr/lib64/libgfrpc.so.0.0.0)
>==2702==    by 0x374C60AF67: rpc_transport_notify (in /usr/lib64/libgfrpc.so.0.0.0)
>==2702==    by 0xE349F74: ??? (in /usr/lib64/glusterfs/3.5qa2/rpc-transport/socket.so)
>==2702== 
>==2702== 9 bytes in 1 blocks are definitely lost in loss record 38 of 400
>==2702==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2702==    by 0x374D2463F2: __gf_calloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D21995C: ??? (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D219B29: dict_set (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0xC32D126: fuse_first_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/mount/fuse.so)
>==2702==    by 0xC32FCE9: fuse_graph_sync (in /usr/lib64/glusterfs/3.5qa2/xlator/mount/fuse.so)
>==2702==    by 0xC32FFAF: ??? (in /usr/lib64/glusterfs/3.5qa2/xlator/mount/fuse.so)
>==2702==    by 0x374BE079D0: start_thread (in /lib64/libpthread-2.12.so)
>==2702==    by 0x12CFE6FF: ???
>==2702== 
>==2702== 9 bytes in 3 blocks are definitely lost in loss record 39 of 400
>==2702==    at 0x4A069EE: malloc (vg_replace_malloc.c:270)
>==2702==    by 0x374D246179: __gf_malloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D2462DC: gf_vasprintf (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D246397: gf_asprintf (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D21A6F2: data_from_uint32 (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D21A77A: dict_set_uint32 (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0xF84B192: dht_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/dht.so)
>==2702==    by 0x374D22147C: default_lookup (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D22147C: default_lookup (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0xFE8F26B: ioc_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/io-cache.so)
>==2702==    by 0x1009E916: qr_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/quick-read.so)
>==2702==    by 0x374D22147C: default_lookup (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702== 
>==2702== 10 bytes in 5 blocks are definitely lost in loss record 43 of 400
>==2702==    at 0x4A069EE: malloc (vg_replace_malloc.c:270)
>==2702==    by 0x374D246179: __gf_malloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D2462DC: gf_vasprintf (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D246397: gf_asprintf (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D21A6F2: data_from_uint32 (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D21A77A: dict_set_uint32 (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0xF84B5AE: dht_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/dht.so)
>==2702==    by 0x374D22147C: default_lookup (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D22147C: default_lookup (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0xFE8F26B: ioc_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/io-cache.so)
>==2702==    by 0x1009E916: qr_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/quick-read.so)
>==2702==    by 0x374D22147C: default_lookup (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702== 
>==2702== 14 bytes in 1 blocks are definitely lost in loss record 51 of 400
>==2702==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2702==    by 0x374D2463F2: __gf_calloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D21995C: ??? (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D219B29: dict_set (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x405E99: ??? (in /usr/sbin/glusterfsd)
>==2702==    by 0x40611A: create_fuse_mount (in /usr/sbin/glusterfsd)
>==2702==    by 0x40792E: main (in /usr/sbin/glusterfsd)
>==2702== 
>==2702== 15 bytes in 5 blocks are definitely lost in loss record 56 of 400
>==2702==    at 0x4A069EE: malloc (vg_replace_malloc.c:270)
>==2702==    by 0x374D246179: __gf_malloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D2462DC: gf_vasprintf (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D246397: gf_asprintf (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D21A6F2: data_from_uint32 (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D21A77A: dict_set_uint32 (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0xF84B57E: dht_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/dht.so)
>==2702==    by 0x374D22147C: default_lookup (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D22147C: default_lookup (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0xFE8F26B: ioc_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/io-cache.so)
>==2702==    by 0x1009E916: qr_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/quick-read.so)
>==2702==    by 0x374D22147C: default_lookup (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702== 
>==2702== 16 bytes in 8 blocks are definitely lost in loss record 76 of 400
>==2702==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2702==    by 0x374D2463F2: __gf_calloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0xF607847: afr_transaction_local_init (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/afr.so)
>==2702==    by 0xF5F4969: afr_transaction (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/afr.so)
>==2702==    by 0xF5ECF23: afr_do_writev (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/afr.so)
>==2702==    by 0xF5ED64A: afr_writev (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/afr.so)
>==2702==    by 0xF858C48: dht_writev (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/dht.so)
>==2702==    by 0xFA76EF3: wb_fulfill_head (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/write-behind.so)
>==2702==    by 0xFA7708A: wb_fulfill (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/write-behind.so)
>==2702==    by 0xFA78A91: wb_process_queue (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/write-behind.so)
>==2702==    by 0xFA7A20F: wb_fsync (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/write-behind.so)
>==2702==    by 0xFC82A99: ra_fsync (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/read-ahead.so)
>==2702== 
>==2702== 16 bytes in 8 blocks are definitely lost in loss record 77 of 400
>==2702==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2702==    by 0x374D2463F2: __gf_calloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0xF607865: afr_transaction_local_init (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/afr.so)
>==2702==    by 0xF5F4969: afr_transaction (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/afr.so)
>==2702==    by 0xF5ECF23: afr_do_writev (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/afr.so)
>==2702==    by 0xF5ED64A: afr_writev (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/afr.so)
>==2702==    by 0xF858C48: dht_writev (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/dht.so)
>==2702==    by 0xFA76EF3: wb_fulfill_head (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/write-behind.so)
>==2702==    by 0xFA7708A: wb_fulfill (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/write-behind.so)
>==2702==    by 0xFA78A91: wb_process_queue (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/write-behind.so)
>==2702==    by 0xFA7A20F: wb_fsync (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/write-behind.so)
>==2702==    by 0xFC82A99: ra_fsync (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/read-ahead.so)
>==2702== 
>==2702== 17 bytes in 1 blocks are definitely lost in loss record 80 of 400
>==2702==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2702==    by 0x374D2463F2: __gf_calloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D21995C: ??? (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D219B29: dict_set (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x4059AC: ??? (in /usr/sbin/glusterfsd)
>==2702==    by 0x40611A: create_fuse_mount (in /usr/sbin/glusterfsd)
>==2702==    by 0x40792E: main (in /usr/sbin/glusterfsd)
>==2702== 
>==2702== 18 bytes in 3 blocks are definitely lost in loss record 85 of 400
>==2702==    at 0x4A069EE: malloc (vg_replace_malloc.c:270)
>==2702==    by 0x374D246179: __gf_malloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D2462DC: gf_vasprintf (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D246397: gf_asprintf (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D21A7F4: data_from_uint64 (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x1009E977: qr_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/quick-read.so)
>==2702==    by 0x374D22147C: default_lookup (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x104B518D: mdc_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/md-cache.so)
>==2702==    by 0x106C1BC6: io_stats_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/debug/io-stats.so)
>==2702==    by 0xC3232FB: ??? (in /usr/lib64/glusterfs/3.5qa2/xlator/mount/fuse.so)
>==2702==    by 0xC330017: ??? (in /usr/lib64/glusterfs/3.5qa2/xlator/mount/fuse.so)
>==2702==    by 0x374BE079D0: start_thread (in /lib64/libpthread-2.12.so)
>==2702== 
>==2702== 25 bytes in 1 blocks are definitely lost in loss record 107 of 400
>==2702==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2702==    by 0x374D2463F2: __gf_calloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D21995C: ??? (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D219B29: dict_set (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374C60AC7D: rpc_transport_inet_options_build (in /usr/lib64/libgfrpc.so.0.0.0)
>==2702==    by 0x407D2E: glusterfs_mgmt_init (in /usr/sbin/glusterfsd)
>==2702==    by 0x405175: glusterfs_volumes_init (in /usr/sbin/glusterfsd)
>==2702==    by 0x40796D: main (in /usr/sbin/glusterfsd)
>==2702== 
>==2702== 25 bytes in 1 blocks are definitely lost in loss record 108 of 400
>==2702==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2702==    by 0x374D2463F2: __gf_calloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D21995C: ??? (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D219B29: dict_set (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0xF84AF0C: dht_check_and_set_acl_xattr_req (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/dht.so)
>==2702==    by 0xF84B5BF: dht_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/dht.so)
>==2702==    by 0x374D22147C: default_lookup (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D22147C: default_lookup (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0xFE8F26B: ioc_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/io-cache.so)
>==2702==    by 0x1009E916: qr_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/quick-read.so)
>==2702==    by 0x374D22147C: default_lookup (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x104B518D: mdc_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/md-cache.so)
>==2702== 
>==2702== 29 bytes in 1 blocks are definitely lost in loss record 112 of 400
>==2702==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2702==    by 0x374D2463F2: __gf_calloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D21995C: ??? (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D219B29: dict_set (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0xF84B596: dht_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/dht.so)
>==2702==    by 0x374D22147C: default_lookup (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D22147C: default_lookup (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0xFE8F26B: ioc_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/io-cache.so)
>==2702==    by 0x1009E916: qr_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/quick-read.so)
>==2702==    by 0x374D22147C: default_lookup (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x104B518D: mdc_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/md-cache.so)
>==2702==    by 0x106C1BC6: io_stats_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/debug/io-stats.so)
>==2702== 
>==2702== 30 bytes in 2 blocks are definitely lost in loss record 114 of 400
>==2702==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2702==    by 0x374D2463F2: __gf_calloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D237415: ??? (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D237AE9: inode_link (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0xC318E2D: fuse_resolve_entry_cbk (in /usr/lib64/glusterfs/3.5qa2/xlator/mount/fuse.so)
>==2702==    by 0x106CA292: io_stats_lookup_cbk (in /usr/lib64/glusterfs/3.5qa2/xlator/debug/io-stats.so)
>==2702==    by 0x104B37AB: mdc_lookup_cbk (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/md-cache.so)
>==2702==    by 0x1009F563: qr_lookup_cbk (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/quick-read.so)
>==2702==    by 0xFE927A0: ioc_lookup_cbk (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/io-cache.so)
>==2702==    by 0xF850742: dht_lookup_cbk (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/dht.so)
>==2702==    by 0xF60F783: ??? (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/afr.so)
>==2702==    by 0xF610EAD: afr_lookup_entry_heal (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/afr.so)
>==2702== 
>==2702== 32 bytes in 4 blocks are definitely lost in loss record 128 of 400
>==2702==    at 0x4A069EE: malloc (vg_replace_malloc.c:270)
>==2702==    by 0x374D246179: __gf_malloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D2462DC: gf_vasprintf (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D246397: gf_asprintf (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D21AAD2: data_from_int32 (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D21AB5A: dict_set_int32 (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0xF3BD828: client_setvolume (in /usr/lib64/glusterfs/3.5qa2/xlator/protocol/client.so)
>==2702==    by 0xF3BF8CB: client_dump_version_cbk (in /usr/lib64/glusterfs/3.5qa2/xlator/protocol/client.so)
>==2702==    by 0x374C60DF74: rpc_clnt_handle_reply (in /usr/lib64/libgfrpc.so.0.0.0)
>==2702==    by 0x374C60F58E: rpc_clnt_notify (in /usr/lib64/libgfrpc.so.0.0.0)
>==2702==    by 0x374C60AF67: rpc_transport_notify (in /usr/lib64/libgfrpc.so.0.0.0)
>==2702==    by 0xE349F74: ??? (in /usr/lib64/glusterfs/3.5qa2/rpc-transport/socket.so)
>==2702== 
>==2702== 35 bytes in 1 blocks are definitely lost in loss record 131 of 400
>==2702==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2702==    by 0x374D2463F2: __gf_calloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374C60AA40: ??? (in /usr/lib64/libgfrpc.so.0.0.0)
>==2702==    by 0x374C60AB27: rpc_transport_inet_options_build (in /usr/lib64/libgfrpc.so.0.0.0)
>==2702==    by 0x407D2E: glusterfs_mgmt_init (in /usr/sbin/glusterfsd)
>==2702==    by 0x405175: glusterfs_volumes_init (in /usr/sbin/glusterfsd)
>==2702==    by 0x40796D: main (in /usr/sbin/glusterfsd)
>==2702== 
>==2702== 36 bytes in 2 blocks are definitely lost in loss record 133 of 400
>==2702==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2702==    by 0x374D2463F2: __gf_calloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D21995C: ??? (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D219B29: dict_set (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x1009E98B: qr_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/quick-read.so)
>==2702==    by 0x374D22147C: default_lookup (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x104B518D: mdc_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/md-cache.so)
>==2702==    by 0x106C1BC6: io_stats_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/debug/io-stats.so)
>==2702==    by 0xC318071: fuse_resolve_entry (in /usr/lib64/glusterfs/3.5qa2/xlator/mount/fuse.so)
>==2702==    by 0xC3182C7: fuse_resolve_parent (in /usr/lib64/glusterfs/3.5qa2/xlator/mount/fuse.so)
>==2702==    by 0xC3185A7: ??? (in /usr/lib64/glusterfs/3.5qa2/xlator/mount/fuse.so)
>==2702==    by 0xC31886D: ??? (in /usr/lib64/glusterfs/3.5qa2/xlator/mount/fuse.so)
>==2702== 
>==2702== 36 bytes in 4 blocks are definitely lost in loss record 134 of 400
>==2702==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2702==    by 0x374D2463F2: __gf_calloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D21995C: ??? (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D219B29: dict_set (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0xC32312E: ??? (in /usr/lib64/glusterfs/3.5qa2/xlator/mount/fuse.so)
>==2702==    by 0xC330017: ??? (in /usr/lib64/glusterfs/3.5qa2/xlator/mount/fuse.so)
>==2702==    by 0x374BE079D0: start_thread (in /lib64/libpthread-2.12.so)
>==2702==    by 0x12CFE6FF: ???
>==2702== 
>==2702== 44 bytes in 2 blocks are definitely lost in loss record 138 of 400
>==2702==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2702==    by 0x374D2463F2: __gf_calloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D21995C: ??? (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D219B29: dict_set (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0xF84B192: dht_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/dht.so)
>==2702==    by 0x374D22147C: default_lookup (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D22147C: default_lookup (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0xFE8F26B: ioc_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/io-cache.so)
>==2702==    by 0x1009E916: qr_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/quick-read.so)
>==2702==    by 0x374D22147C: default_lookup (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x104B518D: mdc_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/md-cache.so)
>==2702==    by 0x106C1BC6: io_stats_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/debug/io-stats.so)
>==2702== 
>==2702== 48 bytes in 2 blocks are definitely lost in loss record 142 of 400
>==2702==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2702==    by 0x374D2463F2: __gf_calloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D21995C: ??? (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D219B29: dict_set (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0xF84B5AE: dht_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/dht.so)
>==2702==    by 0x374D22147C: default_lookup (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D22147C: default_lookup (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0xFE8F26B: ioc_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/io-cache.so)
>==2702==    by 0x1009E916: qr_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/quick-read.so)
>==2702==    by 0x374D22147C: default_lookup (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x104B518D: mdc_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/md-cache.so)
>==2702==    by 0x106C1BC6: io_stats_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/debug/io-stats.so)
>==2702== 
>==2702== 48 bytes in 4 blocks are definitely lost in loss record 143 of 400
>==2702==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2702==    by 0x374D2463F2: __gf_calloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D21995C: ??? (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D219B29: dict_set (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0xF3BD923: client_setvolume (in /usr/lib64/glusterfs/3.5qa2/xlator/protocol/client.so)
>==2702==    by 0xF3BF8CB: client_dump_version_cbk (in /usr/lib64/glusterfs/3.5qa2/xlator/protocol/client.so)
>==2702==    by 0x374C60DF74: rpc_clnt_handle_reply (in /usr/lib64/libgfrpc.so.0.0.0)
>==2702==    by 0x374C60F58E: rpc_clnt_notify (in /usr/lib64/libgfrpc.so.0.0.0)
>==2702==    by 0x374C60AF67: rpc_transport_notify (in /usr/lib64/libgfrpc.so.0.0.0)
>==2702==    by 0xE349F74: ??? (in /usr/lib64/glusterfs/3.5qa2/rpc-transport/socket.so)
>==2702==    by 0xE34B95C: ??? (in /usr/lib64/glusterfs/3.5qa2/rpc-transport/socket.so)
>==2702==    by 0x374D26CEE6: ??? (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702== 
>==2702== 52 bytes in 4 blocks are definitely lost in loss record 151 of 400
>==2702==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2702==    by 0x374D2463F2: __gf_calloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D21995C: ??? (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D219B29: dict_set (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0xF3BD8D3: client_setvolume (in /usr/lib64/glusterfs/3.5qa2/xlator/protocol/client.so)
>==2702==    by 0xF3BF8CB: client_dump_version_cbk (in /usr/lib64/glusterfs/3.5qa2/xlator/protocol/client.so)
>==2702==    by 0x374C60DF74: rpc_clnt_handle_reply (in /usr/lib64/libgfrpc.so.0.0.0)
>==2702==    by 0x374C60F58E: rpc_clnt_notify (in /usr/lib64/libgfrpc.so.0.0.0)
>==2702==    by 0x374C60AF67: rpc_transport_notify (in /usr/lib64/libgfrpc.so.0.0.0)
>==2702==    by 0xE349F74: ??? (in /usr/lib64/glusterfs/3.5qa2/rpc-transport/socket.so)
>==2702==    by 0xE34B95C: ??? (in /usr/lib64/glusterfs/3.5qa2/rpc-transport/socket.so)
>==2702==    by 0x374D26CEE6: ??? (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702== 
>==2702== 54 bytes in 3 blocks are definitely lost in loss record 153 of 400
>==2702==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2702==    by 0x374D2463F2: __gf_calloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D21995C: ??? (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D219B29: dict_set (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x1009E98B: qr_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/quick-read.so)
>==2702==    by 0x374D22147C: default_lookup (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x104B518D: mdc_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/md-cache.so)
>==2702==    by 0x106C1BC6: io_stats_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/debug/io-stats.so)
>==2702==    by 0xC3232FB: ??? (in /usr/lib64/glusterfs/3.5qa2/xlator/mount/fuse.so)
>==2702==    by 0xC330017: ??? (in /usr/lib64/glusterfs/3.5qa2/xlator/mount/fuse.so)
>==2702==    by 0x374BE079D0: start_thread (in /lib64/libpthread-2.12.so)
>==2702==    by 0x12CFE6FF: ???
>==2702== 
>==2702== 64 bytes in 4 blocks are definitely lost in loss record 168 of 400
>==2702==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2702==    by 0x374D2463F2: __gf_calloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D21995C: ??? (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D219B29: dict_set (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0xF3BD969: client_setvolume (in /usr/lib64/glusterfs/3.5qa2/xlator/protocol/client.so)
>==2702==    by 0xF3BF8CB: client_dump_version_cbk (in /usr/lib64/glusterfs/3.5qa2/xlator/protocol/client.so)
>==2702==    by 0x374C60DF74: rpc_clnt_handle_reply (in /usr/lib64/libgfrpc.so.0.0.0)
>==2702==    by 0x374C60F58E: rpc_clnt_notify (in /usr/lib64/libgfrpc.so.0.0.0)
>==2702==    by 0x374C60AF67: rpc_transport_notify (in /usr/lib64/libgfrpc.so.0.0.0)
>==2702==    by 0xE349F74: ??? (in /usr/lib64/glusterfs/3.5qa2/rpc-transport/socket.so)
>==2702==    by 0xE34B95C: ??? (in /usr/lib64/glusterfs/3.5qa2/rpc-transport/socket.so)
>==2702==    by 0x374D26CEE6: ??? (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702== 
>==2702== 74 bytes in 3 blocks are definitely lost in loss record 177 of 400
>==2702==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2702==    by 0x374D2463F2: __gf_calloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D26E414: graphyyparse (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D26EFD6: glusterfs_graph_construct (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x404FF7: glusterfs_process_volfp (in /usr/sbin/glusterfsd)
>==2702==    by 0x40BB9F: mgmt_getspec_cbk (in /usr/sbin/glusterfsd)
>==2702==    by 0x374C60DF74: rpc_clnt_handle_reply (in /usr/lib64/libgfrpc.so.0.0.0)
>==2702==    by 0x374C60F58E: rpc_clnt_notify (in /usr/lib64/libgfrpc.so.0.0.0)
>==2702==    by 0x374C60AF67: rpc_transport_notify (in /usr/lib64/libgfrpc.so.0.0.0)
>==2702==    by 0xE349F74: ??? (in /usr/lib64/glusterfs/3.5qa2/rpc-transport/socket.so)
>==2702==    by 0xE34B95C: ??? (in /usr/lib64/glusterfs/3.5qa2/rpc-transport/socket.so)
>==2702==    by 0x374D26CEE6: ??? (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702== 
>==2702== 88 bytes in 1 blocks are possibly lost in loss record 189 of 400
>==2702==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2702==    by 0x374D2463F2: __gf_calloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374C60D12D: rpcclnt_cbk_program_register (in /usr/lib64/libgfrpc.so.0.0.0)
>==2702==    by 0x407DA4: glusterfs_mgmt_init (in /usr/sbin/glusterfsd)
>==2702==    by 0x405175: glusterfs_volumes_init (in /usr/sbin/glusterfsd)
>==2702==    by 0x40796D: main (in /usr/sbin/glusterfsd)
>==2702== 
>==2702== 88 bytes in 4 blocks are definitely lost in loss record 190 of 400
>==2702==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2702==    by 0x374D2463F2: __gf_calloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D21995C: ??? (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D219B29: dict_set (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0xF84B57E: dht_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/dht.so)
>==2702==    by 0x374D22147C: default_lookup (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D22147C: default_lookup (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0xFE8F26B: ioc_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/io-cache.so)
>==2702==    by 0x1009E916: qr_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/quick-read.so)
>==2702==    by 0x374D22147C: default_lookup (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x104B518D: mdc_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/md-cache.so)
>==2702==    by 0x106C1BC6: io_stats_lookup (in /usr/lib64/glusterfs/3.5qa2/xlator/debug/io-stats.so)
>==2702== 
>==2702== 118 bytes in 7 blocks are definitely lost in loss record 217 of 400
>==2702==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2702==    by 0x374D2463F2: __gf_calloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D21995C: ??? (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D219B29: dict_set (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D26E46A: graphyyparse (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D26EFD6: glusterfs_graph_construct (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x404FF7: glusterfs_process_volfp (in /usr/sbin/glusterfsd)
>==2702==    by 0x40BB9F: mgmt_getspec_cbk (in /usr/sbin/glusterfsd)
>==2702==    by 0x374C60DF74: rpc_clnt_handle_reply (in /usr/lib64/libgfrpc.so.0.0.0)
>==2702==    by 0x374C60F58E: rpc_clnt_notify (in /usr/lib64/libgfrpc.so.0.0.0)
>==2702==    by 0x374C60AF67: rpc_transport_notify (in /usr/lib64/libgfrpc.so.0.0.0)
>==2702==    by 0xE349F74: ??? (in /usr/lib64/glusterfs/3.5qa2/rpc-transport/socket.so)
>==2702== 
>==2702== 168 bytes in 1 blocks are possibly lost in loss record 248 of 400
>==2702==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2702==    by 0x374D2463F2: __gf_calloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0xFE967EF: ioc_inode_update (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/io-cache.so)
>==2702==    by 0xFE92831: ioc_lookup_cbk (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/io-cache.so)
>==2702==    by 0xF84DCBC: dht_lookup_dir_cbk (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/dht.so)
>==2702==    by 0xF60F2E5: afr_discover_cbk (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/afr.so)
>==2702==    by 0xF3B10C6: client3_3_lookup_cbk (in /usr/lib64/glusterfs/3.5qa2/xlator/protocol/client.so)
>==2702==    by 0x374C60DF74: rpc_clnt_handle_reply (in /usr/lib64/libgfrpc.so.0.0.0)
>==2702==    by 0x374C60F58E: rpc_clnt_notify (in /usr/lib64/libgfrpc.so.0.0.0)
>==2702==    by 0x374C60AF67: rpc_transport_notify (in /usr/lib64/libgfrpc.so.0.0.0)
>==2702==    by 0xE349F74: ??? (in /usr/lib64/glusterfs/3.5qa2/rpc-transport/socket.so)
>==2702==    by 0xE34B95C: ??? (in /usr/lib64/glusterfs/3.5qa2/rpc-transport/socket.so)
>==2702== 
>==2702== 168 bytes in 1 blocks are possibly lost in loss record 249 of 400
>==2702==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2702==    by 0x374D2463F2: __gf_calloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0xFE967EF: ioc_inode_update (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/io-cache.so)
>==2702==    by 0xFE92831: ioc_lookup_cbk (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/io-cache.so)
>==2702==    by 0xF850742: dht_lookup_cbk (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/dht.so)
>==2702==    by 0xF60F783: ??? (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/afr.so)
>==2702==    by 0xF610EAD: afr_lookup_entry_heal (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/afr.so)
>==2702==    by 0xF6112B5: afr_lookup_cbk (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/afr.so)
>==2702==    by 0xF3B10C6: client3_3_lookup_cbk (in /usr/lib64/glusterfs/3.5qa2/xlator/protocol/client.so)
>==2702==    by 0x374C60DF74: rpc_clnt_handle_reply (in /usr/lib64/libgfrpc.so.0.0.0)
>==2702==    by 0x374C60F58E: rpc_clnt_notify (in /usr/lib64/libgfrpc.so.0.0.0)
>==2702==    by 0x374C60AF67: rpc_transport_notify (in /usr/lib64/libgfrpc.so.0.0.0)
>==2702== 
>==2702== 168 bytes in 1 blocks are definitely lost in loss record 250 of 400
>==2702==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2702==    by 0x374D2463F2: __gf_calloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0xC32FE51: ??? (in /usr/lib64/glusterfs/3.5qa2/xlator/mount/fuse.so)
>==2702==    by 0x374BE079D0: start_thread (in /lib64/libpthread-2.12.so)
>==2702==    by 0x12CFE6FF: ???
>==2702== 
>==2702== 272 bytes in 1 blocks are possibly lost in loss record 278 of 400
>==2702==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2702==    by 0x374B2118A2: _dl_allocate_tls (in /lib64/ld-2.12.so)
>==2702==    by 0x374BE071E8: pthread_create@@GLIBC_2.2.5 (in /lib64/libpthread-2.12.so)
>==2702==    by 0x4048D3: glusterfs_signals_setup (in /usr/sbin/glusterfsd)
>==2702==    by 0x404C07: daemonize (in /usr/sbin/glusterfsd)
>==2702==    by 0x407940: main (in /usr/sbin/glusterfsd)
>==2702== 
>==2702== 304 bytes in 1 blocks are possibly lost in loss record 282 of 400
>==2702==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2702==    by 0x374B2118A2: _dl_allocate_tls (in /lib64/ld-2.12.so)
>==2702==    by 0x374BE071E8: pthread_create@@GLIBC_2.2.5 (in /lib64/libpthread-2.12.so)
>==2702==    by 0x374D22F1C4: gf_thread_create (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D2346C4: gf_timer_registry_init (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D234A2E: gf_timer_call_after (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374C60E189: rpc_clnt_reconnect (in /usr/lib64/libgfrpc.so.0.0.0)
>==2702==    by 0x374C60E206: rpc_clnt_start (in /usr/lib64/libgfrpc.so.0.0.0)
>==2702==    by 0x407E19: glusterfs_mgmt_init (in /usr/sbin/glusterfsd)
>==2702==    by 0x405175: glusterfs_volumes_init (in /usr/sbin/glusterfsd)
>==2702==    by 0x40796D: main (in /usr/sbin/glusterfsd)
>==2702== 
>==2702== 304 bytes in 1 blocks are possibly lost in loss record 283 of 400
>==2702==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2702==    by 0x374B2118A2: _dl_allocate_tls (in /lib64/ld-2.12.so)
>==2702==    by 0x374BE071E8: pthread_create@@GLIBC_2.2.5 (in /lib64/libpthread-2.12.so)
>==2702==    by 0x374D22F1C4: gf_thread_create (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0xC324DBE: notify (in /usr/lib64/glusterfs/3.5qa2/xlator/mount/fuse.so)
>==2702==    by 0x374D21B195: xlator_notify (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D22108F: default_notify (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x106CC64D: notify (in /usr/lib64/glusterfs/3.5qa2/xlator/debug/io-stats.so)
>==2702==    by 0x374D21B195: xlator_notify (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D221022: default_notify (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D21B195: xlator_notify (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D221022: default_notify (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702== 
>==2702== 304 bytes in 1 blocks are possibly lost in loss record 284 of 400
>==2702==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2702==    by 0x374B2118A2: _dl_allocate_tls (in /lib64/ld-2.12.so)
>==2702==    by 0x374BE071E8: pthread_create@@GLIBC_2.2.5 (in /lib64/libpthread-2.12.so)
>==2702==    by 0x374D22F1C4: gf_thread_create (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0xC3199C6: ??? (in /usr/lib64/glusterfs/3.5qa2/xlator/mount/fuse.so)
>==2702==    by 0xC330017: ??? (in /usr/lib64/glusterfs/3.5qa2/xlator/mount/fuse.so)
>==2702==    by 0x374BE079D0: start_thread (in /lib64/libpthread-2.12.so)
>==2702==    by 0x12CFE6FF: ???
>==2702== 
>==2702== 352 bytes in 4 blocks are possibly lost in loss record 289 of 400
>==2702==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2702==    by 0x374D2463F2: __gf_calloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374C60D12D: rpcclnt_cbk_program_register (in /usr/lib64/libgfrpc.so.0.0.0)
>==2702==    by 0xF397F61: client_init_rpc (in /usr/lib64/glusterfs/3.5qa2/xlator/protocol/client.so)
>==2702==    by 0xF39834F: init (in /usr/lib64/glusterfs/3.5qa2/xlator/protocol/client.so)
>==2702==    by 0x374D21B701: xlator_init (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D251720: glusterfs_graph_init (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D251841: glusterfs_graph_activate (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x4050DA: glusterfs_process_volfp (in /usr/sbin/glusterfsd)
>==2702==    by 0x40BB9F: mgmt_getspec_cbk (in /usr/sbin/glusterfsd)
>==2702==    by 0x374C60DF74: rpc_clnt_handle_reply (in /usr/lib64/libgfrpc.so.0.0.0)
>==2702==    by 0x374C60F58E: rpc_clnt_notify (in /usr/lib64/libgfrpc.so.0.0.0)
>==2702== 
>==2702== 544 bytes in 2 blocks are possibly lost in loss record 311 of 400
>==2702==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2702==    by 0x374B2118A2: _dl_allocate_tls (in /lib64/ld-2.12.so)
>==2702==    by 0x374BE071E8: pthread_create@@GLIBC_2.2.5 (in /lib64/libpthread-2.12.so)
>==2702==    by 0x374D22F1C4: gf_thread_create (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D255591: syncenv_new (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x407955: main (in /usr/sbin/glusterfsd)
>==2702== 
>==2702== 576 (360 direct, 216 indirect) bytes in 1 blocks are definitely lost in loss record 313 of 400
>==2702==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2702==    by 0x374D2463F2: __gf_calloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D2358FA: ??? (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D237914: inode_table_new (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0xC324BF3: fuse_graph_setup (in /usr/lib64/glusterfs/3.5qa2/xlator/mount/fuse.so)
>==2702==    by 0xC324D82: notify (in /usr/lib64/glusterfs/3.5qa2/xlator/mount/fuse.so)
>==2702==    by 0x374D21B195: xlator_notify (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D22108F: default_notify (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x106CC64D: notify (in /usr/lib64/glusterfs/3.5qa2/xlator/debug/io-stats.so)
>==2702==    by 0x374D21B195: xlator_notify (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D221022: default_notify (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D21B195: xlator_notify (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702== 
>==2702== 632 (360 direct, 272 indirect) bytes in 1 blocks are definitely lost in loss record 316 of 400
>==2702==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2702==    by 0x374D2463F2: __gf_calloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D2358FA: ??? (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D235CE2: inode_new (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0xC317EA5: fuse_resolve_entry (in /usr/lib64/glusterfs/3.5qa2/xlator/mount/fuse.so)
>==2702==    by 0xC3182C7: fuse_resolve_parent (in /usr/lib64/glusterfs/3.5qa2/xlator/mount/fuse.so)
>==2702==    by 0xC3185A7: ??? (in /usr/lib64/glusterfs/3.5qa2/xlator/mount/fuse.so)
>==2702==    by 0xC31886D: ??? (in /usr/lib64/glusterfs/3.5qa2/xlator/mount/fuse.so)
>==2702==    by 0xC3188B7: fuse_resolve_and_resume (in /usr/lib64/glusterfs/3.5qa2/xlator/mount/fuse.so)
>==2702==    by 0xC330017: ??? (in /usr/lib64/glusterfs/3.5qa2/xlator/mount/fuse.so)
>==2702==    by 0x374BE079D0: start_thread (in /lib64/libpthread-2.12.so)
>==2702==    by 0x12CFE6FF: ???
>==2702== 
>==2702== 720 bytes in 2 blocks are definitely lost in loss record 322 of 400
>==2702==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2702==    by 0x374D2463F2: __gf_calloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D2358FA: ??? (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D237914: inode_table_new (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0xF603178: afr_selfheal_daemon_init (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/afr.so)
>==2702==    by 0xF6070F1: init (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/afr.so)
>==2702==    by 0x374D21B701: xlator_init (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D251720: glusterfs_graph_init (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D251841: glusterfs_graph_activate (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x4050DA: glusterfs_process_volfp (in /usr/sbin/glusterfsd)
>==2702==    by 0x40BB9F: mgmt_getspec_cbk (in /usr/sbin/glusterfsd)
>==2702==    by 0x374C60DF74: rpc_clnt_handle_reply (in /usr/lib64/libgfrpc.so.0.0.0)
>==2702== 
>==2702== 2,128 (336 direct, 1,792 indirect) bytes in 14 blocks are definitely lost in loss record 340 of 400
>==2702==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2702==    by 0x374D2463F2: __gf_calloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D247684: iobref_new (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0xC3303F7: fuse_write_resume (in /usr/lib64/glusterfs/3.5qa2/xlator/mount/fuse.so)
>==2702==    by 0xC318845: ??? (in /usr/lib64/glusterfs/3.5qa2/xlator/mount/fuse.so)
>==2702==    by 0xC318575: ??? (in /usr/lib64/glusterfs/3.5qa2/xlator/mount/fuse.so)
>==2702==    by 0xC31888D: ??? (in /usr/lib64/glusterfs/3.5qa2/xlator/mount/fuse.so)
>==2702==    by 0xC318910: fuse_resolve_continue (in /usr/lib64/glusterfs/3.5qa2/xlator/mount/fuse.so)
>==2702==    by 0xC31851D: ??? (in /usr/lib64/glusterfs/3.5qa2/xlator/mount/fuse.so)
>==2702==    by 0xC31886D: ??? (in /usr/lib64/glusterfs/3.5qa2/xlator/mount/fuse.so)
>==2702==    by 0xC3188B7: fuse_resolve_and_resume (in /usr/lib64/glusterfs/3.5qa2/xlator/mount/fuse.so)
>==2702==    by 0xC330017: ??? (in /usr/lib64/glusterfs/3.5qa2/xlator/mount/fuse.so)
>==2702== 
>==2702== 6,332 bytes in 3,166 blocks are definitely lost in loss record 350 of 400
>==2702==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2702==    by 0x374D2463F2: __gf_calloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0xF607847: afr_transaction_local_init (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/afr.so)
>==2702==    by 0xF5F4969: afr_transaction (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/afr.so)
>==2702==    by 0xF5ECF23: afr_do_writev (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/afr.so)
>==2702==    by 0xF5ED64A: afr_writev (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/afr.so)
>==2702==    by 0xF858C48: dht_writev (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/dht.so)
>==2702==    by 0xFA76EF3: wb_fulfill_head (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/write-behind.so)
>==2702==    by 0xFA7708A: wb_fulfill (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/write-behind.so)
>==2702==    by 0xFA78A91: wb_process_queue (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/write-behind.so)
>==2702==    by 0xFA79397: wb_writev (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/write-behind.so)
>==2702==    by 0xFC8257B: ra_writev (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/read-ahead.so)
>==2702== 
>==2702== 6,334 bytes in 3,167 blocks are definitely lost in loss record 351 of 400
>==2702==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2702==    by 0x374D2463F2: __gf_calloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0xF607865: afr_transaction_local_init (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/afr.so)
>==2702==    by 0xF5F4969: afr_transaction (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/afr.so)
>==2702==    by 0xF5ECF23: afr_do_writev (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/afr.so)
>==2702==    by 0xF5ED64A: afr_writev (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/afr.so)
>==2702==    by 0xF858C48: dht_writev (in /usr/lib64/glusterfs/3.5qa2/xlator/cluster/dht.so)
>==2702==    by 0xFA76EF3: wb_fulfill_head (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/write-behind.so)
>==2702==    by 0xFA7708A: wb_fulfill (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/write-behind.so)
>==2702==    by 0xFA78A91: wb_process_queue (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/write-behind.so)
>==2702==    by 0xFA79397: wb_writev (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/write-behind.so)
>==2702==    by 0xFC8257B: ra_writev (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/read-ahead.so)
>==2702== 
>==2702== 478,952 (75,624 direct, 403,328 indirect) bytes in 3,151 blocks are definitely lost in loss record 386 of 400
>==2702==    at 0x4A0577B: calloc (vg_replace_malloc.c:593)
>==2702==    by 0x374D2463F2: __gf_calloc (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D247684: iobref_new (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0xFA786F4: __wb_collapse_small_writes (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/write-behind.so)
>==2702==    by 0xFA789CA: __wb_preprocess_winds (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/write-behind.so)
>==2702==    by 0xFA78A4F: wb_process_queue (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/write-behind.so)
>==2702==    by 0xFA79397: wb_writev (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/write-behind.so)
>==2702==    by 0xFC8257B: ra_writev (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/read-ahead.so)
>==2702==    by 0xFE921A5: ioc_writev (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/io-cache.so)
>==2702==    by 0x1009EFDF: qr_writev (in /usr/lib64/glusterfs/3.5qa2/xlator/performance/quick-read.so)
>==2702==    by 0x374D226EC1: default_writev_resume (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702==    by 0x374D23A081: call_resume (in /usr/lib64/libglusterfs.so.0.0.0)
>==2702== 
>==2702== LEAK SUMMARY:
>==2702==    definitely lost: 91,216 bytes in 9,595 blocks
>==2702==    indirectly lost: 405,608 bytes in 3,171 blocks
>==2702==      possibly lost: 2,506 bytes in 14 blocks
>==2702==    still reachable: 39,065,701 bytes in 3,092 blocks
>==2702==         suppressed: 0 bytes in 0 blocks
>==2702== Reachable blocks (those to which a pointer was found) are not shown.
>==2702== To see them, rerun with: --leak-check=full --show-reachable=yes
>==2702== 
>==2702== For counts of detected and suppressed errors, rerun with: -v
>==2702== Use --track-origins=yes to see where uninitialised values come from
>==2702== ERROR SUMMARY: 55 errors from 50 contexts (suppressed: 61 from 9)
>

Actions: View

Attachments on bug 1083709: 883770 | 883773