Created attachment 982763 [details]
coredump and logs

Description of problem:

After reboot of some nodes in replicated volume "heal info command" caused a crash:


gluster> volume heal vm_storage_volume info
*** glibc detected *** /usr/sbin/glfsheal: malloc(): memory corruption (fast): 0x00007f42344ff540 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x78db6)[0x7f42307a2db6]
/lib/x86_64-linux-gnu/libc.so.6(+0x7af81)[0x7f42307a4f81]
/lib/x86_64-linux-gnu/libc.so.6(__libc_calloc+0xd0)[0x7f42307a7a10]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(__gf_calloc+0xbe)[0x7f423233c02e]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x16755)[0x7f4232312755]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_set+0x4a)[0x7f423231290a]
/usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/cluster/replicate.so(afr_self_heal_lookup_unwind+0x1b5)[0x7f422c2020a5]
/usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/cluster/replicate.so(afr_self_heal_completion_cbk+0x13d)[0x7f422c1e083d]
/usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/cluster/replicate.so(afr_sh_data_done+0xe)[0x7f422c1d6bfe]
/usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/cluster/replicate.so(afr_sh_data_close+0x29d)[0x7f422c1d6ead]
/usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/cluster/replicate.so(+0x48018)[0x7f422c1ec018]
/usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/cluster/replicate.so(+0x48295)[0x7f422c1ec295]
/usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/protocol/client.so(client3_3_inodelk_cbk+0x13a)[0x7f422c43d25a]
/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_handle_reply+0xa4)[0x7f4231848ae4]
/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0xcd)[0x7f4231848e6d]
/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7f4231845173]
/usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/rpc-transport/socket.so(+0x85a4)[0x7f422ced35a4]
/usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/rpc-transport/socket.so(+0xad9c)[0x7f422ced5d9c]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x6935a)[0x7f423236535a]
/usr/lib/x86_64-linux-gnu/libgfapi.so.0(+0x75b4)[0x7f42316255b4]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7db4)[0x7f4230ea1db4]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f42308024fd]
======= Memory map: ========
7f4224000000-7f4224021000 rw-p 00000000 00:00 0 
7f4224021000-7f4228000000 ---p 00000000 00:00 0 
7f42288f0000-7f422b0e6000 rw-p 00000000 00:00 0 
7f422b0e6000-7f422b0ff000 r-xp 00000000 fc:00 5395225                    /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/debug/io-stats.so
7f422b0ff000-7f422b2fe000 ---p 00019000 fc:00 5395225                    /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/debug/io-stats.so
7f422b2fe000-7f422b2ff000 r--p 00018000 fc:00 5395225                    /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/debug/io-stats.so
7f422b2ff000-7f422b301000 rw-p 00019000 fc:00 5395225                    /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/debug/io-stats.so
7f422b301000-7f422b30e000 r-xp 00000000 fc:00 5395204                    /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/md-cache.so
7f422b30e000-7f422b50e000 ---p 0000d000 fc:00 5395204                    /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/md-cache.so
7f422b50e000-7f422b50f000 r--p 0000d000 fc:00 5395204                    /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/md-cache.so
7f422b50f000-7f422b510000 rw-p 0000e000 fc:00 5395204                    /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/md-cache.so
7f422b510000-7f422b517000 r-xp 00000000 fc:00 5395206                    /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/open-behind.so
7f422b517000-7f422b717000 ---p 00007000 fc:00 5395206                    /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/open-behind.so
7f422b717000-7f422b718000 r--p 00007000 fc:00 5395206                    /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/open-behind.so
7f422b718000-7f422b719000 rw-p 00008000 fc:00 5395206                    /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/open-behind.so
7f422b719000-7f422b720000 r-xp 00000000 fc:00 5395210                    /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/quick-read.so
7f422b720000-7f422b91f000 ---p 00007000 fc:00 5395210                    /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/quick-read.so
7f422b91f000-7f422b920000 r--p 00006000 fc:00 5395210                    /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/quick-read.so
7f422b920000-7f422b921000 rw-p 00007000 fc:00 5395210                    /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/quick-read.so
7f422b921000-7f422b931000 r-xp 00000000 fc:00 5395213                    /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/io-cache.so
7f422b931000-7f422bb31000 ---p 00010000 fc:00 5395213                    /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/io-cache.so
7f422bb31000-7f422bb32000 r--p 00010000 fc:00 5395213                    /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/io-cache.so
7f422bb32000-7f422bb34000 rw-p 00011000 fc:00 5395213                    /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/io-cache.so
7f422bb34000-7f422bb3f000 r-xp 00000000 fc:00 5395209                    /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/read-ahead.so
7f422bb3f000-7f422bd3e000 ---p 0000b000 fc:00 5395209                    /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/read-ahead.soBrick svm1:/srv/vol/
/vm_images_and_config/vm17.img - Possibly undergoing heal
/vm_images_and_config/vm12.img - Possibly undergoing heal
/vm_images_and_config/vm1.img - Possibly undergoing heal
/vm_images_and_config/vm9.img - Possibly undergoing heal
/vm_images_and_config/vm20.img - Possibly undergoing heal
/vm_images_and_config/vm15.img - Possibly undergoing heal
/vm_images_and_config/vm5.img - Possibly undergoing heal
/vm_images_and_config/vm7.img - Possibly undergoing heal
/vm_images_and_config/vm13.img - Possibly undergoing heal
/users/admvs/.mozilla/firefox/admvs.default
/vm_images_and_config/vm2.img - Possibly undergoing heal
/vm_images_and_config/vm19.img - Possibly undergoing heal
/vm_images_and_config/vm-clone.img - Possibly undergoing heal
/vm_images_and_config/vm8.img - Possibly undergoing heal
/vm_images_and_config/vm18.img - Possibly undergoing heal
/vm_images_and_config/vm16.img - Possibly undergoing heal
/vm_images_and_config/vm6.img - Possibly undergoing heal

How reproducible:
Our configuration is distributed replica on 7 bricks
We had such crashes 3 times. Core dump, config and logs are in the attachment.