Bug 1184883
| Summary: | gluster "volume heal info" command caused crash | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Community] GlusterFS | Reporter: | Anatoly Belikov <awbelikov> | ||||
| Component: | replicate | Assignee: | Anuradha <atalur> | ||||
| Status: | CLOSED EOL | QA Contact: | |||||
| Severity: | unspecified | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 3.5.3 | CC: | bugs, smohan, vbellur | ||||
| Target Milestone: | --- | Keywords: | Triaged | ||||
| Target Release: | --- | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2016-06-17 15:58:07 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
This bug is getting closed because the 3.5 is marked End-Of-Life. There will be no further updates to this version. Please open a new bug against a version that still receives bugfixes if you are still facing this issue in a more current release. |
Created attachment 982763 [details] coredump and logs Description of problem: After reboot of some nodes in replicated volume "heal info command" caused a crash: gluster> volume heal vm_storage_volume info *** glibc detected *** /usr/sbin/glfsheal: malloc(): memory corruption (fast): 0x00007f42344ff540 *** ======= Backtrace: ========= /lib/x86_64-linux-gnu/libc.so.6(+0x78db6)[0x7f42307a2db6] /lib/x86_64-linux-gnu/libc.so.6(+0x7af81)[0x7f42307a4f81] /lib/x86_64-linux-gnu/libc.so.6(__libc_calloc+0xd0)[0x7f42307a7a10] /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(__gf_calloc+0xbe)[0x7f423233c02e] /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x16755)[0x7f4232312755] /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_set+0x4a)[0x7f423231290a] /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/cluster/replicate.so(afr_self_heal_lookup_unwind+0x1b5)[0x7f422c2020a5] /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/cluster/replicate.so(afr_self_heal_completion_cbk+0x13d)[0x7f422c1e083d] /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/cluster/replicate.so(afr_sh_data_done+0xe)[0x7f422c1d6bfe] /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/cluster/replicate.so(afr_sh_data_close+0x29d)[0x7f422c1d6ead] /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/cluster/replicate.so(+0x48018)[0x7f422c1ec018] /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/cluster/replicate.so(+0x48295)[0x7f422c1ec295] /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/protocol/client.so(client3_3_inodelk_cbk+0x13a)[0x7f422c43d25a] /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_handle_reply+0xa4)[0x7f4231848ae4] /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0xcd)[0x7f4231848e6d] /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7f4231845173] /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/rpc-transport/socket.so(+0x85a4)[0x7f422ced35a4] /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/rpc-transport/socket.so(+0xad9c)[0x7f422ced5d9c] /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x6935a)[0x7f423236535a] /usr/lib/x86_64-linux-gnu/libgfapi.so.0(+0x75b4)[0x7f42316255b4] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7db4)[0x7f4230ea1db4] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f42308024fd] ======= Memory map: ======== 7f4224000000-7f4224021000 rw-p 00000000 00:00 0 7f4224021000-7f4228000000 ---p 00000000 00:00 0 7f42288f0000-7f422b0e6000 rw-p 00000000 00:00 0 7f422b0e6000-7f422b0ff000 r-xp 00000000 fc:00 5395225 /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/debug/io-stats.so 7f422b0ff000-7f422b2fe000 ---p 00019000 fc:00 5395225 /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/debug/io-stats.so 7f422b2fe000-7f422b2ff000 r--p 00018000 fc:00 5395225 /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/debug/io-stats.so 7f422b2ff000-7f422b301000 rw-p 00019000 fc:00 5395225 /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/debug/io-stats.so 7f422b301000-7f422b30e000 r-xp 00000000 fc:00 5395204 /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/md-cache.so 7f422b30e000-7f422b50e000 ---p 0000d000 fc:00 5395204 /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/md-cache.so 7f422b50e000-7f422b50f000 r--p 0000d000 fc:00 5395204 /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/md-cache.so 7f422b50f000-7f422b510000 rw-p 0000e000 fc:00 5395204 /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/md-cache.so 7f422b510000-7f422b517000 r-xp 00000000 fc:00 5395206 /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/open-behind.so 7f422b517000-7f422b717000 ---p 00007000 fc:00 5395206 /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/open-behind.so 7f422b717000-7f422b718000 r--p 00007000 fc:00 5395206 /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/open-behind.so 7f422b718000-7f422b719000 rw-p 00008000 fc:00 5395206 /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/open-behind.so 7f422b719000-7f422b720000 r-xp 00000000 fc:00 5395210 /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/quick-read.so 7f422b720000-7f422b91f000 ---p 00007000 fc:00 5395210 /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/quick-read.so 7f422b91f000-7f422b920000 r--p 00006000 fc:00 5395210 /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/quick-read.so 7f422b920000-7f422b921000 rw-p 00007000 fc:00 5395210 /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/quick-read.so 7f422b921000-7f422b931000 r-xp 00000000 fc:00 5395213 /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/io-cache.so 7f422b931000-7f422bb31000 ---p 00010000 fc:00 5395213 /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/io-cache.so 7f422bb31000-7f422bb32000 r--p 00010000 fc:00 5395213 /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/io-cache.so 7f422bb32000-7f422bb34000 rw-p 00011000 fc:00 5395213 /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/io-cache.so 7f422bb34000-7f422bb3f000 r-xp 00000000 fc:00 5395209 /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/read-ahead.so 7f422bb3f000-7f422bd3e000 ---p 0000b000 fc:00 5395209 /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/read-ahead.soBrick svm1:/srv/vol/ /vm_images_and_config/vm17.img - Possibly undergoing heal /vm_images_and_config/vm12.img - Possibly undergoing heal /vm_images_and_config/vm1.img - Possibly undergoing heal /vm_images_and_config/vm9.img - Possibly undergoing heal /vm_images_and_config/vm20.img - Possibly undergoing heal /vm_images_and_config/vm15.img - Possibly undergoing heal /vm_images_and_config/vm5.img - Possibly undergoing heal /vm_images_and_config/vm7.img - Possibly undergoing heal /vm_images_and_config/vm13.img - Possibly undergoing heal /users/admvs/.mozilla/firefox/admvs.default /vm_images_and_config/vm2.img - Possibly undergoing heal /vm_images_and_config/vm19.img - Possibly undergoing heal /vm_images_and_config/vm-clone.img - Possibly undergoing heal /vm_images_and_config/vm8.img - Possibly undergoing heal /vm_images_and_config/vm18.img - Possibly undergoing heal /vm_images_and_config/vm16.img - Possibly undergoing heal /vm_images_and_config/vm6.img - Possibly undergoing heal How reproducible: Our configuration is distributed replica on 7 bricks We had such crashes 3 times. Core dump, config and logs are in the attachment.