Bug 1046318

Summary: Memory corruption and crash in libgfapi
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Lalatendu Mohanty <lmohanty>
Component: glusterfsAssignee: Poornima G <pgurusid>
Status: CLOSED DUPLICATE QA Contact: Lalatendu Mohanty <lmohanty>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 2.1CC: pgurusid, pkarampu, rtalur, sbhaloth, sdharane, vagarwal, vbellur
Target Milestone: ---Keywords: ZStream
Target Release: RHGS 2.1.2   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-01-03 10:19:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1046837    

Description Lalatendu Mohanty 2013-12-24 13:51:36 UTC 
Description of problem:

libgfapi is crashing and generating core for memory corruption while running 
glusterfs/api/examples/glfsxmp.c. 

glusterfs/api/examples/glfsxmp.c does tests fops using libgfapi on GliusterFS volume.

[root@rhsauto057 examples]# ./glfsxmp testvol localhost
glfs_init: returned 0
glfs_init: returned 0
/filename2: (0) Success
/filename2: (0x1385df0) Structure needs cleaning
/filename2: (0x1385df0) Structure needs cleaning
read 32, hi there
/filename3: (0) Success
/filename3: (0) Success
rename(/filename3): (0) Structure needs cleaning
unlink(/filename4): (0) Structure needs cleaning
/dirname2: (0) Structure needs cleaning
lstat(/dirname2): (0) Success
rmdir(/dirname2): (0) Structure needs cleaning
Entries:
.: 8
..: 12
topdir: 24
testdir: 2048
filename2: 25
linkdir: 37
setxattr(/filename2): 0 (Structure needs cleaning)
setxattr(/filename2): 0 (Structure needs cleaning)
listxattr(/filename2): 27 (Success)
key=user.testkey2
key=user.testkey
mkdir(/topdir): File exists
Found test directory /testdir to be existing
Cleanup test directory and restart tests
*** glibc detected *** ./glfsxmp: malloc(): memory corruption: 0x00007fab537f5010 ***
*** glibc detected *** ./glfsxmp: free(): corrupted unsorted chunks: 0x0000000001384f60 ***
Segmentation fault (core dumped)

The BackTrace
##############

(gdb) thread apply all bt

Thread 9 (Thread 0x7fab50c8f700 (LWP 19765)):
#0  0x000000366c60b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000000366d66221b in ?? () from /usr/lib64/libglusterfs.so.0
#2  0x000000366e6063f4 in ?? () from /usr/lib64/libgfapi.so.0
#3  0x000000366c607851 in start_thread () from /lib64/libpthread.so.0
#4  0x000000366c2e894d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7fab3ffff700 (LWP 19769)):
#0  0x000000366c60cb1b in pthread_once () from /lib64/libpthread.so.0
#1  0x000000366c2fe994 in backtrace () from /lib64/libc.so.6
#2  0x000000366c27080b in __libc_message () from /lib64/libc.so.6
#3  0x000000366c276126 in malloc_printerr () from /lib64/libc.so.6
#4  0x000000366c278c53 in _int_free () from /lib64/libc.so.6
#5  0x000000366e20f0d3 in rpc_clnt_connection_cleanup () from /usr/lib64/libgfrpc.so.0
#6  0x000000366e20f594 in rpc_clnt_notify () from /usr/lib64/libgfrpc.so.0
#7  0x000000366e20adf8 in rpc_transport_notify () from /usr/lib64/libgfrpc.so.0
#8  0x00007fab50086641 in ?? () from /usr/lib64/glusterfs/3.4.0.52rhs/rpc-transport/socket.so
#9  0x000000366d662387 in ?? () from /usr/lib64/libglusterfs.so.0
#10 0x000000366e6063f4 in ?? () from /usr/lib64/libgfapi.so.0
#11 0x000000366c607851 in start_thread () from /lib64/libpthread.so.0
#12 0x000000366c2e894d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7fab3f5fe700 (LWP 19770)):
#0  0x000000366c2acbcd in nanosleep () from /lib64/libc.so.6
#1  0x000000366d62cffc in gf_timer_proc () from /usr/lib64/libglusterfs.so.0
#2  0x000000366c607851 in start_thread () from /lib64/libpthread.so.0
#3  0x000000366c2e894d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7fab4fa2a700 (LWP 19766)):
#0  0x000000366c2acbcd in nanosleep () from /lib64/libc.so.6
#1  0x000000366d62cffc in gf_timer_proc () from /usr/lib64/libglusterfs.so.0
#2  0x000000366c607851 in start_thread () from /lib64/libpthread.so.0
#3  0x000000366c2e894d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7fab52126700 (LWP 19764)):
#0  0x000000366c60b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#1  0x000000366d64b16f in syncenv_task () from /usr/lib64/libglusterfs.so.0
#2  0x000000366d650050 in syncenv_processor () from /usr/lib64/libglusterfs.so.0
#3  0x000000366c607851 in start_thread () from /lib64/libpthread.so.0
#4  0x000000366c2e894d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7fab44f9b700 (LWP 19768)):
#0  0x000000366c60b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000000366d64b16f in syncenv_task () from /usr/lib64/libglusterfs.so.0
#2  0x000000366d650050 in syncenv_processor () from /usr/lib64/libglusterfs.so.0
#3  0x000000366c607851 in start_thread () from /lib64/libpthread.so.0
#4  0x000000366c2e894d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7fab52b27700 (LWP 19763)):
#0  0x000000366c60b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000000366d64b16f in syncenv_task () from /usr/lib64/libglusterfs.so.0
#2  0x000000366d650050 in syncenv_processor () from /usr/lib64/libglusterfs.so.0
#3  0x000000366c607851 in start_thread () from /lib64/libpthread.so.0
#4  0x000000366c2e894d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7fab4599c700 (LWP 19767)):
#0  0x000000366c60b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000000366d64b16f in syncenv_task () from /usr/lib64/libglusterfs.so.0
#2  0x000000366d650050 in syncenv_processor () from /usr/lib64/libglusterfs.so.0
#3  0x000000366c607851 in start_thread () from /lib64/libpthread.so.0
#4  0x000000366c2e894d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7fab537ea700 (LWP 19762)):
#0  0x000000366c2761b5 in malloc_consolidate () from /lib64/libc.so.6
#1  0x000000366c2793c5 in _int_malloc () from /lib64/libc.so.6
#2  0x000000366c27a5e6 in calloc () from /lib64/libc.so.6
#3  0x000000366ba0ad1f in _dl_new_object () from /lib64/ld-linux-x86-64.so.2
#4  0x000000366ba071ae in _dl_map_object_from_fd () from /lib64/ld-linux-x86-64.so.2
#5  0x000000366ba0836a in _dl_map_object () from /lib64/ld-linux-x86-64.so.2
#6  0x000000366ba12a34 in dl_open_worker () from /lib64/ld-linux-x86-64.so.2
#7  0x000000366ba0e1a6 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#8  0x000000366ba124ea in _dl_open () from /lib64/ld-linux-x86-64.so.2
#9  0x000000366c326340 in do_dlopen () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---
#10 0x000000366ba0e1a6 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#11 0x000000366c326497 in __libc_dlopen_mode () from /lib64/libc.so.6
#12 0x000000366c2fe895 in init () from /lib64/libc.so.6
#13 0x000000366c60cb23 in pthread_once () from /lib64/libpthread.so.0
#14 0x000000366c2fe994 in backtrace () from /lib64/libc.so.6
#15 0x000000366c27080b in __libc_message () from /lib64/libc.so.6
#16 0x000000366c276126 in malloc_printerr () from /lib64/libc.so.6
#17 0x000000366c279ba4 in _int_malloc () from /lib64/libc.so.6
#18 0x000000366c27a951 in malloc () from /lib64/libc.so.6
#19 0x000000366c301e1b in __vasprintf_chk () from /lib64/libc.so.6
#20 0x000000366d61c9b1 in _gf_log () from /usr/lib64/libglusterfs.so.0
#21 0x00007fab4edf809e in notify () from /usr/lib64/glusterfs/3.4.0.52rhs/xlator/protocol/client.so
#22 0x000000366d618fb6 in xlator_notify () from /usr/lib64/libglusterfs.so.0
#23 0x000000366d61d66f in default_notify () from /usr/lib64/libglusterfs.so.0
#24 0x00007fab4ebca4c6 in afr_notify () from /usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so
#25 0x00007fab4ebca639 in notify () from /usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so
#26 0x000000366d618fb6 in xlator_notify () from /usr/lib64/libglusterfs.so.0
#27 0x000000366d61d66f in default_notify () from /usr/lib64/libglusterfs.so.0
#28 0x00007fab4e9487b6 in dht_notify () from /usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so
#29 0x000000366d618fb6 in xlator_notify () from /usr/lib64/libglusterfs.so.0
#30 0x000000366d61d66f in default_notify () from /usr/lib64/libglusterfs.so.0
#31 0x000000366d618fb6 in xlator_notify () from /usr/lib64/libglusterfs.so.0
#32 0x000000366d61d66f in default_notify () from /usr/lib64/libglusterfs.so.0
#33 0x000000366d618fb6 in xlator_notify () from /usr/lib64/libglusterfs.so.0
#34 0x000000366d61d66f in default_notify () from /usr/lib64/libglusterfs.so.0
#35 0x000000366d618fb6 in xlator_notify () from /usr/lib64/libglusterfs.so.0
#36 0x000000366d61d66f in default_notify () from /usr/lib64/libglusterfs.so.0
#37 0x000000366d618fb6 in xlator_notify () from /usr/lib64/libglusterfs.so.0
#38 0x000000366d61d66f in default_notify () from /usr/lib64/libglusterfs.so.0
#39 0x000000366d618fb6 in xlator_notify () from /usr/lib64/libglusterfs.so.0
#40 0x000000366d61d66f in default_notify () from /usr/lib64/libglusterfs.so.0
#41 0x00007fab4dcea699 in notify () from /usr/lib64/glusterfs/3.4.0.52rhs/xlator/debug/io-stats.so
#42 0x000000366d618fb6 in xlator_notify () from /usr/lib64/libglusterfs.so.0
#43 0x000000366e606317 in glfs_fini () from /usr/lib64/libgfapi.so.0
#44 0x0000000000404adb in main (argc=<value optimized out>, argv=0x7fff7b2f8318) at glfsxmp.c:1595
(gdb) 


Version-Release number of selected component (if applicable):

glusterfs-api-3.4.0.52rhs-1.el6rhs

How reproducible:

Frequently

Steps to Reproduce:
1. Compile glusterfs/api/examples/glfsxmp.c on a rhs/gluster node
2. run "./glfsxmp <volume name> <localhost>" (multiple times)

To compile glusterfs/api/examples/glfsxmp.c install glusterfs-api-devel,  glusterfs-api-devel packages.

Actual results:


Expected results:


Additional info:
Comment 2 Raghavendra Talur 2013-12-31 07:22:25 UTC 
Patch merged downstream at https://code.engineering.redhat.com/gerrit/#/c/17893/
Comment 3 Lalatendu Mohanty 2014-01-02 12:41:09 UTC 
Shouldn't be this bug a duplicate of BZ 1043519 as the same patch fixes both the bugs?
Comment 4 Poornima G 2014-01-03 10:18:00 UTC 
Yes, marking it as duplicate
Comment 5 Poornima G 2014-01-03 10:19:08 UTC 

*** This bug has been marked as a duplicate of bug 1046564 ***