Bug 765345 (GLUSTER-3613)

last night's  valgrind run crashed with stripe setup on branch "release-3.2"
(83a8938fec225268a3afdbcad1011f0e06e54d16)

Complete log and core file can be found at root.12.137:/mnt/nightly_valgrind/logs_stripe/2011-09-22

valgrind log-
---------
==9770== Invalid read of size 4
==9770==    at 0x3D92C08DC1: pthread_mutex_lock (in /lib64/libpthread-2.10.2.so)
==9770==    by 0x4C41E7C: inode_link (inode.c:823)
==9770==    by 0x62D310B: fuse_entry_cbk (fuse-bridge.c:148)
==9770==    by 0x62D3645: fuse_lookup_cbk (fuse-bridge.c:230)
==9770==    by 0x7DC9FB2: io_stats_lookup_cbk (io-stats.c:1391)
==9770==    by 0x7BA261D: sp_lookup_cbk (stat-prefetch.c:919)
==9770==    by 0x7986A0E: qr_lookup_cbk (quick-read.c:439)
==9770==    by 0x6F03E19: client3_1_mkdir_cbk (client3_1-fops.c:291)
==9770==    by 0x4E93FE9: rpc_clnt_handle_reply (rpc-clnt.c:741)
==9770==    by 0x4E94348: rpc_clnt_notify (rpc-clnt.c:854)
==9770==    by 0x4E90701: rpc_transport_notify (rpc-transport.c:919)
==9770==    by 0x83E1B0A: socket_event_poll_in (socket.c:1647)
==9770==  Address 0x363837396e75723f is not stack'd, malloc'd or (recently) free'd

fuse - log
---------
[2011-09-22 16:43:05.530472] W [client3_1-fops.c:2669:client3_1_stat] 0-stripe-client-3: failed to send the fop Transport endpoint is not connected
[2011-09-22 16:43:05.550762] W [client3_1-fops.c:3790:client3_1_fstat] 0-stripe-client-3: (5136460): failed to get fd ctx. EBADFD
[2011-09-22 16:43:05.551455] W [client3_1-fops.c:3807:client3_1_fstat] 0-stripe-client-3: failed to send the fop: File descriptor in bad state
[2011-09-22 16:43:05.559279] W [client3_1-fops.c:3662:client3_1_flush] 0-stripe-client-3: (5136460): failed to get fd ctx. EBADFD
[2011-09-22 16:43:05.559546] W [client3_1-fops.c:3692:client3_1_flush] 0-stripe-client-3: failed to send the fop: File descriptor in bad state
[2011-09-22 16:43:05.567815] W [client3_1-fops.c:2669:client3_1_stat] 0-stripe-client-3: failed to send the fop Transport endpoint is not connected
[2011-09-22 16:43:05.572282] W [client3_1-fops.c:2669:client3_1_stat] 0-stripe-client-3: failed to send the fop Transport endpoint is not connected
[2011-09-22 16:43:05.581215] W [client3_1-fops.c:2606:client3_1_lookup] 0-stripe-client-3: failed to send the fop: Transport endpoint is not connected
[2011-09-22 16:43:05.592059] W [client3_1-fops.c:2606:client3_1_lookup] 0-stripe-client-3: failed to send the fop: Transport endpoint is not connected
[2011-09-22 16:43:05.749684] W [client3_1-fops.c:2606:client3_1_lookup] 0-stripe-client-3: failed to send the fop: Transport endpoint is not connected
[2011-09-22 16:43:05.758565] W [client3_1-fops.c:2606:client3_1_lookup] 0-stripe-client-3: failed to send the fop: Transport endpoint is not connected
[2011-09-22 16:43:05.764022] W [client3_1-fops.c:3301:client3_1_mkdir] 0-stripe-client-3: failed to send the fop: Transport endpoint is not connected
[2011-09-22 16:43:33.630252] I [client-handshake.c:1090:select_server_supported_programs] 0-stripe-client-3: Using Program GlusterFS 3.2git, Num (1298437), Version (310)
[2011-09-22 16:43:33.632832] I [client-handshake.c:913:client_setvolume_cbk] 0-stripe-client-3: Connected to 10.1.12.137:8003, attached to remote volume '/data/export4'.
[2011-09-22 16:43:33.633088] I [client-handshake.c:779:client_post_handshake] 0-stripe-client-3: 4 fds open - Delaying child_up until they are re-opened
[2011-09-22 16:43:33.633426] W [inode.c:1044:inode_path] 0-stripe/inode: no dentry for non-root inode 5136463: 6279c771-3c61-4afe-ab15-14daca3634a3
[2011-09-22 16:43:33.633707] W [client-handshake.c:683:protocol_client_reopen] 0-stripe-client-3: couldn't build path from inode 6279c771-3c61-4afe-ab15-14daca3634a3
[2011-09-22 16:43:33.633934] E [client-handshake.c:722:protocol_client_reopen] 0-: failed to send the re-open request
[2011-09-22 16:43:33.634225] W [inode.c:1044:inode_path] 0-stripe/inode: no dentry for non-root inode 5136488: 35d8be3e-df9f-4315-bfc7-b454ce6c9b34
[2011-09-22 16:43:33.634488] W [client-handshake.c:683:protocol_client_reopen] 0-stripe-client-3: couldn't build path from inode 35d8be3e-df9f-4315-bfc7-b454ce6c9b34
[2011-09-22 16:43:33.634718] E [client-handshake.c:722:protocol_client_reopen] 0-: failed to send the re-open request
[2011-09-22 16:43:33.634993] W [inode.c:1044:inode_path] 0-stripe/inode: no dentry for non-root inode 5136455: 1a23309b-053a-4a78-b47f-96796ef04561
[2011-09-22 16:43:33.635281] W [client-handshake.c:683:protocol_client_reopen] 0-stripe-client-3: couldn't build path from inode 1a23309b-053a-4a78-b47f-96796ef04561
[2011-09-22 16:43:33.635523] E [client-handshake.c:722:protocol_client_reopen] 0-: failed to send the re-open request
[2011-09-22 16:43:33.637434] I [client-lk.c:617:decrement_reopen_fd_count] 0-stripe-client-3: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2011-09-22 16:43:33.637726] I [client3_1-fops.c:2364:client_fdctx_destroy] 0-stripe-client-3: sending release on fd
[2011-09-22 16:43:33.693397] W [fuse-bridge.c:184:fuse_entry_cbk] 0-glusterfs-fuse: 1476580: MKDIR() /run9786/openssl-1.0.0d => -1 (Transport endpoint is not connected)
pending frames:
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)


patchset: git://git.gluster.com/glusterfs.git
signal received: 11
time of crash: 2011-09-22 16:43:33
configuration details:
argp 1
backtrace 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.2git
/lib64/libc.so.6[0x3d920332f0]
/lib64/libpthread.so.0(pthread_mutex_lock+0x1)[0x3d92c08dc1]
/opt/glusterfs/nightly_valgrind/lib/libglusterfs.so.0(inode_link+0x9b)[0x4c41e7d]
/opt/glusterfs/nightly_valgrind/lib/glusterfs/3.2git/xlator/mount/fuse.so[0x62d310c]
/opt/glusterfs/nightly_valgrind/lib/glusterfs/3.2git/xlator/mount/fuse.so[0x62d3646]
/opt/glusterfs/nightly_valgrind/lib/glusterfs/3.2git/xlator/debug/io-stats.so(io_stats_lookup_cbk+0x23c)[0x7dc9fb3]
/opt/glusterfs/nightly_valgrind/lib/glusterfs/3.2git/xlator/performance/stat-prefetch.so(sp_lookup_cbk+0x434)[0x7ba261e]
/opt/glusterfs/nightly_valgrind/lib/glusterfs/3.2git/xlator/performance/quick-read.so(qr_lookup_cbk+0x6fc)[0x7986a0f]
/opt/glusterfs/nightly_valgrind/lib/glusterfs/3.2git/xlator/protocol/client.so(client3_1_mkdir_cbk+0x3c9)[0x6f03e1a]
/opt/glusterfs/nightly_valgrind/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211)[0x4e93fea]
/opt/glusterfs/nightly_valgrind/lib/libgfrpc.so.0(rpc_clnt_notify+0x285)[0x4e94349]
/opt/glusterfs/nightly_valgrind/lib/libgfrpc.so.0(rpc_transport_notify+0x130)[0x4e90702]
/opt/glusterfs/nightly_valgrind/lib/glusterfs/3.2git/rpc-transport/socket.so(socket_event_poll_in+0x54)[0x83e1b0b]
/opt/glusterfs/nightly_valgrind/lib/glusterfs/3.2git/rpc-transport/socket.so(socket_event_handler+0x21d)[0x83e208f]
/opt/glusterfs/nightly_valgrind/lib/libglusterfs.so.0[0x4c58b0c]
/opt/glusterfs/nightly_valgrind/lib/libglusterfs.so.0[0x4c58d2f]
/opt/glusterfs/nightly_valgrind/lib/libglusterfs.so.0(event_dispatch+0x88)[0x4c590ba]
/opt/glusterfs/nightly_valgrind/sbin/glusterfs(main+0x1b7)[0x406fd6]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x3d9201ea4d]
/opt/glusterfs/nightly_valgrind/sbin/glusterfs[0x4036d9]
---------

release is 3.2.4

*** This bug has been marked as a duplicate of bug 3505 ***