Bug 765489 (GLUSTER-3757)

Created a 4 way striped volume. Mounted via nfs and started running dbench from client. Was running 'volume profile' command in a loop from one of the server.
gnfs process in one of the server crashed and dbench failed in warmup stage itself with following error.

  17        35     6.27 MB/sec  warmup  12 sec  latency 11903.011 ms
  17        35     5.78 MB/sec  warmup  13 sec  latency 12903.107 ms
  17        35     5.37 MB/sec  warmup  14 sec  latency 13903.219 ms
[71] open ./clients/client34/filler.003 failed for handle 9941 (File exists)
(72) ERROR: handle 9941 was not found
Child failed with status 1
[root@RHEL6 mnt]# 


back trace of of core file is 

(gdb) bt
#0  0x00000034c23267d5 in __strrchr_sse42 () from /lib64/libc.so.6
#1  0x00007f2f9f5ec095 in stripe_readdirp_cbk (frame=0x7f2fa073fd18, cookie=0x7f2fa073fdbc, this=0x1b485c0, op_ret=8, op_errno=2, orig_entries=0x7fffa7608060) at stripe.c:4093
#2  0x00007f2f9f810b45 in client3_1_readdirp_cbk (req=0x7f2f9c2bcfb4, iov=0x7f2f9c2bcff4, count=1, myframe=0x7f2fa073fdbc) at client3_1-fops.c:1939
#3  0x00007f2fa1469eb5 in rpc_clnt_handle_reply (clnt=0x1b9a4e0, pollin=0x1b9bbe0) at rpc-clnt.c:741
#4  0x00007f2fa146a216 in rpc_clnt_notify (trans=0x1b9a690, mydata=0x1b9a510, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x1b9bbe0) at rpc-clnt.c:854
#5  0x00007f2fa14666ac in rpc_transport_notify (this=0x1b9a690, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x1b9bbe0) at rpc-transport.c:919
#6  0x00007f2f9ca7aaf9 in socket_event_poll_in (this=0x1b9a690) at socket.c:1647
#7  0x00007f2f9ca7b07d in socket_event_handler (fd=11, idx=0, data=0x1b9a690, poll_in=1, poll_out=0, poll_err=0) at socket.c:1762
#8  0x00007f2fa16c1b04 in event_dispatch_epoll_handler (event_pool=0x1b3f3f0, events=0x1b9bed0, i=0) at event.c:794
#9  0x00007f2fa16c1d27 in event_dispatch_epoll (event_pool=0x1b3f3f0) at event.c:856
#10 0x00007f2fa16c20b2 in event_dispatch (event_pool=0x1b3f3f0) at event.c:956
#11 0x000000000040700c in main (argc=7, argv=0x7fffa7608698) at glusterfsd.c:1509
(gdb) 


I see following errors in nfs log.

[2011-10-25 05:47:23.725774] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-hosdu-client-3: changing port to 24009 (from 0)
[2011-10-25 05:47:25.661006] E [nfs3.c:1308:nfs3_lookup] 0-nfs-nfsv3: Volume is disabled: hosdu
[2011-10-25 05:47:25.661107] E [nfs3.c:4206:nfs3_readdir] 0-nfs-nfsv3: Volume is disabled: hosdu
[2011-10-25 05:47:25.661176] E [nfs3.c:1452:nfs3_access] 0-nfs-nfsv3: Volume is disabled: hosdu
[2011-10-25 05:47:25.661219] E [nfs3.c:1452:nfs3_access] 0-nfs-nfsv3: Volume is disabled: hosdu
[2011-10-25 05:47:25.661244] E [nfs3.c:4206:nfs3_readdir] 0-nfs-nfsv3: Volume is disabled: hosdu
[2011-10-25 05:47:25.661293] E [nfs3.c:2516:nfs3_create] 0-nfs-nfsv3: Volume is disabled: hosdu
[2011-10-25 05:47:25.661361] E [nfs3.c:2516:nfs3_create] 0-nfs-nfsv3: Volume is disabled: hosdu
[2011-10-25 05:47:25.661568] E [nfs3.c:2109:nfs3_write] 0-nfs-nfsv3: Volume is disabled: hosdu
[2011-10-25 05:47:25.661807] E [nfs3.c:2109:nfs3_write] 0-nfs-nfsv3: Volume is disabled: hosdu
[2011-10-25 05:47:25.662039] E [nfs3.c:2109:nfs3_write] 0-nfs-nfsv3: Volume is disabled: hosdu
[2011-10-25 05:47:25.662337] E [nfs3.c:2109:nfs3_write] 0-nfs-nfsv3: Volume is disabled: hosdu
[2011-10-25 05:47:25.662527] E [nfs3.c:2109:nfs3_write] 0-nfs-nfsv3: Volume is disabled: hosdu
[2011-10-25 05:47:25.662716] E [nfs3.c:2109:nfs3_write] 0-nfs-nfsv3: Volume is disabled: hosdu
[2011-10-25 05:47:25.662876] E [nfs3.c:2109:nfs3_write] 0-nfs-nfsv3: Volume is disabled: hosdu
[2011-10-25 05:47:25.662984] E [nfs3.c:2109:nfs3_write] 0-nfs-nfsv3: Volume is disabled: hosdu
[2011-10-25 05:47:25.663063] E [nfs3.c:2109:nfs3_write] 0-nfs-nfsv3: Volume is disabled: hosdu
[2011-10-25 05:47:27.720352] I [client-handshake.c:1090:select_server_supported_programs] 0-hosdu-client-0: Using Program GlusterFS 3.2.5qa2, Num (1298437), Version (310)
[2011-10-25 05:47:27.720637] I [client-handshake.c:913:client_setvolume_cbk] 0-hosdu-client-0: Connected to 10.1.11.113:24009, attached to remote volume '/data/brick'.
[2011-10-25 05:47:27.722900] I [client-handshake.c:1090:select_server_supported_programs] 0-hosdu-client-1: Using Program GlusterFS 3.2.5qa2, Num (1298437), Version (310)
[2011-10-25 05:47:27.723168] I [client-handshake.c:913:client_setvolume_cbk] 0-hosdu-client-1: Connected to 10.1.11.114:24009, attached to remote volume '/data/brick'.
[2011-10-25 05:47:27.725393] I [client-handshake.c:1090:select_server_supported_programs] 0-hosdu-client-2: Using Program GlusterFS 3.2.5qa2, Num (1298437), Version (310)
[2011-10-25 05:47:27.725591] I [client-handshake.c:913:client_setvolume_cbk] 0-hosdu-client-2: Connected to 10.1.11.136:24009, attached to remote volume '/data/brick'.
[2011-10-25 05:47:27.728327] I [client-handshake.c:1090:select_server_supported_programs] 0-hosdu-client-3: Using Program GlusterFS 3.2.5qa2, Num (1298437), Version (310)
[2011-10-25 05:47:27.728601] I [client-handshake.c:913:client_setvolume_cbk] 0-hosdu-client-3: Connected to 10.1.11.137:24009, attached to remote volume '/data/brick'.
[2011-10-25 05:48:25.695789] W [inode.c:1044:inode_path] 0-hosdu/inode: no dentry for non-root inode 33578016: b1f0e3b1-fac5-4163-8d48-b4f740ef3e81
pending frames:

I have attached the nfs log and archived the core file.