Hide Forgot
Created a 4 way striped volume. Mounted via nfs and started running dbench from client. Was running 'volume profile' command in a loop from one of the server. gnfs process in one of the server crashed and dbench failed in warmup stage itself with following error. 17 35 6.27 MB/sec warmup 12 sec latency 11903.011 ms 17 35 5.78 MB/sec warmup 13 sec latency 12903.107 ms 17 35 5.37 MB/sec warmup 14 sec latency 13903.219 ms [71] open ./clients/client34/filler.003 failed for handle 9941 (File exists) (72) ERROR: handle 9941 was not found Child failed with status 1 [root@RHEL6 mnt]# back trace of of core file is (gdb) bt #0 0x00000034c23267d5 in __strrchr_sse42 () from /lib64/libc.so.6 #1 0x00007f2f9f5ec095 in stripe_readdirp_cbk (frame=0x7f2fa073fd18, cookie=0x7f2fa073fdbc, this=0x1b485c0, op_ret=8, op_errno=2, orig_entries=0x7fffa7608060) at stripe.c:4093 #2 0x00007f2f9f810b45 in client3_1_readdirp_cbk (req=0x7f2f9c2bcfb4, iov=0x7f2f9c2bcff4, count=1, myframe=0x7f2fa073fdbc) at client3_1-fops.c:1939 #3 0x00007f2fa1469eb5 in rpc_clnt_handle_reply (clnt=0x1b9a4e0, pollin=0x1b9bbe0) at rpc-clnt.c:741 #4 0x00007f2fa146a216 in rpc_clnt_notify (trans=0x1b9a690, mydata=0x1b9a510, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x1b9bbe0) at rpc-clnt.c:854 #5 0x00007f2fa14666ac in rpc_transport_notify (this=0x1b9a690, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x1b9bbe0) at rpc-transport.c:919 #6 0x00007f2f9ca7aaf9 in socket_event_poll_in (this=0x1b9a690) at socket.c:1647 #7 0x00007f2f9ca7b07d in socket_event_handler (fd=11, idx=0, data=0x1b9a690, poll_in=1, poll_out=0, poll_err=0) at socket.c:1762 #8 0x00007f2fa16c1b04 in event_dispatch_epoll_handler (event_pool=0x1b3f3f0, events=0x1b9bed0, i=0) at event.c:794 #9 0x00007f2fa16c1d27 in event_dispatch_epoll (event_pool=0x1b3f3f0) at event.c:856 #10 0x00007f2fa16c20b2 in event_dispatch (event_pool=0x1b3f3f0) at event.c:956 #11 0x000000000040700c in main (argc=7, argv=0x7fffa7608698) at glusterfsd.c:1509 (gdb) I see following errors in nfs log. [2011-10-25 05:47:23.725774] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-hosdu-client-3: changing port to 24009 (from 0) [2011-10-25 05:47:25.661006] E [nfs3.c:1308:nfs3_lookup] 0-nfs-nfsv3: Volume is disabled: hosdu [2011-10-25 05:47:25.661107] E [nfs3.c:4206:nfs3_readdir] 0-nfs-nfsv3: Volume is disabled: hosdu [2011-10-25 05:47:25.661176] E [nfs3.c:1452:nfs3_access] 0-nfs-nfsv3: Volume is disabled: hosdu [2011-10-25 05:47:25.661219] E [nfs3.c:1452:nfs3_access] 0-nfs-nfsv3: Volume is disabled: hosdu [2011-10-25 05:47:25.661244] E [nfs3.c:4206:nfs3_readdir] 0-nfs-nfsv3: Volume is disabled: hosdu [2011-10-25 05:47:25.661293] E [nfs3.c:2516:nfs3_create] 0-nfs-nfsv3: Volume is disabled: hosdu [2011-10-25 05:47:25.661361] E [nfs3.c:2516:nfs3_create] 0-nfs-nfsv3: Volume is disabled: hosdu [2011-10-25 05:47:25.661568] E [nfs3.c:2109:nfs3_write] 0-nfs-nfsv3: Volume is disabled: hosdu [2011-10-25 05:47:25.661807] E [nfs3.c:2109:nfs3_write] 0-nfs-nfsv3: Volume is disabled: hosdu [2011-10-25 05:47:25.662039] E [nfs3.c:2109:nfs3_write] 0-nfs-nfsv3: Volume is disabled: hosdu [2011-10-25 05:47:25.662337] E [nfs3.c:2109:nfs3_write] 0-nfs-nfsv3: Volume is disabled: hosdu [2011-10-25 05:47:25.662527] E [nfs3.c:2109:nfs3_write] 0-nfs-nfsv3: Volume is disabled: hosdu [2011-10-25 05:47:25.662716] E [nfs3.c:2109:nfs3_write] 0-nfs-nfsv3: Volume is disabled: hosdu [2011-10-25 05:47:25.662876] E [nfs3.c:2109:nfs3_write] 0-nfs-nfsv3: Volume is disabled: hosdu [2011-10-25 05:47:25.662984] E [nfs3.c:2109:nfs3_write] 0-nfs-nfsv3: Volume is disabled: hosdu [2011-10-25 05:47:25.663063] E [nfs3.c:2109:nfs3_write] 0-nfs-nfsv3: Volume is disabled: hosdu [2011-10-25 05:47:27.720352] I [client-handshake.c:1090:select_server_supported_programs] 0-hosdu-client-0: Using Program GlusterFS 3.2.5qa2, Num (1298437), Version (310) [2011-10-25 05:47:27.720637] I [client-handshake.c:913:client_setvolume_cbk] 0-hosdu-client-0: Connected to 10.1.11.113:24009, attached to remote volume '/data/brick'. [2011-10-25 05:47:27.722900] I [client-handshake.c:1090:select_server_supported_programs] 0-hosdu-client-1: Using Program GlusterFS 3.2.5qa2, Num (1298437), Version (310) [2011-10-25 05:47:27.723168] I [client-handshake.c:913:client_setvolume_cbk] 0-hosdu-client-1: Connected to 10.1.11.114:24009, attached to remote volume '/data/brick'. [2011-10-25 05:47:27.725393] I [client-handshake.c:1090:select_server_supported_programs] 0-hosdu-client-2: Using Program GlusterFS 3.2.5qa2, Num (1298437), Version (310) [2011-10-25 05:47:27.725591] I [client-handshake.c:913:client_setvolume_cbk] 0-hosdu-client-2: Connected to 10.1.11.136:24009, attached to remote volume '/data/brick'. [2011-10-25 05:47:27.728327] I [client-handshake.c:1090:select_server_supported_programs] 0-hosdu-client-3: Using Program GlusterFS 3.2.5qa2, Num (1298437), Version (310) [2011-10-25 05:47:27.728601] I [client-handshake.c:913:client_setvolume_cbk] 0-hosdu-client-3: Connected to 10.1.11.137:24009, attached to remote volume '/data/brick'. [2011-10-25 05:48:25.695789] W [inode.c:1044:inode_path] 0-hosdu/inode: no dentry for non-root inode 33578016: b1f0e3b1-fac5-4163-8d48-b4f740ef3e81 pending frames: I have attached the nfs log and archived the core file.
CHANGE: http://review.gluster.com/640 (Change-Id: I9bbdfe79664c1339b66819a6c7ea4b7698beb5c6) merged in release-3.2 by Vijay Bellur (vijay)
Checked with 3.2.5qa4. Now the crash doesn't occur, but dbench still fails. There is another bug open for that so closing the bug since crash doesn't happen any more.
as per last comment