Bug 765489 (GLUSTER-3757) - [glusterfs-3.2.5qa2] - gnfs process crashed while running dbench on striped volume
Summary: [glusterfs-3.2.5qa2] - gnfs process crashed while running dbench on striped v...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: GLUSTER-3757
Product: GlusterFS
Classification: Community
Component: stripe
Version: pre-release
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: ---
Assignee: Raghavendra Bhat
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2011-10-25 10:07 UTC by M S Vishwanath Bhat
Modified: 2016-06-01 01:55 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions: 3.2.5qa4


Attachments (Terms of Use)
gnfs server log (15.02 KB, text/x-log)
2011-10-25 07:07 UTC, M S Vishwanath Bhat
no flags Details

Description M S Vishwanath Bhat 2011-10-25 10:07:51 UTC
Created a 4 way striped volume. Mounted via nfs and started running dbench from client. Was running 'volume profile' command in a loop from one of the server.
gnfs process in one of the server crashed and dbench failed in warmup stage itself with following error.

  17        35     6.27 MB/sec  warmup  12 sec  latency 11903.011 ms
  17        35     5.78 MB/sec  warmup  13 sec  latency 12903.107 ms
  17        35     5.37 MB/sec  warmup  14 sec  latency 13903.219 ms
[71] open ./clients/client34/filler.003 failed for handle 9941 (File exists)
(72) ERROR: handle 9941 was not found
Child failed with status 1
[root@RHEL6 mnt]# 


back trace of of core file is 

(gdb) bt
#0  0x00000034c23267d5 in __strrchr_sse42 () from /lib64/libc.so.6
#1  0x00007f2f9f5ec095 in stripe_readdirp_cbk (frame=0x7f2fa073fd18, cookie=0x7f2fa073fdbc, this=0x1b485c0, op_ret=8, op_errno=2, orig_entries=0x7fffa7608060) at stripe.c:4093
#2  0x00007f2f9f810b45 in client3_1_readdirp_cbk (req=0x7f2f9c2bcfb4, iov=0x7f2f9c2bcff4, count=1, myframe=0x7f2fa073fdbc) at client3_1-fops.c:1939
#3  0x00007f2fa1469eb5 in rpc_clnt_handle_reply (clnt=0x1b9a4e0, pollin=0x1b9bbe0) at rpc-clnt.c:741
#4  0x00007f2fa146a216 in rpc_clnt_notify (trans=0x1b9a690, mydata=0x1b9a510, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x1b9bbe0) at rpc-clnt.c:854
#5  0x00007f2fa14666ac in rpc_transport_notify (this=0x1b9a690, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x1b9bbe0) at rpc-transport.c:919
#6  0x00007f2f9ca7aaf9 in socket_event_poll_in (this=0x1b9a690) at socket.c:1647
#7  0x00007f2f9ca7b07d in socket_event_handler (fd=11, idx=0, data=0x1b9a690, poll_in=1, poll_out=0, poll_err=0) at socket.c:1762
#8  0x00007f2fa16c1b04 in event_dispatch_epoll_handler (event_pool=0x1b3f3f0, events=0x1b9bed0, i=0) at event.c:794
#9  0x00007f2fa16c1d27 in event_dispatch_epoll (event_pool=0x1b3f3f0) at event.c:856
#10 0x00007f2fa16c20b2 in event_dispatch (event_pool=0x1b3f3f0) at event.c:956
#11 0x000000000040700c in main (argc=7, argv=0x7fffa7608698) at glusterfsd.c:1509
(gdb) 


I see following errors in nfs log.

[2011-10-25 05:47:23.725774] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-hosdu-client-3: changing port to 24009 (from 0)
[2011-10-25 05:47:25.661006] E [nfs3.c:1308:nfs3_lookup] 0-nfs-nfsv3: Volume is disabled: hosdu
[2011-10-25 05:47:25.661107] E [nfs3.c:4206:nfs3_readdir] 0-nfs-nfsv3: Volume is disabled: hosdu
[2011-10-25 05:47:25.661176] E [nfs3.c:1452:nfs3_access] 0-nfs-nfsv3: Volume is disabled: hosdu
[2011-10-25 05:47:25.661219] E [nfs3.c:1452:nfs3_access] 0-nfs-nfsv3: Volume is disabled: hosdu
[2011-10-25 05:47:25.661244] E [nfs3.c:4206:nfs3_readdir] 0-nfs-nfsv3: Volume is disabled: hosdu
[2011-10-25 05:47:25.661293] E [nfs3.c:2516:nfs3_create] 0-nfs-nfsv3: Volume is disabled: hosdu
[2011-10-25 05:47:25.661361] E [nfs3.c:2516:nfs3_create] 0-nfs-nfsv3: Volume is disabled: hosdu
[2011-10-25 05:47:25.661568] E [nfs3.c:2109:nfs3_write] 0-nfs-nfsv3: Volume is disabled: hosdu
[2011-10-25 05:47:25.661807] E [nfs3.c:2109:nfs3_write] 0-nfs-nfsv3: Volume is disabled: hosdu
[2011-10-25 05:47:25.662039] E [nfs3.c:2109:nfs3_write] 0-nfs-nfsv3: Volume is disabled: hosdu
[2011-10-25 05:47:25.662337] E [nfs3.c:2109:nfs3_write] 0-nfs-nfsv3: Volume is disabled: hosdu
[2011-10-25 05:47:25.662527] E [nfs3.c:2109:nfs3_write] 0-nfs-nfsv3: Volume is disabled: hosdu
[2011-10-25 05:47:25.662716] E [nfs3.c:2109:nfs3_write] 0-nfs-nfsv3: Volume is disabled: hosdu
[2011-10-25 05:47:25.662876] E [nfs3.c:2109:nfs3_write] 0-nfs-nfsv3: Volume is disabled: hosdu
[2011-10-25 05:47:25.662984] E [nfs3.c:2109:nfs3_write] 0-nfs-nfsv3: Volume is disabled: hosdu
[2011-10-25 05:47:25.663063] E [nfs3.c:2109:nfs3_write] 0-nfs-nfsv3: Volume is disabled: hosdu
[2011-10-25 05:47:27.720352] I [client-handshake.c:1090:select_server_supported_programs] 0-hosdu-client-0: Using Program GlusterFS 3.2.5qa2, Num (1298437), Version (310)
[2011-10-25 05:47:27.720637] I [client-handshake.c:913:client_setvolume_cbk] 0-hosdu-client-0: Connected to 10.1.11.113:24009, attached to remote volume '/data/brick'.
[2011-10-25 05:47:27.722900] I [client-handshake.c:1090:select_server_supported_programs] 0-hosdu-client-1: Using Program GlusterFS 3.2.5qa2, Num (1298437), Version (310)
[2011-10-25 05:47:27.723168] I [client-handshake.c:913:client_setvolume_cbk] 0-hosdu-client-1: Connected to 10.1.11.114:24009, attached to remote volume '/data/brick'.
[2011-10-25 05:47:27.725393] I [client-handshake.c:1090:select_server_supported_programs] 0-hosdu-client-2: Using Program GlusterFS 3.2.5qa2, Num (1298437), Version (310)
[2011-10-25 05:47:27.725591] I [client-handshake.c:913:client_setvolume_cbk] 0-hosdu-client-2: Connected to 10.1.11.136:24009, attached to remote volume '/data/brick'.
[2011-10-25 05:47:27.728327] I [client-handshake.c:1090:select_server_supported_programs] 0-hosdu-client-3: Using Program GlusterFS 3.2.5qa2, Num (1298437), Version (310)
[2011-10-25 05:47:27.728601] I [client-handshake.c:913:client_setvolume_cbk] 0-hosdu-client-3: Connected to 10.1.11.137:24009, attached to remote volume '/data/brick'.
[2011-10-25 05:48:25.695789] W [inode.c:1044:inode_path] 0-hosdu/inode: no dentry for non-root inode 33578016: b1f0e3b1-fac5-4163-8d48-b4f740ef3e81
pending frames:

I have attached the nfs log and archived the core file.

Comment 1 Anand Avati 2011-10-28 08:15:03 UTC
CHANGE: http://review.gluster.com/640 (Change-Id: I9bbdfe79664c1339b66819a6c7ea4b7698beb5c6) merged in release-3.2 by Vijay Bellur (vijay)

Comment 2 M S Vishwanath Bhat 2011-10-31 09:39:26 UTC
Checked with 3.2.5qa4. Now the crash doesn't occur, but dbench still fails. There is another bug open for that so closing the bug since crash doesn't happen any more.

Comment 3 Amar Tumballi 2011-11-17 07:03:11 UTC
as per last comment


Note You need to log in before you can comment on or make changes to this bug.