Bug 762943 (GLUSTER-1211) - Crash in rpcsvc_conn_submit
Summary: Crash in rpcsvc_conn_submit
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: GLUSTER-1211
Product: GlusterFS
Classification: Community
Component: transport
Version: mainline
Hardware: All
OS: Linux
low
high
Target Milestone: ---
Assignee: Amar Tumballi
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2010-07-23 13:44 UTC by Anush Shetty
Modified: 2015-12-01 16:45 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:


Attachments (Terms of Use)
Valgrind log (131.64 KB, text/x-log)
2010-07-26 08:35 UTC, Anush Shetty
no flags Details

Description Anush Shetty 2010-07-23 13:44:34 UTC
Was running dbench on a 2-replica setup. 

(gdb) bt
#0  0x00007f3c37a2bc27 in ?? () from /lib/libgcc_s.so.1
#1  0x00007f3c37a2c48b in _Unwind_Backtrace () from /lib/libgcc_s.so.1
#2  0x00007f3c3a56ecce in *__GI___backtrace (array=<value optimized out>, size=200) at ../sysdeps/x86_64/../ia64/backtrace.c:85
#3  0x00007f3c3b04f8d3 in gf_print_trace (signum=11) at common-utils.c:363
#4  <signal handler called>
#5  0x0000000000000451 in ?? ()
#6  0x00007f3c3ae218b6 in rpcsvc_conn_submit (conn=0x1306278, hdrvec=<value optimized out>, hdrcount=1, proghdr=0x7f3c3b3e9b40, proghdrcount=1, 
    progpayload=0x0, progpayloadcount=0, iobref=0x7f3c30004198, priv=0x451) at rpcsvc.c:1369
#7  0x00007f3c3ae2275e in rpcsvc_submit_generic (req=0x7f3c37f12cc8, proghdr=0x7f3c3b3e9b40, hdrcount=1, payload=0x0, payloadcount=0, iobref=0x1306278)
    at rpcsvc.c:1530
#8  0x00007f3c38bb7a5e in server_submit_reply (frame=0x132c240, req=0x7f3c37f12cc8, arg=<value optimized out>, payload=<value optimized out>, 
    payloadcount=<value optimized out>, iobref=0x7f3c30004198, sfunc=0x7f3c3ac11fe0 <xdr_serialize_writev_rsp>) at server.c:123
#9  0x00007f3c38bc580a in server_writev_cbk (frame=0x132c240, cookie=<value optimized out>, this=0x188, op_ret=65536, op_errno=0, 
    prebuf=<value optimized out>, postbuf=0x7f3c3b3e9de0) at server3_1-fops.c:1235
#10 0x00007f3c38dddb89 in iot_writev_cbk (frame=0x132c5a8, cookie=<value optimized out>, this=<value optimized out>, op_ret=65536, op_errno=0, 
    prebuf=<value optimized out>, postbuf=0x7f3c3b3e9de0) at io-threads.c:945
#11 0x00007f3c38fee989 in pl_writev_cbk (frame=0x7f3c30000c88, cookie=<value optimized out>, this=<value optimized out>, op_ret=65536, op_errno=0, 
    prebuf=<value optimized out>, postbuf=0x7f3c3b3e9de0) at posix.c:497
#12 0x00007f3c39203fc0 in posix_writev (frame=0x7f3c30001328, this=<value optimized out>, fd=<value optimized out>, vector=<value optimized out>, 
    count=<value optimized out>, offset=<value optimized out>, iobref=0x132c148) at posix.c:2548
#13 0x00007f3c38ff0412 in pl_writev (frame=<value optimized out>, this=0x130ba18, fd=0x1329288, vector=0x132c0d8, count=<value optimized out>, 
    offset=<value optimized out>, iobref=0x132c148) at posix.c:736
#14 0x00007f3c38de083d in iot_writev_wrapper (frame=<value optimized out>, this=0x130cdc8, fd=<value optimized out>, vector=0x132c0d8, count=1, 
    offset=983040, iobref=0x132c148) at io-threads.c:955
#15 0x00007f3c3b05367c in call_resume_wind (stub=0x132c648) at call-stub.c:2233
#16 call_resume (stub=0x132c648) at call-stub.c:3852
#17 0x00007f3c38de589a in iot_worker (data=0x13120c8) at io-threads.c:118
#18 0x00007f3c3a7eda04 in start_thread (arg=<value optimized out>) at pthread_create.c:300
#19 0x00007f3c3a556d4d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#20 0x0000000000000000 in ?? ()



configuration details:
argp 1
backtrace 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.1.0git
/lib/libc.so.6[0x7fce6d755530]
/lib/libpthread.so.0(pthread_mutex_lock+0x1)[0x7fce6da9b081]
/gluster/3.1/lib/libglusterfs.so.0(inode_unref+0x1c)[0x7fce6e2fd68c]
/gluster/3.1/lib/glusterfs/3.1.0git/xlator/protocol/client.so(client3_1_releasedir+0x116)[0x7fce6c292ef6]
/gluster/3.1/lib/glusterfs/3.1.0git/xlator/protocol/client.so(client_releasedir+0x156)[0x7fce6c28ba36]
/gluster/3.1/lib/libglusterfs.so.0(fd_unref+0x28f)[0x7fce6e308eaf]
/gluster/3.1/lib/glusterfs/3.1.0git/xlator/cluster/replicate.so(afr_sh_entry_done+0x2c)[0x7fce6c06c14c]
/gluster/3.1/lib/glusterfs/3.1.0git/xlator/cluster/replicate.so(afr_sh_entry_unlck_cbk+0x7d)[0x7fce6c06d64d]
/gluster/3.1/lib/glusterfs/3.1.0git/xlator/protocol/client.so(client3_1_entrylk_cbk+0xb0)[0x7fce6c294400]
/gluster/3.1/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0xab)[0x7fce6e0d2e5b]
/gluster/3.1/lib/libgfrpc.so.0(rpc_clnt_notify+0x7b)[0x7fce6e0d3c2b]
/gluster/3.1/lib/libgfrpc.so.0(rpc_transport_notify+0x26)[0x7fce6e0cf226]
/gluster/3.1/lib/glusterfs/3.1.0git/rpc-transport/socket.so(socket_event_poll_in+0x34)[0x7fce6b3e90d4]
/gluster/3.1/lib/glusterfs/3.1.0git/rpc-transport/socket.so(socket_event_handler+0xd8)[0x7fce6b3e91c8]
/gluster/3.1/lib/libglusterfs.so.0[0x7fce6e30aab4]
/gluster/3.1/sbin/glusterfs(main+0x1e2)[0x404b72]
/lib/libc.so.6(__libc_start_main+0xfd)[0x7fce6d740abd]
/gluster/3.1/sbin/glusterfs[0x402d39]

Comment 1 Anush Shetty 2010-07-26 08:35:19 UTC
Created attachment 260 [details]
this is the bad .conf file that linuxconf made. Apache no go

Comment 2 Amar Tumballi 2010-08-04 08:27:54 UTC
Are you sure the backtrace and the log are same?? because the backtrace points it to server process, whereas the log is showing the client side crash..

Comment 3 Amar Tumballi 2010-08-04 09:01:55 UTC
this shows that 'conn' is surely corrupted. Suspect that there is a disconnect, but server is not clearing out variables properly.

Comment 4 Amar Tumballi 2010-08-13 10:28:18 UTC
this could have been fixed with other rpc fixes.. need to varify with qa6.

Comment 5 Amar Tumballi 2010-09-01 03:51:23 UTC
need to reopen/ open new bug if reproduced. Currently with latest git, with all the rpc fixes this is not happening anymore.


Note You need to log in before you can comment on or make changes to this bug.