| Summary: | Crash in rpcsvc_conn_submit | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Community] GlusterFS | Reporter: | Anush Shetty <anush> | ||||
| Component: | transport | Assignee: | Amar Tumballi <amarts> | ||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | |||||
| Severity: | high | Docs Contact: | |||||
| Priority: | low | ||||||
| Version: | mainline | CC: | amarts, gluster-bugs, tejas, vraman | ||||
| Target Milestone: | --- | ||||||
| Target Release: | --- | ||||||
| Hardware: | All | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | Type: | --- | |||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Attachments: |
|
||||||
Created attachment 260 [details]
this is the bad .conf file that linuxconf made. Apache no go
Are you sure the backtrace and the log are same?? because the backtrace points it to server process, whereas the log is showing the client side crash.. this shows that 'conn' is surely corrupted. Suspect that there is a disconnect, but server is not clearing out variables properly. this could have been fixed with other rpc fixes.. need to varify with qa6. need to reopen/ open new bug if reproduced. Currently with latest git, with all the rpc fixes this is not happening anymore. |
Was running dbench on a 2-replica setup. (gdb) bt #0 0x00007f3c37a2bc27 in ?? () from /lib/libgcc_s.so.1 #1 0x00007f3c37a2c48b in _Unwind_Backtrace () from /lib/libgcc_s.so.1 #2 0x00007f3c3a56ecce in *__GI___backtrace (array=<value optimized out>, size=200) at ../sysdeps/x86_64/../ia64/backtrace.c:85 #3 0x00007f3c3b04f8d3 in gf_print_trace (signum=11) at common-utils.c:363 #4 <signal handler called> #5 0x0000000000000451 in ?? () #6 0x00007f3c3ae218b6 in rpcsvc_conn_submit (conn=0x1306278, hdrvec=<value optimized out>, hdrcount=1, proghdr=0x7f3c3b3e9b40, proghdrcount=1, progpayload=0x0, progpayloadcount=0, iobref=0x7f3c30004198, priv=0x451) at rpcsvc.c:1369 #7 0x00007f3c3ae2275e in rpcsvc_submit_generic (req=0x7f3c37f12cc8, proghdr=0x7f3c3b3e9b40, hdrcount=1, payload=0x0, payloadcount=0, iobref=0x1306278) at rpcsvc.c:1530 #8 0x00007f3c38bb7a5e in server_submit_reply (frame=0x132c240, req=0x7f3c37f12cc8, arg=<value optimized out>, payload=<value optimized out>, payloadcount=<value optimized out>, iobref=0x7f3c30004198, sfunc=0x7f3c3ac11fe0 <xdr_serialize_writev_rsp>) at server.c:123 #9 0x00007f3c38bc580a in server_writev_cbk (frame=0x132c240, cookie=<value optimized out>, this=0x188, op_ret=65536, op_errno=0, prebuf=<value optimized out>, postbuf=0x7f3c3b3e9de0) at server3_1-fops.c:1235 #10 0x00007f3c38dddb89 in iot_writev_cbk (frame=0x132c5a8, cookie=<value optimized out>, this=<value optimized out>, op_ret=65536, op_errno=0, prebuf=<value optimized out>, postbuf=0x7f3c3b3e9de0) at io-threads.c:945 #11 0x00007f3c38fee989 in pl_writev_cbk (frame=0x7f3c30000c88, cookie=<value optimized out>, this=<value optimized out>, op_ret=65536, op_errno=0, prebuf=<value optimized out>, postbuf=0x7f3c3b3e9de0) at posix.c:497 #12 0x00007f3c39203fc0 in posix_writev (frame=0x7f3c30001328, this=<value optimized out>, fd=<value optimized out>, vector=<value optimized out>, count=<value optimized out>, offset=<value optimized out>, iobref=0x132c148) at posix.c:2548 #13 0x00007f3c38ff0412 in pl_writev (frame=<value optimized out>, this=0x130ba18, fd=0x1329288, vector=0x132c0d8, count=<value optimized out>, offset=<value optimized out>, iobref=0x132c148) at posix.c:736 #14 0x00007f3c38de083d in iot_writev_wrapper (frame=<value optimized out>, this=0x130cdc8, fd=<value optimized out>, vector=0x132c0d8, count=1, offset=983040, iobref=0x132c148) at io-threads.c:955 #15 0x00007f3c3b05367c in call_resume_wind (stub=0x132c648) at call-stub.c:2233 #16 call_resume (stub=0x132c648) at call-stub.c:3852 #17 0x00007f3c38de589a in iot_worker (data=0x13120c8) at io-threads.c:118 #18 0x00007f3c3a7eda04 in start_thread (arg=<value optimized out>) at pthread_create.c:300 #19 0x00007f3c3a556d4d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #20 0x0000000000000000 in ?? () configuration details: argp 1 backtrace 1 dlfcn 1 fdatasync 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 3.1.0git /lib/libc.so.6[0x7fce6d755530] /lib/libpthread.so.0(pthread_mutex_lock+0x1)[0x7fce6da9b081] /gluster/3.1/lib/libglusterfs.so.0(inode_unref+0x1c)[0x7fce6e2fd68c] /gluster/3.1/lib/glusterfs/3.1.0git/xlator/protocol/client.so(client3_1_releasedir+0x116)[0x7fce6c292ef6] /gluster/3.1/lib/glusterfs/3.1.0git/xlator/protocol/client.so(client_releasedir+0x156)[0x7fce6c28ba36] /gluster/3.1/lib/libglusterfs.so.0(fd_unref+0x28f)[0x7fce6e308eaf] /gluster/3.1/lib/glusterfs/3.1.0git/xlator/cluster/replicate.so(afr_sh_entry_done+0x2c)[0x7fce6c06c14c] /gluster/3.1/lib/glusterfs/3.1.0git/xlator/cluster/replicate.so(afr_sh_entry_unlck_cbk+0x7d)[0x7fce6c06d64d] /gluster/3.1/lib/glusterfs/3.1.0git/xlator/protocol/client.so(client3_1_entrylk_cbk+0xb0)[0x7fce6c294400] /gluster/3.1/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0xab)[0x7fce6e0d2e5b] /gluster/3.1/lib/libgfrpc.so.0(rpc_clnt_notify+0x7b)[0x7fce6e0d3c2b] /gluster/3.1/lib/libgfrpc.so.0(rpc_transport_notify+0x26)[0x7fce6e0cf226] /gluster/3.1/lib/glusterfs/3.1.0git/rpc-transport/socket.so(socket_event_poll_in+0x34)[0x7fce6b3e90d4] /gluster/3.1/lib/glusterfs/3.1.0git/rpc-transport/socket.so(socket_event_handler+0xd8)[0x7fce6b3e91c8] /gluster/3.1/lib/libglusterfs.so.0[0x7fce6e30aab4] /gluster/3.1/sbin/glusterfs(main+0x1e2)[0x404b72] /lib/libc.so.6(__libc_start_main+0xfd)[0x7fce6d740abd] /gluster/3.1/sbin/glusterfs[0x402d39]