This service will be undergoing maintenance at 00:00 UTC, 2016-08-01. It is expected to last about 1 hours

Bug 763724 (GLUSTER-1992)

Summary: glusterd crashed while submitting a callback.
Product: [Community] GlusterFS Reporter: Raghavendra G <raghavendra>
Component: rdmaAssignee: Raghavendra G <raghavendra>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: medium Docs Contact:
Priority: low    
Version: mainlineCC: anush, gluster-bugs
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: RTNR Mount Type: fuse
Documentation: DNR CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:

Description Raghavendra G 2010-10-21 01:35:00 EDT
#0  0x00002aaaaafb7b5e in __rdma_ioq_churn_request (peer=0x2aaab03b7ce0, entry=0x2aaab05575ac, post=0xb9c1268)
    at ../../../../../rpc/rpc-transport/rdma/src/rdma.c:1015
#1  0x00002aaaaafb961a in __rdma_ioq_churn_entry (peer=0x2aaab03b7ce0, entry=0x2aaab05575ac)
    at ../../../../../rpc/rpc-transport/rdma/src/rdma.c:1677
#2  0x00002aaaaafb9774 in rdma_writev (this=0x2aaab03b79e8, entry=0x2aaab05575ac)
    at ../../../../../rpc/rpc-transport/rdma/src/rdma.c:1744
#3  0x00002aaaaafb9ccc in rdma_submit_request (this=0x2aaab03b79e8, req=0x7fff786c2ab0)
    at ../../../../../rpc/rpc-transport/rdma/src/rdma.c:1871
#4  0x00002b9550b448b3 in rpc_transport_submit_request (this=0x2aaab03b79e8, req=0x7fff786c2ab0)
    at ../../../../rpc/rpc-lib/src/rpc-transport.c:1007
#5  0x00002b9550b3f6cb in rpcsvc_callback_submit (rpc=0xb9b2378, trans=0x2aaab03b79e8, prog=0x2aaaaad115e0, 
    procnum=1, proghdr=0x0, proghdrcount=0) at ../../../../rpc/rpc-lib/src/rpcsvc.c:1334
#6  0x00002aaaaaabcff7 in glusterd_fetchspec_notify (this=0xb9afe48)
    at ../../../../../xlators/mgmt/glusterd/src/glusterd.c:119
#7  0x00002aaaaaafe4a9 in glusterd_create_volfiles (volinfo=0x2aaab042f8d8)
    at ../../../../../xlators/mgmt/glusterd/src/glusterd-volgen.c:1250
#8  0x00002aaaaaadcfac in glusterd_op_create_volume (req=0x2aaab018e6f8, op_errstr=0x7fff786c2d18)
    at ../../../../../xlators/mgmt/glusterd/src/glusterd-op-sm.c:1817
#9  0x00002aaaaaae6066 in glusterd_op_commit_perform (req=0x2aaab018e6f8, op_errstr=0x7fff786c2d18, rsp_dict=0x0)
    at ../../../../../xlators/mgmt/glusterd/src/glusterd-op-sm.c:4824
#10 0x00002aaaaaaf39f1 in glusterd3_1_commit_op (frame=0x0, this=0xb9afe48, data=0x0)
    at ../../../../../xlators/mgmt/glusterd/src/glusterd3_1-mops.c:1302
#11 0x00002aaaaaae404f in glusterd_op_ac_send_commit_op (event=0x2aaab017ff28, ctx=0x0)
    at ../../../../../xlators/mgmt/glusterd/src/glusterd-op-sm.c:4090
#12 0x00002aaaaaae6637 in glusterd_op_sm () at ../../../../../xlators/mgmt/glusterd/src/glusterd-op-sm.c:5118
#13 0x00002aaaaaaf1b0e in glusterd3_1_stage_op_cbk (req=0x2aaaae41a280, iov=0x2aaaae41a2c0, count=1, 
    myframe=0x2b95519f00c4) at ../../../../../xlators/mgmt/glusterd/src/glusterd3_1-mops.c:640
#14 0x00002b9550b473f5 in rpc_clnt_handle_reply (clnt=0xb9dfd28, pollin=0x2aaab017fea8)
    at ../../../../rpc/rpc-lib/src/rpc-clnt.c:752
#15 0x00002b9550b4772c in rpc_clnt_notify (trans=0xb9dff48, mydata=0xb9dfd58, event=RPC_TRANSPORT_MSG_RECEIVED, 
    data=0x2aaab017fea8) at ../../../../rpc/rpc-lib/src/rpc-clnt.c:865
#16 0x00002b9550b44ec6 in rpc_transport_notify (this=0xb9dff48, event=RPC_TRANSPORT_MSG_RECEIVED, 
    data=0x2aaab017fea8) at ../../../../rpc/rpc-lib/src/rpc-transport.c:1142
#17 0x00002aaaaada8549 in socket_event_poll_in (this=0xb9dff48)
    at ../../../../../rpc/rpc-transport/socket/src/socket.c:1619
#18 0x00002aaaaada88ba in socket_event_handler (fd=14, idx=3, data=0xb9dff48, poll_in=1, poll_out=0, poll_err=0)
    at ../../../../../rpc/rpc-transport/socket/src/socket.c:1733
#19 0x00002b9550902178 in event_dispatch_epoll_handler (event_pool=0xb9ab198, events=0xb9dd378, i=0)
    at ../../../libglusterfs/src/event.c:812
#20 0x00002b9550902367 in event_dispatch_epoll (event_pool=0xb9ab198) at ../../../libglusterfs/src/event.c:876
#21 0x00002b95509026c3 in event_dispatch (event_pool=0xb9ab198) at ../../../libglusterfs/src/event.c:984
#22 0x0000000000405e29 in main (argc=1, argv=0x7fff786c3418) at ../../../glusterfsd/src/glusterfsd.c:1410
(gdb) info thr
  5 Thread 0x41105940 (LWP 30244)  0x00000037fa20e4e8 in do_sigwait () from /lib64/libpthread.so.0
  4 Thread 0x423f1940 (LWP 30245)  0x00000037fa20d5cb in read () from /lib64/libpthread.so.0
  3 Thread 0x42df2940 (LWP 30246)  0x00000037fa20d5cb in read () from /lib64/libpthread.so.0
  2 Thread 0x437f3940 (LWP 30252)  0x00000037f9a9a0b1 in nanosleep () from /lib64/libc.so.6
* 1 Thread 0x2b9550f7d190 (LWP 30243)  0x00002aaaaafb7b5e in __rdma_ioq_churn_request (peer=0x2aaab03b7ce0, 
    entry=0x2aaab05575ac, post=0xb9c1268) at ../../../../../rpc/rpc-transport/rdma/src/rdma.c:1015
(gdb) l
1015            entry->msg.request.rpc_req->conn_private = request_ctx;
1016            if (entry->msg.request.rsp_iobref != NULL) {
1017                    request_ctx->rsp_iobref
1018                            = iobref_ref (entry->msg.request.rsp_iobref);
1019            }
1020
1021            rpc_msg = (struct rpc_msg *) entry->rpchdr[0].iov_base;
1022
1023            hdr->rm_xid    = rpc_msg->rm_xid; /* no need of hton32(rpc_msg->rm_xid),
1024                                               * since rpc_msg->rm_xid is already
(gdb) p entry->msg.request.rpc_req
$19 = (struct rpc_req *) 0x0
Comment 1 Anand Avati 2010-10-26 03:02:18 EDT
PATCH: http://patches.gluster.com/patch/5553 in master (rpc-transport/rdma: create request_context only if message contains read chunks or write chunks.)
Comment 2 Anand Avati 2010-11-07 20:15:16 EST
PATCH: http://patches.gluster.com/patch/5638 in master (rpc-transport/rdma: create request_context only if message contains read chunks or write chunks.)