Bug 763724 (GLUSTER-1992)
| Summary: | glusterd crashed while submitting a callback. | ||
|---|---|---|---|
| Product: | [Community] GlusterFS | Reporter: | Raghavendra G <raghavendra> |
| Component: | rdma | Assignee: | Raghavendra G <raghavendra> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | |
| Severity: | medium | Docs Contact: | |
| Priority: | low | ||
| Version: | mainline | CC: | anush, gluster-bugs |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | Type: | --- | |
| Regression: | RTNR | Mount Type: | fuse |
| Documentation: | DNR | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
PATCH: http://patches.gluster.com/patch/5553 in master (rpc-transport/rdma: create request_context only if message contains read chunks or write chunks.) PATCH: http://patches.gluster.com/patch/5638 in master (rpc-transport/rdma: create request_context only if message contains read chunks or write chunks.) |
#0 0x00002aaaaafb7b5e in __rdma_ioq_churn_request (peer=0x2aaab03b7ce0, entry=0x2aaab05575ac, post=0xb9c1268) at ../../../../../rpc/rpc-transport/rdma/src/rdma.c:1015 #1 0x00002aaaaafb961a in __rdma_ioq_churn_entry (peer=0x2aaab03b7ce0, entry=0x2aaab05575ac) at ../../../../../rpc/rpc-transport/rdma/src/rdma.c:1677 #2 0x00002aaaaafb9774 in rdma_writev (this=0x2aaab03b79e8, entry=0x2aaab05575ac) at ../../../../../rpc/rpc-transport/rdma/src/rdma.c:1744 #3 0x00002aaaaafb9ccc in rdma_submit_request (this=0x2aaab03b79e8, req=0x7fff786c2ab0) at ../../../../../rpc/rpc-transport/rdma/src/rdma.c:1871 #4 0x00002b9550b448b3 in rpc_transport_submit_request (this=0x2aaab03b79e8, req=0x7fff786c2ab0) at ../../../../rpc/rpc-lib/src/rpc-transport.c:1007 #5 0x00002b9550b3f6cb in rpcsvc_callback_submit (rpc=0xb9b2378, trans=0x2aaab03b79e8, prog=0x2aaaaad115e0, procnum=1, proghdr=0x0, proghdrcount=0) at ../../../../rpc/rpc-lib/src/rpcsvc.c:1334 #6 0x00002aaaaaabcff7 in glusterd_fetchspec_notify (this=0xb9afe48) at ../../../../../xlators/mgmt/glusterd/src/glusterd.c:119 #7 0x00002aaaaaafe4a9 in glusterd_create_volfiles (volinfo=0x2aaab042f8d8) at ../../../../../xlators/mgmt/glusterd/src/glusterd-volgen.c:1250 #8 0x00002aaaaaadcfac in glusterd_op_create_volume (req=0x2aaab018e6f8, op_errstr=0x7fff786c2d18) at ../../../../../xlators/mgmt/glusterd/src/glusterd-op-sm.c:1817 #9 0x00002aaaaaae6066 in glusterd_op_commit_perform (req=0x2aaab018e6f8, op_errstr=0x7fff786c2d18, rsp_dict=0x0) at ../../../../../xlators/mgmt/glusterd/src/glusterd-op-sm.c:4824 #10 0x00002aaaaaaf39f1 in glusterd3_1_commit_op (frame=0x0, this=0xb9afe48, data=0x0) at ../../../../../xlators/mgmt/glusterd/src/glusterd3_1-mops.c:1302 #11 0x00002aaaaaae404f in glusterd_op_ac_send_commit_op (event=0x2aaab017ff28, ctx=0x0) at ../../../../../xlators/mgmt/glusterd/src/glusterd-op-sm.c:4090 #12 0x00002aaaaaae6637 in glusterd_op_sm () at ../../../../../xlators/mgmt/glusterd/src/glusterd-op-sm.c:5118 #13 0x00002aaaaaaf1b0e in glusterd3_1_stage_op_cbk (req=0x2aaaae41a280, iov=0x2aaaae41a2c0, count=1, myframe=0x2b95519f00c4) at ../../../../../xlators/mgmt/glusterd/src/glusterd3_1-mops.c:640 #14 0x00002b9550b473f5 in rpc_clnt_handle_reply (clnt=0xb9dfd28, pollin=0x2aaab017fea8) at ../../../../rpc/rpc-lib/src/rpc-clnt.c:752 #15 0x00002b9550b4772c in rpc_clnt_notify (trans=0xb9dff48, mydata=0xb9dfd58, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x2aaab017fea8) at ../../../../rpc/rpc-lib/src/rpc-clnt.c:865 #16 0x00002b9550b44ec6 in rpc_transport_notify (this=0xb9dff48, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x2aaab017fea8) at ../../../../rpc/rpc-lib/src/rpc-transport.c:1142 #17 0x00002aaaaada8549 in socket_event_poll_in (this=0xb9dff48) at ../../../../../rpc/rpc-transport/socket/src/socket.c:1619 #18 0x00002aaaaada88ba in socket_event_handler (fd=14, idx=3, data=0xb9dff48, poll_in=1, poll_out=0, poll_err=0) at ../../../../../rpc/rpc-transport/socket/src/socket.c:1733 #19 0x00002b9550902178 in event_dispatch_epoll_handler (event_pool=0xb9ab198, events=0xb9dd378, i=0) at ../../../libglusterfs/src/event.c:812 #20 0x00002b9550902367 in event_dispatch_epoll (event_pool=0xb9ab198) at ../../../libglusterfs/src/event.c:876 #21 0x00002b95509026c3 in event_dispatch (event_pool=0xb9ab198) at ../../../libglusterfs/src/event.c:984 #22 0x0000000000405e29 in main (argc=1, argv=0x7fff786c3418) at ../../../glusterfsd/src/glusterfsd.c:1410 (gdb) info thr 5 Thread 0x41105940 (LWP 30244) 0x00000037fa20e4e8 in do_sigwait () from /lib64/libpthread.so.0 4 Thread 0x423f1940 (LWP 30245) 0x00000037fa20d5cb in read () from /lib64/libpthread.so.0 3 Thread 0x42df2940 (LWP 30246) 0x00000037fa20d5cb in read () from /lib64/libpthread.so.0 2 Thread 0x437f3940 (LWP 30252) 0x00000037f9a9a0b1 in nanosleep () from /lib64/libc.so.6 * 1 Thread 0x2b9550f7d190 (LWP 30243) 0x00002aaaaafb7b5e in __rdma_ioq_churn_request (peer=0x2aaab03b7ce0, entry=0x2aaab05575ac, post=0xb9c1268) at ../../../../../rpc/rpc-transport/rdma/src/rdma.c:1015 (gdb) l 1015 entry->msg.request.rpc_req->conn_private = request_ctx; 1016 if (entry->msg.request.rsp_iobref != NULL) { 1017 request_ctx->rsp_iobref 1018 = iobref_ref (entry->msg.request.rsp_iobref); 1019 } 1020 1021 rpc_msg = (struct rpc_msg *) entry->rpchdr[0].iov_base; 1022 1023 hdr->rm_xid = rpc_msg->rm_xid; /* no need of hton32(rpc_msg->rm_xid), 1024 * since rpc_msg->rm_xid is already (gdb) p entry->msg.request.rpc_req $19 = (struct rpc_req *) 0x0