Description of problem: Compile glusterfs with debug enabled glusterd crashes by assertion abord if rpc connection is failed . please find attached core file : #0 0x0000003c76616332 in _dl_runtime_resolve () from /lib64/ld-linux-x86-64.so.2 #1 0x0000000000408bb4 in glusterfsd_print_trace (signum=6) at glusterfsd.c:1766 #2 <signal handler called> #3 0x0000003c76e348c7 in raise () from /lib64/libc.so.6 #4 0x0000003c76e3652a in abort () from /lib64/libc.so.6 #5 0x0000003c76e2d46d in __assert_fail_base () from /lib64/libc.so.6 #6 0x0000003c76e2d522 in __assert_fail () from /lib64/libc.so.6 #7 0x00007f436d59b0eb in gd_syncop_mgmt_v3_unlock_cbk_fn (req=0x7f4360005a8c, iov=0x0, count=0, myframe=0x7f4360005994) at glusterd-syncop.c:418 #8 0x00007f436d542b0d in glusterd_big_locked_cbk (req=0x7f4360005a8c, iov=0x0, count=0, myframe=0x7f4360005994, fn=0x7f436d59b047 <gd_syncop_mgmt_v3_unlock_cbk_fn>) at glusterd-rpc-ops.c:209 #9 0x00007f436d59b256 in gd_syncop_mgmt_v3_unlock_cbk (req=0x7f4360005a8c, iov=0x0, count=0, myframe=0x7f4360005994) at glusterd-syncop.c:457 #10 0x00007f4376cc239f in rpc_clnt_submit (rpc=0x7f4360003980, prog=0x7f436d82c580 <gd_mgmt_v3_prog>, procnum=6, cbkfn=0x7f436d59b21e <gd_syncop_mgmt_v3_unlock_cbk>, proghdr=0x1f0cb10, proghdrcount=1, progpayload=0x0, progpayloadcount=0, iobref=0x7f43600023e0, frame=0x7f4360005994, rsphdr=0x0, rsphdr_count=0, rsp_payload=0x0, rsp_payload_count=0, rsp_iobref=0x0) at rpc-clnt.c:1601 #11 0x00007f436d59a9e3 in gd_syncop_submit_request (rpc=0x7f4360003980, req=0x1f0cbb0, local=0x1f0cc40, cookie=0x7f4360004d50, prog=0x7f436d82c580 <gd_mgmt_v3_prog>, procnum=6, cbkfn=0x7f436d59b21e <gd_syncop_mgmt_v3_unlock_cbk>, xdrproc=0x7f4376aa3429 <xdr_gd1_mgmt_v3_unlock_req>) at glusterd-syncop.c:199 #12 0x00007f436d59b3c5 in gd_syncop_mgmt_v3_unlock (op_ctx=0x7f4358000f7c, peerinfo=0x7f4360004d50, args=0x1f0cc40, my_uuid=0x1d07628 "`\346A\207`RL\256\200\344\"{\237\356n\370/var/lib/glusterd", recv_uuid=0x1f0d3b0 "", txn_id=0x7f4358002fc0 "\032X\237\033\201\fJo\230\rq\302^\211/\250", <incomplete sequence \360\255\272>) at glusterd-syncop.c:483 #13 0x00007f436d59d92b in gd_unlock_op_phase (conf=0x1d075d0, op=GD_OP_CREATE_VOLUME, op_ret=0x1f0d4f4, req=0x1d0bf3c, op_ctx=0x7f4358000f7c, op_errstr=0x7f43600034a0 "Locking failed on 10.70.43.120. Please check log file for details.", npeers=1, volname=0x7f4358002590 "Vol1", is_acquired=_gf_true, txn_id=0x7f4358002fc0 "\032X\237\033\201\fJo\230\rq\302^\211/\250", <incomplete sequence \360\255\272>, peers=0x1f0d4b0) at glusterd-syncop.c:1397 #14 0x00007f436d59e556 in gd_sync_task_begin (op_ctx=0x7f4358000f7c, req=0x1d0bf3c) at glusterd-syncop.c:1694 #15 0x00007f436d59e6c7 in glusterd_op_begin_synctask (req=0x1d0bf3c, op=GD_OP_CREATE_VOLUME, dict=0x7f4358000f7c) at glusterd-syncop.c:1740 #16 0x00007f436d589bb7 in __glusterd_handle_create_volume (req=0x1d0bf3c) at glusterd-volume-ops.c:388 #17 0x00007f436d5022f4 in glusterd_big_locked_handler (req=0x1d0bf3c, actor_fn=0x7f436d5892da <__glusterd_handle_create_volume>) at glusterd-handler.c:80 #18 0x00007f436d589c96 in glusterd_handle_create_volume (req=0x1d0bf3c) at glusterd-volume-ops.c:412 #19 0x00007f4376f4147b in synctask_wrap (old_task=0x1d0ca40) at syncop.c:335 #20 0x0000003c76e47ff0 in ?? () from /lib64/libc.so.6 #21 0x0000000000000000 in ?? () (gdb) f 6 #6 0x0000003c76e2d522 in __assert_fail () from /lib64/libc.so.6 (gdb) f 7 #7 0x00007f436d59b0eb in gd_syncop_mgmt_v3_unlock_cbk_fn (req=0x7f4360005a8c, iov=0x0, count=0, myframe=0x7f4360005994) at glusterd-syncop.c:418 418 GF_ASSERT(iov); (gdb) p iov $1 = (struct iovec *) 0x0
unable add core file
core file size is 74MB so unable to adde core file .
compress the core file? Sometimes they compress quite well? Or note where the core file can be found? Thanks
REVIEW: http://review.gluster.org/9449 (glusterd: Added error checking in proper places.) posted (#1) for review on master by Anand Nekkunti (anekkunt)
REVIEW: http://review.gluster.org/9449 (glusterd: Added error checking in proper places.) posted (#2) for review on master by Anand Nekkunti (anekkunt)
REVIEW: http://review.gluster.org/9449 (glusterd: Added iov error checking in rpc callbacks.) posted (#3) for review on master by Anand Nekkunti (anekkunt)
REVIEW: http://review.gluster.org/9449 (glusterd: Added iov error checking in rpc callbacks.) posted (#4) for review on master by Anand Nekkunti (anekkunt)
REVIEW: http://review.gluster.org/9449 (glusterd: Added iov error checking in rpc callbacks.) posted (#5) for review on master by Anand Nekkunti (anekkunt)
COMMIT: http://review.gluster.org/9449 committed in master by Krishnan Parthasarathi (kparthas) ------ commit f6a2f152aa9c8a66768e4ba0d1f66737c081639b Author: Anand <anekkunt> Date: Wed Jan 14 22:23:05 2015 +0530 glusterd: Added iov error checking in rpc callbacks. Problem : glusterd was crashing with SIGABRT if rpc connection is failed in debug mode. Reason : It was happening due to iov is passing to assert() before checking rpc status in rpc call back function (rpc is calling callback function with setting rpc status as -1 and passing NULL to iov if connection is failed). Fix : Error checking for iov added after checking the rpc status verified and error messages are added properly . Change-Id: I35c05c438444d0454aadac4e45524565a7be68a8 BUG: 1181543 Signed-off-by: Anand <anekkunt> Reviewed-on: http://review.gluster.org/9449 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Atin Mukherjee <amukherj> Reviewed-by: Krishnan Parthasarathi <kparthas> Tested-by: Krishnan Parthasarathi <kparthas>
Reducing the severity to MEDIUM as this problem is seen while running glusterd in debug mode
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report. glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939 [2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user