Description of problem: Gluster crashed during sysbench and replace brick start status commit start status abort test. Version-Release number of selected component (if applicable): 3.4.1 How reproducible: I haven't reproduced it again so far. Steps to Reproduce: 1. start doing on replicate-distribute volume sysbench --num-threads=1 --max-requests=0 --init-rng=on --max-time=0 --test=fileio --file-total-size=$((1*10000))M --file-num=1000 --file-test-mode=seqwr run 2. start replacing bricks around 3. randomly choose to abort replication Actual results: [2013-10-04 12:26:32.878559] E [socket.c:2788:socket_connect] 0-management: connection attempt failed (Connection refused) [2013-10-04 12:26:35.879044] E [socket.c:2788:socket_connect] 0-management: connection attempt failed (Connection refused) [2013-10-04 12:26:38.879475] E [socket.c:2788:socket_connect] 0-management: connection attempt failed (Connection refused) [2013-10-04 12:26:39.815428] W [socket.c:514:__socket_rwv] 0-management: readv failed (No data available) [2013-10-04 12:26:39.817722] E [glusterd-utils.c:1493:glusterd_brick_unlink_socket_file] 0-management: Failed to remove /var/run/b86e8c2ca7bb1a2d7c781ecc68b19f23.socket error: Resource temporarily unavailable [2013-10-04 12:26:41.948948] W [socket.c:514:__socket_rwv] 0-socket.management: writev failed (Broken pipe) [2013-10-04 12:26:41.949051] W [rpcsvc.c:173:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) [2013-10-04 12:26:41.949065] E [rpcsvc.c:450:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully [2013-10-04 12:26:41.950319] W [rpcsvc.c:173:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) [2013-10-04 12:26:41.950335] E [rpcsvc.c:450:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully [2013-10-04 12:26:41.951769] W [rpcsvc.c:173:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) [2013-10-04 12:26:41.951785] E [rpcsvc.c:450:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully [2013-10-04 12:26:41.951820] W [rpcsvc.c:173:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) [2013-10-04 12:26:41.951836] E [rpcsvc.c:450:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully [2013-10-04 12:26:41.951874] W [rpcsvc.c:173:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) [2013-10-04 12:26:41.951885] E [rpcsvc.c:450:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully [2013-10-04 12:26:41.951911] W [rpcsvc.c:173:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) [2013-10-04 12:26:41.951921] E [rpcsvc.c:450:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully [2013-10-04 12:26:41.951944] W [rpcsvc.c:173:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) [2013-10-04 12:26:41.952460] E [rpcsvc.c:450:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully [2013-10-04 12:26:41.952577] W [rpcsvc.c:173:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) [2013-10-04 12:26:41.952599] E [rpcsvc.c:450:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully [2013-10-04 12:26:41.952626] W [rpcsvc.c:173:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) [2013-10-04 12:26:41.952635] E [rpcsvc.c:450:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully [2013-10-04 12:26:41.952659] W [rpcsvc.c:173:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) [2013-10-04 12:26:41.952668] E [rpcsvc.c:450:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully [2013-10-04 12:26:41.952690] W [rpcsvc.c:173:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) [2013-10-04 12:26:41.952699] E [rpcsvc.c:450:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully [2013-10-04 12:26:41.952723] W [rpcsvc.c:173:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) [2013-10-04 12:26:41.952731] E [rpcsvc.c:450:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully [2013-10-04 12:26:41.952743] W [socket.c:514:__socket_rwv] 0-socket.management: writev failed (Broken pipe) [2013-10-04 12:26:41.953293] W [rpcsvc.c:173:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) [2013-10-04 12:26:41.953309] E [rpcsvc.c:450:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully [2013-10-04 12:26:44.881902] E [socket.c:2788:socket_connect] 0-management: connection attempt failed (Connection refused) [2013-10-04 12:26:48.022183] E [socket.c:2788:socket_connect] 0-management: connection attempt failed (Connection refused) [2013-10-04 12:26:51.027141] E [socket.c:2788:socket_connect] 0-management: connection attempt failed (Connection refused) [2013-10-04 12:26:54.027648] E [socket.c:2788:socket_connect] 0-management: connection attempt failed (Connection refused) [2013-10-04 12:26:57.028147] E [socket.c:2788:socket_connect] 0-management: connection attempt failed (Connection refused) [2013-10-04 12:27:00.028649] E [socket.c:2788:socket_connect] 0-management: connection attempt failed (Connection refused) [2013-10-04 12:27:03.032863] E [socket.c:2788:socket_connect] 0-management: connection attempt failed (Connection refused) [2013-10-04 12:27:06.033506] E [socket.c:2788:socket_connect] 0-management: connection attempt failed (Connection refused) [2013-10-04 12:27:07.307871] W [glusterd-op-sm.c:3170:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2013-10-04 12:27:07.318216] W [glusterd-op-sm.c:3115:glusterd_op_modify_op_ctx] 0-management: Failed uuid to hostname conversion [2013-10-04 12:27:07.318234] W [glusterd-op-sm.c:3170:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2013-10-04 12:27:07.325132] W [glusterd-op-sm.c:3115:glusterd_op_modify_op_ctx] 0-management: Failed uuid to hostname conversion [2013-10-04 12:27:07.325166] W [glusterd-op-sm.c:3170:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2013-10-04 12:27:07.332520] W [glusterd-op-sm.c:3115:glusterd_op_modify_op_ctx] 0-management: Failed uuid to hostname conversion [2013-10-04 12:27:07.332539] W [glusterd-op-sm.c:3170:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2013-10-04 12:27:07.340944] W [glusterd-op-sm.c:3115:glusterd_op_modify_op_ctx] 0-management: Failed uuid to hostname conversion [2013-10-04 12:27:07.340962] W [glusterd-op-sm.c:3170:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2013-10-04 12:27:07.343860] W [glusterd-op-sm.c:3115:glusterd_op_modify_op_ctx] 0-management: Failed uuid to hostname conversion [2013-10-04 12:27:07.343875] W [glusterd-op-sm.c:3170:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2013-10-04 12:27:07.356378] W [glusterd-op-sm.c:3115:glusterd_op_modify_op_ctx] 0-management: Failed uuid to hostname conversion [2013-10-04 12:27:07.356422] W [glusterd-op-sm.c:3170:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2013-10-04 12:27:07.364330] W [glusterd-op-sm.c:3115:glusterd_op_modify_op_ctx] 0-management: Failed uuid to hostname conversion [2013-10-04 12:27:07.364348] W [glusterd-op-sm.c:3170:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2013-10-04 12:27:07.426657] W [glusterd-op-sm.c:3115:glusterd_op_modify_op_ctx] 0-management: Failed uuid to hostname conversion [2013-10-04 12:27:07.426682] W [glusterd-op-sm.c:3170:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2013-10-04 12:27:07.428996] W [glusterd-op-sm.c:3115:glusterd_op_modify_op_ctx] 0-management: Failed uuid to hostname conversion [2013-10-04 12:27:07.429009] W [glusterd-op-sm.c:3170:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2013-10-04 12:27:09.043113] E [socket.c:2788:socket_connect] 0-management: connection attempt failed (Connection refused) [2013-10-04 12:27:12.045260] E [socket.c:2788:socket_connect] 0-management: connection attempt failed (Connection refused) [2013-10-04 12:27:12.294977] W [glusterd-op-sm.c:3170:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2013-10-04 12:27:12.297498] W [glusterd-op-sm.c:3115:glusterd_op_modify_op_ctx] 0-management: Failed uuid to hostname conversion [2013-10-04 12:27:12.297516] W [glusterd-op-sm.c:3170:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2013-10-04 12:27:12.298783] W [glusterd-op-sm.c:3115:glusterd_op_modify_op_ctx] 0-management: Failed uuid to hostname conversion [2013-10-04 12:27:12.298797] W [glusterd-op-sm.c:3170:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2013-10-04 12:27:12.300007] W [glusterd-op-sm.c:3115:glusterd_op_modify_op_ctx] 0-management: Failed uuid to hostname conversion [2013-10-04 12:27:12.300020] W [glusterd-op-sm.c:3170:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2013-10-04 12:27:12.301308] W [glusterd-op-sm.c:3115:glusterd_op_modify_op_ctx] 0-management: Failed uuid to hostname conversion [2013-10-04 12:27:12.301323] W [glusterd-op-sm.c:3170:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2013-10-04 12:27:12.302892] W [glusterd-op-sm.c:3115:glusterd_op_modify_op_ctx] 0-management: Failed uuid to hostname conversion [2013-10-04 12:27:12.302908] W [glusterd-op-sm.c:3170:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2013-10-04 12:27:12.305382] W [glusterd-op-sm.c:3115:glusterd_op_modify_op_ctx] 0-management: Failed uuid to hostname conversion [2013-10-04 12:27:12.305417] W [glusterd-op-sm.c:3170:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2013-10-04 12:27:12.307688] W [glusterd-op-sm.c:3115:glusterd_op_modify_op_ctx] 0-management: Failed uuid to hostname conversion [2013-10-04 12:27:12.307704] W [glusterd-op-sm.c:3170:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2013-10-04 12:27:12.312481] W [glusterd-op-sm.c:3115:glusterd_op_modify_op_ctx] 0-management: Failed uuid to hostname conversion [2013-10-04 12:27:12.312497] W [glusterd-op-sm.c:3170:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2013-10-04 12:27:12.314657] W [glusterd-op-sm.c:3115:glusterd_op_modify_op_ctx] 0-management: Failed uuid to hostname conversion [2013-10-04 12:27:12.314673] W [glusterd-op-sm.c:3170:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2013-10-04 12:27:15.048443] E [socket.c:2788:socket_connect] 0-management: connection attempt failed (Connection refused) [2013-10-04 12:27:18.060765] E [socket.c:2788:socket_connect] 0-management: connection attempt failed (Connection refused) [2013-10-04 12:27:21.061600] E [socket.c:2788:socket_connect] 0-management: connection attempt failed (Connection refused) [2013-10-04 12:27:24.062185] E [socket.c:2788:socket_connect] 0-management: connection attempt failed (Connection refused) [2013-10-04 12:27:24.514946] E [glusterd-utils.c:1493:glusterd_brick_unlink_socket_file] 0-management: Failed to remove /var/run/b86e8c2ca7bb1a2d7c781ecc68b19f23.socket error: Resource temporarily unavailable [2013-10-04 12:27:24.515481] W [socket.c:514:__socket_rwv] 0-management: readv failed (No data available) [2013-10-04 12:27:26.578373] W [socket.c:514:__socket_rwv] 0-socket.management: writev failed (Connection reset by peer) [2013-10-04 12:27:27.736232] W [rpcsvc.c:173:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) [2013-10-04 12:27:27.736274] E [rpcsvc.c:450:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully [2013-10-04 12:27:27.737993] W [rpcsvc.c:173:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) [2013-10-04 12:27:27.738010] E [rpcsvc.c:450:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully [2013-10-04 12:27:27.741726] W [rpcsvc.c:173:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) [2013-10-04 12:27:27.741746] E [rpcsvc.c:450:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully [2013-10-04 12:27:27.741780] W [rpcsvc.c:173:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) [2013-10-04 12:27:27.741795] E [rpcsvc.c:450:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully [2013-10-04 12:27:27.741834] W [rpcsvc.c:173:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) [2013-10-04 12:27:27.741845] E [rpcsvc.c:450:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully [2013-10-04 12:27:27.741872] W [rpcsvc.c:173:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) [2013-10-04 12:27:27.741882] E [rpcsvc.c:450:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully [2013-10-04 12:27:27.741920] W [rpcsvc.c:173:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) [2013-10-04 12:27:27.741930] E [rpcsvc.c:450:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully [2013-10-04 12:27:27.741955] W [rpcsvc.c:173:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) [2013-10-04 12:27:27.741965] E [rpcsvc.c:450:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully [2013-10-04 12:27:27.741990] W [rpcsvc.c:173:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) [2013-10-04 12:27:27.741999] E [rpcsvc.c:450:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully [2013-10-04 12:27:27.742025] W [rpcsvc.c:173:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) [2013-10-04 12:27:27.742035] E [rpcsvc.c:450:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully [2013-10-04 12:27:27.743923] W [rpcsvc.c:173:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) [2013-10-04 12:27:27.743940] E [rpcsvc.c:450:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully [2013-10-04 12:27:27.744003] W [rpcsvc.c:173:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) [2013-10-04 12:27:27.744015] E [rpcsvc.c:450:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully [2013-10-04 12:27:27.744034] W [socket.c:514:__socket_rwv] 0-socket.management: writev failed (Broken pipe) [2013-10-04 12:27:27.753584] W [socket.c:514:__socket_rwv] 0-management: readv failed (No data available) [2013-10-04 12:27:27.753608] W [socket.c:1962:__socket_proto_state_machine] 0-management: reading from socket failed. Error (No data available), peer (/var/run/3cfb5e29e1ae61f0cc499296479de533.socket) [2013-10-04 12:27:30.064792] E [socket.c:2788:socket_connect] 0-management: connection attempt failed (Connection refused) [2013-10-04 12:27:30.974983] W [glusterd-op-sm.c:3170:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2013-10-04 12:27:30.977402] W [glusterd-op-sm.c:3115:glusterd_op_modify_op_ctx] 0-management: Failed uuid to hostname conversion [2013-10-04 12:27:30.977431] W [glusterd-op-sm.c:3170:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2013-10-04 12:27:30.979020] W [glusterd-op-sm.c:3115:glusterd_op_modify_op_ctx] 0-management: Failed uuid to hostname conversion [2013-10-04 12:27:30.979035] W [glusterd-op-sm.c:3170:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2013-10-04 12:27:30.981078] W [glusterd-op-sm.c:3115:glusterd_op_modify_op_ctx] 0-management: Failed uuid to hostname conversion [2013-10-04 12:27:30.981093] W [glusterd-op-sm.c:3170:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2013-10-04 12:27:30.983241] W [glusterd-op-sm.c:3115:glusterd_op_modify_op_ctx] 0-management: Failed uuid to hostname conversion [2013-10-04 12:27:30.983255] W [glusterd-op-sm.c:3170:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2013-10-04 12:27:30.985737] W [glusterd-op-sm.c:3115:glusterd_op_modify_op_ctx] 0-management: Failed uuid to hostname conversion [2013-10-04 12:27:30.985751] W [glusterd-op-sm.c:3170:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2013-10-04 12:27:30.987781] W [glusterd-op-sm.c:3115:glusterd_op_modify_op_ctx] 0-management: Failed uuid to hostname conversion [2013-10-04 12:27:30.987796] W [glusterd-op-sm.c:3170:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2013-10-04 12:27:30.989591] W [glusterd-op-sm.c:3115:glusterd_op_modify_op_ctx] 0-management: Failed uuid to hostname conversion [2013-10-04 12:27:30.989606] W [glusterd-op-sm.c:3170:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2013-10-04 12:27:31.007437] E [glusterd-op-sm.c:1903:_add_brick_name_to_dict] (-->/usr/lib64/glusterfs/3.4.1/xlator/mgmt/glusterd.so(glusterd_op_commit_perform+0x1c6) [0x7fdeba762ed6] (-->/usr/lib64/glusterfs/3.4.1/xlator/mgmt/glusterd.so(+0x3b708) [0x7fdeba762708] (-->/usr/lib64/glusterfs/3.4.1/xlator/mgmt/glusterd.so(+0x3b057) [0x7fdeba762057]))) 0-: Assertion failed: brick pending frames: frame : type(0) op(0) patchset: git://git.gluster.com/glusterfs.git signal received: 11 time of crash: 2013-10-04 12:27:31configuration details: argp 1 backtrace 1 dlfcn 1 fdatasync 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 3.4.1 /lib64/libc.so.6[0x3e8a832920] /lib64/libc.so.6(_IO_vfprintf+0x3e5c)[0x3e8a8480ac] /lib64/libc.so.6(__vsnprintf_chk+0xb0)[0x3e8a8ffa80] /lib64/libc.so.6(__snprintf_chk+0x7a)[0x3e8a8ff9ba] /usr/lib64/glusterfs/3.4.1/xlator/mgmt/glusterd.so(+0x3a920)[0x7fdeba761920] /usr/lib64/glusterfs/3.4.1/xlator/mgmt/glusterd.so(+0x3b057)[0x7fdeba762057] /usr/lib64/glusterfs/3.4.1/xlator/mgmt/glusterd.so(+0x3b708)[0x7fdeba762708] /usr/lib64/glusterfs/3.4.1/xlator/mgmt/glusterd.so(glusterd_op_commit_perform+0x1c6)[0x7fdeba762ed6] /usr/lib64/glusterfs/3.4.1/xlator/mgmt/glusterd.so(gd_commit_op_phase+0xb7)[0x7fdeba7aebe7] /usr/lib64/glusterfs/3.4.1/xlator/mgmt/glusterd.so(gd_sync_task_begin+0x2c2)[0x7fdeba7af8e2] /usr/lib64/glusterfs/3.4.1/xlator/mgmt/glusterd.so(glusterd_op_begin_synctask+0x3b)[0x7fdeba7afa1b] /usr/lib64/glusterfs/3.4.1/xlator/mgmt/glusterd.so(__glusterd_handle_status_volume+0x216)[0x7fdeba749ae6] /usr/lib64/glusterfs/3.4.1/xlator/mgmt/glusterd.so(glusterd_big_locked_handler+0x3f)[0x7fdeba74ab4f] /usr/lib64/libglusterfs.so.0(synctask_wrap+0x12)[0x7fdebe1b29c2] /lib64/libc.so.6[0x3e8a843b70] --------- Expected results: successful abort of replace-brick operation Additional info:
pre-release version is ambiguous and about to be removed as a choice. If you believe this is still a bug, please change the status back to NEW and choose the appropriate, applicable version for it.