Bug 797729

Description of problem:
Suppose in a replace-brick operation, the source brick crashes, the replace-brick abort cannot be given (the command just hangs and then returns without giving any message), and further volume related operations such as volume stop fails saying replace-brick is in progress.

 gluster volume stop test
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
Replace brick is in progress on volume test. Please retry after replace-brick operation is committed or aborted
[root@node130 src]# gluster volume stop mirror
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
Replace brick is in progress on volume mirror. Please retry after replace-brick operation is committed or aborted
[root@node130 src]# 


Version-Release number of selected component (if applicable):


How reproducible:

Whenever the source brick is crashed

Steps to Reproduce:
1. create and start a volume
2. mount the volume and do some i/o on the mount point
3. do replace brick
  
Actual results:

after source brick crash, replace-brick cannot be aborted (even after volume start force) and volume stop commands does not work.

Expected results:

after the source brick crash, replace-brick should be able to be aborted.

Additional info:

glusterd logs.

[2012-02-27 01:58:01.306839] E [socket.c:2121:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-02-27 01:58:04.318727] E [socket.c:2121:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-02-27 01:58:07.332177] E [socket.c:2121:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-02-27 01:58:10.344481] E [socket.c:2121:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-02-27 01:58:13.355091] E [socket.c:2121:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-02-27 01:58:16.365892] E [socket.c:2121:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-02-27 01:58:19.376598] E [socket.c:2121:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-02-27 01:58:22.387863] E [socket.c:2121:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-02-27 01:58:25.399657] E [socket.c:2121:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-02-27 01:58:28.411238] E [socket.c:2121:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-02-27 01:58:31.422155] E [socket.c:2121:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-02-27 01:58:34.434952] E [socket.c:2121:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-02-27 01:58:37.445701] E [socket.c:2121:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-02-27 01:58:40.456460] E [socket.c:2121:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-02-27 01:58:43.467230] E [socket.c:2121:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-02-27 01:58:46.478046] E [socket.c:2121:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-02-27 01:58:49.489197] E [socket.c:2121:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-02-27 01:58:52.500438] E [socket.c:2121:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-02-27 01:58:55.511128] E [socket.c:2121:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-02-27 01:58:58.521764] E [socket.c:2121:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-02-27 01:59:01.532598] E [socket.c:2121:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-02-27 01:59:04.545361] E [socket.c:2121:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-02-27 01:59:05.926555] I [glusterd-volume-ops.c:353:glusterd_handle_cli_stop_volume] 0-glusterd: Received stop vol reqfor volume test
[2012-02-27 01:59:05.926622] I [glusterd-utils.c:267:glusterd_lock] 0-glusterd: Cluster lock held by 771bfee1-3f30-477f-8f7c-3c9b91f70307
[2012-02-27 01:59:05.926668] I [glusterd-handler.c:453:glusterd_op_txn_begin] 0-management: Acquired local lock
[2012-02-27 01:59:05.926750] E [glusterd-volume-ops.c:910:glusterd_op_stage_stop_volume] 0-management: Replace brick is in progress on volume test. Please retry after replace-brick operation is committed or aborted
[2012-02-27 01:59:05.926771] E [glusterd-op-sm.c:1685:glusterd_op_ac_send_stage_op] 0-: Staging failed
[2012-02-27 01:59:05.926786] I [glusterd-op-sm.c:1725:glusterd_op_ac_send_stage_op] 0-glusterd: Sent op req to 0 peers
[2012-02-27 01:59:05.926807] I [glusterd-op-sm.c:2107:glusterd_op_txn_complete] 0-glusterd: Cleared local lock
[2012-02-27 01:59:07.556972] E [socket.c:2121:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-02-27 01:59:10.570813] E [socket.c:2121:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-02-27 01:59:12.572551] I [glusterd-volume-ops.c:353:glusterd_handle_cli_stop_volume] 0-glusterd: Received stop vol reqfor volume mirror
[2012-02-27 01:59:12.572600] I [glusterd-utils.c:267:glusterd_lock] 0-glusterd: Cluster lock held by 771bfee1-3f30-477f-8f7c-3c9b91f70307
[2012-02-27 01:59:12.572612] I [glusterd-handler.c:453:glusterd_op_txn_begin] 0-management: Acquired local lock
[2012-02-27 01:59:12.572659] E [glusterd-volume-ops.c:910:glusterd_op_stage_stop_volume] 0-management: Replace brick is in progress on volume mirror. Please retry after replace-brick operation is committed or aborted
[2012-02-27 01:59:12.572683] E [glusterd-op-sm.c:1685:glusterd_op_ac_send_stage_op] 0-: Staging failed
[2012-02-27 01:59:12.572695] I [glusterd-op-sm.c:1725:glusterd_op_ac_send_stage_op] 0-glusterd: Sent op req to 0 peers
[2012-02-27 01:59:12.572709] I [glusterd-op-sm.c:2107:glusterd_op_txn_complete] 0-glusterd: Cleared local lock
[2012-02-27 01:59:13.581461] E [socket.c:2121:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-02-27 01:59:16.594401] E [socket.c:2121:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-02-27 01:59:19.605427] E [socket.c:2121:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-02-27 01:59:22.617023] E [socket.c:2121:socket_connect] 0-management: connection attempt failed (Connection refused)
(END)