Description of problem:
Replicate volume with replica count 3. 1 fuse and 1 nfs client. fuse client running sanity script and nfs client running rdd. Gave replace-brick.
replace brick operation start said it started. status said that source brick is not online.

Source brick did not restart with pump loaded (log file said address already in use). So gave volume start force. And then gave replace-brick abort, during which the command hung. 


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.
  
Actual results:

Replace-brick commands are failing (i.e. gluster cli is hanging till timeout and returning the prompt without any message).

Expected results:
Replace-brick operations given by cli should succeed.

Additional info:

Log file information:

GLUSTERD LOGS:

012-02-03 03:44:39.273765] I [glusterd-op-sm.c:2085:glusterd_op_txn_complete] 0-glusterd: Cleared local lock
[2012-02-03 03:44:49.250147] I [glusterd-replace-brick.c:98:glusterd_handle_replace_brick] 0-glusterd: Received replace brick req
[2012-02-03 03:44:49.250197] I [glusterd-replace-brick.c:147:glusterd_handle_replace_brick] 0-glusterd: Received replace brick abort request
[2012-02-03 03:44:49.256549] I [glusterd-utils.c:262:glusterd_lock] 0-glusterd: Cluster lock held by 64565e7a-656f-494e-9fbc-8fc298b4d34d
[2012-02-03 03:44:49.256566] I [glusterd-handler.c:448:glusterd_op_txn_begin] 0-management: Acquired local lock
[2012-02-03 03:44:49.257018] I [glusterd-rpc-ops.c:534:glusterd3_1_cluster_lock_cbk] 0-glusterd: Received ACC from uuid: 85b4f943-e2fa-4ca5-899a-78248455727c
[2012-02-03 03:44:49.257075] I [glusterd-rpc-ops.c:534:glusterd3_1_cluster_lock_cbk] 0-glusterd: Received ACC from uuid: 8045dfcd-f3c7-4979-aadd-ef6a907bf3b7
[2012-02-03 03:44:49.257108] I [glusterd-rpc-ops.c:534:glusterd3_1_cluster_lock_cbk] 0-glusterd: Received ACC from uuid: 20479dd8-3dd4-4766-a949-aa1c8b5a89aa
[2012-02-03 03:44:49.264321] I [glusterd-utils.c:781:glusterd_volume_brickinfo_get_by_brick] 0-: brick: 10.1.11.130:/export-xfs/mirror
[2012-02-03 03:44:49.264514] I [glusterd-utils.c:218:glusterd_is_local_addr] 0-glusterd: 10.1.11.130 is local
[2012-02-03 03:44:49.264547] I [glusterd-utils.c:741:glusterd_volume_brickinfo_get] 0-management: Found brick
[2012-02-03 03:44:49.264592] I [glusterd-utils.c:218:glusterd_is_local_addr] 0-glusterd: 10.1.11.130 is local
[2012-02-03 03:44:49.264766] I [glusterd-op-sm.c:1703:glusterd_op_ac_send_stage_op] 0-glusterd: Sent op req to 3 peers
[2012-02-03 03:44:49.266664] I [glusterd-rpc-ops.c:863:glusterd3_1_stage_op_cbk] 0-glusterd: Received ACC from uuid: 20479dd8-3dd4-4766-a949-aa1c8b5a89aa
[2012-02-03 03:44:49.337955] I [glusterd-rpc-ops.c:863:glusterd3_1_stage_op_cbk] 0-glusterd: Received ACC from uuid: 8045dfcd-f3c7-4979-aadd-ef6a907bf3b7
[2012-02-03 03:44:50.001625] I [glusterd-rpc-ops.c:863:glusterd3_1_stage_op_cbk] 0-glusterd: Received ACC from uuid: 85b4f943-e2fa-4ca5-899a-78248455727c
[2012-02-03 03:44:50.001720] I [glusterd-utils.c:781:glusterd_volume_brickinfo_get_by_brick] 0-: brick: 10.1.11.130:/export-xfs/mirror
[2012-02-03 03:44:50.001898] I [glusterd-utils.c:218:glusterd_is_local_addr] 0-glusterd: 10.1.11.130 is local
[2012-02-03 03:44:50.001921] I [glusterd-utils.c:741:glusterd_volume_brickinfo_get] 0-management: Found brick
[2012-02-03 03:44:50.001955] I [glusterd-utils.c:218:glusterd_is_local_addr] 0-glusterd: 10.1.11.130 is local
[2012-02-03 03:44:50.001970] I [glusterd-replace-brick.c:1243:rb_update_srcbrick_port] 0-: adding src-brick port no
(END) 

SOURCE BRICK LOGS:

[2012-02-03 02:34:07.965108] I [server-handshake.c:540:server_setvolume] 0-mirror-server: accepted client from 10.1.11.104:1021 (version: 3.3.
0qa21)
[2012-02-03 02:34:07.969485] I [server-handshake.c:540:server_setvolume] 0-mirror-server: accepted client from 10.1.11.104:1020 (version: 3.3.
0qa21)
[2012-02-03 02:34:09.085444] W [dict.c:1220:data_to_str] (-->/usr/local/lib/glusterfs/3.3.0qa21/rpc-transport/socket.so(socket_connect+0x261) 
[0x7fde82b2a38c] (-->/usr/local/lib/glusterfs/3.3.0qa21/rpc-transport/socket.so(socket_client_get_remote_sockaddr+0x16a) [0x7fde82b2dcb7] (-->
/usr/local/lib/glusterfs/3.3.0qa21/rpc-transport/socket.so(client_fill_address_family+0x135) [0x7fde82b2ccc8]))) 0-dict: data is NULL
[2012-02-03 02:34:09.085487] W [dict.c:1220:data_to_str] (-->/usr/local/lib/glusterfs/3.3.0qa21/rpc-transport/socket.so(socket_connect+0x261) 
[0x7fde82b2a38c] (-->/usr/local/lib/glusterfs/3.3.0qa21/rpc-transport/socket.so(socket_client_get_remote_sockaddr+0x16a) [0x7fde82b2dcb7] (-->
/usr/local/lib/glusterfs/3.3.0qa21/rpc-transport/socket.so(client_fill_address_family+0x144) [0x7fde82b2ccd7]))) 0-dict: data is NULL
[2012-02-03 02:34:09.085500] E [name.c:151:client_fill_address_family] 0-mirror-replace-brick: transport.address-family not specified and not 
able to determine the same from other options (remote-host:(null) and transport.unix.connect-path:(null))
[2012-02-03 02:34:09.303922] I [server3_1-fops.c:345:server_entrylk_cbk] 0-mirror-server: 362555: ENTRYLK (null) (--) ==> -1 (No such file or 
directory)
[2012-02-03 02:34:09.304856] I [server3_1-fops.c:345:server_entrylk_cbk] 0-mirror-server: 362556: ENTRYLK (null) (--) ==> -1 (No such file or 
directory)
[2012-02-03 02:34:09.305446] I [server3_1-fops.c:345:server_entrylk_cbk] 0-mirror-server: 362557: ENTRYLK (null) (--) ==> -1 (No such file or 
directory)
[2012-02-03 02:34:09.310385] I [server3_1-fops.c:262:server_inodelk_cbk] 0-mirror-server: 362558: INODELK (null) (--) ==> -1 (No such file or 
directory)
[2012-02-03 02:34:09.310976] I [server3_1-fops.c:262:server_inodelk_cbk] 0-mirror-server: 362559: INODELK (null) (--) ==> -1 (No such file or 
directory)
[2012-02-03 02:34:09.320655] I [server3_1-fops.c:262:server_inodelk_cbk] 0-mirror-server: 362561: INODELK (null) (--) ==> -1 (No such file or 
directory)
[2012-02-03 02:34:09.321510] I [server3_1-fops.c:262:server_inodelk_cbk] 0-mirror-server: 362562: INODELK (null) (--) ==> -1 (No such file or 
directory)
[2012-02-03 02:34:09.327338] I [server3_1-fops.c:345:server_entrylk_cbk] 0-mirror-server: 362564: ENTRYLK (null) (--) ==> -1 (No such file or 
directory)
[2012-02-03 02:34:09.328149] I [server3_1-fops.c:345:server_entrylk_cbk] 0-mirror-server: 362565: ENTRYLK (null) (--) ==> -1 (No such file or 
directory)
[2012-02-03 02:34:09.474724] I [server-handshake.c:540:server_setvolume] 0-mirror-server: accepted client from 10.1.11.130:998 (version: 3.3.0
qa21)
[2012-02-03 02:34:09.730825] I [server3_1-fops.c:345:server_entrylk_cbk] 0-mirror-server: 362566: ENTRYLK (null) (--) ==> -1 (No such file or 
directory)
[2012-02-03 02:34:09.731448] I [server3_1-fops.c:345:server_entrylk_cbk] 0-mirror-server: 362567: ENTRYLK (null) (--) ==> -1 (No such file or 
directory)
[2012-02-03 02:34:09.737098] I [server3_1-fops.c:345:server_entrylk_cbk] 0-mirror-server: 362568: ENTRYLK (null) (--) ==> -1 (No such file or 
directory)
[2012-02-03 02:34:09.737644] I [server3_1-fops.c:345:server_entrylk_cbk] 0-mirror-server: 362569: ENTRYLK (null) (--) ==> -1 (No such file or 
directory)


[2012-02-03 02:34:12.085812] W [dict.c:1220:data_to_str] (-->/usr/local/lib/glusterfs/3.3.0qa21/rpc-transport/socket.so(socket_connect+0x261) 
[0x7fde82b2a38c] (-->/usr/local/lib/glusterfs/3.3.0qa21/rpc-transport/socket.so(socket_client_get_remote_sockaddr+0x16a) [0x7fde82b2dcb7] (-->
/usr/local/lib/glusterfs/3.3.0qa21/rpc-transport/socket.so(client_fill_address_family+0x135) [0x7fde82b2ccc8]))) 0-dict: data is NULL
[2012-02-03 02:34:12.085860] W [dict.c:1220:data_to_str] (-->/usr/local/lib/glusterfs/3.3.0qa21/rpc-transport/socket.so(socket_connect+0x261) [0x7fde82b2a38c] (-->/usr/local/lib/glusterfs/3.3.0qa21/rpc-transport/socket.so(socket_client_get_remote_sockaddr+0x16a) [0x7fde82b2dcb7] (-->/usr/local/lib/glusterfs/3.3.0qa21/rpc-transport/socket.so(client_fill_address_family+0x144) [0x7fde82b2ccd7]))) 0-dict: data is NULL
[2012-02-03 02:34:12.085880] E [name.c:151:client_fill_address_family] 0-mirror-replace-brick: transport.address-family not specified and not able to determine the same from other options (remote-host:(null) and transport.unix.connect-path:(null))
[2012-02-03 02:34:12.147971] I [server-handshake.c:540:server_setvolume] 0-mirror-server: accepted client from 10.1.11.130:988 (version: 3.3.0qa21)
[2012-02-03 02:34:15.086311] W [dict.c:1220:data_to_str] (-->/usr/local/lib/glusterfs/3.3.0qa21/rpc-transport/socket.so(socket_connect+0x261) [0x7fde82b2a38c] (-->/usr/local/lib/glusterfs/3.3.0qa21/rpc-transport/socket.so(socket_client_get_remote_sockaddr+0x16a) [0x7fde82b2dcb7] (-->/usr/local/lib/glusterfs/3.3.0qa21/rpc-transport/socket.so(client_fill_address_family+0x135) [0x7fde82b2ccc8]))) 0-dict: data is NULL
[2012-02-03 02:34:15.086366] W [dict.c:1220:data_to_str] (-->/usr/local/lib/glusterfs/3.3.0qa21/rpc-transport/socket.so(socket_connect+0x261) [0x7fde82b2a38c] (-->/usr/local/lib/glusterfs/3.3.0qa21/rpc-transport/socket.so(socket_client_get_remote_sockaddr+0x16a) [0x7fde82b2dcb7] (-->/usr/local/lib/glusterfs/3.3.0qa21/rpc-transport/socket.so(client_fill_address_family+0x144) [0x7fde82b2ccd7]))) 0-dict: data is NULL
[2012-02-03 02:34:15.086389] E [name.c:151:client_fill_address_family] 0-mirror-replace-brick: transport.address-family not specified and not able to determine the same from other options (remote-host:(null) and transport.unix.connect-path:(null))
[2012-02-03 02:34:16.847291] I [server-handshake.c:540:server_setvolume] 0-mirror-server: accepted client from 10.1.11.130:996 (version: 3.3.0qa21)
[2012-02-03 02:34:16.866889] I [server.c:556:server_rpc_notify] 0-mirror-server: disconnected connection from 10.1.11.130:996
[2012-02-03 02:34:16.866923] I [server-helpers.c:763:server_connection_destroy] 0-mirror-server: destroyed connection of node130-1643-2012/02/03-02:34:16:791421-mnt-client
[2012-02-03 02:34:18.086703] W [dict.c:1220:data_to_str] (-->/usr/local/lib/glusterfs/3.3.0qa21/rpc-transport/socket.so(socket_connect+0x261) [0x7fde82b2a38c] (-->/usr/local/lib/glusterfs/3.3.0qa21/rpc-transport/socket.so(socket_client_get_remote_sockaddr+0x16a) [0x7fde82b2dcb7] (-->/usr/local/lib/glusterfs/3.3.0qa21/rpc-transport/socket.so(client_fill_address_family+0x135) [0x7fde82b2ccc8]))) 0-dict: data is NULL
[2012-02-03 02:34:18.086750] W [dict.c:1220:data_to_str] (-->/usr/local/lib/glusterfs/3.3.0qa21/rpc-transport/socket.so(socket_connect+0x261) [0x7fde82b2a38c] (-->/usr/local/lib/glusterfs/3.3.0qa21/rpc-transport/socket.so(socket_client_get_remote_sockaddr+0x16a) [0x7fde82b2dcb7] (-->/usr/local/lib/glusterfs/3.3.0qa21/rpc-transport/socket.so(client_fill_address_family+0x144) [0x7fde82b2ccd7]))) 0-dict: data is NULL
[2012-02-03 02:34:18.086763] E [name.c:151:client_fill_address_family] 0-mirror-replace-brick: transport.address-family not specified and not able to determine the same from other options (remote-host:(null) and transport.unix.connect-path:(null))
[2012-02-03 02:34:19.684061] I [server-handshake.c:540:server_setvolume] 0-mirror-server: accepted client from 10.1.11.130:987 (version: 3.3.0qa21)
[2012-02-03 02:34:19.695127] I [server.c:556:server_rpc_notify] 0-mirror-server: disconnected connection from 10.1.11.130:987
[2012-02-03 02:34:19.695168] I [server-helpers.c:763:server_connection_destroy] 0-mirror-server: destroyed connection of node130-1658-2012/02/03-02:34:19:637790-mnt-client
[2012-02-03 02:34:21.087067] W [dict.c:1220:data_to_str] (-->/usr/local/lib/glusterfs/3.3.0qa21/rpc-transport/socket.so(socket_connect+0x261) [0x7fde82b2a38c] (-->/usr/local/lib/glusterfs/3.3.0qa21/rpc-transport/socket.so(socket_client_get_remote_sockaddr+0x16a) [0x7fde82b2dcb7] (-->/usr/local/lib/glusterfs/3.3.0qa21/rpc-transport/socket.so(client_fill_address_family+0x135) [0x7fde82b2ccc8]))) 0-dict: data is NULL
[2012-02-03 02:34:21.087115] W [dict.c:1220:data_to_str] (-->/usr/local/lib/glusterfs/3.3.0qa21/rpc-transport/socket.so(socket_connect+0x261) [0x7fde82b2a38c] (-->/usr/local/lib/glusterfs/3.3.0qa21/rpc-transport/socket.so(socket_client_get_remote_sockaddr+0x16a) [0x7fde82b2dcb7] (-->/usr/local/lib/glusterfs/3.3.0qa21/rpc-transport/socket.so(client_fill_address_family+0x144) [0x7fde82b2ccd7]))) 0-dict: data is NULL