Bug 763388 (GLUSTER-1656)

Summary: replace-brick from non-source machine fails.
Product: [Community] GlusterFS Reporter: Lakshmipathi G <lakshmipathi>
Component: glusterdAssignee: shishir gowda <sgowda>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: high Docs Contact:
Priority: low    
Version: 3.1-alphaCC: gluster-bugs, nsathyan, vijay
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: RTP Mount Type: All
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Lakshmipathi G 2010-09-21 08:19:12 UTC 
Replace didn't work with qa26. Here is the log

172.24.21.135#gluster volume replace-brick dht-simple 172.24.21.136:/mnt/dht/export1 172.24.21.138:/mnt/dht/export1 start
replace-brick failed to start
Replacing brick from Volume dht-simple failed.

on source brick (136) - no new process is started.
on target brick (138) - one process is running with rb_dst_brick.vol.

---------

cat usr-local-etc-glusterfs-glusterd.vol.log
[2010-09-21 01:02:20.460201] I [glusterd-op-sm.c:3885:glusterd_op_set_cli_op] : Returning 0
[2010-09-21 01:02:20.460309] I [glusterd-handler.c:1565:glusterd_handle_replace_brick] glusterd: Received replace brick req
[2010-09-21 01:02:20.460358] I [glusterd-utils.c:221:glusterd_lock] glusterd: Cluster lock held by 6ce01949-12c7-4974-b363-375ba47fbabc
[2010-09-21 01:02:20.460371] I [glusterd-handler.c:2776:glusterd_op_txn_begin] glusterd: Acquired local lock
[2010-09-21 01:02:20.460385] I [glusterd-op-sm.c:3747:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 1
[2010-09-21 01:02:20.460397] I [glusterd-handler.c:2780:glusterd_op_txn_begin] glusterd: Returning 0
[2010-09-21 01:02:20.460475] I [glusterd3_1-mops.c:979:glusterd3_1_cluster_lock] glusterd: Sent lock req to 2 peers
[2010-09-21 01:02:20.460490] I [glusterd-op-sm.c:3452:glusterd_op_sm_transition_state] : Transitioning from 0 to 1
[2010-09-21 01:02:20.460503] W [glusterd3_1-mops.c:1358:glusterd_handle_rpc_msg] : Returning 0
[2010-09-21 01:02:20.460747] I [glusterd3_1-mops.c:402:glusterd3_1_cluster_lock_cbk] glusterd: Received ACC from uuid: f880e25b-cb5e-4806-85db-db10ef448b31
[2010-09-21 01:02:20.460783] I [glusterd-handler.c:114:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-09-21 01:02:20.460799] I [glusterd-op-sm.c:3747:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-09-21 01:02:20.460818] I [glusterd-op-sm.c:3452:glusterd_op_sm_transition_state] : Transitioning from 1 to 1
[2010-09-21 01:02:20.460855] I [glusterd3_1-mops.c:402:glusterd3_1_cluster_lock_cbk] glusterd: Received ACC from uuid: 1f3d9cbd-4412-4af7-9be0-58777cfe14c2
[2010-09-21 01:02:20.460869] I [glusterd-handler.c:114:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-09-21 01:02:20.460881] I [glusterd-op-sm.c:3747:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-09-21 01:02:20.460894] I [glusterd-op-sm.c:3747:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 4
[2010-09-21 01:02:20.460905] I [glusterd-op-sm.c:3452:glusterd_op_sm_transition_state] : Transitioning from 1 to 1
[2010-09-21 01:02:20.460938] I [glusterd-utils.c:684:glusterd_brickinfo_get] : brick: 172.24.21.136:/mnt/dht/export1/
[2010-09-21 01:02:20.460953] I [glusterd-utils.c:711:glusterd_brickinfo_get] : Found brick
[2010-09-21 01:02:20.461046] I [glusterd3_1-mops.c:1114:glusterd3_1_stage_op] glusterd: Sent op req to 2 peers
[2010-09-21 01:02:20.461066] I [glusterd-op-sm.c:3452:glusterd_op_sm_transition_state] : Transitioning from 1 to 3
[2010-09-21 01:02:20.461387] I [glusterd3_1-mops.c:521:glusterd3_1_stage_op_cbk] glusterd: Received ACC from uuid: f880e25b-cb5e-4806-85db-db10ef448b31
[2010-09-21 01:02:20.461404] I [glusterd-handler.c:114:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-09-21 01:02:20.461419] I [glusterd-op-sm.c:3747:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-09-21 01:02:20.461434] I [glusterd-op-sm.c:3452:glusterd_op_sm_transition_state] : Transitioning from 3 to 3
[2010-09-21 01:02:20.461467] I [glusterd3_1-mops.c:521:glusterd3_1_stage_op_cbk] glusterd: Received ACC from uuid: 1f3d9cbd-4412-4af7-9be0-58777cfe14c2
[2010-09-21 01:02:20.461497] I [glusterd-handler.c:114:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-09-21 01:02:20.461510] I [glusterd-op-sm.c:3747:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-09-21 01:02:20.461523] I [glusterd-op-sm.c:3747:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 5
[2010-09-21 01:02:20.461535] I [glusterd-op-sm.c:3452:glusterd_op_sm_transition_state] : Transitioning from 3 to 3
[2010-09-21 01:02:20.461559] I [glusterd-utils.c:684:glusterd_brickinfo_get] : brick: 172.24.21.136:/mnt/dht/export1/
[2010-09-21 01:02:20.461573] I [glusterd-utils.c:711:glusterd_brickinfo_get] : Found brick
[2010-09-21 01:02:20.461722] I [glusterd3_1-mops.c:1205:glusterd3_1_commit_op] glusterd: Sent op req to 2 peers
[2010-09-21 01:02:20.461738] I [glusterd-op-sm.c:3452:glusterd_op_sm_transition_state] : Transitioning from 3 to 5
[2010-09-21 01:02:20.462085] I [glusterd3_1-mops.c:648:glusterd3_1_commit_op_cbk] glusterd: Received RJT from uuid: f880e25b-cb5e-4806-85db-db10ef448b31
[2010-09-21 01:02:20.462115] I [glusterd-handler.c:114:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-09-21 01:02:20.462132] I [glusterd-op-sm.c:3747:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 7
[2010-09-21 01:02:20.462147] I [glusterd-op-sm.c:3747:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 11
[2010-09-21 01:02:20.462161] I [glusterd-op-sm.c:3452:glusterd_op_sm_transition_state] : Transitioning from 5 to 5
[2010-09-21 01:02:20.462206] I [glusterd3_1-mops.c:1030:glusterd3_1_cluster_unlock] glusterd: Sent unlock req to 2 peers
[2010-09-21 01:02:20.462219] I [glusterd-op-sm.c:3452:glusterd_op_sm_transition_state] : Transitioning from 5 to 7
[2010-09-21 01:02:20.462381] I [glusterd3_1-mops.c:461:glusterd3_1_cluster_unlock_cbk] glusterd: Received ACC from uuid: f880e25b-cb5e-4806-85db-db10ef448b31
[2010-09-21 01:02:20.462398] I [glusterd-handler.c:114:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-09-21 01:02:20.462418] I [glusterd-op-sm.c:3747:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-09-21 01:02:20.462433] I [glusterd-op-sm.c:3452:glusterd_op_sm_transition_state] : Transitioning from 7 to 7
[2010-09-21 01:02:20.494629] I [glusterd3_1-mops.c:648:glusterd3_1_commit_op_cbk] glusterd: Received RJT from uuid: 1f3d9cbd-4412-4af7-9be0-58777cfe14c2
[2010-09-21 01:02:20.494645] I [glusterd-handler.c:114:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-09-21 01:02:20.494662] I [glusterd-op-sm.c:3747:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 7
[2010-09-21 01:02:20.494674] I [glusterd-op-sm.c:3452:glusterd_op_sm_transition_state] : Transitioning from 7 to 7
[2010-09-21 01:02:20.494828] I [glusterd3_1-mops.c:461:glusterd3_1_cluster_unlock_cbk] glusterd: Received ACC from uuid: 1f3d9cbd-4412-4af7-9be0-58777cfe14c2
[2010-09-21 01:02:20.494844] I [glusterd-handler.c:114:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-09-21 01:02:20.494865] I [glusterd-op-sm.c:3747:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-09-21 01:02:20.494880] I [glusterd-op-sm.c:3747:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 4
[2010-09-21 01:02:20.494893] I [glusterd-op-sm.c:3452:glusterd_op_sm_transition_state] : Transitioning from 7 to 7
[2010-09-21 01:02:20.494910] I [glusterd-op-sm.c:3312:glusterd_op_txn_complete] glusterd: Cleared local lock
[2010-09-21 01:02:20.494965] I [glusterd-op-sm.c:3276:glusterd_op_send_cli_response] : Returning 0
[2010-09-21 01:02:20.494981] I [glusterd-op-sm.c:3347:glusterd_op_txn_complete] glusterd: Returning 0
[2010-09-21 01:02:20.494993] I [glusterd-op-sm.c:3452:glusterd_op_sm_transition_state] : Transitioning from 7 to 0
Comment 1 shishir gowda 2010-09-24 02:11:42 UTC 
Fixed as part of bg 1235.

patch: http://patches.gluster.com/patch/4947
Comment 2 Lakshmipathi G 2010-09-24 03:36:34 UTC 
twice it failed. It starts the process the target-brick but commit said-
"Replacing brick from Volume nfsdhtbeta failed".

third time,it passed !!!

#gluster volume replace-brick BETARB  10.192.134.144:/mnt/beta 10.214.231.112:/mnt/beta/ start
replace-brick started successfully
10.192.141.187#gluster volume replace-brick BETARB  10.192.134.144:/mnt/beta 10.214.231.112:/mnt/beta/ status

10.192.141.187#gluster volume replace-brick BETARB  10.192.134.144:/mnt/beta 10.214.231.112:/mnt/beta/ commit
replace-brick commit successful

creating new files on mount pt - distributed to added brick.

issues - 
-it didn't copy  existing files from old-brick 10.192.134.144 to  new-brick 10.214.231.112.
----