Replace didn't work with qa26. Here is the log 172.24.21.135#gluster volume replace-brick dht-simple 172.24.21.136:/mnt/dht/export1 172.24.21.138:/mnt/dht/export1 start replace-brick failed to start Replacing brick from Volume dht-simple failed. on source brick (136) - no new process is started. on target brick (138) - one process is running with rb_dst_brick.vol. --------- cat usr-local-etc-glusterfs-glusterd.vol.log [2010-09-21 01:02:20.460201] I [glusterd-op-sm.c:3885:glusterd_op_set_cli_op] : Returning 0 [2010-09-21 01:02:20.460309] I [glusterd-handler.c:1565:glusterd_handle_replace_brick] glusterd: Received replace brick req [2010-09-21 01:02:20.460358] I [glusterd-utils.c:221:glusterd_lock] glusterd: Cluster lock held by 6ce01949-12c7-4974-b363-375ba47fbabc [2010-09-21 01:02:20.460371] I [glusterd-handler.c:2776:glusterd_op_txn_begin] glusterd: Acquired local lock [2010-09-21 01:02:20.460385] I [glusterd-op-sm.c:3747:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 1 [2010-09-21 01:02:20.460397] I [glusterd-handler.c:2780:glusterd_op_txn_begin] glusterd: Returning 0 [2010-09-21 01:02:20.460475] I [glusterd3_1-mops.c:979:glusterd3_1_cluster_lock] glusterd: Sent lock req to 2 peers [2010-09-21 01:02:20.460490] I [glusterd-op-sm.c:3452:glusterd_op_sm_transition_state] : Transitioning from 0 to 1 [2010-09-21 01:02:20.460503] W [glusterd3_1-mops.c:1358:glusterd_handle_rpc_msg] : Returning 0 [2010-09-21 01:02:20.460747] I [glusterd3_1-mops.c:402:glusterd3_1_cluster_lock_cbk] glusterd: Received ACC from uuid: f880e25b-cb5e-4806-85db-db10ef448b31 [2010-09-21 01:02:20.460783] I [glusterd-handler.c:114:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3 [2010-09-21 01:02:20.460799] I [glusterd-op-sm.c:3747:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3 [2010-09-21 01:02:20.460818] I [glusterd-op-sm.c:3452:glusterd_op_sm_transition_state] : Transitioning from 1 to 1 [2010-09-21 01:02:20.460855] I [glusterd3_1-mops.c:402:glusterd3_1_cluster_lock_cbk] glusterd: Received ACC from uuid: 1f3d9cbd-4412-4af7-9be0-58777cfe14c2 [2010-09-21 01:02:20.460869] I [glusterd-handler.c:114:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3 [2010-09-21 01:02:20.460881] I [glusterd-op-sm.c:3747:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3 [2010-09-21 01:02:20.460894] I [glusterd-op-sm.c:3747:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 4 [2010-09-21 01:02:20.460905] I [glusterd-op-sm.c:3452:glusterd_op_sm_transition_state] : Transitioning from 1 to 1 [2010-09-21 01:02:20.460938] I [glusterd-utils.c:684:glusterd_brickinfo_get] : brick: 172.24.21.136:/mnt/dht/export1/ [2010-09-21 01:02:20.460953] I [glusterd-utils.c:711:glusterd_brickinfo_get] : Found brick [2010-09-21 01:02:20.461046] I [glusterd3_1-mops.c:1114:glusterd3_1_stage_op] glusterd: Sent op req to 2 peers [2010-09-21 01:02:20.461066] I [glusterd-op-sm.c:3452:glusterd_op_sm_transition_state] : Transitioning from 1 to 3 [2010-09-21 01:02:20.461387] I [glusterd3_1-mops.c:521:glusterd3_1_stage_op_cbk] glusterd: Received ACC from uuid: f880e25b-cb5e-4806-85db-db10ef448b31 [2010-09-21 01:02:20.461404] I [glusterd-handler.c:114:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3 [2010-09-21 01:02:20.461419] I [glusterd-op-sm.c:3747:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3 [2010-09-21 01:02:20.461434] I [glusterd-op-sm.c:3452:glusterd_op_sm_transition_state] : Transitioning from 3 to 3 [2010-09-21 01:02:20.461467] I [glusterd3_1-mops.c:521:glusterd3_1_stage_op_cbk] glusterd: Received ACC from uuid: 1f3d9cbd-4412-4af7-9be0-58777cfe14c2 [2010-09-21 01:02:20.461497] I [glusterd-handler.c:114:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3 [2010-09-21 01:02:20.461510] I [glusterd-op-sm.c:3747:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3 [2010-09-21 01:02:20.461523] I [glusterd-op-sm.c:3747:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 5 [2010-09-21 01:02:20.461535] I [glusterd-op-sm.c:3452:glusterd_op_sm_transition_state] : Transitioning from 3 to 3 [2010-09-21 01:02:20.461559] I [glusterd-utils.c:684:glusterd_brickinfo_get] : brick: 172.24.21.136:/mnt/dht/export1/ [2010-09-21 01:02:20.461573] I [glusterd-utils.c:711:glusterd_brickinfo_get] : Found brick [2010-09-21 01:02:20.461722] I [glusterd3_1-mops.c:1205:glusterd3_1_commit_op] glusterd: Sent op req to 2 peers [2010-09-21 01:02:20.461738] I [glusterd-op-sm.c:3452:glusterd_op_sm_transition_state] : Transitioning from 3 to 5 [2010-09-21 01:02:20.462085] I [glusterd3_1-mops.c:648:glusterd3_1_commit_op_cbk] glusterd: Received RJT from uuid: f880e25b-cb5e-4806-85db-db10ef448b31 [2010-09-21 01:02:20.462115] I [glusterd-handler.c:114:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3 [2010-09-21 01:02:20.462132] I [glusterd-op-sm.c:3747:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 7 [2010-09-21 01:02:20.462147] I [glusterd-op-sm.c:3747:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 11 [2010-09-21 01:02:20.462161] I [glusterd-op-sm.c:3452:glusterd_op_sm_transition_state] : Transitioning from 5 to 5 [2010-09-21 01:02:20.462206] I [glusterd3_1-mops.c:1030:glusterd3_1_cluster_unlock] glusterd: Sent unlock req to 2 peers [2010-09-21 01:02:20.462219] I [glusterd-op-sm.c:3452:glusterd_op_sm_transition_state] : Transitioning from 5 to 7 [2010-09-21 01:02:20.462381] I [glusterd3_1-mops.c:461:glusterd3_1_cluster_unlock_cbk] glusterd: Received ACC from uuid: f880e25b-cb5e-4806-85db-db10ef448b31 [2010-09-21 01:02:20.462398] I [glusterd-handler.c:114:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3 [2010-09-21 01:02:20.462418] I [glusterd-op-sm.c:3747:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3 [2010-09-21 01:02:20.462433] I [glusterd-op-sm.c:3452:glusterd_op_sm_transition_state] : Transitioning from 7 to 7 [2010-09-21 01:02:20.494629] I [glusterd3_1-mops.c:648:glusterd3_1_commit_op_cbk] glusterd: Received RJT from uuid: 1f3d9cbd-4412-4af7-9be0-58777cfe14c2 [2010-09-21 01:02:20.494645] I [glusterd-handler.c:114:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3 [2010-09-21 01:02:20.494662] I [glusterd-op-sm.c:3747:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 7 [2010-09-21 01:02:20.494674] I [glusterd-op-sm.c:3452:glusterd_op_sm_transition_state] : Transitioning from 7 to 7 [2010-09-21 01:02:20.494828] I [glusterd3_1-mops.c:461:glusterd3_1_cluster_unlock_cbk] glusterd: Received ACC from uuid: 1f3d9cbd-4412-4af7-9be0-58777cfe14c2 [2010-09-21 01:02:20.494844] I [glusterd-handler.c:114:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3 [2010-09-21 01:02:20.494865] I [glusterd-op-sm.c:3747:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3 [2010-09-21 01:02:20.494880] I [glusterd-op-sm.c:3747:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 4 [2010-09-21 01:02:20.494893] I [glusterd-op-sm.c:3452:glusterd_op_sm_transition_state] : Transitioning from 7 to 7 [2010-09-21 01:02:20.494910] I [glusterd-op-sm.c:3312:glusterd_op_txn_complete] glusterd: Cleared local lock [2010-09-21 01:02:20.494965] I [glusterd-op-sm.c:3276:glusterd_op_send_cli_response] : Returning 0 [2010-09-21 01:02:20.494981] I [glusterd-op-sm.c:3347:glusterd_op_txn_complete] glusterd: Returning 0 [2010-09-21 01:02:20.494993] I [glusterd-op-sm.c:3452:glusterd_op_sm_transition_state] : Transitioning from 7 to 0
Fixed as part of bg 1235. patch: http://patches.gluster.com/patch/4947
twice it failed. It starts the process the target-brick but commit said- "Replacing brick from Volume nfsdhtbeta failed". third time,it passed !!! #gluster volume replace-brick BETARB 10.192.134.144:/mnt/beta 10.214.231.112:/mnt/beta/ start replace-brick started successfully 10.192.141.187#gluster volume replace-brick BETARB 10.192.134.144:/mnt/beta 10.214.231.112:/mnt/beta/ status 10.192.141.187#gluster volume replace-brick BETARB 10.192.134.144:/mnt/beta 10.214.231.112:/mnt/beta/ commit replace-brick commit successful creating new files on mount pt - distributed to added brick. issues - -it didn't copy existing files from old-brick 10.192.134.144 to new-brick 10.214.231.112. ----