For example if you have 48 bricks :- Remove 5 bricks and add 5 bricks, adding bricks are not sequential and never maintain proper topology. This topology or order of sub-volumes is important in case of distribute. Such a change in topology results in a live i/o to exit. But its a problem as the order of sub-volumes has changed, distribute tries to rehash everything and writes new layout. This case was tested @ customer site during LUN migration. Current solution is to use replace-brick over a stopped volume to do in-place brick replacement.
I am not clear about what the actual problem is. Are we adding a new brick, or are we adding the old ones back (with no change in data but with a change in the brick parameters - like change in hostname, change in path etc) ? Can you please elaborate on the LUN migration setup ? And, elucidate what the actual problem is ? Do you mean that the problem in such a scenario is that the rehashing will cause in creation of lot of link files?
Can you please respond to the update?
(In reply to comment #2) > Can you please respond to the update? Resolving this since, replace-brick fixes the issue.