Description of problem:

I have been experimenting with server replacement procedures in a twin-replicated system say, between server1 and server2. I do the following:
- shutdown on server2.
- On server1, remove-brick of the bricks on server2 for all replicated volumes.  
- On server1, detach peer server2.

After this, if you 
- replace server2 with server3, 
- make bricks on server 3, 
- add them in replica 2 mode to the (now) distributed volumes on server1, 

and then attempt to mount the volumes, you might find that the mount.glusterfs program is stalled. Also, any file-system operation on the mount hangs, and cannot be interrupted (but kill -9 does terminate it). 

Here's an example:

root     21891 21890  0 16:47 ?        00:00:00 /bin/mount -t glusterfs server3:testvol /mnt
root     21892 21891  0 16:47 ?        00:00:00 /bin/sh /sbin/mount.glusterfs server3:testvol /mnt -o rw
root     21931 21892  0 16:47 ?        00:00:00 /bin/sh /sbin/mount.glusterfs server3:testvol /mnt -o rw
root     21932 21931  0 16:47 ?        00:00:00 stat -c %i /mnt

I checked the wchan for 21932 (stat) and found that it is 'fuse_get_req'. The funny thing here is that if I try to set any volume option, the problem immediately corrects itself. For example, I tried to set the following 2 options for debug: server.statedump-path and diagnostics.client-log-level and each time this caused the problem to get resolved. 
  
Version-Release number of selected component (if applicable):
glusterfs 3.4.2 over linux kernel 2.6.34.

How reproducible:
Intermittent.

Steps to Reproduce:
1. Take two servers, server1 and server2. Probe each other. 
2. Create bricks server1:/bricks/testvol and server2:/bricks/testvol
3. gluster volume create testvol replica 2 server1:/bricks/testvol and server2:/bricks/testvol
4. gluster volume start testvol
5. On both servers, mount -t glusterfs server<n>:testvol /mnt
6. Turn off server2.
7. gluster volume remove-brick testvol replica 1 server2:/bricks/testvol
8. gluster peer detach server2.
9. Introduce server3.
10. On server3, gluster peer probe server1.
11. Create server3:/bricks/testvol
12. On server3, gluster volume add-brick testvol replica 2 server3:/bricks/testvol 
13 On server 3, mount -t glusterfs server3:testvol /mnt


Actual results:

At this point, the mount command might hang indefinitely just so:

root     21891 21890  0 16:47 ?        00:00:00 /bin/mount -t glusterfs server3:testvol /mnt
root     21892 21891  0 16:47 ?        00:00:00 /bin/sh /sbin/mount.glusterfs server3:testvol /mnt -o rw
root     21931 21892  0 16:47 ?        00:00:00 /bin/sh /sbin/mount.glusterfs server3:testvol /mnt -o rw
root     21932 21931  0 16:47 ?        00:00:00 stat -c %i /mnt

Any command on the file-system such as ls/stat/df will also hang (and so will the strace of these commands). If you set a volume option on testvol using the 'gluster volume set ...' command, the problem disappears immediately.


Expected results:
Mount should complete successfully.

Additional info:

If you have two volumes like testvol and testvol2, then if the issue is observed on testvol, it will be also observed for testvol2. However, resolving it for testvol using volume set commands will not cause it to automatically go away for testvol2.