Created attachment 1018395 [details]
gfs1 and gfs3 debug logging

Initial data:
All nodes have the same:
1) os distribution (proxmox v3.3, kernel 2.6.32-32-pve)
2) disabled selinux
3) default iptables with ALLOW action
4) version of glusterfs
      - ii  glusterfs-client                 3.5.2-1
      - ii  glusterfs-common                 3.5.2-1
      - ii  glusterfs-server                 3.5.2-1
5) correct and working DNS forward and reverse resolution

6) 
192.168.9.53/gfs3 \ 
192.168.9.54/gfs4  - already in cluster
192.168.9.56/gfs6 /

192.168.9.51/gfs1 - want to add to the cluster
192.168.9.52/gfs2 - want to add to the cluster

gfs1 and gfs2 previously were part of another glusterfs cluster, but I stopped all services and removed /var/lib/glusterd directory.

7) cluster contains several production distributed-replicated volumes

Description of problem: 
When I do gluster peer probe gfs1 or gfs2 from any node in the cluster the command hangs and after timeout new peer has status "Probe Sent to Peer":
gfs3#gluster peer status
Number of Peers: 3

Hostname: gfs6
Uuid: 6bd6ee25-e257-4703-b500-330741b90471
State: Peer in Cluster (Connected)

Hostname: gfs4
Uuid: bb1bed20-25bf-43b0-8faa-49f1b5b9ae59
State: Peer in Cluster (Connected)

Hostname: gfs1
Uuid: c5cd8152-c239-474a-977b-9c6b35edd857
State: Probe Sent to Peer (Connected) 

-- gfs3 added peer gfs1 ------------
gfs3#cat /var/lib/glusterd/peers/c5cd8152-c239-474a-977b-9c6b35edd857
uuid=c5cd8152-c239-474a-977b-9c6b35edd857
state=1
hostname1=gfs1

-- but gfs1 did not add gfs3
gfs1#cat /var/lib/glusterd/peers/192.168.9.53
uuid=00000000-0000-0000-0000-000000000000
state=8
hostname1=192.168.9.53


Debug logging showed that gfs3 sent req, gfs1 sent resp, gfs3 received response and after that gfs3 does not do anything until command timeouts (see attach).
So gfs3 does not complete peer handshake with gfs1.