1404128 – Volume creation fails with error "host is not in 'Peer in Cluster' state"

Bug 1404128 - Volume creation fails with error "host is not in 'Peer in Cluster' state"

Summary: Volume creation fails with error "host is not in 'Peer in Cluster' state"

Keywords:
Status:	CLOSED NOTABUG
Alias:	None
Product:	GlusterFS
Classification:	Community
Component:	glusterd
Sub Component:
Version:	3.8
Hardware:	x86_64
OS:	Linux
Priority:	unspecified
Severity:	high
Target Milestone:	---
Assignee:	Raghavendra Talur
QA Contact:
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2016-12-13 07:06 UTC by Ankit Rana
Modified:	2016-12-20 09:13 UTC (History)
CC List:	2 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2016-12-20 09:13:50 UTC
Regression:	---
Mount Type:	---
Documentation:	---
CRM:
Verified Versions:
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Description Ankit Rana 2016-12-13 07:06:01 UTC

Description of problem:
While setting up a two node cluster,I am unable to create new volume.

Version-Release number of selected component (if applicable):
[root@localhost ~]# yum list | grep gluster 
centos-release-gluster38.noarch         1.0-1.el7.centos               @extras  
glusterfs.x86_64                        3.8.5-1.el7                    @centos-gluster38
glusterfs-api.x86_64                    3.8.5-1.el7                    @centos-gluster38
glusterfs-cli.x86_64                    3.8.5-1.el7                    @centos-gluster38
glusterfs-client-xlators.x86_64         3.8.5-1.el7                    @centos-gluster38
glusterfs-fuse.x86_64                   3.8.5-1.el7                    @centos-gluster38
glusterfs-libs.x86_64                   3.8.5-1.el7                    @centos-gluster38
glusterfs-server.x86_64                 3.8.5-1.el7                    @centos-gluster38
userspace-rcu.x86_64                    0.7.16-1.el7                   @centos-gluster38


How reproducible:
I tried steps from https://wiki.centos.org/HowTos/GlusterFSonCentOS on 2 nodes over vmware


Actual results:
[arana@localhost ~]$ sudo gluster peer status
Number of Peers: 1

Hostname: gnode-2
Uuid: 59da0310-9fc3-4aac-ac5f-74b1726d39d0
State: Peer in Cluster (Connected)

[arana@localhost ~]$ sudo gluster volume create vol1 replica 2 transport tcp gnode-1:/bricks/brick1/brick gnode-2:/bricks/brick1/brick
volume create: vol1: failed: Staging failed on gnode-2. Error: Host gnode-2 is not in ' Peer in Cluster’ state

[arana@localhost ~]$ sudo gluster volume create vol1 replica 2 transport tcp gnode-1:/bricks/brick1/brick gnode-2:/bricks/brick1/brick
volume create: vol1: failed: /bricks/brick1/brick is already part of a volume

Expected results:
[arana@localhost ~]$ sudo gluster peer status
Number of Peers: 1

Hostname: gnode-2
Uuid: 59da0310-9fc3-4aac-ac5f-74b1726d39d0
State: Peer in Cluster (Connected)

[arana@localhost ~]$ sudo gluster volume create vol1 replica 2 transport tcp gnode-1:/bricks/brick1/brick gnode-2:/bricks/brick1/brick
volume create: vol1: success: please start the volume to access data

Additional info:
there are two nodes in cluster gnode-1 and gnode-2

gnode-1
[root@localhost ~]# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere            
INPUT_direct  all  --  anywhere             anywhere            
INPUT_ZONES_SOURCE  all  --  anywhere             anywhere            
INPUT_ZONES  all  --  anywhere             anywhere            
ACCEPT     icmp --  anywhere             anywhere            
REJECT     all  --  anywhere             anywhere             reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere            
FORWARD_direct  all  --  anywhere             anywhere            
FORWARD_IN_ZONES_SOURCE  all  --  anywhere             anywhere            
FORWARD_IN_ZONES  all  --  anywhere             anywhere            
FORWARD_OUT_ZONES_SOURCE  all  --  anywhere             anywhere            
FORWARD_OUT_ZONES  all  --  anywhere             anywhere            
ACCEPT     icmp --  anywhere             anywhere            
REJECT     all  --  anywhere             anywhere             reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
OUTPUT_direct  all  --  anywhere             anywhere            

Chain FORWARD_IN_ZONES (1 references)
target     prot opt source               destination         
FWDI_public  all  --  anywhere             anywhere            [goto] 
FWDI_public  all  --  anywhere             anywhere            [goto] 

Chain FORWARD_IN_ZONES_SOURCE (1 references)
target     prot opt source               destination         

Chain FORWARD_OUT_ZONES (1 references)
target     prot opt source               destination         
FWDO_public  all  --  anywhere             anywhere            [goto] 
FWDO_public  all  --  anywhere             anywhere            [goto] 

Chain FORWARD_OUT_ZONES_SOURCE (1 references)
target     prot opt source               destination         

Chain FORWARD_direct (1 references)
target     prot opt source               destination         

Chain FWDI_public (2 references)
target     prot opt source               destination         
FWDI_public_log  all  --  anywhere             anywhere            
FWDI_public_deny  all  --  anywhere             anywhere            
FWDI_public_allow  all  --  anywhere             anywhere            

Chain FWDI_public_allow (1 references)
target     prot opt source               destination         

Chain FWDI_public_deny (1 references)
target     prot opt source               destination         

Chain FWDI_public_log (1 references)
target     prot opt source               destination         

Chain FWDO_public (2 references)
target     prot opt source               destination         
FWDO_public_log  all  --  anywhere             anywhere            
FWDO_public_deny  all  --  anywhere             anywhere            
FWDO_public_allow  all  --  anywhere             anywhere            

Chain FWDO_public_allow (1 references)
target     prot opt source               destination         

Chain FWDO_public_deny (1 references)
target     prot opt source               destination         

Chain FWDO_public_log (1 references)
target     prot opt source               destination         

Chain INPUT_ZONES (1 references)
target     prot opt source               destination         
IN_public  all  --  anywhere             anywhere            [goto] 
IN_public  all  --  anywhere             anywhere            [goto] 

Chain INPUT_ZONES_SOURCE (1 references)
target     prot opt source               destination         

Chain INPUT_direct (1 references)
target     prot opt source               destination         

Chain IN_public (2 references)
target     prot opt source               destination         
IN_public_log  all  --  anywhere             anywhere            
IN_public_deny  all  --  anywhere             anywhere            
IN_public_allow  all  --  anywhere             anywhere            

Chain IN_public_allow (1 references)
target     prot opt source               destination         
ACCEPT     udp  --  anywhere             anywhere             udp dpt:netbios-ns ctstate NEW
ACCEPT     udp  --  anywhere             anywhere             udp dpt:netbios-dgm ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:netbios-ssn ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:microsoft-ds ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:nfs ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ssh ctstate NEW
ACCEPT     udp  --  anywhere             anywhere             udp dpt:netbios-ns ctstate NEW
ACCEPT     udp  --  anywhere             anywhere             udp dpt:netbios-dgm ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpts:24007:24008 ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:netbios-ssn ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpts:38465:38469 ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ipp ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:965 ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:24009 ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:microsoft-ds ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:nfs ctstate NEW
ACCEPT     udp  --  anywhere             anywhere             udp dpt:sunrpc ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:sunrpc ctstate NEW
ACCEPT     udp  --  anywhere             anywhere             udp dpt:963 ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpts:49152:49251 ctstate NEW

Chain IN_public_deny (1 references)
target     prot opt source               destination         

Chain IN_public_log (1 references)
target     prot opt source               destination         

Chain OUTPUT_direct (1 references)
target     prot opt source               destination 

[root@localhost ~]# ls -la /bricks/brick1/bricktotal 0
drwxr-xr-x 2 root root  6 Dec 13 00:37 .
drwxr-xr-x 4 root root 35 Dec 13 01:18 ..


gnode-2
[root@localhost ~]# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere            
INPUT_direct  all  --  anywhere             anywhere            
INPUT_ZONES_SOURCE  all  --  anywhere             anywhere            
INPUT_ZONES  all  --  anywhere             anywhere            
ACCEPT     icmp --  anywhere             anywhere            
REJECT     all  --  anywhere             anywhere             reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere            
FORWARD_direct  all  --  anywhere             anywhere            
FORWARD_IN_ZONES_SOURCE  all  --  anywhere             anywhere            
FORWARD_IN_ZONES  all  --  anywhere             anywhere            
FORWARD_OUT_ZONES_SOURCE  all  --  anywhere             anywhere            
FORWARD_OUT_ZONES  all  --  anywhere             anywhere            
ACCEPT     icmp --  anywhere             anywhere            
REJECT     all  --  anywhere             anywhere             reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
OUTPUT_direct  all  --  anywhere             anywhere            

Chain FORWARD_IN_ZONES (1 references)
target     prot opt source               destination         
FWDI_public  all  --  anywhere             anywhere            [goto] 
FWDI_public  all  --  anywhere             anywhere            [goto] 

Chain FORWARD_IN_ZONES_SOURCE (1 references)
target     prot opt source               destination         

Chain FORWARD_OUT_ZONES (1 references)
target     prot opt source               destination         
FWDO_public  all  --  anywhere             anywhere            [goto] 
FWDO_public  all  --  anywhere             anywhere            [goto] 

Chain FORWARD_OUT_ZONES_SOURCE (1 references)
target     prot opt source               destination         

Chain FORWARD_direct (1 references)
target     prot opt source               destination         

Chain FWDI_public (2 references)
target     prot opt source               destination         
FWDI_public_log  all  --  anywhere             anywhere            
FWDI_public_deny  all  --  anywhere             anywhere            
FWDI_public_allow  all  --  anywhere             anywhere            

Chain FWDI_public_allow (1 references)
target     prot opt source               destination         

Chain FWDI_public_deny (1 references)
target     prot opt source               destination         

Chain FWDI_public_log (1 references)
target     prot opt source               destination         

Chain FWDO_public (2 references)
target     prot opt source               destination         
FWDO_public_log  all  --  anywhere             anywhere            
FWDO_public_deny  all  --  anywhere             anywhere            
FWDO_public_allow  all  --  anywhere             anywhere            

Chain FWDO_public_allow (1 references)
target     prot opt source               destination         

Chain FWDO_public_deny (1 references)
target     prot opt source               destination         

Chain FWDO_public_log (1 references)
target     prot opt source               destination         

Chain INPUT_ZONES (1 references)
target     prot opt source               destination         
IN_public  all  --  anywhere             anywhere            [goto] 
IN_public  all  --  anywhere             anywhere            [goto] 

Chain INPUT_ZONES_SOURCE (1 references)
target     prot opt source               destination         

Chain INPUT_direct (1 references)
target     prot opt source               destination         

Chain IN_public (2 references)
target     prot opt source               destination         
IN_public_log  all  --  anywhere             anywhere            
IN_public_deny  all  --  anywhere             anywhere            
IN_public_allow  all  --  anywhere             anywhere            

Chain IN_public_allow (1 references)
target     prot opt source               destination         
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ssh ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpts:24007:24008 ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:24009 ctstate NEW

Chain IN_public_deny (1 references)
target     prot opt source               destination         

Chain IN_public_log (1 references)
target     prot opt source               destination         

Chain OUTPUT_direct (1 references)
target     prot opt source               destination 
  
[root@localhost ~]#  ls -la /bricks/brick1/brick
total 0
drwxr-xr-x 2 root root  6 Dec 13 00:38 .
drwxr-xr-x 3 root root 18 Dec 13 01:18 ..

Comment 1 Ankit Rana 2016-12-14 10:07:51 UTC

Updating /etc/hosts on both nodes solved my problem:
gnode-1:
[root@localhost ~]# cat  /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4 gnode-1
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.213.156 gnode-2

gnode-2:
[arana@localhost ~]$ cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4 gnode-2
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.213.155 gnode-1

Comment 2 SATHEESARAN 2016-12-20 09:13:50 UTC

(In reply to Ankit Rana from comment #1)
> Updating /etc/hosts on both nodes solved my problem:
> gnode-1:
> [root@localhost ~]# cat  /etc/hosts
> 127.0.0.1   localhost localhost.localdomain localhost4
> localhost4.localdomain4 gnode-1
> ::1         localhost localhost.localdomain localhost6
> localhost6.localdomain6
> 192.168.213.156 gnode-2
> 
> gnode-2:
> [arana@localhost ~]$ cat /etc/hosts
> 127.0.0.1   localhost localhost.localdomain localhost4
> localhost4.localdomain4 gnode-2
> ::1         localhost localhost.localdomain localhost6
> localhost6.localdomain6
> 192.168.213.155 gnode-1

This means that the hostnames are not resolvable from both the hosts.

Gluster expects that the hostname should be resolvable from all the nodes of the cluster. This is the prerequisite.

Also Gluster provides the firewall service named as 'glusterfs' which helps open all the required ports on CentOS 7

# firewall-cmd --add-service=glusterfs --permanent
# firewall-cmd --add-service=glusterfs

Since this is not a genuine issue, I am closing this bug.

Feel free re-open the bug, if you again face the problems

Note You need to log in before you can comment on or make changes to this bug.