Description of problem: I am having machines with names, rhs-client21.lab.eng.blr.redhat.com rhs-client23.lab.eng.blr.redhat.com rhs-client36.lab.eng.blr.redhat.com and I am trying to bring up nfs-ganesha on these machines, the ganesha comes up on these machines but the pcs cluster fails. Version-Release number of selected component (if applicable): glusterfs-3.7.0-2.el6rhs.x86_64 nfs-ganesha-2.2.0-0.el6.x86_64 How reproducible: always Actual results: root@rhs-client21 ganesha]# bash -x ganesha-ha.sh setup /etc/ganesha + HA_NUM_SERVERS=0 + HA_SERVERS= + HA_CONFDIR= + HA_VOL_NAME=gluster_shared_storage + HA_VOL_MNT=/var/run/gluster/shared_storage + SERVICE_MAN=DISTRO_NOT_FOUND ++ cat /etc/sysconfig/ganesha ++ grep CONFFILE ++ cut -f 2 -d = + CONF=/etc/ganesha/ganesha.conf + RHEL6_PCS_CNAME_OPTION=--name + main setup /etc/ganesha + local cmd=setup + shift + HA_CONFDIR=/etc/ganesha + shift + local node= + local vip= + . /etc/ganesha/ganesha-ha.conf ++ HA_NAME=clamper ++ HA_VOL_SERVER=rhs-client21.lab.eng.blr.redhat.com ++ HA_CLUSTER_NODES=rhs-client21.lab.eng.blr.redhat.com,rhs-client23.lab.eng.blr.redhat.com,rhs-client36.lab.eng.blr.redhat.com ++ VIP_rhs-client21.lab.eng.blr.redhat.com=10.70.36.80 /etc/ganesha/ganesha-ha.conf: line 11: VIP_rhs-client21.lab.eng.blr.redhat.com=10.70.36.80: command not found ++ VIP_rhs-client23.lab.eng.blr.redhat.com=10.70.36.81 /etc/ganesha/ganesha-ha.conf: line 12: VIP_rhs-client23.lab.eng.blr.redhat.com=10.70.36.81: command not found ++ VIP_rhs-client36.lab.eng.blr.redhat.com=10.70.36.82 /etc/ganesha/ganesha-ha.conf: line 13: VIP_rhs-client36.lab.eng.blr.redhat.com=10.70.36.82: command not found + '[' -e /etc/os-release ']' + case "${cmd}" in + logger 'setting up clamper' + check_cluster_exists clamper + local name=clamper + local cluster_name= + '[' -e /var/run/corosync.pid ']' ++ pcs status ++ grep 'Cluster name:' ++ cut -d ' ' -f 3 Error: cluster is not currently running on this node + cluster_name= + '[' -a = clamper ']' + determine_servers setup + local cmd=setup + local num_servers=0 + local 'tmp_ifs= ' + local ha_servers= + [[ XsetupX != \X\s\e\t\u\p\X ]] + IFS=, + for server in '${HA_CLUSTER_NODES}' ++ expr 0 + 1 + num_servers=1 + for server in '${HA_CLUSTER_NODES}' ++ expr 1 + 1 + num_servers=2 + for server in '${HA_CLUSTER_NODES}' ++ expr 2 + 1 + num_servers=3 + IFS=' ' + HA_NUM_SERVERS=3 + HA_SERVERS='rhs-client21.lab.eng.blr.redhat.com rhs-client23.lab.eng.blr.redhat.com rhs-client36.lab.eng.blr.redhat.com' + '[' X3X '!=' X1X ']' + setup_cluster clamper 3 'rhs-client21.lab.eng.blr.redhat.com rhs-client23.lab.eng.blr.redhat.com rhs-client36.lab.eng.blr.redhat.com' + local name=clamper + local num_servers=3 + local 'servers=rhs-client21.lab.eng.blr.redhat.com rhs-client23.lab.eng.blr.redhat.com rhs-client36.lab.eng.blr.redhat.com' + local unclean= + logger 'setting up cluster clamper with the following rhs-client21.lab.eng.blr.redhat.com rhs-client23.lab.eng.blr.redhat.com rhs-client36.lab.eng.blr.redhat.com' + pcs cluster auth rhs-client21.lab.eng.blr.redhat.com rhs-client23.lab.eng.blr.redhat.com rhs-client36.lab.eng.blr.redhat.com rhs-client21.lab.eng.blr.redhat.com: Already authorized rhs-client23.lab.eng.blr.redhat.com: Already authorized rhs-client36.lab.eng.blr.redhat.com: Already authorized + pcs cluster setup --name clamper rhs-client21.lab.eng.blr.redhat.com rhs-client23.lab.eng.blr.redhat.com rhs-client36.lab.eng.blr.redhat.com rhs-client21.lab.eng.blr.redhat.com: Updated cluster.conf... rhs-client23.lab.eng.blr.redhat.com: Updated cluster.conf... rhs-client36.lab.eng.blr.redhat.com: Updated cluster.conf... + '[' 0 -ne 0 ']' + pcs cluster start --all rhs-client21.lab.eng.blr.redhat.com: Starting Cluster... rhs-client23.lab.eng.blr.redhat.com: Starting Cluster... rhs-client36.lab.eng.blr.redhat.com: Starting Cluster... + '[' 0 -ne 0 ']' + sleep 3 ++ pcs status ++ grep -u UNCLEAN + unclean= + [[ X = \U\N\C\L\E\A\N\X ]] + sleep 1 + '[' 3 -lt 3 ']' + pcs property set stonith-enabled=false + '[' 0 -ne 0 ']' + setup_create_resources rhs-client21.lab.eng.blr.redhat.com rhs-client23.lab.eng.blr.redhat.com rhs-client36.lab.eng.blr.redhat.com ++ mktemp -u + local cibfile=/tmp/tmp.GRq4seYRox + pcs resource create nfs_start ganesha_nfsd ha_vol_mnt=/var/run/gluster/shared_storage --clone + '[' 0 -ne 0 ']' + sleep 1 + pcs resource delete nfs_start-clone Attempting to stop: nfs_start...Stopped Deleting Resource - nfs_start + '[' 0 -ne 0 ']' + pcs resource create nfs-mon ganesha_mon --clone + '[' 0 -ne 0 ']' + pcs resource create nfs-grace ganesha_grace --clone + '[' 0 -ne 0 ']' + pcs cluster cib /tmp/tmp.GRq4seYRox + [[ -n rhs-client21.lab.eng.blr.redhat.com ]] + ipaddrx=VIP_rhs_client21.lab.eng.blr.redhat.com + ipaddr= + pcs -f /tmp/tmp.GRq4seYRox resource create rhs-client21.lab.eng.blr.redhat.com-cluster_ip-1 ocf:heartbeat:IPaddr ip= cidr_netmask=32 op monitor interval=15s + '[' 0 -ne 0 ']' + pcs -f /tmp/tmp.GRq4seYRox resource create rhs-client21.lab.eng.blr.redhat.com-trigger_ip-1 ocf:heartbeat:Dummy + '[' 0 -ne 0 ']' + pcs -f /tmp/tmp.GRq4seYRox constraint colocation add rhs-client21.lab.eng.blr.redhat.com-cluster_ip-1 with rhs-client21.lab.eng.blr.redhat.com-trigger_ip-1 + '[' 0 -ne 0 ']' + pcs -f /tmp/tmp.GRq4seYRox constraint order rhs-client21.lab.eng.blr.redhat.com-trigger_ip-1 then nfs-grace-clone Adding rhs-client21.lab.eng.blr.redhat.com-trigger_ip-1 nfs-grace-clone (kind: Mandatory) (Options: first-action=start then-action=start) + '[' 0 -ne 0 ']' + pcs -f /tmp/tmp.GRq4seYRox constraint order nfs-grace-clone then rhs-client21.lab.eng.blr.redhat.com-cluster_ip-1 Adding nfs-grace-clone rhs-client21.lab.eng.blr.redhat.com-cluster_ip-1 (kind: Mandatory) (Options: first-action=start then-action=start) + '[' 0 -ne 0 ']' + shift + [[ -n rhs-client23.lab.eng.blr.redhat.com ]] + ipaddrx=VIP_rhs_client23.lab.eng.blr.redhat.com + ipaddr= + pcs -f /tmp/tmp.GRq4seYRox resource create rhs-client23.lab.eng.blr.redhat.com-cluster_ip-1 ocf:heartbeat:IPaddr ip= cidr_netmask=32 op monitor interval=15s + '[' 0 -ne 0 ']' + pcs -f /tmp/tmp.GRq4seYRox resource create rhs-client23.lab.eng.blr.redhat.com-trigger_ip-1 ocf:heartbeat:Dummy + '[' 0 -ne 0 ']' + pcs -f /tmp/tmp.GRq4seYRox constraint colocation add rhs-client23.lab.eng.blr.redhat.com-cluster_ip-1 with rhs-client23.lab.eng.blr.redhat.com-trigger_ip-1 + '[' 0 -ne 0 ']' + pcs -f /tmp/tmp.GRq4seYRox constraint order rhs-client23.lab.eng.blr.redhat.com-trigger_ip-1 then nfs-grace-clone Adding rhs-client23.lab.eng.blr.redhat.com-trigger_ip-1 nfs-grace-clone (kind: Mandatory) (Options: first-action=start then-action=start) + '[' 0 -ne 0 ']' + pcs -f /tmp/tmp.GRq4seYRox constraint order nfs-grace-clone then rhs-client23.lab.eng.blr.redhat.com-cluster_ip-1 Adding nfs-grace-clone rhs-client23.lab.eng.blr.redhat.com-cluster_ip-1 (kind: Mandatory) (Options: first-action=start then-action=start) + '[' 0 -ne 0 ']' + shift + [[ -n rhs-client36.lab.eng.blr.redhat.com ]] + ipaddrx=VIP_rhs_client36.lab.eng.blr.redhat.com + ipaddr= + pcs -f /tmp/tmp.GRq4seYRox resource create rhs-client36.lab.eng.blr.redhat.com-cluster_ip-1 ocf:heartbeat:IPaddr ip= cidr_netmask=32 op monitor interval=15s + '[' 0 -ne 0 ']' + pcs -f /tmp/tmp.GRq4seYRox resource create rhs-client36.lab.eng.blr.redhat.com-trigger_ip-1 ocf:heartbeat:Dummy + '[' 0 -ne 0 ']' + pcs -f /tmp/tmp.GRq4seYRox constraint colocation add rhs-client36.lab.eng.blr.redhat.com-cluster_ip-1 with rhs-client36.lab.eng.blr.redhat.com-trigger_ip-1 + '[' 0 -ne 0 ']' + pcs -f /tmp/tmp.GRq4seYRox constraint order rhs-client36.lab.eng.blr.redhat.com-trigger_ip-1 then nfs-grace-clone Adding rhs-client36.lab.eng.blr.redhat.com-trigger_ip-1 nfs-grace-clone (kind: Mandatory) (Options: first-action=start then-action=start) + '[' 0 -ne 0 ']' + pcs -f /tmp/tmp.GRq4seYRox constraint order nfs-grace-clone then rhs-client36.lab.eng.blr.redhat.com-cluster_ip-1 Adding nfs-grace-clone rhs-client36.lab.eng.blr.redhat.com-cluster_ip-1 (kind: Mandatory) (Options: first-action=start then-action=start) + '[' 0 -ne 0 ']' + shift + [[ -n '' ]] + create_virt_ip_constraints /tmp/tmp.GRq4seYRox rhs-client21.lab.eng.blr.redhat.com rhs-client23.lab.eng.blr.redhat.com rhs-client36.lab.eng.blr.redhat.com + local cibfile=/tmp/tmp.GRq4seYRox + shift + [[ -n rhs-client21.lab.eng.blr.redhat.com ]] + wrap_create_virt_ip_constraints /tmp/tmp.GRq4seYRox rhs-client21.lab.eng.blr.redhat.com rhs-client21.lab.eng.blr.redhat.com rhs-client23.lab.eng.blr.redhat.com rhs-client36.lab.eng.blr.redhat.com + local cibfile=/tmp/tmp.GRq4seYRox + shift + local primary=rhs-client21.lab.eng.blr.redhat.com + shift + local head= + local tail= + [[ -n rhs-client21.lab.eng.blr.redhat.com ]] + '[' rhs-client21.lab.eng.blr.redhat.com = rhs-client21.lab.eng.blr.redhat.com ']' + shift + [[ -n rhs-client23.lab.eng.blr.redhat.com ]] + tail=' rhs-client23.lab.eng.blr.redhat.com' + shift + [[ -n rhs-client36.lab.eng.blr.redhat.com ]] + tail=' rhs-client23.lab.eng.blr.redhat.com rhs-client36.lab.eng.blr.redhat.com' + shift + [[ -n '' ]] + shift + [[ -n '' ]] + do_create_virt_ip_constraints /tmp/tmp.GRq4seYRox rhs-client21.lab.eng.blr.redhat.com rhs-client23.lab.eng.blr.redhat.com rhs-client36.lab.eng.blr.redhat.com + local cibfile=/tmp/tmp.GRq4seYRox + shift + local primary=rhs-client21.lab.eng.blr.redhat.com + shift + local weight=1000 + pcs -f /tmp/tmp.GRq4seYRox constraint location rhs-client21.lab.eng.blr.redhat.com-cluster_ip-1 rule score=-INFINITY ganesha-active ne 1 + '[' 0 -ne 0 ']' + [[ -n rhs-client23.lab.eng.blr.redhat.com ]] + pcs -f /tmp/tmp.GRq4seYRox constraint location rhs-client21.lab.eng.blr.redhat.com-cluster_ip-1 prefers rhs-client23.lab.eng.blr.redhat.com=1000 + '[' 0 -ne 0 ']' ++ expr 1000 + 1000 + weight=2000 + shift + [[ -n rhs-client36.lab.eng.blr.redhat.com ]] + pcs -f /tmp/tmp.GRq4seYRox constraint location rhs-client21.lab.eng.blr.redhat.com-cluster_ip-1 prefers rhs-client36.lab.eng.blr.redhat.com=2000 + '[' 0 -ne 0 ']' ++ expr 2000 + 1000 + weight=3000 + shift + [[ -n '' ]] + pcs -f /tmp/tmp.GRq4seYRox constraint location rhs-client21.lab.eng.blr.redhat.com-cluster_ip-1 prefers rhs-client21.lab.eng.blr.redhat.com=3000 + '[' 0 -ne 0 ']' + shift + [[ -n rhs-client23.lab.eng.blr.redhat.com ]] + wrap_create_virt_ip_constraints /tmp/tmp.GRq4seYRox rhs-client23.lab.eng.blr.redhat.com rhs-client21.lab.eng.blr.redhat.com rhs-client23.lab.eng.blr.redhat.com rhs-client36.lab.eng.blr.redhat.com + local cibfile=/tmp/tmp.GRq4seYRox + shift + local primary=rhs-client23.lab.eng.blr.redhat.com + shift + local head= + local tail= + [[ -n rhs-client21.lab.eng.blr.redhat.com ]] + '[' rhs-client21.lab.eng.blr.redhat.com = rhs-client23.lab.eng.blr.redhat.com ']' + head=' rhs-client21.lab.eng.blr.redhat.com' + shift + [[ -n rhs-client23.lab.eng.blr.redhat.com ]] + '[' rhs-client23.lab.eng.blr.redhat.com = rhs-client23.lab.eng.blr.redhat.com ']' + shift + [[ -n rhs-client36.lab.eng.blr.redhat.com ]] + tail=' rhs-client36.lab.eng.blr.redhat.com' + shift + [[ -n '' ]] + shift + [[ -n '' ]] + do_create_virt_ip_constraints /tmp/tmp.GRq4seYRox rhs-client23.lab.eng.blr.redhat.com rhs-client36.lab.eng.blr.redhat.com rhs-client21.lab.eng.blr.redhat.com + local cibfile=/tmp/tmp.GRq4seYRox + shift + local primary=rhs-client23.lab.eng.blr.redhat.com + shift + local weight=1000 + pcs -f /tmp/tmp.GRq4seYRox constraint location rhs-client23.lab.eng.blr.redhat.com-cluster_ip-1 rule score=-INFINITY ganesha-active ne 1 + '[' 0 -ne 0 ']' + [[ -n rhs-client36.lab.eng.blr.redhat.com ]] + pcs -f /tmp/tmp.GRq4seYRox constraint location rhs-client23.lab.eng.blr.redhat.com-cluster_ip-1 prefers rhs-client36.lab.eng.blr.redhat.com=1000 + '[' 0 -ne 0 ']' ++ expr 1000 + 1000 + weight=2000 + shift + [[ -n rhs-client21.lab.eng.blr.redhat.com ]] + pcs -f /tmp/tmp.GRq4seYRox constraint location rhs-client23.lab.eng.blr.redhat.com-cluster_ip-1 prefers rhs-client21.lab.eng.blr.redhat.com=2000 + '[' 0 -ne 0 ']' ++ expr 2000 + 1000 + weight=3000 + shift + [[ -n '' ]] + pcs -f /tmp/tmp.GRq4seYRox constraint location rhs-client23.lab.eng.blr.redhat.com-cluster_ip-1 prefers rhs-client23.lab.eng.blr.redhat.com=3000 + '[' 0 -ne 0 ']' + shift + [[ -n rhs-client36.lab.eng.blr.redhat.com ]] + wrap_create_virt_ip_constraints /tmp/tmp.GRq4seYRox rhs-client36.lab.eng.blr.redhat.com rhs-client21.lab.eng.blr.redhat.com rhs-client23.lab.eng.blr.redhat.com rhs-client36.lab.eng.blr.redhat.com + local cibfile=/tmp/tmp.GRq4seYRox + shift + local primary=rhs-client36.lab.eng.blr.redhat.com + shift + local head= + local tail= + [[ -n rhs-client21.lab.eng.blr.redhat.com ]] + '[' rhs-client21.lab.eng.blr.redhat.com = rhs-client36.lab.eng.blr.redhat.com ']' + head=' rhs-client21.lab.eng.blr.redhat.com' + shift + [[ -n rhs-client23.lab.eng.blr.redhat.com ]] + '[' rhs-client23.lab.eng.blr.redhat.com = rhs-client36.lab.eng.blr.redhat.com ']' + head=' rhs-client21.lab.eng.blr.redhat.com rhs-client23.lab.eng.blr.redhat.com' + shift + [[ -n rhs-client36.lab.eng.blr.redhat.com ]] + '[' rhs-client36.lab.eng.blr.redhat.com = rhs-client36.lab.eng.blr.redhat.com ']' + shift + [[ -n '' ]] + shift + [[ -n '' ]] + do_create_virt_ip_constraints /tmp/tmp.GRq4seYRox rhs-client36.lab.eng.blr.redhat.com rhs-client21.lab.eng.blr.redhat.com rhs-client23.lab.eng.blr.redhat.com + local cibfile=/tmp/tmp.GRq4seYRox + shift + local primary=rhs-client36.lab.eng.blr.redhat.com + shift + local weight=1000 + pcs -f /tmp/tmp.GRq4seYRox constraint location rhs-client36.lab.eng.blr.redhat.com-cluster_ip-1 rule score=-INFINITY ganesha-active ne 1 + '[' 0 -ne 0 ']' + [[ -n rhs-client21.lab.eng.blr.redhat.com ]] + pcs -f /tmp/tmp.GRq4seYRox constraint location rhs-client36.lab.eng.blr.redhat.com-cluster_ip-1 prefers rhs-client21.lab.eng.blr.redhat.com=1000 + '[' 0 -ne 0 ']' ++ expr 1000 + 1000 + weight=2000 + shift + [[ -n rhs-client23.lab.eng.blr.redhat.com ]] + pcs -f /tmp/tmp.GRq4seYRox constraint location rhs-client36.lab.eng.blr.redhat.com-cluster_ip-1 prefers rhs-client23.lab.eng.blr.redhat.com=2000 + '[' 0 -ne 0 ']' ++ expr 2000 + 1000 + weight=3000 + shift + [[ -n '' ]] + pcs -f /tmp/tmp.GRq4seYRox constraint location rhs-client36.lab.eng.blr.redhat.com-cluster_ip-1 prefers rhs-client36.lab.eng.blr.redhat.com=3000 + '[' 0 -ne 0 ']' + shift + [[ -n '' ]] + pcs cluster cib-push /tmp/tmp.GRq4seYRox CIB updated + '[' 0 -ne 0 ']' + rm -f /tmp/tmp.GRq4seYRox + setup_state_volume rhs-client21.lab.eng.blr.redhat.com rhs-client23.lab.eng.blr.redhat.com rhs-client36.lab.eng.blr.redhat.com + local mnt=/var/run/gluster/shared_storage + local longname= + local shortname= + local dname= ++ hostname + longname=rhs-client21.lab.eng.blr.redhat.com ++ hostname -s + dname=.lab.eng.blr.redhat.com + [[ -n rhs-client21.lab.eng.blr.redhat.com ]] + '[' '!' -d /var/run/gluster/shared_storage/nfs-ganesha/rhs-client21.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com ']' + '[' '!' -d /var/run/gluster/shared_storage/nfs-ganesha/rhs-client21.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs ']' + '[' '!' -d /var/run/gluster/shared_storage/nfs-ganesha/rhs-client21.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/ganesha ']' + '[' '!' -d /var/run/gluster/shared_storage/nfs-ganesha/rhs-client21.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/statd ']' + touch /var/run/gluster/shared_storage/nfs-ganesha/rhs-client21.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/state + '[' '!' -d /var/run/gluster/shared_storage/nfs-ganesha/rhs-client21.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/ganesha/v4recov ']' + '[' '!' -d /var/run/gluster/shared_storage/nfs-ganesha/rhs-client21.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/ganesha/v4old ']' + '[' '!' -d /var/run/gluster/shared_storage/nfs-ganesha/rhs-client21.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/statd/sm ']' + '[' '!' -d /var/run/gluster/shared_storage/nfs-ganesha/rhs-client21.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/statd/sm.bak ']' + '[' '!' -d /var/run/gluster/shared_storage/nfs-ganesha/rhs-client21.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/statd/state ']' + for server in '${HA_SERVERS}' + '[' rhs-client21.lab.eng.blr.redhat.com '!=' rhs-client21.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com ']' + ln -s /var/run/gluster/shared_storage/nfs-ganesha/rhs-client21.lab.eng.blr.redhat.com/nfs/ganesha /var/run/gluster/shared_storage/nfs-ganesha/rhs-client21.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/ganesha/rhs-client21.lab.eng.blr.redhat.com ln: creating symbolic link `/var/run/gluster/shared_storage/nfs-ganesha/rhs-client21.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/ganesha/rhs-client21.lab.eng.blr.redhat.com': File exists + ln -s /var/run/gluster/shared_storage/nfs-ganesha/rhs-client21.lab.eng.blr.redhat.com/nfs/statd /var/run/gluster/shared_storage/nfs-ganesha/rhs-client21.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/statd/rhs-client21.lab.eng.blr.redhat.com ln: creating symbolic link `/var/run/gluster/shared_storage/nfs-ganesha/rhs-client21.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/statd/rhs-client21.lab.eng.blr.redhat.com': File exists + for server in '${HA_SERVERS}' + '[' rhs-client23.lab.eng.blr.redhat.com '!=' rhs-client21.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com ']' + ln -s /var/run/gluster/shared_storage/nfs-ganesha/rhs-client23.lab.eng.blr.redhat.com/nfs/ganesha /var/run/gluster/shared_storage/nfs-ganesha/rhs-client21.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/ganesha/rhs-client23.lab.eng.blr.redhat.com ln: creating symbolic link `/var/run/gluster/shared_storage/nfs-ganesha/rhs-client21.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/ganesha/rhs-client23.lab.eng.blr.redhat.com': File exists + ln -s /var/run/gluster/shared_storage/nfs-ganesha/rhs-client23.lab.eng.blr.redhat.com/nfs/statd /var/run/gluster/shared_storage/nfs-ganesha/rhs-client21.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/statd/rhs-client23.lab.eng.blr.redhat.com ln: creating symbolic link `/var/run/gluster/shared_storage/nfs-ganesha/rhs-client21.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/statd/rhs-client23.lab.eng.blr.redhat.com': File exists + for server in '${HA_SERVERS}' + '[' rhs-client36.lab.eng.blr.redhat.com '!=' rhs-client21.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com ']' + ln -s /var/run/gluster/shared_storage/nfs-ganesha/rhs-client36.lab.eng.blr.redhat.com/nfs/ganesha /var/run/gluster/shared_storage/nfs-ganesha/rhs-client21.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/ganesha/rhs-client36.lab.eng.blr.redhat.com ln: creating symbolic link `/var/run/gluster/shared_storage/nfs-ganesha/rhs-client21.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/ganesha/rhs-client36.lab.eng.blr.redhat.com': File exists + ln -s /var/run/gluster/shared_storage/nfs-ganesha/rhs-client36.lab.eng.blr.redhat.com/nfs/statd /var/run/gluster/shared_storage/nfs-ganesha/rhs-client21.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/statd/rhs-client36.lab.eng.blr.redhat.com ln: creating symbolic link `/var/run/gluster/shared_storage/nfs-ganesha/rhs-client21.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/statd/rhs-client36.lab.eng.blr.redhat.com': File exists + shift + [[ -n rhs-client23.lab.eng.blr.redhat.com ]] + '[' '!' -d /var/run/gluster/shared_storage/nfs-ganesha/rhs-client23.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com ']' + '[' '!' -d /var/run/gluster/shared_storage/nfs-ganesha/rhs-client23.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs ']' + '[' '!' -d /var/run/gluster/shared_storage/nfs-ganesha/rhs-client23.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/ganesha ']' + '[' '!' -d /var/run/gluster/shared_storage/nfs-ganesha/rhs-client23.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/statd ']' + touch /var/run/gluster/shared_storage/nfs-ganesha/rhs-client23.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/state + '[' '!' -d /var/run/gluster/shared_storage/nfs-ganesha/rhs-client23.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/ganesha/v4recov ']' + '[' '!' -d /var/run/gluster/shared_storage/nfs-ganesha/rhs-client23.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/ganesha/v4old ']' + '[' '!' -d /var/run/gluster/shared_storage/nfs-ganesha/rhs-client23.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/statd/sm ']' + '[' '!' -d /var/run/gluster/shared_storage/nfs-ganesha/rhs-client23.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/statd/sm.bak ']' + '[' '!' -d /var/run/gluster/shared_storage/nfs-ganesha/rhs-client23.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/statd/state ']' + for server in '${HA_SERVERS}' + '[' rhs-client21.lab.eng.blr.redhat.com '!=' rhs-client23.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com ']' + ln -s /var/run/gluster/shared_storage/nfs-ganesha/rhs-client21.lab.eng.blr.redhat.com/nfs/ganesha /var/run/gluster/shared_storage/nfs-ganesha/rhs-client23.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/ganesha/rhs-client21.lab.eng.blr.redhat.com ln: creating symbolic link `/var/run/gluster/shared_storage/nfs-ganesha/rhs-client23.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/ganesha/rhs-client21.lab.eng.blr.redhat.com': File exists + ln -s /var/run/gluster/shared_storage/nfs-ganesha/rhs-client21.lab.eng.blr.redhat.com/nfs/statd /var/run/gluster/shared_storage/nfs-ganesha/rhs-client23.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/statd/rhs-client21.lab.eng.blr.redhat.com ln: creating symbolic link `/var/run/gluster/shared_storage/nfs-ganesha/rhs-client23.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/statd/rhs-client21.lab.eng.blr.redhat.com': File exists + for server in '${HA_SERVERS}' + '[' rhs-client23.lab.eng.blr.redhat.com '!=' rhs-client23.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com ']' + ln -s /var/run/gluster/shared_storage/nfs-ganesha/rhs-client23.lab.eng.blr.redhat.com/nfs/ganesha /var/run/gluster/shared_storage/nfs-ganesha/rhs-client23.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/ganesha/rhs-client23.lab.eng.blr.redhat.com ln: creating symbolic link `/var/run/gluster/shared_storage/nfs-ganesha/rhs-client23.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/ganesha/rhs-client23.lab.eng.blr.redhat.com': File exists + ln -s /var/run/gluster/shared_storage/nfs-ganesha/rhs-client23.lab.eng.blr.redhat.com/nfs/statd /var/run/gluster/shared_storage/nfs-ganesha/rhs-client23.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/statd/rhs-client23.lab.eng.blr.redhat.com ln: creating symbolic link `/var/run/gluster/shared_storage/nfs-ganesha/rhs-client23.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/statd/rhs-client23.lab.eng.blr.redhat.com': File exists + for server in '${HA_SERVERS}' + '[' rhs-client36.lab.eng.blr.redhat.com '!=' rhs-client23.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com ']' + ln -s /var/run/gluster/shared_storage/nfs-ganesha/rhs-client36.lab.eng.blr.redhat.com/nfs/ganesha /var/run/gluster/shared_storage/nfs-ganesha/rhs-client23.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/ganesha/rhs-client36.lab.eng.blr.redhat.com ln: creating symbolic link `/var/run/gluster/shared_storage/nfs-ganesha/rhs-client23.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/ganesha/rhs-client36.lab.eng.blr.redhat.com': File exists + ln -s /var/run/gluster/shared_storage/nfs-ganesha/rhs-client36.lab.eng.blr.redhat.com/nfs/statd /var/run/gluster/shared_storage/nfs-ganesha/rhs-client23.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/statd/rhs-client36.lab.eng.blr.redhat.com ln: creating symbolic link `/var/run/gluster/shared_storage/nfs-ganesha/rhs-client23.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/statd/rhs-client36.lab.eng.blr.redhat.com': File exists + shift + [[ -n rhs-client36.lab.eng.blr.redhat.com ]] + '[' '!' -d /var/run/gluster/shared_storage/nfs-ganesha/rhs-client36.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com ']' + '[' '!' -d /var/run/gluster/shared_storage/nfs-ganesha/rhs-client36.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs ']' + '[' '!' -d /var/run/gluster/shared_storage/nfs-ganesha/rhs-client36.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/ganesha ']' + '[' '!' -d /var/run/gluster/shared_storage/nfs-ganesha/rhs-client36.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/statd ']' + touch /var/run/gluster/shared_storage/nfs-ganesha/rhs-client36.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/state + '[' '!' -d /var/run/gluster/shared_storage/nfs-ganesha/rhs-client36.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/ganesha/v4recov ']' + '[' '!' -d /var/run/gluster/shared_storage/nfs-ganesha/rhs-client36.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/ganesha/v4old ']' + '[' '!' -d /var/run/gluster/shared_storage/nfs-ganesha/rhs-client36.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/statd/sm ']' + '[' '!' -d /var/run/gluster/shared_storage/nfs-ganesha/rhs-client36.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/statd/sm.bak ']' + '[' '!' -d /var/run/gluster/shared_storage/nfs-ganesha/rhs-client36.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/statd/state ']' + for server in '${HA_SERVERS}' + '[' rhs-client21.lab.eng.blr.redhat.com '!=' rhs-client36.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com ']' + ln -s /var/run/gluster/shared_storage/nfs-ganesha/rhs-client21.lab.eng.blr.redhat.com/nfs/ganesha /var/run/gluster/shared_storage/nfs-ganesha/rhs-client36.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/ganesha/rhs-client21.lab.eng.blr.redhat.com ln: creating symbolic link `/var/run/gluster/shared_storage/nfs-ganesha/rhs-client36.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/ganesha/rhs-client21.lab.eng.blr.redhat.com': File exists + ln -s /var/run/gluster/shared_storage/nfs-ganesha/rhs-client21.lab.eng.blr.redhat.com/nfs/statd /var/run/gluster/shared_storage/nfs-ganesha/rhs-client36.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/statd/rhs-client21.lab.eng.blr.redhat.com ln: creating symbolic link `/var/run/gluster/shared_storage/nfs-ganesha/rhs-client36.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/statd/rhs-client21.lab.eng.blr.redhat.com': File exists + for server in '${HA_SERVERS}' + '[' rhs-client23.lab.eng.blr.redhat.com '!=' rhs-client36.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com ']' + ln -s /var/run/gluster/shared_storage/nfs-ganesha/rhs-client23.lab.eng.blr.redhat.com/nfs/ganesha /var/run/gluster/shared_storage/nfs-ganesha/rhs-client36.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/ganesha/rhs-client23.lab.eng.blr.redhat.com ln: creating symbolic link `/var/run/gluster/shared_storage/nfs-ganesha/rhs-client36.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/ganesha/rhs-client23.lab.eng.blr.redhat.com': File exists + ln -s /var/run/gluster/shared_storage/nfs-ganesha/rhs-client23.lab.eng.blr.redhat.com/nfs/statd /var/run/gluster/shared_storage/nfs-ganesha/rhs-client36.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/statd/rhs-client23.lab.eng.blr.redhat.com ln: creating symbolic link `/var/run/gluster/shared_storage/nfs-ganesha/rhs-client36.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/statd/rhs-client23.lab.eng.blr.redhat.com': File exists + for server in '${HA_SERVERS}' + '[' rhs-client36.lab.eng.blr.redhat.com '!=' rhs-client36.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com ']' + ln -s /var/run/gluster/shared_storage/nfs-ganesha/rhs-client36.lab.eng.blr.redhat.com/nfs/ganesha /var/run/gluster/shared_storage/nfs-ganesha/rhs-client36.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/ganesha/rhs-client36.lab.eng.blr.redhat.com ln: creating symbolic link `/var/run/gluster/shared_storage/nfs-ganesha/rhs-client36.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/ganesha/rhs-client36.lab.eng.blr.redhat.com': File exists + ln -s /var/run/gluster/shared_storage/nfs-ganesha/rhs-client36.lab.eng.blr.redhat.com/nfs/statd /var/run/gluster/shared_storage/nfs-ganesha/rhs-client36.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/statd/rhs-client36.lab.eng.blr.redhat.com ln: creating symbolic link `/var/run/gluster/shared_storage/nfs-ganesha/rhs-client36.lab.eng.blr.redhat.com.lab.eng.blr.redhat.com/nfs/statd/rhs-client36.lab.eng.blr.redhat.com': File exists + shift + [[ -n '' ]] + setup_copy_config rhs-client21.lab.eng.blr.redhat.com rhs-client23.lab.eng.blr.redhat.com rhs-client36.lab.eng.blr.redhat.com ++ hostname -s + local short_host=rhs-client21 + '[' -e /var/lib/glusterd/nfs/secret.pem ']' + logger 'warning: scp ganesha-ha.conf to rhs-client21.lab.eng.blr.redhat.com failed' + setup_finalize + local cibfile= + local stopped= ++ pcs status ++ grep -u Stopped + stopped= + [[ X = \S\t\o\p\p\e\d\X ]] + pcs status + grep dead_ip-1 + sort [root@rhs-client21 ganesha]# Expected results: The pcs cluster should be successful Additional info:
short term work-around: use short names in /etc/ganesha/ganesha-ha.conf and substitute '_' for '-' in the VIP_rhs-clientXX=... lines. Make sure /etc/hosts has matching short names for each FQ name. e.g. # Name of the HA cluster created. HA_NAME="clamper" # The server from which you intend to mount # the shared volume. HA_VOL_SERVER="rhs-client21.lab.eng.blr.redhat.com" # The subset of nodes of the Gluster Trusted Pool # that forms the ganesha HA cluster. IP/Hostname # is specified. HA_CLUSTER_NODES="rhs-client21,rhs-client23,rhs-client36" # Virtual IPs of each of the nodes specified above. VIP_rhs_client21="10.70.36.80" VIP_rhs_client23="10.70.36.81" VIP_rhs_client36="10.70.36.82"
(re)fixed in rhgs-3.1.0, commit 7c16f45e7b2700e3e3563a46e8f75a265d3d6cd7 BZ 1227311
Able to setup 4 node ganesha cluster on rhel 6.7 with hostnames: vm1-server.lab.eng.blr.redhat.com vm2-server.lab.eng.blr.redhat.com vm3-server.lab.eng.blr.redhat.com vm4-server.lab.eng.blr.redhat.com
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2015-1495.html