Bug 1566462 - [RFE] [Gdeploy+Ganesha] Check for HA node status after adding node to existing ganesha cluster via gdeploy
Summary: [RFE] [Gdeploy+Ganesha] Check for HA node status after adding node to existin...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: gdeploy
Version: rhgs-3.4
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: ---
Assignee: Sachidananda Urs
QA Contact: Manisha Saini
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-12 11:18 UTC by Manisha Saini
Modified: 2018-11-15 06:51 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-11-15 06:51:09 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Manisha Saini 2018-04-12 11:18:15 UTC
Description of problem:

Post adding the node to existing ganesha cluster,Gdeploy should check for Ganesha HA status to validate if the cluster is in HEALTHY state and none of the nodes are in failover state.

There can be a case wherein the VIP is failed to assigned to the eth of the new node and the ganesha cluster goes to FAILOVER state.Currently in that case,Gdeploy logs doesn't show any failures.

From usability point of view,its good to check the ganesha HA status post adding node to ganesha cluster

Ganesha status and be validated from the command-

#  /usr/libexec/ganesha/ganesha-ha.sh --status /var/run/gluster/shared_storage/nfs-ganesha
Online: [ dhcp37-121.lab.eng.blr.redhat.com dhcp46-116.lab.eng.blr.redhat.com dhcp46-184.lab.eng.blr.redhat.com dhcp47-193.lab.eng.blr.redhat.com dhcp47-2.lab.eng.blr.redhat.com ]

dhcp46-116.lab.eng.blr.redhat.com-cluster_ip-1 dhcp46-116.lab.eng.blr.redhat.com
dhcp46-184.lab.eng.blr.redhat.com-cluster_ip-1 dhcp46-184.lab.eng.blr.redhat.com
dhcp47-193.lab.eng.blr.redhat.com-cluster_ip-1 dhcp47-193.lab.eng.blr.redhat.com
dhcp47-2.lab.eng.blr.redhat.com-cluster_ip-1 dhcp47-2.lab.eng.blr.redhat.com
dhcp37-121.lab.eng.blr.redhat.com-cluster_ip-1 dhcp47-2.lab.eng.blr.redhat.com

Cluster HA Status: FAILOVER



Version-Release number of selected component (if applicable):

# rpm -qa | grep gdeploy
gdeploy-2.0.2-24.el7rhgs.noarch

How reproducible:


Steps to Reproduce:
1.Create 4 node Ganesha cluster via Gdeploy
2.Add the additional node to ganesha cluster via Gdeploy

[hosts]
dhcp47-193.lab.eng.blr.redhat.com
dhcp46-116.lab.eng.blr.redhat.com
dhcp46-184.lab.eng.blr.redhat.com
dhcp47-2.lab.eng.blr.redhat.com
dhcp37-121.lab.eng.blr.redhat.com

[firewalld]
action=add
ports=111/tcp,2049/tcp,54321/tcp,5900/tcp,5900-6923/tcp,5666/tcp,16514/tcp
services=glusterfs,nlm,nfs,rpc-bind,high-availability,mountd,rquota

[peer]
action=probe


[clients]
action=mount
volname=dhcp37-121.lab.eng.blr.redhat.com:gluster_shared_storage
hosts=dhcp37-121.lab.eng.blr.redhat.com
fstype=glusterfs
client_mount_points=/var/run/gluster/shared_storage/


[nfs-ganesha]
action=add-node
cluster_nodes=dhcp47-193.lab.eng.blr.redhat.com,dhcp46-116.lab.eng.blr.redhat.com,dhcp46-184.lab.eng.blr.redhat.com,dhcp47-2.lab.eng.blr.redhat.com
nodes=dhcp37-121.lab.eng.blr.redhat.com
vip=10.70.47.31



Actual results:
Gdeploy is not reflecting the HA HEALTH status post adding node to existing cluster


Expected results:
Gdeploy should reflect the HA HEALTH status after adding node to cluster.


Additional info:


Gdeploy add node output currently-

[root@dhcp47-193 ganesha_8_node]# gdeploy  -c add_node.conf 

PLAY [gluster_servers] *******************************************************************************************************************************

TASK [Start firewalld if not already started] ********************************************************************************************************
ok: [dhcp37-121.lab.eng.blr.redhat.com]
ok: [dhcp47-193.lab.eng.blr.redhat.com]
ok: [dhcp46-116.lab.eng.blr.redhat.com]
ok: [dhcp47-2.lab.eng.blr.redhat.com]
ok: [dhcp46-184.lab.eng.blr.redhat.com]

TASK [Add/Delete services to firewalld rules] ********************************************************************************************************
ok: [dhcp37-121.lab.eng.blr.redhat.com] => (item=glusterfs)
ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=glusterfs)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=glusterfs)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=glusterfs)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=glusterfs)
ok: [dhcp37-121.lab.eng.blr.redhat.com] => (item=nlm)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=nlm)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=nlm)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=nlm)
ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=nlm)
ok: [dhcp37-121.lab.eng.blr.redhat.com] => (item=nfs)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=nfs)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=nfs)
ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=nfs)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=nfs)
ok: [dhcp37-121.lab.eng.blr.redhat.com] => (item=rpc-bind)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=rpc-bind)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=rpc-bind)
ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=rpc-bind)
ok: [dhcp37-121.lab.eng.blr.redhat.com] => (item=high-availability)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=rpc-bind)
ok: [dhcp37-121.lab.eng.blr.redhat.com] => (item=mountd)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=high-availability)
ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=high-availability)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=high-availability)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=high-availability)
ok: [dhcp37-121.lab.eng.blr.redhat.com] => (item=rquota)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=mountd)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=mountd)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=mountd)
ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=mountd)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=rquota)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=rquota)
ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=rquota)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=rquota)

TASK [Reloads the firewall] **************************************************************************************************************************
changed: [dhcp37-121.lab.eng.blr.redhat.com]
changed: [dhcp46-184.lab.eng.blr.redhat.com]
changed: [dhcp47-193.lab.eng.blr.redhat.com]
changed: [dhcp47-2.lab.eng.blr.redhat.com]
changed: [dhcp46-116.lab.eng.blr.redhat.com]

PLAY RECAP *******************************************************************************************************************************************
dhcp37-121.lab.eng.blr.redhat.com : ok=3    changed=1    unreachable=0    failed=0   
dhcp46-116.lab.eng.blr.redhat.com : ok=3    changed=1    unreachable=0    failed=0   
dhcp46-184.lab.eng.blr.redhat.com : ok=3    changed=1    unreachable=0    failed=0   
dhcp47-193.lab.eng.blr.redhat.com : ok=3    changed=1    unreachable=0    failed=0   
dhcp47-2.lab.eng.blr.redhat.com : ok=3    changed=1    unreachable=0    failed=0   


PLAY [gluster_servers] *******************************************************************************************************************************

TASK [Start firewalld if not already started] ********************************************************************************************************
ok: [dhcp37-121.lab.eng.blr.redhat.com]
ok: [dhcp47-193.lab.eng.blr.redhat.com]
ok: [dhcp46-184.lab.eng.blr.redhat.com]
ok: [dhcp46-116.lab.eng.blr.redhat.com]
ok: [dhcp47-2.lab.eng.blr.redhat.com]

TASK [Open/Close firewalld ports] ********************************************************************************************************************
ok: [dhcp37-121.lab.eng.blr.redhat.com] => (item=111/tcp)
ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=111/tcp)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=111/tcp)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=111/tcp)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=111/tcp)
ok: [dhcp37-121.lab.eng.blr.redhat.com] => (item=2049/tcp)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=2049/tcp)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=2049/tcp)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=2049/tcp)
ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=2049/tcp)
ok: [dhcp37-121.lab.eng.blr.redhat.com] => (item=54321/tcp)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=54321/tcp)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=54321/tcp)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=54321/tcp)
ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=54321/tcp)
ok: [dhcp37-121.lab.eng.blr.redhat.com] => (item=5900/tcp)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=5900/tcp)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=5900/tcp)
ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=5900/tcp)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=5900/tcp)
ok: [dhcp37-121.lab.eng.blr.redhat.com] => (item=5900-6923/tcp)
ok: [dhcp37-121.lab.eng.blr.redhat.com] => (item=5666/tcp)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=5900-6923/tcp)
ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=5900-6923/tcp)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=5900-6923/tcp)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=5900-6923/tcp)
ok: [dhcp37-121.lab.eng.blr.redhat.com] => (item=16514/tcp)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=5666/tcp)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=5666/tcp)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=5666/tcp)
ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=5666/tcp)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=16514/tcp)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=16514/tcp)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=16514/tcp)
ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=16514/tcp)

TASK [Reloads the firewall] **************************************************************************************************************************
changed: [dhcp37-121.lab.eng.blr.redhat.com]
changed: [dhcp47-2.lab.eng.blr.redhat.com]
changed: [dhcp46-116.lab.eng.blr.redhat.com]
changed: [dhcp47-193.lab.eng.blr.redhat.com]
changed: [dhcp46-184.lab.eng.blr.redhat.com]

PLAY RECAP *******************************************************************************************************************************************
dhcp37-121.lab.eng.blr.redhat.com : ok=3    changed=1    unreachable=0    failed=0   
dhcp46-116.lab.eng.blr.redhat.com : ok=3    changed=1    unreachable=0    failed=0   
dhcp46-184.lab.eng.blr.redhat.com : ok=3    changed=1    unreachable=0    failed=0   
dhcp47-193.lab.eng.blr.redhat.com : ok=3    changed=1    unreachable=0    failed=0   
dhcp47-2.lab.eng.blr.redhat.com : ok=3    changed=1    unreachable=0    failed=0   


PLAY [master] ****************************************************************************************************************************************

TASK [Creates a Trusted Storage Pool] ****************************************************************************************************************
changed: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [Pause for a few seconds] ***********************************************************************************************************************
Pausing for 5 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
ok: [dhcp47-193.lab.eng.blr.redhat.com]

PLAY RECAP *******************************************************************************************************************************************
dhcp47-193.lab.eng.blr.redhat.com : ok=2    changed=1    unreachable=0    failed=0   


PLAY [clients] ***************************************************************************************************************************************

TASK [Create the dir to mount the volume, skips if present] ******************************************************************************************
ok: [dhcp37-121.lab.eng.blr.redhat.com] => (item={u'mountpoint': u'/var/run/gluster/shared_storage/', u'fstype': u'fuse'})

PLAY RECAP *******************************************************************************************************************************************
dhcp37-121.lab.eng.blr.redhat.com : ok=1    changed=0    unreachable=0    failed=0   


PLAY [clients] ***************************************************************************************************************************************

TASK [Mount the volumes, if fstype is glusterfs] *****************************************************************************************************
changed: [dhcp37-121.lab.eng.blr.redhat.com] => (item={u'mountpoint': u'/var/run/gluster/shared_storage/', u'fstype': u'fuse'})

PLAY RECAP *******************************************************************************************************************************************
dhcp37-121.lab.eng.blr.redhat.com : ok=1    changed=1    unreachable=0    failed=0   


PLAY [clients] ***************************************************************************************************************************************

TASK [Gathering Facts] *******************************************************************************************************************************
ok: [dhcp37-121.lab.eng.blr.redhat.com]

TASK [Uncomment STATD_PORT for rpc.statd to listen on] ***********************************************************************************************
skipping: [dhcp37-121.lab.eng.blr.redhat.com] => (item={u'mountpoint': u'/var/run/gluster/shared_storage/', u'fstype': u'fuse'}) 

TASK [Uncomment LOCKD_TCPPORT for rpc.lockd to listen on] ********************************************************************************************
skipping: [dhcp37-121.lab.eng.blr.redhat.com] => (item={u'mountpoint': u'/var/run/gluster/shared_storage/', u'fstype': u'fuse'}) 

TASK [Uncomment LOCKD_UDPPORT for rpc.lockd to listen on] ********************************************************************************************
skipping: [dhcp37-121.lab.eng.blr.redhat.com] => (item={u'mountpoint': u'/var/run/gluster/shared_storage/', u'fstype': u'fuse'}) 

TASK [Uncomment MOUNTD_PORT for rpc.mountd to listen on] *********************************************************************************************
skipping: [dhcp37-121.lab.eng.blr.redhat.com] => (item={u'mountpoint': u'/var/run/gluster/shared_storage/', u'fstype': u'fuse'}) 

TASK [Restart nfs service (RHEL 6 only)] *************************************************************************************************************
skipping: [dhcp37-121.lab.eng.blr.redhat.com] => (item={u'mountpoint': u'/var/run/gluster/shared_storage/', u'fstype': u'fuse'}) 

TASK [Restart rpc-statd service] *********************************************************************************************************************
skipping: [dhcp37-121.lab.eng.blr.redhat.com] => (item={u'mountpoint': u'/var/run/gluster/shared_storage/', u'fstype': u'fuse'}) 

TASK [Restart nfs-config service] ********************************************************************************************************************
skipping: [dhcp37-121.lab.eng.blr.redhat.com] => (item={u'mountpoint': u'/var/run/gluster/shared_storage/', u'fstype': u'fuse'}) 

TASK [Restart nfs-mountd service] ********************************************************************************************************************
skipping: [dhcp37-121.lab.eng.blr.redhat.com] => (item={u'mountpoint': u'/var/run/gluster/shared_storage/', u'fstype': u'fuse'}) 

TASK [Restart nfslock service (RHEL 6 & 7)] **********************************************************************************************************
skipping: [dhcp37-121.lab.eng.blr.redhat.com] => (item={u'mountpoint': u'/var/run/gluster/shared_storage/', u'fstype': u'fuse'}) 

TASK [Mount the volumes if fstype is NFS] ************************************************************************************************************
skipping: [dhcp37-121.lab.eng.blr.redhat.com] => (item={u'mountpoint': u'/var/run/gluster/shared_storage/', u'fstype': u'fuse'}) 

PLAY RECAP *******************************************************************************************************************************************
dhcp37-121.lab.eng.blr.redhat.com : ok=1    changed=0    unreachable=0    failed=0   


PLAY [clients] ***************************************************************************************************************************************

TASK [Mount the volumes, if fstype is CIFS] **********************************************************************************************************
skipping: [dhcp37-121.lab.eng.blr.redhat.com] => (item={u'mountpoint': u'/var/run/gluster/shared_storage/', u'fstype': u'fuse'}) 

PLAY RECAP *******************************************************************************************************************************************
dhcp37-121.lab.eng.blr.redhat.com : ok=0    changed=0    unreachable=0    failed=0   


PLAY [master_node] ***********************************************************************************************************************************

TASK [Gathering Facts] *******************************************************************************************************************************
ok: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [Copy the public key to the local] **************************************************************************************************************
changed: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [Copy the private key to the local] *************************************************************************************************************
changed: [dhcp47-193.lab.eng.blr.redhat.com]

PLAY RECAP *******************************************************************************************************************************************
dhcp47-193.lab.eng.blr.redhat.com : ok=3    changed=2    unreachable=0    failed=0   


PLAY [new_nodes] *************************************************************************************************************************************

TASK [Gathering Facts] *******************************************************************************************************************************
ok: [dhcp37-121.lab.eng.blr.redhat.com]

TASK [Check if nfs-ganesha is installed] *************************************************************************************************************
 [WARNING]: Consider using yum, dnf or zypper module rather than running rpm

changed: [dhcp37-121.lab.eng.blr.redhat.com]

TASK [fail] ******************************************************************************************************************************************
skipping: [dhcp37-121.lab.eng.blr.redhat.com]

TASK [Check if corosync is installed] ****************************************************************************************************************
changed: [dhcp37-121.lab.eng.blr.redhat.com]

TASK [fail] ******************************************************************************************************************************************
skipping: [dhcp37-121.lab.eng.blr.redhat.com]

TASK [Check if pacemaker is installed] ***************************************************************************************************************
changed: [dhcp37-121.lab.eng.blr.redhat.com]

TASK [fail] ******************************************************************************************************************************************
skipping: [dhcp37-121.lab.eng.blr.redhat.com]

TASK [Check if libntirpc is installed] ***************************************************************************************************************
changed: [dhcp37-121.lab.eng.blr.redhat.com]

TASK [fail] ******************************************************************************************************************************************
skipping: [dhcp37-121.lab.eng.blr.redhat.com]

TASK [Check if pcs is installed] *********************************************************************************************************************
changed: [dhcp37-121.lab.eng.blr.redhat.com]

TASK [fail] ******************************************************************************************************************************************
skipping: [dhcp37-121.lab.eng.blr.redhat.com]

TASK [Stop kernel NFS] *******************************************************************************************************************************
ok: [dhcp37-121.lab.eng.blr.redhat.com]

TASK [Stop network manager service] ******************************************************************************************************************
ok: [dhcp37-121.lab.eng.blr.redhat.com]

TASK [Disable network manager service] ***************************************************************************************************************
ok: [dhcp37-121.lab.eng.blr.redhat.com]

TASK [Start network service] *************************************************************************************************************************
ok: [dhcp37-121.lab.eng.blr.redhat.com]

TASK [Enable network service] ************************************************************************************************************************
ok: [dhcp37-121.lab.eng.blr.redhat.com]

TASK [Start pcsd service] ****************************************************************************************************************************
ok: [dhcp37-121.lab.eng.blr.redhat.com]

TASK [Enable pcsd service] ***************************************************************************************************************************
ok: [dhcp37-121.lab.eng.blr.redhat.com]

TASK [Enable pacemaker service] **********************************************************************************************************************
changed: [dhcp37-121.lab.eng.blr.redhat.com]

TASK [Create a user hacluster on new nodes] **********************************************************************************************************
ok: [dhcp37-121.lab.eng.blr.redhat.com]

TASK [Set the hacluster user the same password on new nodes] *****************************************************************************************
changed: [dhcp37-121.lab.eng.blr.redhat.com]

TASK [Copy the public key to remote nodes] ***********************************************************************************************************
changed: [dhcp37-121.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com)

TASK [Copy the private key to remote node] ***********************************************************************************************************
changed: [dhcp37-121.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com)

TASK [Deploy the pubkey ~/root/.ssh/authorized_keys on all nodes] ************************************************************************************
changed: [dhcp37-121.lab.eng.blr.redhat.com]

TASK [Define service port] ***************************************************************************************************************************
changed: [dhcp37-121.lab.eng.blr.redhat.com]

TASK [Restart statd service (RHEL 6 only)] ***********************************************************************************************************
skipping: [dhcp37-121.lab.eng.blr.redhat.com]

TASK [Restart nfs-config service] ********************************************************************************************************************
changed: [dhcp37-121.lab.eng.blr.redhat.com]

TASK [Restart rpc-statd service] *********************************************************************************************************************
changed: [dhcp37-121.lab.eng.blr.redhat.com]

TASK [Pcs cluster authenticate the hacluster on new nodes] *******************************************************************************************
changed: [dhcp37-121.lab.eng.blr.redhat.com] => (item=dhcp37-121.lab.eng.blr.redhat.com)

TASK [Pcs cluster authenticate the hacluster on existing nodes] **************************************************************************************
changed: [dhcp37-121.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com)
changed: [dhcp37-121.lab.eng.blr.redhat.com] => (item=dhcp46-116.lab.eng.blr.redhat.com)
changed: [dhcp37-121.lab.eng.blr.redhat.com] => (item=dhcp46-184.lab.eng.blr.redhat.com)
changed: [dhcp37-121.lab.eng.blr.redhat.com] => (item=dhcp47-2.lab.eng.blr.redhat.com)

TASK [Pause for a few seconds after pcs auth] ********************************************************************************************************
Pausing for 20 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
ok: [dhcp37-121.lab.eng.blr.redhat.com]

TASK [Enable GlusterFS Share Storage service] ********************************************************************************************************
ok: [dhcp37-121.lab.eng.blr.redhat.com]

TASK [Enable nfs-ganesha service] ********************************************************************************************************************
ok: [dhcp37-121.lab.eng.blr.redhat.com]

PLAY RECAP *******************************************************************************************************************************************
dhcp37-121.lab.eng.blr.redhat.com : ok=27   changed=15   unreachable=0    failed=0   


PLAY [cluster_nodes] *********************************************************************************************************************************

TASK [Pcs cluster authenticate the hacluster on new nodes] *******************************************************************************************
changed: [dhcp46-184.lab.eng.blr.redhat.com] => (item=dhcp37-121.lab.eng.blr.redhat.com)
failed: [dhcp47-193.lab.eng.blr.redhat.com] (item=dhcp37-121.lab.eng.blr.redhat.com) => {"changed": true, "cmd": "pcs cluster auth -u hacluster -p hacluster dhcp37-121.lab.eng.blr.redhat.com", "delta": "0:00:06.787876", "end": "2018-04-12 16:31:25.676369", "item": "dhcp37-121.lab.eng.blr.redhat.com", "msg": "non-zero return code", "rc": 1, "start": "2018-04-12 16:31:18.888493", "stderr": "Error: Some nodes had a newer tokens than the local node. Local node's tokens were updated. Please repeat the authentication if needed.\nError: Unable to communicate with pcsd", "stderr_lines": ["Error: Some nodes had a newer tokens than the local node. Local node's tokens were updated. Please repeat the authentication if needed.", "Error: Unable to communicate with pcsd"], "stdout": "", "stdout_lines": []}
changed: [dhcp47-2.lab.eng.blr.redhat.com] => (item=dhcp37-121.lab.eng.blr.redhat.com)
failed: [dhcp46-116.lab.eng.blr.redhat.com] (item=dhcp37-121.lab.eng.blr.redhat.com) => {"changed": true, "cmd": "pcs cluster auth -u hacluster -p hacluster dhcp37-121.lab.eng.blr.redhat.com", "delta": "0:00:06.993237", "end": "2018-04-12 16:31:26.496938", "item": "dhcp37-121.lab.eng.blr.redhat.com", "msg": "non-zero return code", "rc": 1, "start": "2018-04-12 16:31:19.503701", "stderr": "Error: Some nodes had a newer tokens than the local node. Local node's tokens were updated. Please repeat the authentication if needed.\nError: Unable to communicate with pcsd", "stderr_lines": ["Error: Some nodes had a newer tokens than the local node. Local node's tokens were updated. Please repeat the authentication if needed.", "Error: Unable to communicate with pcsd"], "stdout": "", "stdout_lines": []}

TASK [Pause for a few seconds] ***********************************************************************************************************************
Pausing for 5 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
ok: [dhcp46-184.lab.eng.blr.redhat.com]
	to retry, use: --limit @/tmp/tmpNFpwX1/ganesha-pcs-auth-new-nodes.retry

PLAY RECAP *******************************************************************************************************************************************
dhcp46-116.lab.eng.blr.redhat.com : ok=0    changed=0    unreachable=0    failed=1   
dhcp46-184.lab.eng.blr.redhat.com : ok=2    changed=1    unreachable=0    failed=0   
dhcp47-193.lab.eng.blr.redhat.com : ok=0    changed=0    unreachable=0    failed=1   
dhcp47-2.lab.eng.blr.redhat.com : ok=1    changed=1    unreachable=0    failed=0   

Ignoring errors...

PLAY [master] ****************************************************************************************************************************************

TASK [Adds a node to the cluster] ********************************************************************************************************************
changed: [dhcp47-193.lab.eng.blr.redhat.com] => (item={u'host': u'dhcp37-121.lab.eng.blr.redhat.com', u'vip': u'10.70.47.31'})

TASK [Report ganesha add-node status] ****************************************************************************************************************
ok: [dhcp47-193.lab.eng.blr.redhat.com] => {
    "msg": [
        "Disabling SBD service...", 
        "dhcp37-121.lab.eng.blr.redhat.com: sbd disabled", 
        "Sending remote node configuration files to 'dhcp37-121.lab.eng.blr.redhat.com'", 
        "dhcp37-121.lab.eng.blr.redhat.com: successful distribution of the file 'pacemaker_remote authkey'", 
        "dhcp47-193.lab.eng.blr.redhat.com: Corosync updated", 
        "dhcp46-116.lab.eng.blr.redhat.com: Corosync updated", 
        "dhcp46-184.lab.eng.blr.redhat.com: Corosync updated", 
        "dhcp47-2.lab.eng.blr.redhat.com: Corosync updated", 
        "Setting up corosync...", 
        "dhcp37-121.lab.eng.blr.redhat.com: Succeeded", 
        "Synchronizing pcsd certificates on nodes dhcp37-121.lab.eng.blr.redhat.com...", 
        "dhcp37-121.lab.eng.blr.redhat.com: Success", 
        "Restarting pcsd on the nodes in order to reload the certificates...", 
        "dhcp37-121.lab.eng.blr.redhat.com: Success", 
        "dhcp37-121.lab.eng.blr.redhat.com: Starting Cluster...", 
        "Removing group: dhcp46-116.lab.eng.blr.redhat.com-group (and all resources within group)", 
        "Stopping all resources in group: dhcp46-116.lab.eng.blr.redhat.com-group...", 
        "Deleting Resource - dhcp46-116.lab.eng.blr.redhat.com-nfs_block", 
        "Removing Constraint - order-nfs-grace-clone-dhcp46-116.lab.eng.blr.redhat.com-cluster_ip-1-mandatory", 
        "Deleting Resource - dhcp46-116.lab.eng.blr.redhat.com-cluster_ip-1", 
        "Removing Constraint - location-dhcp46-116.lab.eng.blr.redhat.com-group", 
        "Removing Constraint - location-dhcp46-116.lab.eng.blr.redhat.com-group-dhcp46-184.lab.eng.blr.redhat.com-1000", 
        "Removing Constraint - location-dhcp46-116.lab.eng.blr.redhat.com-group-dhcp47-2.lab.eng.blr.redhat.com-2000", 
        "Removing Constraint - location-dhcp46-116.lab.eng.blr.redhat.com-group-dhcp47-193.lab.eng.blr.redhat.com-3000", 
        "Removing Constraint - location-dhcp46-116.lab.eng.blr.redhat.com-group-dhcp46-116.lab.eng.blr.redhat.com-4000", 
        "Deleting Resource (and group) - dhcp46-116.lab.eng.blr.redhat.com-nfs_unblock", 
        "Removing group: dhcp46-184.lab.eng.blr.redhat.com-group (and all resources within group)", 
        "Stopping all resources in group: dhcp46-184.lab.eng.blr.redhat.com-group...", 
        "Deleting Resource - dhcp46-184.lab.eng.blr.redhat.com-nfs_block", 
        "Removing Constraint - order-nfs-grace-clone-dhcp46-184.lab.eng.blr.redhat.com-cluster_ip-1-mandatory", 
        "Deleting Resource - dhcp46-184.lab.eng.blr.redhat.com-cluster_ip-1", 
        "Removing Constraint - location-dhcp46-184.lab.eng.blr.redhat.com-group", 
        "Removing Constraint - location-dhcp46-184.lab.eng.blr.redhat.com-group-dhcp47-2.lab.eng.blr.redhat.com-1000", 
        "Removing Constraint - location-dhcp46-184.lab.eng.blr.redhat.com-group-dhcp47-193.lab.eng.blr.redhat.com-2000", 
        "Removing Constraint - location-dhcp46-184.lab.eng.blr.redhat.com-group-dhcp46-116.lab.eng.blr.redhat.com-3000", 
        "Removing Constraint - location-dhcp46-184.lab.eng.blr.redhat.com-group-dhcp46-184.lab.eng.blr.redhat.com-4000", 
        "Deleting Resource (and group) - dhcp46-184.lab.eng.blr.redhat.com-nfs_unblock", 
        "Removing group: dhcp47-193.lab.eng.blr.redhat.com-group (and all resources within group)", 
        "Stopping all resources in group: dhcp47-193.lab.eng.blr.redhat.com-group...", 
        "Deleting Resource - dhcp47-193.lab.eng.blr.redhat.com-nfs_block", 
        "Removing Constraint - order-nfs-grace-clone-dhcp47-193.lab.eng.blr.redhat.com-cluster_ip-1-mandatory", 
        "Deleting Resource - dhcp47-193.lab.eng.blr.redhat.com-cluster_ip-1", 
        "Removing Constraint - location-dhcp47-193.lab.eng.blr.redhat.com-group", 
        "Removing Constraint - location-dhcp47-193.lab.eng.blr.redhat.com-group-dhcp46-116.lab.eng.blr.redhat.com-1000", 
        "Removing Constraint - location-dhcp47-193.lab.eng.blr.redhat.com-group-dhcp46-184.lab.eng.blr.redhat.com-2000", 
        "Removing Constraint - location-dhcp47-193.lab.eng.blr.redhat.com-group-dhcp47-2.lab.eng.blr.redhat.com-3000", 
        "Removing Constraint - location-dhcp47-193.lab.eng.blr.redhat.com-group-dhcp47-193.lab.eng.blr.redhat.com-4000", 
        "Deleting Resource (and group) - dhcp47-193.lab.eng.blr.redhat.com-nfs_unblock", 
        "Removing group: dhcp47-2.lab.eng.blr.redhat.com-group (and all resources within group)", 
        "Stopping all resources in group: dhcp47-2.lab.eng.blr.redhat.com-group...", 
        "Deleting Resource - dhcp47-2.lab.eng.blr.redhat.com-nfs_block", 
        "Removing Constraint - order-nfs-grace-clone-dhcp47-2.lab.eng.blr.redhat.com-cluster_ip-1-mandatory", 
        "Deleting Resource - dhcp47-2.lab.eng.blr.redhat.com-cluster_ip-1", 
        "Removing Constraint - location-dhcp47-2.lab.eng.blr.redhat.com-group", 
        "Removing Constraint - location-dhcp47-2.lab.eng.blr.redhat.com-group-dhcp47-193.lab.eng.blr.redhat.com-1000", 
        "Removing Constraint - location-dhcp47-2.lab.eng.blr.redhat.com-group-dhcp46-116.lab.eng.blr.redhat.com-2000", 
        "Removing Constraint - location-dhcp47-2.lab.eng.blr.redhat.com-group-dhcp46-184.lab.eng.blr.redhat.com-3000", 
        "Removing Constraint - location-dhcp47-2.lab.eng.blr.redhat.com-group-dhcp47-2.lab.eng.blr.redhat.com-4000", 
        "Deleting Resource (and group) - dhcp47-2.lab.eng.blr.redhat.com-nfs_unblock", 
        "Adding nfs-grace-clone dhcp46-116.lab.eng.blr.redhat.com-cluster_ip-1 (kind: Mandatory) (Options: first-action=start then-action=start)", 
        "Adding nfs-grace-clone dhcp46-184.lab.eng.blr.redhat.com-cluster_ip-1 (kind: Mandatory) (Options: first-action=start then-action=start)", 
        "Adding nfs-grace-clone dhcp47-193.lab.eng.blr.redhat.com-cluster_ip-1 (kind: Mandatory) (Options: first-action=start then-action=start)", 
        "Adding nfs-grace-clone dhcp47-2.lab.eng.blr.redhat.com-cluster_ip-1 (kind: Mandatory) (Options: first-action=start then-action=start)", 
        "Adding nfs-grace-clone dhcp37-121.lab.eng.blr.redhat.com-cluster_ip-1 (kind: Mandatory) (Options: first-action=start then-action=start)", 
        "CIB updated"
    ]
}

PLAY RECAP *******************************************************************************************************************************************
dhcp47-193.lab.eng.blr.redhat.com : ok=2    changed=1    unreachable=0    failed=0

Comment 3 Sachidananda Urs 2018-11-15 06:51:09 UTC
gdeploy will be deprecated in favour of gluster-ansible, this issue will be addressed in gluster-ansible.


Note You need to log in before you can comment on or make changes to this bug.