This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 1461114 - [Ganesha+Gdeploy] While Adding a node to existing ganesha cluster "Unable to communicate with pcsd" messages are being observed [NEEDINFO]
[Ganesha+Gdeploy] While Adding a node to existing ganesha cluster "Unable to ...
Status: POST
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: gdeploy (Show other bugs)
3.3
Unspecified Unspecified
high Severity unspecified
: ---
: ---
Assigned To: Devyani Kota
Manisha Saini
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-13 10:14 EDT by Manisha Saini
Modified: 2017-10-17 08:21 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
amukherj: needinfo? (msaini)


Attachments (Terms of Use)

  None (edit)
Description Manisha Saini 2017-06-13 10:14:35 EDT
Description of problem:


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 2 Manisha Saini 2017-06-13 10:26:36 EDT
Submitted the bug without steps by mistake.


Description of problem:
While Adding a node to existing ganesha cluster "Unable to communicate with pcsd" messages are being observed 

Version-Release number of selected component (if applicable):
# rpm -qa | grep gdeploy
gdeploy-2.0.2-10.el7rhgs.noarch


How reproducible:
intermittent

Steps to Reproduce:
1.Create 4 node ganesha cluster.
2.Add a new node to existing ganesha cluster via gdeploy

# cat add_node.conf 
[hosts]
dhcp42-125.lab.eng.blr.redhat.com
dhcp42-127.lab.eng.blr.redhat.com
dhcp42-129.lab.eng.blr.redhat.com
dhcp42-119.lab.eng.blr.redhat.com
dhcp42-114.lab.eng.blr.redhat.com

[peer]
action=probe

[clients]
action=mount
volname=dhcp42-114.lab.eng.blr.redhat.com:gluster_shared_storage
hosts=dhcp42-114.lab.eng.blr.redhat.com
fstype=glusterfs
client_mount_points=/var/run/gluster/shared_storage/

[nfs-ganesha]
action=add-node
cluster_nodes=dhcp42-125.lab.eng.blr.redhat.com,dhcp42-127.lab.eng.blr.redhat.com,dhcp42-129.lab.eng.blr.redhat.com,dhcp42-119.lab.eng.blr.redhat.com
nodes=dhcp42-114.lab.eng.blr.redhat.com
vip=10.70.42.44


Actual results:
"Unable to communicate with pcsd" messages are observed while adding a node to cluster via gdeploy

"Error: Some nodes had a newer tokens than the local node. Local node's tokens were updated. Please repeat the authentication if needed.\nError: Unable to communicate with pcsd", "stdout": "", "stdout_lines": [], "warnings": []}
"

Expected results:
No such messages should be observed

Additional info:


# gdeploy -c add_node.conf 

PLAY [master] ******************************************************************

TASK [Creates a Trusted Storage Pool] ******************************************
changed: [dhcp42-125.lab.eng.blr.redhat.com]

TASK [Pause for a few seconds] *************************************************
Pausing for 5 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
ok: [dhcp42-125.lab.eng.blr.redhat.com]

PLAY RECAP *********************************************************************
dhcp42-125.lab.eng.blr.redhat.com : ok=2    changed=1    unreachable=0    failed=0   


PLAY [clients] *****************************************************************

TASK [Create the dir to mount the volume, skips if present] ********************
ok: [dhcp42-114.lab.eng.blr.redhat.com] => (item={u'mountpoint': u'/var/run/gluster/shared_storage/', u'fstype': u'fuse'})

PLAY RECAP *********************************************************************
dhcp42-114.lab.eng.blr.redhat.com : ok=1    changed=0    unreachable=0    failed=0   


PLAY [clients] *****************************************************************

TASK [Mount the volumes, if fstype is glusterfs] *******************************
ok: [dhcp42-114.lab.eng.blr.redhat.com] => (item={u'mountpoint': u'/var/run/gluster/shared_storage/', u'fstype': u'fuse'})

PLAY RECAP *********************************************************************
dhcp42-114.lab.eng.blr.redhat.com : ok=1    changed=0    unreachable=0    failed=0   


PLAY [clients] *****************************************************************

TASK [setup] *******************************************************************
ok: [dhcp42-114.lab.eng.blr.redhat.com]

TASK [Uncomment STATD_PORT for rpc.statd to listen on] *************************
skipping: [dhcp42-114.lab.eng.blr.redhat.com] => (item={u'mountpoint': u'/var/run/gluster/shared_storage/', u'fstype': u'fuse'}) 

TASK [Uncomment LOCKD_TCPPORT for rpc.lockd to listen on] **********************
skipping: [dhcp42-114.lab.eng.blr.redhat.com] => (item={u'mountpoint': u'/var/run/gluster/shared_storage/', u'fstype': u'fuse'}) 

TASK [Uncomment LOCKD_UDPPORT for rpc.lockd to listen on] **********************
skipping: [dhcp42-114.lab.eng.blr.redhat.com] => (item={u'mountpoint': u'/var/run/gluster/shared_storage/', u'fstype': u'fuse'}) 

TASK [Uncomment MOUNTD_PORT for rpc.mountd to listen on] ***********************
skipping: [dhcp42-114.lab.eng.blr.redhat.com] => (item={u'mountpoint': u'/var/run/gluster/shared_storage/', u'fstype': u'fuse'}) 

TASK [Restart nfs service (RHEL 6 only)] ***************************************
skipping: [dhcp42-114.lab.eng.blr.redhat.com] => (item={u'mountpoint': u'/var/run/gluster/shared_storage/', u'fstype': u'fuse'}) 

TASK [Restart rpc-statd service] ***********************************************
skipping: [dhcp42-114.lab.eng.blr.redhat.com] => (item={u'mountpoint': u'/var/run/gluster/shared_storage/', u'fstype': u'fuse'}) 

TASK [Restart nfs-config service] **********************************************
skipping: [dhcp42-114.lab.eng.blr.redhat.com] => (item={u'mountpoint': u'/var/run/gluster/shared_storage/', u'fstype': u'fuse'}) 

TASK [Restart nfs-mountd service] **********************************************
skipping: [dhcp42-114.lab.eng.blr.redhat.com] => (item={u'mountpoint': u'/var/run/gluster/shared_storage/', u'fstype': u'fuse'}) 

TASK [Restart nfslock service (RHEL 6 & 7)] ************************************
skipping: [dhcp42-114.lab.eng.blr.redhat.com] => (item={u'mountpoint': u'/var/run/gluster/shared_storage/', u'fstype': u'fuse'}) 

TASK [Mount the volumes if fstype is NFS] **************************************
skipping: [dhcp42-114.lab.eng.blr.redhat.com] => (item={u'mountpoint': u'/var/run/gluster/shared_storage/', u'fstype': u'fuse'}) 

PLAY RECAP *********************************************************************
dhcp42-114.lab.eng.blr.redhat.com : ok=1    changed=0    unreachable=0    failed=0   


PLAY [clients] *****************************************************************

TASK [Mount the volumes, if fstype is CIFS] ************************************
skipping: [dhcp42-114.lab.eng.blr.redhat.com] => (item={u'mountpoint': u'/var/run/gluster/shared_storage/', u'fstype': u'fuse'}) 

PLAY RECAP *********************************************************************
dhcp42-114.lab.eng.blr.redhat.com : ok=0    changed=0    unreachable=0    failed=0   


PLAY [master_node] *************************************************************

TASK [setup] *******************************************************************
ok: [dhcp42-125.lab.eng.blr.redhat.com]

TASK [Copy the public key to the local] ****************************************
changed: [dhcp42-125.lab.eng.blr.redhat.com]

TASK [Copy the private key to the local] ***************************************
changed: [dhcp42-125.lab.eng.blr.redhat.com]

PLAY RECAP *********************************************************************
dhcp42-125.lab.eng.blr.redhat.com : ok=3    changed=2    unreachable=0    failed=0   


PLAY [new_nodes] ***************************************************************

TASK [setup] *******************************************************************
ok: [dhcp42-114.lab.eng.blr.redhat.com]

TASK [Check if nfs-ganesha is installed] ***************************************
changed: [dhcp42-114.lab.eng.blr.redhat.com]
 [WARNING]: Consider using yum, dnf or zypper module rather than running rpm


TASK [fail] ********************************************************************
skipping: [dhcp42-114.lab.eng.blr.redhat.com]

TASK [Check if corosync is installed] ******************************************
changed: [dhcp42-114.lab.eng.blr.redhat.com]

TASK [fail] ********************************************************************
skipping: [dhcp42-114.lab.eng.blr.redhat.com]

TASK [Check if pacemaker is installed] *****************************************
changed: [dhcp42-114.lab.eng.blr.redhat.com]

TASK [fail] ********************************************************************
skipping: [dhcp42-114.lab.eng.blr.redhat.com]

TASK [Check if libntirpc is installed] *****************************************
changed: [dhcp42-114.lab.eng.blr.redhat.com]

TASK [fail] ********************************************************************
skipping: [dhcp42-114.lab.eng.blr.redhat.com]

TASK [Check if pcs is installed] ***********************************************
changed: [dhcp42-114.lab.eng.blr.redhat.com]

TASK [fail] ********************************************************************
skipping: [dhcp42-114.lab.eng.blr.redhat.com]

TASK [Stop kernel NFS] *********************************************************
ok: [dhcp42-114.lab.eng.blr.redhat.com]

TASK [Stop network manager service] ********************************************
ok: [dhcp42-114.lab.eng.blr.redhat.com]

TASK [Disable network manager service] *****************************************
ok: [dhcp42-114.lab.eng.blr.redhat.com]

TASK [Start network service] ***************************************************
ok: [dhcp42-114.lab.eng.blr.redhat.com]

TASK [Enable network service] **************************************************
ok: [dhcp42-114.lab.eng.blr.redhat.com]

TASK [Start pcsd service] ******************************************************
ok: [dhcp42-114.lab.eng.blr.redhat.com]

TASK [Enable pcsd service] *****************************************************
ok: [dhcp42-114.lab.eng.blr.redhat.com]

TASK [Enable pacemaker service] ************************************************
changed: [dhcp42-114.lab.eng.blr.redhat.com]

TASK [Create a user hacluster on new nodes] ************************************
ok: [dhcp42-114.lab.eng.blr.redhat.com]

TASK [Set the hacluster user the same password on new nodes] *******************
changed: [dhcp42-114.lab.eng.blr.redhat.com]

TASK [Copy the public key to remote nodes] *************************************
ok: [dhcp42-114.lab.eng.blr.redhat.com] => (item=dhcp42-125.lab.eng.blr.redhat.com)

TASK [Copy the private key to remote node] *************************************
ok: [dhcp42-114.lab.eng.blr.redhat.com] => (item=dhcp42-125.lab.eng.blr.redhat.com)

TASK [Deploy the pubkey ~/root/.ssh/authorized_keys on all nodes] **************
changed: [dhcp42-114.lab.eng.blr.redhat.com]

TASK [Define service port] *****************************************************
changed: [dhcp42-114.lab.eng.blr.redhat.com]

TASK [Restart statd service (RHEL 6 only)] *************************************
skipping: [dhcp42-114.lab.eng.blr.redhat.com]

TASK [Restart nfs-config service] **********************************************
changed: [dhcp42-114.lab.eng.blr.redhat.com]

TASK [Restart rpc-statd service] ***********************************************
changed: [dhcp42-114.lab.eng.blr.redhat.com]

TASK [Pcs cluster authenticate the hacluster on new nodes] *********************
changed: [dhcp42-114.lab.eng.blr.redhat.com] => (item=dhcp42-114.lab.eng.blr.redhat.com)

TASK [Pcs cluster authenticate the hacluster on existing nodes] ****************
changed: [dhcp42-114.lab.eng.blr.redhat.com] => (item=dhcp42-125.lab.eng.blr.redhat.com)
changed: [dhcp42-114.lab.eng.blr.redhat.com] => (item=dhcp42-127.lab.eng.blr.redhat.com)
changed: [dhcp42-114.lab.eng.blr.redhat.com] => (item=dhcp42-129.lab.eng.blr.redhat.com)
changed: [dhcp42-114.lab.eng.blr.redhat.com] => (item=dhcp42-119.lab.eng.blr.redhat.com)

TASK [Pause for a few seconds after pcs auth] **********************************
Pausing for 3 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
ok: [dhcp42-114.lab.eng.blr.redhat.com]

PLAY RECAP *********************************************************************
dhcp42-114.lab.eng.blr.redhat.com : ok=25   changed=13   unreachable=0    failed=0   


PLAY [cluster_nodes] ***********************************************************

TASK [Pcs cluster authenticate the hacluster on new nodes] *********************
changed: [dhcp42-129.lab.eng.blr.redhat.com] => (item=dhcp42-114.lab.eng.blr.redhat.com)
failed: [dhcp42-125.lab.eng.blr.redhat.com] (item=dhcp42-114.lab.eng.blr.redhat.com) => {"changed": true, "cmd": "pcs cluster auth -u hacluster -p hacluster dhcp42-114.lab.eng.blr.redhat.com", "delta": "0:00:05.453812", "end": "2017-06-13 19:08:11.573044", "failed": true, "item": "dhcp42-114.lab.eng.blr.redhat.com", "rc": 1, "start": "2017-06-13 19:08:06.119232", "stderr": "Error: Some nodes had a newer tokens than the local node. Local node's tokens were updated. Please repeat the authentication if needed.\nError: Unable to communicate with pcsd", "stdout": "", "stdout_lines": [], "warnings": []}
changed: [dhcp42-119.lab.eng.blr.redhat.com] => (item=dhcp42-114.lab.eng.blr.redhat.com)
failed: [dhcp42-127.lab.eng.blr.redhat.com] (item=dhcp42-114.lab.eng.blr.redhat.com) => {"changed": true, "cmd": "pcs cluster auth -u hacluster -p hacluster dhcp42-114.lab.eng.blr.redhat.com", "delta": "0:00:05.669351", "end": "2017-06-13 19:08:12.091837", "failed": true, "item": "dhcp42-114.lab.eng.blr.redhat.com", "rc": 1, "start": "2017-06-13 19:08:06.422486", "stderr": "Error: Some nodes had a newer tokens than the local node. Local node's tokens were updated. Please repeat the authentication if needed.\nError: Unable to communicate with pcsd", "stdout": "", "stdout_lines": [], "warnings": []}
	to retry, use: --limit @/tmp/tmpPU0xKa/ganesha-pcs-auth-new-nodes.retry

PLAY RECAP *********************************************************************
dhcp42-119.lab.eng.blr.redhat.com : ok=1    changed=1    unreachable=0    failed=0   
dhcp42-125.lab.eng.blr.redhat.com : ok=0    changed=0    unreachable=0    failed=1   
dhcp42-127.lab.eng.blr.redhat.com : ok=0    changed=0    unreachable=0    failed=1   
dhcp42-129.lab.eng.blr.redhat.com : ok=1    changed=1    unreachable=0    failed=0   

Ignoring errors...

PLAY [master] ******************************************************************

TASK [Adds a node to the cluster] **********************************************
changed: [dhcp42-125.lab.eng.blr.redhat.com] => (item={u'host': u'dhcp42-114.lab.eng.blr.redhat.com', u'vip': u'10.70.42.44'})

TASK [Report ganesha add-node status] ******************************************
ok: [dhcp42-125.lab.eng.blr.redhat.com] => {
    "msg": [
        "Disabling SBD service...", 
        "dhcp42-114.lab.eng.blr.redhat.com: sbd disabled", 
        "dhcp42-125.lab.eng.blr.redhat.com: Corosync updated", 
        "dhcp42-127.lab.eng.blr.redhat.com: Corosync updated", 
        "dhcp42-129.lab.eng.blr.redhat.com: Corosync updated", 
        "dhcp42-119.lab.eng.blr.redhat.com: Corosync updated", 
        "Setting up corosync...", 
        "dhcp42-114.lab.eng.blr.redhat.com: Succeeded", 
        "Synchronizing pcsd certificates on nodes dhcp42-114.lab.eng.blr.redhat.com...", 
        "dhcp42-114.lab.eng.blr.redhat.com: Success", 
        "Restarting pcsd on the nodes in order to reload the certificates...", 
        "dhcp42-114.lab.eng.blr.redhat.com: Success", 
        "dhcp42-114.lab.eng.blr.redhat.com: Stopping Cluster (pacemaker)...", 
        "dhcp42-127.lab.eng.blr.redhat.com: Stopping Cluster (pacemaker)...", 
        "dhcp42-125.lab.eng.blr.redhat.com: Stopping Cluster (pacemaker)...", 
        "dhcp42-129.lab.eng.blr.redhat.com: Stopping Cluster (pacemaker)...", 
        "dhcp42-119.lab.eng.blr.redhat.com: Stopping Cluster (pacemaker)...", 
        "dhcp42-114.lab.eng.blr.redhat.com: Stopping Cluster (corosync)...", 
        "dhcp42-127.lab.eng.blr.redhat.com: Stopping Cluster (corosync)...", 
        "dhcp42-125.lab.eng.blr.redhat.com: Stopping Cluster (corosync)...", 
        "dhcp42-129.lab.eng.blr.redhat.com: Stopping Cluster (corosync)...", 
        "dhcp42-119.lab.eng.blr.redhat.com: Stopping Cluster (corosync)...", 
        "dhcp42-129.lab.eng.blr.redhat.com: Starting Cluster...", 
        "dhcp42-127.lab.eng.blr.redhat.com: Starting Cluster...", 
        "dhcp42-119.lab.eng.blr.redhat.com: Starting Cluster...", 
        "dhcp42-125.lab.eng.blr.redhat.com: Starting Cluster...", 
        "dhcp42-114.lab.eng.blr.redhat.com: Starting Cluster...", 
        "Removing group: dhcp42-119.lab.eng.blr.redhat.com-group (and all resources within group)", 
        "Stopping all resources in group: dhcp42-119.lab.eng.blr.redhat.com-group...", 
        "Deleting Resource - dhcp42-119.lab.eng.blr.redhat.com-nfs_block", 
        "Removing Constraint - order-nfs-grace-clone-dhcp42-119.lab.eng.blr.redhat.com-cluster_ip-1-mandatory", 
        "Deleting Resource - dhcp42-119.lab.eng.blr.redhat.com-cluster_ip-1", 
        "Removing Constraint - location-dhcp42-119.lab.eng.blr.redhat.com-group", 
        "Removing Constraint - location-dhcp42-119.lab.eng.blr.redhat.com-group-dhcp42-125.lab.eng.blr.redhat.com-1000", 
        "Removing Constraint - location-dhcp42-119.lab.eng.blr.redhat.com-group-dhcp42-127.lab.eng.blr.redhat.com-2000", 
        "Removing Constraint - location-dhcp42-119.lab.eng.blr.redhat.com-group-dhcp42-129.lab.eng.blr.redhat.com-3000", 
        "Removing Constraint - location-dhcp42-119.lab.eng.blr.redhat.com-group-dhcp42-119.lab.eng.blr.redhat.com-4000", 
        "Deleting Resource (and group) - dhcp42-119.lab.eng.blr.redhat.com-nfs_unblock", 
        "Removing group: dhcp42-125.lab.eng.blr.redhat.com-group (and all resources within group)", 
        "Stopping all resources in group: dhcp42-125.lab.eng.blr.redhat.com-group...", 
        "Deleting Resource - dhcp42-125.lab.eng.blr.redhat.com-nfs_block", 
        "Removing Constraint - order-nfs-grace-clone-dhcp42-125.lab.eng.blr.redhat.com-cluster_ip-1-mandatory", 
        "Deleting Resource - dhcp42-125.lab.eng.blr.redhat.com-cluster_ip-1", 
        "Removing Constraint - location-dhcp42-125.lab.eng.blr.redhat.com-group", 
        "Removing Constraint - location-dhcp42-125.lab.eng.blr.redhat.com-group-dhcp42-127.lab.eng.blr.redhat.com-1000", 
        "Removing Constraint - location-dhcp42-125.lab.eng.blr.redhat.com-group-dhcp42-129.lab.eng.blr.redhat.com-2000", 
        "Removing Constraint - location-dhcp42-125.lab.eng.blr.redhat.com-group-dhcp42-119.lab.eng.blr.redhat.com-3000", 
        "Removing Constraint - location-dhcp42-125.lab.eng.blr.redhat.com-group-dhcp42-125.lab.eng.blr.redhat.com-4000", 
        "Deleting Resource (and group) - dhcp42-125.lab.eng.blr.redhat.com-nfs_unblock", 
        "Removing group: dhcp42-127.lab.eng.blr.redhat.com-group (and all resources within group)", 
        "Stopping all resources in group: dhcp42-127.lab.eng.blr.redhat.com-group...", 
        "Deleting Resource - dhcp42-127.lab.eng.blr.redhat.com-nfs_block", 
        "Removing Constraint - order-nfs-grace-clone-dhcp42-127.lab.eng.blr.redhat.com-cluster_ip-1-mandatory", 
        "Deleting Resource - dhcp42-127.lab.eng.blr.redhat.com-cluster_ip-1", 
        "Removing Constraint - location-dhcp42-127.lab.eng.blr.redhat.com-group", 
        "Removing Constraint - location-dhcp42-127.lab.eng.blr.redhat.com-group-dhcp42-129.lab.eng.blr.redhat.com-1000", 
        "Removing Constraint - location-dhcp42-127.lab.eng.blr.redhat.com-group-dhcp42-119.lab.eng.blr.redhat.com-2000", 
        "Removing Constraint - location-dhcp42-127.lab.eng.blr.redhat.com-group-dhcp42-125.lab.eng.blr.redhat.com-3000", 
        "Removing Constraint - location-dhcp42-127.lab.eng.blr.redhat.com-group-dhcp42-127.lab.eng.blr.redhat.com-4000", 
        "Deleting Resource (and group) - dhcp42-127.lab.eng.blr.redhat.com-nfs_unblock", 
        "Removing group: dhcp42-129.lab.eng.blr.redhat.com-group (and all resources within group)", 
        "Stopping all resources in group: dhcp42-129.lab.eng.blr.redhat.com-group...", 
        "Deleting Resource - dhcp42-129.lab.eng.blr.redhat.com-nfs_block", 
        "Removing Constraint - order-nfs-grace-clone-dhcp42-129.lab.eng.blr.redhat.com-cluster_ip-1-mandatory", 
        "Deleting Resource - dhcp42-129.lab.eng.blr.redhat.com-cluster_ip-1", 
        "Removing Constraint - location-dhcp42-129.lab.eng.blr.redhat.com-group", 
        "Removing Constraint - location-dhcp42-129.lab.eng.blr.redhat.com-group-dhcp42-119.lab.eng.blr.redhat.com-1000", 
        "Removing Constraint - location-dhcp42-129.lab.eng.blr.redhat.com-group-dhcp42-125.lab.eng.blr.redhat.com-2000", 
        "Removing Constraint - location-dhcp42-129.lab.eng.blr.redhat.com-group-dhcp42-127.lab.eng.blr.redhat.com-3000", 
        "Removing Constraint - location-dhcp42-129.lab.eng.blr.redhat.com-group-dhcp42-129.lab.eng.blr.redhat.com-4000", 
        "Deleting Resource (and group) - dhcp42-129.lab.eng.blr.redhat.com-nfs_unblock", 
        "Adding nfs-grace-clone dhcp42-119.lab.eng.blr.redhat.com-cluster_ip-1 (kind: Mandatory) (Options: first-action=start then-action=start)", 
        "Adding nfs-grace-clone dhcp42-125.lab.eng.blr.redhat.com-cluster_ip-1 (kind: Mandatory) (Options: first-action=start then-action=start)", 
        "Adding nfs-grace-clone dhcp42-127.lab.eng.blr.redhat.com-cluster_ip-1 (kind: Mandatory) (Options: first-action=start then-action=start)", 
        "Adding nfs-grace-clone dhcp42-129.lab.eng.blr.redhat.com-cluster_ip-1 (kind: Mandatory) (Options: first-action=start then-action=start)", 
        "Adding nfs-grace-clone dhcp42-114.lab.eng.blr.redhat.com-cluster_ip-1 (kind: Mandatory) (Options: first-action=start then-action=start)", 
        "CIB updated"
    ]
}

PLAY RECAP *********************************************************************
dhcp42-125.lab.eng.blr.redhat.com : ok=2    changed=1    unreachable=0    failed=0
Comment 5 Manisha Saini 2017-06-19 05:46:47 EDT
Few Points-

The issue is only observed while adding the node to cluster ->Deleting the node from cluster ->Re adding the same node to existing cluster.

Adding and deleting the node from existing is not deleting the token from all the nodes including the new node.

A reference to the bug which has the similar issue reported in rhel 7 -
https://bugzilla.redhat.com/show_bug.cgi?id=1265925
Comment 10 Devyani Kota 2017-10-09 06:18:12 EDT
Hi all,

The issue was not reproducible on 4-5 nodes.
The issue is intermittent while performing add node/delete node/re-add node operations, on scaling upto 7 nodes.
Will try adding more sleep seconds during pcs auth check, to find out if 
it is gdeploy or nfs-ganesha issue.

Thanks,
Devyani Kota
Comment 11 Devyani Kota 2017-10-10 04:40:08 EDT
Hi,

This pull-request[1] should resolve this issue.
[1] https://github.com/gluster/gdeploy/pull/449

Thanks,
Devyani Kota

Note You need to log in before you can comment on or make changes to this bug.