Bug 1393968

Summary: gdeploy: nfs-ganesha cluster creation fails during pcs authentication
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Arthy Loganathan <aloganat>
Component: gdeployAssignee: Sachidananda Urs <surs>
Status: CLOSED ERRATA QA Contact: Arthy Loganathan <aloganat>
Severity: high Docs Contact:
Priority: unspecified    
Version: rhgs-3.2CC: aloganat, rhinduja, rhs-bugs, smohan, storage-qa-internal
Target Milestone: ---   
Target Release: RHGS 3.2.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: gdeploy-2.0.1-4 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-03-23 05:08:49 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1351528    

Description Arthy Loganathan 2016-11-10 18:01:58 UTC
Description of problem:
nfs-ganesha cluster creation fails during pcs authentication due to timing issue.

Version-Release number of selected component (if applicable):
gdeploy-2.0.1-3.el7rhgs.noarch

How reproducible:
Always

Steps to Reproduce:
1. Setup nfs-ganesha using gdeploy.

Actual results:
nfs-ganesha cluster creation fails during pcs authentication

Expected results:
nfs-ganesha cluster should get created successfully.

Additional info:

TASK [Pcs cluster authenticate the hacluster users in all the nodes] ***********
changed: [dhcp46-139.lab.eng.blr.redhat.com] => (item=dhcp46-111.lab.eng.blr.redhat.com)
changed: [dhcp46-115.lab.eng.blr.redhat.com] => (item=dhcp46-111.lab.eng.blr.redhat.com)
failed: [dhcp46-124.lab.eng.blr.redhat.com] (item=dhcp46-111.lab.eng.blr.redhat.com) => {"changed": true, "cmd": "pcs cluster auth -u hacluster -p hacluster dhcp46-111.lab.eng.blr.redhat.com", "delta": "0:00:02.100926", "end": "2016-11-09 17:15:22.376325", "failed": true, "item": "dhcp46-111.lab.eng.blr.redhat.com", "rc": 1, "start": "2016-11-09 17:15:20.275399", "stderr": "Error: Some nodes had a newer tokens than the local node. Local node's tokens were updated. Please repeat the authentication if needed.\nError: Unable to communicate with pcsd", "stdout": "", "stdout_lines": [], "warnings": []}
changed: [dhcp46-111.lab.eng.blr.redhat.com] => (item=dhcp46-111.lab.eng.blr.redhat.com)
changed: [dhcp46-115.lab.eng.blr.redhat.com] => (item=dhcp46-115.lab.eng.blr.redhat.com)
changed: [dhcp46-124.lab.eng.blr.redhat.com] => (item=dhcp46-115.lab.eng.blr.redhat.com)
failed: [dhcp46-139.lab.eng.blr.redhat.com] (item=dhcp46-115.lab.eng.blr.redhat.com) => {"changed": true, "cmd": "pcs cluster auth -u hacluster -p hacluster dhcp46-115.lab.eng.blr.redhat.com", "delta": "0:00:01.840204", "end": "2016-11-09 17:15:24.252541", "failed": true, "item": "dhcp46-115.lab.eng.blr.redhat.com", "rc": 1, "start": "2016-11-09 17:15:22.412337", "stderr": "Error: Some nodes had a newer tokens than the local node. Local node's tokens were updated. Please repeat the authentication if needed.\nError: Unable to communicate with pcsd", "stdout": "", "stdout_lines": [], "warnings": []}

Comment 2 Sachidananda Urs 2016-11-12 16:38:53 UTC
Arthy can you please double check if the proposed solution of pausing a couple of seconds work?

Comment 3 Sachidananda Urs 2016-11-14 07:00:34 UTC
The actual issue is due to: https://bugzilla.redhat.com/show_bug.cgi?id=1265925
(Thanks to Arthy)

This is beyond the scope of gdeploy. However, adding a sleep for 2-3 seconds will
reduce the error frequency.

Comment 5 Sachidananda Urs 2016-11-14 10:58:48 UTC
Commit: https://github.com/gluster/gdeploy/pull/216/commits/e47fad6c fixes the issue.

Comment 7 Arthy Loganathan 2016-11-16 06:30:01 UTC
I have tried with the proposed solution, and the frequency of getting error is reduced(hit 1/5 times).

Comment 8 Arthy Loganathan 2016-11-21 12:52:19 UTC
Verified the fix in build, gdeploy-2.0.1-4.el7rhgs.noarch

Comment 10 errata-xmlrpc 2017-03-23 05:08:49 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2017-0482.html