Bug 1226874

Summary: nfs-ganesha: in case pcs cluster setup fails then nfs-ganesha process should not start
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Saurabh <saujain>
Component: common-haAssignee: Jiffin <jthottan>
Status: CLOSED ERRATA QA Contact: Manisha Saini <msaini>
Severity: urgent Docs Contact:
Priority: high    
Version: rhgs-3.1CC: amukherj, asriram, divya, jthottan, kkeithle, nlevinki, rhinduja, rhs-bugs, sankarshan, sheggodu, skoduri, srmukher, storage-qa-internal
Target Milestone: ---Keywords: FutureFeature, RFE, Triaged, ZStream
Target Release: RHGS 3.4.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: glusterfs-3.12.2-7 Doc Type: Bug Fix
Doc Text:
Previously, the NFS-Ganesha startup script did not clean up the ganesha processes if the set up failed. Hence, the ganesha processes continues to run on the nodes even if the ‘gluster nfs-ganesha enable’ command failed. With this fix, the ganesha processes will be stopped on the nodes if the ‘gluster nfs-ganesha enable’ command fails and the processes are cleaned up.
 Story Points: ---
Clone Of:
: 1563500 (view as bug list) Environment:
Last Closed: 2018-09-04 06:26:56 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1202842, 1216951, 1503132    

Description Saurabh 2015-06-01 11:14:36 UTC 
Description of problem:
As per the new implementation for 3.1 that we are propsoing is HA by default for nfs-ganesha server. The problem is that there have been cases when pcs cluster setup has failed in some scenarios but the point to note is that nfs-ganesha process has spawned on the nodes, whereas nfs-ganesha should also not spawn in case pcs cluster setup has failed. Otherwise we are not providing HA as default with the setup that we are doing

One of the example scenario is BZ 1225507, there may be others.

Version-Release number of selected component (if applicable):
glusterfs-3.7.0-2.el6rhs.x86_64
nfs-ganesha-2.2.0-0.el6.x86_64

How reproducible:
several times


Expected results:
We need to have a check that decides if the pcs cluster is up only then nfs-ganesha should be spawned.

Additional info:
Comment 4 Divya 2015-06-22 10:50:55 UTC 
Hi,

Please provide the doc text for this known issue in the Doc Text field.
Comment 6 monti lawrence 2015-07-22 20:29:21 UTC 
Doc text is edited. Please sign off to be included in Known Issues.
Comment 7 Soumya Koduri 2015-07-27 09:05:13 UTC 
Doc text looks good to me.
Comment 10 Soumya Koduri 2017-06-28 11:28:14 UTC 
Upstream reference : https://review.gluster.org/#/c/16823/
Downstream reference : https://code.engineering.redhat.com/gerrit/104144
Comment 13 Manisha Saini 2018-03-26 09:20:36 UTC 
Tested this with 

# rpm -qa | grep ganesha
glusterfs-ganesha-3.12.2-5.el7rhgs.x86_64
nfs-ganesha-2.5.5-3.el7rhgs.x86_64
nfs-ganesha-gluster-2.5.5-3.el7rhgs.x86_64


To test the fix,I intentionally gave wrong hostname (ivalid domain) so that ganesha cluster creation fails.But even though ganesha enable command is failed,Ganesha service was running on 2 out of 4 nodes.Even symlinks are not been cleaned on these 2 nodes post setup failure.


========================================

ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=nlm)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=nlm)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=nfs)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=nfs)
ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=nfs)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=nfs)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=rpc-bind)
ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=rpc-bind)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=rpc-bind)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=rpc-bind)
ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=high-availability)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=high-availability)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=high-availability)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=high-availability)
ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=mountd)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=mountd)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=mountd)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=mountd)
ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=rquota)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=rquota)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=rquota)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=rquota)

TASK [Reloads the firewall] ************************************************************************************************************************************************************
changed: [dhcp47-2.lab.eng.blr.redhat.com]
changed: [dhcp46-116.lab.eng.blr.redhat.com]
changed: [dhcp46-184.lab.eng.blr.redhat.com]
changed: [dhcp47-193.lab.eng.blr.redhat.com]

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp46-116.lab.eng.blr.redhat.com : ok=3    changed=1    unreachable=0    failed=0   
dhcp46-184.lab.eng.blr.redhat.com : ok=3    changed=1    unreachable=0    failed=0   
dhcp47-193.lab.eng.blr.redhat.com : ok=3    changed=1    unreachable=0    failed=0   
dhcp47-2.lab.eng.blr.redhat.com : ok=3    changed=1    unreachable=0    failed=0   


PLAY [gluster_servers] *****************************************************************************************************************************************************************

TASK [Start firewalld if not already started] ******************************************************************************************************************************************
ok: [dhcp47-2.lab.eng.blr.redhat.com]
ok: [dhcp46-116.lab.eng.blr.redhat.com]
ok: [dhcp47-193.lab.eng.blr.redhat.com]
ok: [dhcp46-184.lab.eng.blr.redhat.com]

TASK [Open/Close firewalld ports] ******************************************************************************************************************************************************
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=111/tcp)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=111/tcp)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=111/tcp)
ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=111/tcp)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=2049/tcp)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=2049/tcp)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=2049/tcp)
ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=2049/tcp)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=54321/tcp)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=54321/tcp)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=54321/tcp)
ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=54321/tcp)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=5900/tcp)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=5900/tcp)
ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=5900/tcp)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=5900/tcp)
ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=5900-6923/tcp)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=5900-6923/tcp)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=5900-6923/tcp)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=5900-6923/tcp)
ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=5666/tcp)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=5666/tcp)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=5666/tcp)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=5666/tcp)
ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=16514/tcp)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=16514/tcp)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=16514/tcp)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=16514/tcp)

TASK [Reloads the firewall] ************************************************************************************************************************************************************
changed: [dhcp47-2.lab.eng.blr.redhat.com]
changed: [dhcp46-184.lab.eng.blr.redhat.com]
changed: [dhcp46-116.lab.eng.blr.redhat.com]
changed: [dhcp47-193.lab.eng.blr.redhat.com]

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp46-116.lab.eng.blr.redhat.com : ok=3    changed=1    unreachable=0    failed=0   
dhcp46-184.lab.eng.blr.redhat.com : ok=3    changed=1    unreachable=0    failed=0   
dhcp47-193.lab.eng.blr.redhat.com : ok=3    changed=1    unreachable=0    failed=0   
dhcp47-2.lab.eng.blr.redhat.com : ok=3    changed=1    unreachable=0    failed=0   


PLAY [cluster_nodes] *******************************************************************************************************************************************************************

TASK [Check if nfs-ganesha is installed] ***********************************************************************************************************************************************
 [WARNING]: Consider using yum, dnf or zypper module rather than running rpm

changed: [dhcp46-184.lab.eng.blr.redhat.com]
changed: [dhcp46-116.lab.eng.blr.redhat.com]
changed: [dhcp47-2.lab.eng.blr.redhat.com]
changed: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [fail] ****************************************************************************************************************************************************************************
skipping: [dhcp47-193.lab.eng.blr.redhat.com]
skipping: [dhcp46-116.lab.eng.blr.redhat.com]
skipping: [dhcp46-184.lab.eng.blr.redhat.com]
skipping: [dhcp47-2.lab.eng.blr.redhat.com]

TASK [Check if corosync is installed] **************************************************************************************************************************************************
changed: [dhcp46-184.lab.eng.blr.redhat.com]
changed: [dhcp47-193.lab.eng.blr.redhat.com]
changed: [dhcp46-116.lab.eng.blr.redhat.com]
changed: [dhcp47-2.lab.eng.blr.redhat.com]

TASK [fail] ****************************************************************************************************************************************************************************
skipping: [dhcp47-193.lab.eng.blr.redhat.com]
skipping: [dhcp46-116.lab.eng.blr.redhat.com]
skipping: [dhcp46-184.lab.eng.blr.redhat.com]
skipping: [dhcp47-2.lab.eng.blr.redhat.com]

TASK [Check if pacemaker is installed] *************************************************************************************************************************************************
changed: [dhcp46-116.lab.eng.blr.redhat.com]
changed: [dhcp47-2.lab.eng.blr.redhat.com]
changed: [dhcp47-193.lab.eng.blr.redhat.com]
changed: [dhcp46-184.lab.eng.blr.redhat.com]

TASK [fail] ****************************************************************************************************************************************************************************
skipping: [dhcp47-193.lab.eng.blr.redhat.com]
skipping: [dhcp46-116.lab.eng.blr.redhat.com]
skipping: [dhcp46-184.lab.eng.blr.redhat.com]
skipping: [dhcp47-2.lab.eng.blr.redhat.com]

TASK [Check if libntirpc is installed] *************************************************************************************************************************************************
changed: [dhcp46-116.lab.eng.blr.redhat.com]
changed: [dhcp46-184.lab.eng.blr.redhat.com]
changed: [dhcp47-193.lab.eng.blr.redhat.com]
changed: [dhcp47-2.lab.eng.blr.redhat.com]

TASK [fail] ****************************************************************************************************************************************************************************
skipping: [dhcp47-193.lab.eng.blr.redhat.com]
skipping: [dhcp46-116.lab.eng.blr.redhat.com]
skipping: [dhcp46-184.lab.eng.blr.redhat.com]
skipping: [dhcp47-2.lab.eng.blr.redhat.com]

TASK [Check if pcs is installed] *******************************************************************************************************************************************************
changed: [dhcp46-116.lab.eng.blr.redhat.com]
changed: [dhcp47-193.lab.eng.blr.redhat.com]
changed: [dhcp46-184.lab.eng.blr.redhat.com]
changed: [dhcp47-2.lab.eng.blr.redhat.com]

TASK [fail] ****************************************************************************************************************************************************************************
skipping: [dhcp47-193.lab.eng.blr.redhat.com]
skipping: [dhcp46-116.lab.eng.blr.redhat.com]
skipping: [dhcp46-184.lab.eng.blr.redhat.com]
skipping: [dhcp47-2.lab.eng.blr.redhat.com]

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp46-116.lab.eng.blr.redhat.com : ok=5    changed=5    unreachable=0    failed=0   
dhcp46-184.lab.eng.blr.redhat.com : ok=5    changed=5    unreachable=0    failed=0   
dhcp47-193.lab.eng.blr.redhat.com : ok=5    changed=5    unreachable=0    failed=0   
dhcp47-2.lab.eng.blr.redhat.com : ok=5    changed=5    unreachable=0    failed=0   


PLAY [cluster_nodes] *******************************************************************************************************************************************************************

TASK [Stop kernel NFS] *****************************************************************************************************************************************************************
ok: [dhcp47-2.lab.eng.blr.redhat.com]
ok: [dhcp46-116.lab.eng.blr.redhat.com]
ok: [dhcp46-184.lab.eng.blr.redhat.com]
ok: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [Stop network manager service] ****************************************************************************************************************************************************
ok: [dhcp46-184.lab.eng.blr.redhat.com]
ok: [dhcp47-193.lab.eng.blr.redhat.com]
ok: [dhcp46-116.lab.eng.blr.redhat.com]
ok: [dhcp47-2.lab.eng.blr.redhat.com]

TASK [Disable network manager service] *************************************************************************************************************************************************
ok: [dhcp47-193.lab.eng.blr.redhat.com]
ok: [dhcp46-116.lab.eng.blr.redhat.com]
ok: [dhcp47-2.lab.eng.blr.redhat.com]
ok: [dhcp46-184.lab.eng.blr.redhat.com]

TASK [Start network service] ***********************************************************************************************************************************************************
ok: [dhcp47-2.lab.eng.blr.redhat.com]
ok: [dhcp46-184.lab.eng.blr.redhat.com]
ok: [dhcp46-116.lab.eng.blr.redhat.com]
ok: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [Enable network service] **********************************************************************************************************************************************************
ok: [dhcp46-184.lab.eng.blr.redhat.com]
ok: [dhcp47-193.lab.eng.blr.redhat.com]
ok: [dhcp47-2.lab.eng.blr.redhat.com]
ok: [dhcp46-116.lab.eng.blr.redhat.com]

TASK [Start pcsd service] **************************************************************************************************************************************************************
ok: [dhcp47-193.lab.eng.blr.redhat.com]
ok: [dhcp47-2.lab.eng.blr.redhat.com]
ok: [dhcp46-116.lab.eng.blr.redhat.com]
ok: [dhcp46-184.lab.eng.blr.redhat.com]

TASK [Enable pcsd service] *************************************************************************************************************************************************************
ok: [dhcp46-116.lab.eng.blr.redhat.com]
ok: [dhcp47-193.lab.eng.blr.redhat.com]
ok: [dhcp46-184.lab.eng.blr.redhat.com]
ok: [dhcp47-2.lab.eng.blr.redhat.com]

TASK [Enable pacemaker service] ********************************************************************************************************************************************************
changed: [dhcp47-193.lab.eng.blr.redhat.com]
changed: [dhcp46-116.lab.eng.blr.redhat.com]
changed: [dhcp47-2.lab.eng.blr.redhat.com]
changed: [dhcp46-184.lab.eng.blr.redhat.com]

TASK [GlusterFS Share Storage service] *************************************************************************************************************************************************
ok: [dhcp46-116.lab.eng.blr.redhat.com]
ok: [dhcp47-193.lab.eng.blr.redhat.com]
ok: [dhcp47-2.lab.eng.blr.redhat.com]
ok: [dhcp46-184.lab.eng.blr.redhat.com]

TASK [Enable nfs-ganesha service] ******************************************************************************************************************************************************
ok: [dhcp46-116.lab.eng.blr.redhat.com]
ok: [dhcp47-193.lab.eng.blr.redhat.com]
ok: [dhcp47-2.lab.eng.blr.redhat.com]
ok: [dhcp46-184.lab.eng.blr.redhat.com]

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp46-116.lab.eng.blr.redhat.com : ok=10   changed=1    unreachable=0    failed=0   
dhcp46-184.lab.eng.blr.redhat.com : ok=10   changed=1    unreachable=0    failed=0   
dhcp47-193.lab.eng.blr.redhat.com : ok=10   changed=1    unreachable=0    failed=0   
dhcp47-2.lab.eng.blr.redhat.com : ok=10   changed=1    unreachable=0    failed=0   


PLAY [master_node] *********************************************************************************************************************************************************************

TASK [Generate ssh key in one of the nodes in HA cluster] ******************************************************************************************************************************
changed: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [Copy the public key to the local] ************************************************************************************************************************************************
changed: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [Copy the private key to the local] ***********************************************************************************************************************************************
changed: [dhcp47-193.lab.eng.blr.redhat.com]

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp47-193.lab.eng.blr.redhat.com : ok=3    changed=3    unreachable=0    failed=0   


PLAY [gluster_servers] *****************************************************************************************************************************************************************

TASK [Copy the public key to remote nodes] *********************************************************************************************************************************************
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com)
changed: [dhcp46-184.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com)
changed: [dhcp47-2.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com)
changed: [dhcp46-116.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com)

TASK [Copy the private key to remote node] *********************************************************************************************************************************************
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com)
changed: [dhcp46-116.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com)
changed: [dhcp46-184.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com)
changed: [dhcp47-2.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com)

TASK [Deploy the pubkey ~/root/.ssh/authorized_keys on all nodes] **********************************************************************************************************************
changed: [dhcp47-2.lab.eng.blr.redhat.com]
changed: [dhcp46-184.lab.eng.blr.redhat.com]
changed: [dhcp47-193.lab.eng.blr.redhat.com]
changed: [dhcp46-116.lab.eng.blr.redhat.com]

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp46-116.lab.eng.blr.redhat.com : ok=3    changed=3    unreachable=0    failed=0   
dhcp46-184.lab.eng.blr.redhat.com : ok=3    changed=3    unreachable=0    failed=0   
dhcp47-193.lab.eng.blr.redhat.com : ok=3    changed=1    unreachable=0    failed=0   
dhcp47-2.lab.eng.blr.redhat.com : ok=3    changed=3    unreachable=0    failed=0   


PLAY [master_node] *********************************************************************************************************************************************************************

TASK [Make the volume a gluster shared volume] *****************************************************************************************************************************************
changed: [dhcp47-193.lab.eng.blr.redhat.com]

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp47-193.lab.eng.blr.redhat.com : ok=1    changed=1    unreachable=0    failed=0   


PLAY [gluster_servers] *****************************************************************************************************************************************************************

TASK [Create a user hacluster in all the cluster nodes if not already present] *********************************************************************************************************
ok: [dhcp47-2.lab.eng.blr.redhat.com]
ok: [dhcp46-116.lab.eng.blr.redhat.com]
ok: [dhcp47-193.lab.eng.blr.redhat.com]
ok: [dhcp46-184.lab.eng.blr.redhat.com]

TASK [Set the hacluster user the same password on all machines] ************************************************************************************************************************
changed: [dhcp47-2.lab.eng.blr.redhat.com]
changed: [dhcp46-116.lab.eng.blr.redhat.com]
changed: [dhcp46-184.lab.eng.blr.redhat.com]
changed: [dhcp47-193.lab.eng.blr.redhat.com]

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp46-116.lab.eng.blr.redhat.com : ok=2    changed=1    unreachable=0    failed=0   
dhcp46-184.lab.eng.blr.redhat.com : ok=2    changed=1    unreachable=0    failed=0   
dhcp47-193.lab.eng.blr.redhat.com : ok=2    changed=1    unreachable=0    failed=0   
dhcp47-2.lab.eng.blr.redhat.com : ok=2    changed=1    unreachable=0    failed=0   


PLAY [gluster_servers] *****************************************************************************************************************************************************************

TASK [Pcs cluster authenticate the hacluster users in all the nodes] *******************************************************************************************************************
changed: [dhcp47-2.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com)
changed: [dhcp46-116.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com)
changed: [dhcp46-184.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com)
changed: [dhcp47-193.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com)
changed: [dhcp47-2.lab.eng.blr.redhat.com] => (item=dhcp46-116.lab.eng.blr.redhat.com)
changed: [dhcp46-116.lab.eng.blr.redhat.com] => (item=dhcp46-116.lab.eng.blr.redhat.com)
changed: [dhcp46-184.lab.eng.blr.redhat.com] => (item=dhcp46-116.lab.eng.blr.redhat.com)
changed: [dhcp47-193.lab.eng.blr.redhat.com] => (item=dhcp46-116.lab.eng.blr.redhat.com)
changed: [dhcp47-2.lab.eng.blr.redhat.com] => (item=dhcp46-184.lab.eng.blr.redhat.com)
changed: [dhcp46-116.lab.eng.blr.redhat.com] => (item=dhcp46-184.lab.eng.blr.redhat.com)
changed: [dhcp46-184.lab.eng.blr.redhat.com] => (item=dhcp46-184.lab.eng.blr.redhat.com)
changed: [dhcp47-193.lab.eng.blr.redhat.com] => (item=dhcp46-184.lab.eng.blr.redhat.com)
changed: [dhcp47-2.lab.eng.blr.redhat.com] => (item=dhcp47-2.lab.eng.blr.redhat.com)
changed: [dhcp46-116.lab.eng.blr.redhat.com] => (item=dhcp47-2.lab.eng.blr.redhat.com)
changed: [dhcp47-193.lab.eng.blr.redhat.com] => (item=dhcp47-2.lab.eng.blr.redhat.com)
changed: [dhcp46-184.lab.eng.blr.redhat.com] => (item=dhcp47-2.lab.eng.blr.redhat.com)

TASK [Pause for a few seconds] *********************************************************************************************************************************************************
Pausing for 3 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
ok: [dhcp47-193.lab.eng.blr.redhat.com]

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp46-116.lab.eng.blr.redhat.com : ok=1    changed=1    unreachable=0    failed=0   
dhcp46-184.lab.eng.blr.redhat.com : ok=1    changed=1    unreachable=0    failed=0   
dhcp47-193.lab.eng.blr.redhat.com : ok=2    changed=1    unreachable=0    failed=0   
dhcp47-2.lab.eng.blr.redhat.com : ok=1    changed=1    unreachable=0    failed=0   


PLAY [master_node] *********************************************************************************************************************************************************************

TASK [Create configuration directory for nfs-ganesha] **********************************************************************************************************************************
changed: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [Copy ganesha.conf to config directory on shared volume] **************************************************************************************************************************
changed: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [Creating ganesha-ha.conf file] ***************************************************************************************************************************************************
changed: [dhcp47-193.lab.eng.blr.redhat.com]

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp47-193.lab.eng.blr.redhat.com : ok=3    changed=3    unreachable=0    failed=0   


PLAY [gluster_servers] *****************************************************************************************************************************************************************

TASK [Define service port] *************************************************************************************************************************************************************
changed: [dhcp46-116.lab.eng.blr.redhat.com]
changed: [dhcp47-2.lab.eng.blr.redhat.com]
changed: [dhcp47-193.lab.eng.blr.redhat.com]
changed: [dhcp46-184.lab.eng.blr.redhat.com]

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp46-116.lab.eng.blr.redhat.com : ok=1    changed=1    unreachable=0    failed=0   
dhcp46-184.lab.eng.blr.redhat.com : ok=1    changed=1    unreachable=0    failed=0   
dhcp47-193.lab.eng.blr.redhat.com : ok=1    changed=1    unreachable=0    failed=0   
dhcp47-2.lab.eng.blr.redhat.com : ok=1    changed=1    unreachable=0    failed=0   


PLAY [master_node] *********************************************************************************************************************************************************************

TASK [Enable nfs-ganesha] **************************************************************************************************************************************************************
fatal: [dhcp47-193.lab.eng.blr.redhat.com]: FAILED! => {"changed": true, "cmd": "gluster nfs-ganesha enable --mode=script", "delta": "0:00:10.702735", "end": "2018-03-26 12:42:44.520213", "failed_when_result": true, "msg": "non-zero return code", "rc": 1, "start": "2018-03-26 12:42:33.817478", "stderr": "nfs-ganesha: failed: NFS-Ganesha failed to start.Please see log file for details", "stderr_lines": ["nfs-ganesha: failed: NFS-Ganesha failed to start.Please see log file for details"], "stdout": "This will take a few minutes to complete. Please wait ..", "stdout_lines": ["This will take a few minutes to complete. Please wait .."]}
...ignoring

TASK [Pause for 30 seconds (takes a while to enable NFS Ganesha)] **********************************************************************************************************************
Pausing for 30 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
ok: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [Check NFS Ganesha status] ********************************************************************************************************************************************************
changed: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [Report NFS Ganesha status] *******************************************************************************************************************************************************
skipping: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [Report NFS Ganesha status (If any errors)] ***************************************************************************************************************************************
ok: [dhcp47-193.lab.eng.blr.redhat.com] => {
    "msg": "Error: cluster is not currently running on this node"
}

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp47-193.lab.eng.blr.redhat.com : ok=4    changed=2    unreachable=0    failed=0   

[root@dhcp47-193 gdeploy]# vi ganesha_setup.conf 
[root@dhcp47-193 gdeploy]# gdeploy  -c ganesha_setup.conf 

PLAY [gluster_servers] *****************************************************************************************************************************************************************

TASK [Start firewalld if not already started] ******************************************************************************************************************************************
fatal: [dhcp47-2.labb.eng.blr.redhat.com]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true}
ok: [dhcp46-116.lab.eng.blr.redhat.com]
ok: [dhcp46-184.lab.eng.blr.redhat.com]
ok: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [Add/Delete services to firewalld rules] ******************************************************************************************************************************************
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=glusterfs)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=glusterfs)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=glusterfs)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=nlm)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=nlm)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=nlm)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=nfs)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=nfs)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=nfs)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=rpc-bind)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=rpc-bind)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=rpc-bind)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=high-availability)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=high-availability)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=high-availability)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=mountd)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=mountd)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=mountd)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=rquota)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=rquota)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=rquota)

TASK [Reloads the firewall] ************************************************************************************************************************************************************
changed: [dhcp46-184.lab.eng.blr.redhat.com]
changed: [dhcp46-116.lab.eng.blr.redhat.com]
changed: [dhcp47-193.lab.eng.blr.redhat.com]
    to retry, use: --limit @/tmp/tmp0hDpxC/firewalld-service-op.retry

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp46-116.lab.eng.blr.redhat.com : ok=3    changed=1    unreachable=0    failed=0   
dhcp46-184.lab.eng.blr.redhat.com : ok=3    changed=1    unreachable=0    failed=0   
dhcp47-193.lab.eng.blr.redhat.com : ok=3    changed=1    unreachable=0    failed=0   
dhcp47-2.labb.eng.blr.redhat.com : ok=0    changed=0    unreachable=1    failed=0   

Ignoring errors...

PLAY [gluster_servers] *****************************************************************************************************************************************************************

TASK [Start firewalld if not already started] ******************************************************************************************************************************************
fatal: [dhcp47-2.labb.eng.blr.redhat.com]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true}
ok: [dhcp47-193.lab.eng.blr.redhat.com]
ok: [dhcp46-184.lab.eng.blr.redhat.com]
ok: [dhcp46-116.lab.eng.blr.redhat.com]

TASK [Open/Close firewalld ports] ******************************************************************************************************************************************************
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=111/tcp)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=111/tcp)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=111/tcp)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=2049/tcp)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=2049/tcp)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=2049/tcp)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=54321/tcp)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=54321/tcp)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=54321/tcp)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=5900/tcp)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=5900/tcp)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=5900/tcp)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=5900-6923/tcp)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=5900-6923/tcp)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=5900-6923/tcp)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=5666/tcp)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=5666/tcp)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=5666/tcp)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=16514/tcp)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=16514/tcp)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=16514/tcp)

TASK [Reloads the firewall] ************************************************************************************************************************************************************
changed: [dhcp46-116.lab.eng.blr.redhat.com]
changed: [dhcp46-184.lab.eng.blr.redhat.com]
changed: [dhcp47-193.lab.eng.blr.redhat.com]
    to retry, use: --limit @/tmp/tmp0hDpxC/firewalld-ports-op.retry

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp46-116.lab.eng.blr.redhat.com : ok=3    changed=1    unreachable=0    failed=0   
dhcp46-184.lab.eng.blr.redhat.com : ok=3    changed=1    unreachable=0    failed=0   
dhcp47-193.lab.eng.blr.redhat.com : ok=3    changed=1    unreachable=0    failed=0   
dhcp47-2.labb.eng.blr.redhat.com : ok=0    changed=0    unreachable=1    failed=0   

Ignoring errors...

PLAY [cluster_nodes] *******************************************************************************************************************************************************************

TASK [Check if nfs-ganesha is installed] ***********************************************************************************************************************************************
fatal: [dhcp47-2.labb.eng.blr.redhat.com]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true}
 [WARNING]: Consider using yum, dnf or zypper module rather than running rpm

changed: [dhcp46-116.lab.eng.blr.redhat.com]
changed: [dhcp47-193.lab.eng.blr.redhat.com]
changed: [dhcp46-184.lab.eng.blr.redhat.com]

TASK [fail] ****************************************************************************************************************************************************************************
skipping: [dhcp47-193.lab.eng.blr.redhat.com]
skipping: [dhcp46-116.lab.eng.blr.redhat.com]
skipping: [dhcp46-184.lab.eng.blr.redhat.com]

TASK [Check if corosync is installed] **************************************************************************************************************************************************
changed: [dhcp47-193.lab.eng.blr.redhat.com]
changed: [dhcp46-116.lab.eng.blr.redhat.com]
changed: [dhcp46-184.lab.eng.blr.redhat.com]

TASK [fail] ****************************************************************************************************************************************************************************
skipping: [dhcp47-193.lab.eng.blr.redhat.com]
skipping: [dhcp46-116.lab.eng.blr.redhat.com]
skipping: [dhcp46-184.lab.eng.blr.redhat.com]

TASK [Check if pacemaker is installed] *************************************************************************************************************************************************
changed: [dhcp46-116.lab.eng.blr.redhat.com]
changed: [dhcp47-193.lab.eng.blr.redhat.com]
changed: [dhcp46-184.lab.eng.blr.redhat.com]

TASK [fail] ****************************************************************************************************************************************************************************
skipping: [dhcp47-193.lab.eng.blr.redhat.com]
skipping: [dhcp46-116.lab.eng.blr.redhat.com]
skipping: [dhcp46-184.lab.eng.blr.redhat.com]

TASK [Check if libntirpc is installed] *************************************************************************************************************************************************
changed: [dhcp46-116.lab.eng.blr.redhat.com]
changed: [dhcp47-193.lab.eng.blr.redhat.com]
changed: [dhcp46-184.lab.eng.blr.redhat.com]

TASK [fail] ****************************************************************************************************************************************************************************
skipping: [dhcp47-193.lab.eng.blr.redhat.com]
skipping: [dhcp46-116.lab.eng.blr.redhat.com]
skipping: [dhcp46-184.lab.eng.blr.redhat.com]

TASK [Check if pcs is installed] *******************************************************************************************************************************************************
changed: [dhcp46-116.lab.eng.blr.redhat.com]
changed: [dhcp47-193.lab.eng.blr.redhat.com]
changed: [dhcp46-184.lab.eng.blr.redhat.com]

TASK [fail] ****************************************************************************************************************************************************************************
skipping: [dhcp47-193.lab.eng.blr.redhat.com]
skipping: [dhcp46-116.lab.eng.blr.redhat.com]
skipping: [dhcp46-184.lab.eng.blr.redhat.com]
    to retry, use: --limit @/tmp/tmp0hDpxC/check_package_installs_ganesha.retry

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp46-116.lab.eng.blr.redhat.com : ok=5    changed=5    unreachable=0    failed=0   
dhcp46-184.lab.eng.blr.redhat.com : ok=5    changed=5    unreachable=0    failed=0   
dhcp47-193.lab.eng.blr.redhat.com : ok=5    changed=5    unreachable=0    failed=0   
dhcp47-2.labb.eng.blr.redhat.com : ok=0    changed=0    unreachable=1    failed=0   

Ignoring errors...

PLAY [cluster_nodes] *******************************************************************************************************************************************************************

TASK [Stop kernel NFS] *****************************************************************************************************************************************************************
fatal: [dhcp47-2.labb.eng.blr.redhat.com]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true}
ok: [dhcp46-116.lab.eng.blr.redhat.com]
ok: [dhcp46-184.lab.eng.blr.redhat.com]
ok: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [Stop network manager service] ****************************************************************************************************************************************************
ok: [dhcp46-116.lab.eng.blr.redhat.com]
ok: [dhcp47-193.lab.eng.blr.redhat.com]
ok: [dhcp46-184.lab.eng.blr.redhat.com]

TASK [Disable network manager service] *************************************************************************************************************************************************
ok: [dhcp46-116.lab.eng.blr.redhat.com]
ok: [dhcp47-193.lab.eng.blr.redhat.com]
ok: [dhcp46-184.lab.eng.blr.redhat.com]

TASK [Start network service] ***********************************************************************************************************************************************************
ok: [dhcp46-116.lab.eng.blr.redhat.com]
ok: [dhcp47-193.lab.eng.blr.redhat.com]
ok: [dhcp46-184.lab.eng.blr.redhat.com]

TASK [Enable network service] **********************************************************************************************************************************************************
ok: [dhcp46-116.lab.eng.blr.redhat.com]
ok: [dhcp46-184.lab.eng.blr.redhat.com]
ok: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [Start pcsd service] **************************************************************************************************************************************************************
ok: [dhcp46-116.lab.eng.blr.redhat.com]
ok: [dhcp46-184.lab.eng.blr.redhat.com]
ok: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [Enable pcsd service] *************************************************************************************************************************************************************
ok: [dhcp46-116.lab.eng.blr.redhat.com]
ok: [dhcp47-193.lab.eng.blr.redhat.com]
ok: [dhcp46-184.lab.eng.blr.redhat.com]

TASK [Enable pacemaker service] ********************************************************************************************************************************************************
changed: [dhcp46-116.lab.eng.blr.redhat.com]
changed: [dhcp47-193.lab.eng.blr.redhat.com]
changed: [dhcp46-184.lab.eng.blr.redhat.com]

TASK [GlusterFS Share Storage service] *************************************************************************************************************************************************
ok: [dhcp46-116.lab.eng.blr.redhat.com]
ok: [dhcp47-193.lab.eng.blr.redhat.com]
ok: [dhcp46-184.lab.eng.blr.redhat.com]

TASK [Enable nfs-ganesha service] ******************************************************************************************************************************************************
ok: [dhcp46-116.lab.eng.blr.redhat.com]
ok: [dhcp47-193.lab.eng.blr.redhat.com]
ok: [dhcp46-184.lab.eng.blr.redhat.com]
    to retry, use: --limit @/tmp/tmp0hDpxC/configure-services.retry

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp46-116.lab.eng.blr.redhat.com : ok=10   changed=1    unreachable=0    failed=0   
dhcp46-184.lab.eng.blr.redhat.com : ok=10   changed=1    unreachable=0    failed=0   
dhcp47-193.lab.eng.blr.redhat.com : ok=10   changed=1    unreachable=0    failed=0   
dhcp47-2.labb.eng.blr.redhat.com : ok=0    changed=0    unreachable=1    failed=0   

Ignoring errors...

PLAY [master_node] *********************************************************************************************************************************************************************

TASK [Generate ssh key in one of the nodes in HA cluster] ******************************************************************************************************************************
changed: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [Copy the public key to the local] ************************************************************************************************************************************************
changed: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [Copy the private key to the local] ***********************************************************************************************************************************************
changed: [dhcp47-193.lab.eng.blr.redhat.com]

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp47-193.lab.eng.blr.redhat.com : ok=3    changed=3    unreachable=0    failed=0   


PLAY [gluster_servers] *****************************************************************************************************************************************************************

TASK [Copy the public key to remote nodes] *********************************************************************************************************************************************
failed: [dhcp47-2.labb.eng.blr.redhat.com] (item=dhcp47-193.lab.eng.blr.redhat.com) => {"item": "dhcp47-193.lab.eng.blr.redhat.com", "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true}
fatal: [dhcp47-2.labb.eng.blr.redhat.com]: UNREACHABLE! => {"changed": false, "msg": "All items completed", "results": [{"_ansible_ignore_errors": null, "_ansible_item_result": true, "item": "dhcp47-193.lab.eng.blr.redhat.com", "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true}]}
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com)
changed: [dhcp46-116.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com)
changed: [dhcp46-184.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com)

TASK [Copy the private key to remote node] *********************************************************************************************************************************************
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com)
changed: [dhcp46-116.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com)
changed: [dhcp46-184.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com)

TASK [Deploy the pubkey ~/root/.ssh/authorized_keys on all nodes] **********************************************************************************************************************
changed: [dhcp46-116.lab.eng.blr.redhat.com]
changed: [dhcp47-193.lab.eng.blr.redhat.com]
changed: [dhcp46-184.lab.eng.blr.redhat.com]
    to retry, use: --limit @/tmp/tmp0hDpxC/copy-ssh-key.retry

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp46-116.lab.eng.blr.redhat.com : ok=3    changed=3    unreachable=0    failed=0   
dhcp46-184.lab.eng.blr.redhat.com : ok=3    changed=3    unreachable=0    failed=0   
dhcp47-193.lab.eng.blr.redhat.com : ok=3    changed=1    unreachable=0    failed=0   
dhcp47-2.labb.eng.blr.redhat.com : ok=0    changed=0    unreachable=1    failed=0   

Ignoring errors...

PLAY [master_node] *********************************************************************************************************************************************************************

TASK [Make the volume a gluster shared volume] *****************************************************************************************************************************************
changed: [dhcp47-193.lab.eng.blr.redhat.com]

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp47-193.lab.eng.blr.redhat.com : ok=1    changed=1    unreachable=0    failed=0   


PLAY [gluster_servers] *****************************************************************************************************************************************************************

TASK [Create a user hacluster in all the cluster nodes if not already present] *********************************************************************************************************
fatal: [dhcp47-2.labb.eng.blr.redhat.com]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true}
ok: [dhcp46-184.lab.eng.blr.redhat.com]
ok: [dhcp46-116.lab.eng.blr.redhat.com]
ok: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [Set the hacluster user the same password on all machines] ************************************************************************************************************************
changed: [dhcp46-116.lab.eng.blr.redhat.com]
changed: [dhcp47-193.lab.eng.blr.redhat.com]
changed: [dhcp46-184.lab.eng.blr.redhat.com]
    to retry, use: --limit @/tmp/tmp0hDpxC/set-pcs-auth-passwd.retry

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp46-116.lab.eng.blr.redhat.com : ok=2    changed=1    unreachable=0    failed=0   
dhcp46-184.lab.eng.blr.redhat.com : ok=2    changed=1    unreachable=0    failed=0   
dhcp47-193.lab.eng.blr.redhat.com : ok=2    changed=1    unreachable=0    failed=0   
dhcp47-2.labb.eng.blr.redhat.com : ok=0    changed=0    unreachable=1    failed=0   

Ignoring errors...

PLAY [gluster_servers] *****************************************************************************************************************************************************************

TASK [Pcs cluster authenticate the hacluster users in all the nodes] *******************************************************************************************************************
failed: [dhcp47-2.labb.eng.blr.redhat.com] (item=dhcp47-193.lab.eng.blr.redhat.com) => {"item": "dhcp47-193.lab.eng.blr.redhat.com", "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true}
failed: [dhcp47-2.labb.eng.blr.redhat.com] (item=dhcp46-116.lab.eng.blr.redhat.com) => {"item": "dhcp46-116.lab.eng.blr.redhat.com", "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true}
failed: [dhcp47-2.labb.eng.blr.redhat.com] (item=dhcp46-184.lab.eng.blr.redhat.com) => {"item": "dhcp46-184.lab.eng.blr.redhat.com", "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true}
failed: [dhcp47-2.labb.eng.blr.redhat.com] (item=dhcp47-2.labb.eng.blr.redhat.com) => {"item": "dhcp47-2.labb.eng.blr.redhat.com", "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true}
fatal: [dhcp47-2.labb.eng.blr.redhat.com]: UNREACHABLE! => {"changed": false, "msg": "All items completed", "results": [{"_ansible_ignore_errors": null, "_ansible_item_result": true, "item": "dhcp47-193.lab.eng.blr.redhat.com", "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true}, {"_ansible_ignore_errors": null, "_ansible_item_result": true, "item": "dhcp46-116.lab.eng.blr.redhat.com", "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true}, {"_ansible_ignore_errors": null, "_ansible_item_result": true, "item": "dhcp46-184.lab.eng.blr.redhat.com", "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true}, {"_ansible_ignore_errors": null, "_ansible_item_result": true, "item": "dhcp47-2.labb.eng.blr.redhat.com", "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true}]}
changed: [dhcp46-116.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com)
changed: [dhcp46-184.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com)
changed: [dhcp47-193.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com)
changed: [dhcp46-116.lab.eng.blr.redhat.com] => (item=dhcp46-116.lab.eng.blr.redhat.com)
changed: [dhcp46-184.lab.eng.blr.redhat.com] => (item=dhcp46-116.lab.eng.blr.redhat.com)
changed: [dhcp47-193.lab.eng.blr.redhat.com] => (item=dhcp46-116.lab.eng.blr.redhat.com)
changed: [dhcp46-116.lab.eng.blr.redhat.com] => (item=dhcp46-184.lab.eng.blr.redhat.com)
changed: [dhcp46-184.lab.eng.blr.redhat.com] => (item=dhcp46-184.lab.eng.blr.redhat.com)
changed: [dhcp47-193.lab.eng.blr.redhat.com] => (item=dhcp46-184.lab.eng.blr.redhat.com)
failed: [dhcp46-116.lab.eng.blr.redhat.com] (item=dhcp47-2.labb.eng.blr.redhat.com) => {"changed": true, "cmd": "pcs cluster auth -u hacluster -p hacluster dhcp47-2.labb.eng.blr.redhat.com", "delta": "0:00:00.827062", "end": "2018-03-26 12:55:42.810885", "item": "dhcp47-2.labb.eng.blr.redhat.com", "msg": "non-zero return code", "rc": 1, "start": "2018-03-26 12:55:41.983823", "stderr": "Error: Unable to communicate with dhcp47-2.labb.eng.blr.redhat.com", "stderr_lines": ["Error: Unable to communicate with dhcp47-2.labb.eng.blr.redhat.com"], "stdout": "", "stdout_lines": []}
failed: [dhcp46-184.lab.eng.blr.redhat.com] (item=dhcp47-2.labb.eng.blr.redhat.com) => {"changed": true, "cmd": "pcs cluster auth -u hacluster -p hacluster dhcp47-2.labb.eng.blr.redhat.com", "delta": "0:00:00.934702", "end": "2018-03-26 12:55:43.317563", "item": "dhcp47-2.labb.eng.blr.redhat.com", "msg": "non-zero return code", "rc": 1, "start": "2018-03-26 12:55:42.382861", "stderr": "Error: Unable to communicate with dhcp47-2.labb.eng.blr.redhat.com", "stderr_lines": ["Error: Unable to communicate with dhcp47-2.labb.eng.blr.redhat.com"], "stdout": "", "stdout_lines": []}
failed: [dhcp47-193.lab.eng.blr.redhat.com] (item=dhcp47-2.labb.eng.blr.redhat.com) => {"changed": true, "cmd": "pcs cluster auth -u hacluster -p hacluster dhcp47-2.labb.eng.blr.redhat.com", "delta": "0:00:01.003482", "end": "2018-03-26 12:55:43.387661", "item": "dhcp47-2.labb.eng.blr.redhat.com", "msg": "non-zero return code", "rc": 1, "start": "2018-03-26 12:55:42.384179", "stderr": "Error: Unable to communicate with dhcp47-2.labb.eng.blr.redhat.com", "stderr_lines": ["Error: Unable to communicate with dhcp47-2.labb.eng.blr.redhat.com"], "stdout": "", "stdout_lines": []}
    to retry, use: --limit @/tmp/tmp0hDpxC/pcs-authentication.retry

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp46-116.lab.eng.blr.redhat.com : ok=0    changed=0    unreachable=0    failed=1   
dhcp46-184.lab.eng.blr.redhat.com : ok=0    changed=0    unreachable=0    failed=1   
dhcp47-193.lab.eng.blr.redhat.com : ok=0    changed=0    unreachable=0    failed=1   
dhcp47-2.labb.eng.blr.redhat.com : ok=0    changed=0    unreachable=1    failed=0   

Ignoring errors...

PLAY [master_node] *********************************************************************************************************************************************************************

TASK [Create configuration directory for nfs-ganesha] **********************************************************************************************************************************
changed: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [Copy ganesha.conf to config directory on shared volume] **************************************************************************************************************************
changed: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [Creating ganesha-ha.conf file] ***************************************************************************************************************************************************
changed: [dhcp47-193.lab.eng.blr.redhat.com]

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp47-193.lab.eng.blr.redhat.com : ok=3    changed=3    unreachable=0    failed=0   


PLAY [gluster_servers] *****************************************************************************************************************************************************************

TASK [Define service port] *************************************************************************************************************************************************************
fatal: [dhcp47-2.labb.eng.blr.redhat.com]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true}
changed: [dhcp46-116.lab.eng.blr.redhat.com]
changed: [dhcp47-193.lab.eng.blr.redhat.com]
changed: [dhcp46-184.lab.eng.blr.redhat.com]
    to retry, use: --limit @/tmp/tmp0hDpxC/define_service_ports.retry

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp46-116.lab.eng.blr.redhat.com : ok=1    changed=1    unreachable=0    failed=0   
dhcp46-184.lab.eng.blr.redhat.com : ok=1    changed=1    unreachable=0    failed=0   
dhcp47-193.lab.eng.blr.redhat.com : ok=1    changed=1    unreachable=0    failed=0   
dhcp47-2.labb.eng.blr.redhat.com : ok=0    changed=0    unreachable=1    failed=0   

Ignoring errors...

PLAY [master_node] *********************************************************************************************************************************************************************

TASK [Enable nfs-ganesha] **************************************************************************************************************************************************************
fatal: [dhcp47-193.lab.eng.blr.redhat.com]: FAILED! => {"changed": true, "cmd": "gluster nfs-ganesha enable --mode=script", "delta": "0:00:49.946201", "end": "2018-03-26 12:56:44.035225", "failed_when_result": true, "msg": "non-zero return code", "rc": 1, "start": "2018-03-26 12:55:54.089024", "stderr": "nfs-ganesha: failed: Failed to set up HA config for NFS-Ganesha. Please check the log file for details", "stderr_lines": ["nfs-ganesha: failed: Failed to set up HA config for NFS-Ganesha. Please check the log file for details"], "stdout": "This will take a few minutes to complete. Please wait ..", "stdout_lines": ["This will take a few minutes to complete. Please wait .."]}
...ignoring

TASK [Pause for 30 seconds (takes a while to enable NFS Ganesha)] **********************************************************************************************************************
Pausing for 30 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
ok: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [Check NFS Ganesha status] ********************************************************************************************************************************************************
changed: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [Report NFS Ganesha status] *******************************************************************************************************************************************************
skipping: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [Report NFS Ganesha status (If any errors)] ***************************************************************************************************************************************
ok: [dhcp47-193.lab.eng.blr.redhat.com] => {
    "msg": "Error: cluster is not currently running on this node"
}

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp47-193.lab.eng.blr.redhat.com : ok=4    changed=2    unreachable=0    failed=0   


=========================================



ganesha-ha.conf


[root@dhcp46-116 ganesha]# cat /var/run/gluster/shared_storage/nfs-ganesha/ganesha-ha.conf
# Provide a unique name for the cluster.
HA_NAME="ganesha-ha-360"

# The subset of nodes of the Gluster Trusted Storage Pool that forms the ganesha
# HA cluster. Hostname should specified, IP addresses are not allowed.
# Maximum number of 16 nodes are supported.

HA_CLUSTER_NODES="dhcp47-193.lab.eng.blr.redhat.com,dhcp46-116.lab.eng.blr.redhat.com,dhcp46-184.lab.eng.blr.redhat.com,dhcp47-2.labb.eng.blr.redhat.com"

# Virtual IPs of each of the nodes specified above.
VIP_dhcp47-193.lab.eng.blr.redhat.com="10.70.46.51"
VIP_dhcp46-116.lab.eng.blr.redhat.com="10.70.46.54"
VIP_dhcp46-184.lab.eng.blr.redhat.com="10.70.46.55"
VIP_dhcp47-2.labb.eng.blr.redhat.com="10.70.46.56"


==========================================



# service nfs-ganesha status
Redirecting to /bin/systemctl status nfs-ganesha.service
● nfs-ganesha.service - NFS-Ganesha file server
   Loaded: loaded (/usr/lib/systemd/system/nfs-ganesha.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2018-03-26 12:56:16 IST; 1h 50min ago
     Docs: http://github.com/nfs-ganesha/nfs-ganesha/wiki
  Process: 4001 ExecStartPost=/bin/bash -c /usr/bin/sleep 2 && /bin/dbus-send --system   --dest=org.ganesha.nfsd --type=method_call /org/ganesha/nfsd/admin  org.ganesha.nfsd.admin.init_fds_limit (code=exited, status=0/SUCCESS)
  Process: 3997 ExecStartPost=/bin/bash -c prlimit --pid $MAINPID --nofile=$NOFILE:$NOFILE (code=exited, status=0/SUCCESS)
  Process: 3995 ExecStart=/bin/bash -c ${NUMACTL} ${NUMAOPTS} /usr/bin/ganesha.nfsd ${OPTIONS} ${EPOCH} (code=exited, status=0/SUCCESS)
 Main PID: 3996 (ganesha.nfsd)
    Tasks: 272
   CGroup: /system.slice/nfs-ganesha.service
           └─3996 /usr/bin/ganesha.nfsd -L /var/log/ganesha/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT -E 6537151427721232384

Mar 26 12:56:14 dhcp46-116.lab.eng.blr.redhat.com systemd[1]: Starting NFS-Ganesha file server...
Mar 26 12:56:16 dhcp46-116.lab.eng.blr.redhat.com systemd[1]: Started NFS-Ganesha file server.



Since the nfs-ganesha service is observed been running post setup creation fails on 2 out of 4 nodes and even symlinks are not cleaned.Based on these 2 observation moving this BZ to Assigned state.
Comment 18 Manisha Saini 2018-04-05 14:30:18 UTC 
Verified this BZ on 

# rpm -qa | grep ganesha
glusterfs-ganesha-3.12.2-7.el7rhgs.x86_64
nfs-ganesha-2.5.5-3.el7rhgs.x86_64
nfs-ganesha-gluster-2.5.5-3.el7rhgs.x86_64

Intentionally tried failing the "gluster nfs-ganesha enable" command.The ganesha service was not found running.Works as expected.
Moving this BZ to verified state.

Have raised a separate Bug[1] for the use case in comment #13.This needs to be fixed in Gdeploy.

[1]https://bugzilla.redhat.com/show_bug.cgi?id=1563923
Comment 24 errata-xmlrpc 2018-09-04 06:26:56 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2607