Bug 1226874 - nfs-ganesha: in case pcs cluster setup fails then nfs-ganesha process should not start
Summary: nfs-ganesha: in case pcs cluster setup fails then nfs-ganesha process should ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: common-ha
Version: rhgs-3.1
Hardware: x86_64
OS: Linux
high
urgent
Target Milestone: ---
: RHGS 3.4.0
Assignee: Jiffin
QA Contact: Manisha Saini
URL:
Whiteboard:
Depends On:
Blocks: 1202842 1216951 1503132
TreeView+ depends on / blocked
 
Reported: 2015-06-01 11:14 UTC by Saurabh
Modified: 2018-09-24 09:53 UTC (History)
13 users (show)

Fixed In Version: glusterfs-3.12.2-7
Doc Type: Bug Fix
Doc Text:
Previously, the NFS-Ganesha startup script did not clean up the ganesha processes if the set up failed. Hence, the ganesha processes continues to run on the nodes even if the ‘gluster nfs-ganesha enable’ command failed. With this fix, the ganesha processes will be stopped on the nodes if the ‘gluster nfs-ganesha enable’ command fails and the processes are cleaned up.
Clone Of:
: 1563500 (view as bug list)
Environment:
Last Closed: 2018-09-04 06:26:56 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2018:2607 None None None 2018-09-04 06:28:39 UTC

Description Saurabh 2015-06-01 11:14:36 UTC
Description of problem:
As per the new implementation for 3.1 that we are propsoing is HA by default for nfs-ganesha server. The problem is that there have been cases when pcs cluster setup has failed in some scenarios but the point to note is that nfs-ganesha process has spawned on the nodes, whereas nfs-ganesha should also not spawn in case pcs cluster setup has failed. Otherwise we are not providing HA as default with the setup that we are doing

One of the example scenario is BZ 1225507, there may be others.

Version-Release number of selected component (if applicable):
glusterfs-3.7.0-2.el6rhs.x86_64
nfs-ganesha-2.2.0-0.el6.x86_64

How reproducible:
several times


Expected results:
We need to have a check that decides if the pcs cluster is up only then nfs-ganesha should be spawned.

Additional info:

Comment 4 Divya 2015-06-22 10:50:55 UTC
Hi,

Please provide the doc text for this known issue in the Doc Text field.

Comment 6 monti lawrence 2015-07-22 20:29:21 UTC
Doc text is edited. Please sign off to be included in Known Issues.

Comment 7 Soumya Koduri 2015-07-27 09:05:13 UTC
Doc text looks good to me.

Comment 10 Soumya Koduri 2017-06-28 11:28:14 UTC
Upstream reference : https://review.gluster.org/#/c/16823/
Downstream reference : https://code.engineering.redhat.com/gerrit/104144

Comment 13 Manisha Saini 2018-03-26 09:20:36 UTC
Tested this with 

# rpm -qa | grep ganesha
glusterfs-ganesha-3.12.2-5.el7rhgs.x86_64
nfs-ganesha-2.5.5-3.el7rhgs.x86_64
nfs-ganesha-gluster-2.5.5-3.el7rhgs.x86_64


To test the fix,I intentionally gave wrong hostname (ivalid domain) so that ganesha cluster creation fails.But even though ganesha enable command is failed,Ganesha service was running on 2 out of 4 nodes.Even symlinks are not been cleaned on these 2 nodes post setup failure.


========================================

ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=nlm)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=nlm)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=nfs)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=nfs)
ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=nfs)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=nfs)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=rpc-bind)
ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=rpc-bind)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=rpc-bind)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=rpc-bind)
ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=high-availability)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=high-availability)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=high-availability)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=high-availability)
ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=mountd)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=mountd)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=mountd)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=mountd)
ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=rquota)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=rquota)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=rquota)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=rquota)

TASK [Reloads the firewall] ************************************************************************************************************************************************************
changed: [dhcp47-2.lab.eng.blr.redhat.com]
changed: [dhcp46-116.lab.eng.blr.redhat.com]
changed: [dhcp46-184.lab.eng.blr.redhat.com]
changed: [dhcp47-193.lab.eng.blr.redhat.com]

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp46-116.lab.eng.blr.redhat.com : ok=3    changed=1    unreachable=0    failed=0   
dhcp46-184.lab.eng.blr.redhat.com : ok=3    changed=1    unreachable=0    failed=0   
dhcp47-193.lab.eng.blr.redhat.com : ok=3    changed=1    unreachable=0    failed=0   
dhcp47-2.lab.eng.blr.redhat.com : ok=3    changed=1    unreachable=0    failed=0   


PLAY [gluster_servers] *****************************************************************************************************************************************************************

TASK [Start firewalld if not already started] ******************************************************************************************************************************************
ok: [dhcp47-2.lab.eng.blr.redhat.com]
ok: [dhcp46-116.lab.eng.blr.redhat.com]
ok: [dhcp47-193.lab.eng.blr.redhat.com]
ok: [dhcp46-184.lab.eng.blr.redhat.com]

TASK [Open/Close firewalld ports] ******************************************************************************************************************************************************
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=111/tcp)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=111/tcp)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=111/tcp)
ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=111/tcp)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=2049/tcp)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=2049/tcp)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=2049/tcp)
ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=2049/tcp)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=54321/tcp)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=54321/tcp)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=54321/tcp)
ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=54321/tcp)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=5900/tcp)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=5900/tcp)
ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=5900/tcp)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=5900/tcp)
ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=5900-6923/tcp)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=5900-6923/tcp)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=5900-6923/tcp)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=5900-6923/tcp)
ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=5666/tcp)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=5666/tcp)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=5666/tcp)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=5666/tcp)
ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=16514/tcp)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=16514/tcp)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=16514/tcp)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=16514/tcp)

TASK [Reloads the firewall] ************************************************************************************************************************************************************
changed: [dhcp47-2.lab.eng.blr.redhat.com]
changed: [dhcp46-184.lab.eng.blr.redhat.com]
changed: [dhcp46-116.lab.eng.blr.redhat.com]
changed: [dhcp47-193.lab.eng.blr.redhat.com]

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp46-116.lab.eng.blr.redhat.com : ok=3    changed=1    unreachable=0    failed=0   
dhcp46-184.lab.eng.blr.redhat.com : ok=3    changed=1    unreachable=0    failed=0   
dhcp47-193.lab.eng.blr.redhat.com : ok=3    changed=1    unreachable=0    failed=0   
dhcp47-2.lab.eng.blr.redhat.com : ok=3    changed=1    unreachable=0    failed=0   


PLAY [cluster_nodes] *******************************************************************************************************************************************************************

TASK [Check if nfs-ganesha is installed] ***********************************************************************************************************************************************
 [WARNING]: Consider using yum, dnf or zypper module rather than running rpm

changed: [dhcp46-184.lab.eng.blr.redhat.com]
changed: [dhcp46-116.lab.eng.blr.redhat.com]
changed: [dhcp47-2.lab.eng.blr.redhat.com]
changed: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [fail] ****************************************************************************************************************************************************************************
skipping: [dhcp47-193.lab.eng.blr.redhat.com]
skipping: [dhcp46-116.lab.eng.blr.redhat.com]
skipping: [dhcp46-184.lab.eng.blr.redhat.com]
skipping: [dhcp47-2.lab.eng.blr.redhat.com]

TASK [Check if corosync is installed] **************************************************************************************************************************************************
changed: [dhcp46-184.lab.eng.blr.redhat.com]
changed: [dhcp47-193.lab.eng.blr.redhat.com]
changed: [dhcp46-116.lab.eng.blr.redhat.com]
changed: [dhcp47-2.lab.eng.blr.redhat.com]

TASK [fail] ****************************************************************************************************************************************************************************
skipping: [dhcp47-193.lab.eng.blr.redhat.com]
skipping: [dhcp46-116.lab.eng.blr.redhat.com]
skipping: [dhcp46-184.lab.eng.blr.redhat.com]
skipping: [dhcp47-2.lab.eng.blr.redhat.com]

TASK [Check if pacemaker is installed] *************************************************************************************************************************************************
changed: [dhcp46-116.lab.eng.blr.redhat.com]
changed: [dhcp47-2.lab.eng.blr.redhat.com]
changed: [dhcp47-193.lab.eng.blr.redhat.com]
changed: [dhcp46-184.lab.eng.blr.redhat.com]

TASK [fail] ****************************************************************************************************************************************************************************
skipping: [dhcp47-193.lab.eng.blr.redhat.com]
skipping: [dhcp46-116.lab.eng.blr.redhat.com]
skipping: [dhcp46-184.lab.eng.blr.redhat.com]
skipping: [dhcp47-2.lab.eng.blr.redhat.com]

TASK [Check if libntirpc is installed] *************************************************************************************************************************************************
changed: [dhcp46-116.lab.eng.blr.redhat.com]
changed: [dhcp46-184.lab.eng.blr.redhat.com]
changed: [dhcp47-193.lab.eng.blr.redhat.com]
changed: [dhcp47-2.lab.eng.blr.redhat.com]

TASK [fail] ****************************************************************************************************************************************************************************
skipping: [dhcp47-193.lab.eng.blr.redhat.com]
skipping: [dhcp46-116.lab.eng.blr.redhat.com]
skipping: [dhcp46-184.lab.eng.blr.redhat.com]
skipping: [dhcp47-2.lab.eng.blr.redhat.com]

TASK [Check if pcs is installed] *******************************************************************************************************************************************************
changed: [dhcp46-116.lab.eng.blr.redhat.com]
changed: [dhcp47-193.lab.eng.blr.redhat.com]
changed: [dhcp46-184.lab.eng.blr.redhat.com]
changed: [dhcp47-2.lab.eng.blr.redhat.com]

TASK [fail] ****************************************************************************************************************************************************************************
skipping: [dhcp47-193.lab.eng.blr.redhat.com]
skipping: [dhcp46-116.lab.eng.blr.redhat.com]
skipping: [dhcp46-184.lab.eng.blr.redhat.com]
skipping: [dhcp47-2.lab.eng.blr.redhat.com]

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp46-116.lab.eng.blr.redhat.com : ok=5    changed=5    unreachable=0    failed=0   
dhcp46-184.lab.eng.blr.redhat.com : ok=5    changed=5    unreachable=0    failed=0   
dhcp47-193.lab.eng.blr.redhat.com : ok=5    changed=5    unreachable=0    failed=0   
dhcp47-2.lab.eng.blr.redhat.com : ok=5    changed=5    unreachable=0    failed=0   


PLAY [cluster_nodes] *******************************************************************************************************************************************************************

TASK [Stop kernel NFS] *****************************************************************************************************************************************************************
ok: [dhcp47-2.lab.eng.blr.redhat.com]
ok: [dhcp46-116.lab.eng.blr.redhat.com]
ok: [dhcp46-184.lab.eng.blr.redhat.com]
ok: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [Stop network manager service] ****************************************************************************************************************************************************
ok: [dhcp46-184.lab.eng.blr.redhat.com]
ok: [dhcp47-193.lab.eng.blr.redhat.com]
ok: [dhcp46-116.lab.eng.blr.redhat.com]
ok: [dhcp47-2.lab.eng.blr.redhat.com]

TASK [Disable network manager service] *************************************************************************************************************************************************
ok: [dhcp47-193.lab.eng.blr.redhat.com]
ok: [dhcp46-116.lab.eng.blr.redhat.com]
ok: [dhcp47-2.lab.eng.blr.redhat.com]
ok: [dhcp46-184.lab.eng.blr.redhat.com]

TASK [Start network service] ***********************************************************************************************************************************************************
ok: [dhcp47-2.lab.eng.blr.redhat.com]
ok: [dhcp46-184.lab.eng.blr.redhat.com]
ok: [dhcp46-116.lab.eng.blr.redhat.com]
ok: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [Enable network service] **********************************************************************************************************************************************************
ok: [dhcp46-184.lab.eng.blr.redhat.com]
ok: [dhcp47-193.lab.eng.blr.redhat.com]
ok: [dhcp47-2.lab.eng.blr.redhat.com]
ok: [dhcp46-116.lab.eng.blr.redhat.com]

TASK [Start pcsd service] **************************************************************************************************************************************************************
ok: [dhcp47-193.lab.eng.blr.redhat.com]
ok: [dhcp47-2.lab.eng.blr.redhat.com]
ok: [dhcp46-116.lab.eng.blr.redhat.com]
ok: [dhcp46-184.lab.eng.blr.redhat.com]

TASK [Enable pcsd service] *************************************************************************************************************************************************************
ok: [dhcp46-116.lab.eng.blr.redhat.com]
ok: [dhcp47-193.lab.eng.blr.redhat.com]
ok: [dhcp46-184.lab.eng.blr.redhat.com]
ok: [dhcp47-2.lab.eng.blr.redhat.com]

TASK [Enable pacemaker service] ********************************************************************************************************************************************************
changed: [dhcp47-193.lab.eng.blr.redhat.com]
changed: [dhcp46-116.lab.eng.blr.redhat.com]
changed: [dhcp47-2.lab.eng.blr.redhat.com]
changed: [dhcp46-184.lab.eng.blr.redhat.com]

TASK [GlusterFS Share Storage service] *************************************************************************************************************************************************
ok: [dhcp46-116.lab.eng.blr.redhat.com]
ok: [dhcp47-193.lab.eng.blr.redhat.com]
ok: [dhcp47-2.lab.eng.blr.redhat.com]
ok: [dhcp46-184.lab.eng.blr.redhat.com]

TASK [Enable nfs-ganesha service] ******************************************************************************************************************************************************
ok: [dhcp46-116.lab.eng.blr.redhat.com]
ok: [dhcp47-193.lab.eng.blr.redhat.com]
ok: [dhcp47-2.lab.eng.blr.redhat.com]
ok: [dhcp46-184.lab.eng.blr.redhat.com]

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp46-116.lab.eng.blr.redhat.com : ok=10   changed=1    unreachable=0    failed=0   
dhcp46-184.lab.eng.blr.redhat.com : ok=10   changed=1    unreachable=0    failed=0   
dhcp47-193.lab.eng.blr.redhat.com : ok=10   changed=1    unreachable=0    failed=0   
dhcp47-2.lab.eng.blr.redhat.com : ok=10   changed=1    unreachable=0    failed=0   


PLAY [master_node] *********************************************************************************************************************************************************************

TASK [Generate ssh key in one of the nodes in HA cluster] ******************************************************************************************************************************
changed: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [Copy the public key to the local] ************************************************************************************************************************************************
changed: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [Copy the private key to the local] ***********************************************************************************************************************************************
changed: [dhcp47-193.lab.eng.blr.redhat.com]

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp47-193.lab.eng.blr.redhat.com : ok=3    changed=3    unreachable=0    failed=0   


PLAY [gluster_servers] *****************************************************************************************************************************************************************

TASK [Copy the public key to remote nodes] *********************************************************************************************************************************************
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com)
changed: [dhcp46-184.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com)
changed: [dhcp47-2.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com)
changed: [dhcp46-116.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com)

TASK [Copy the private key to remote node] *********************************************************************************************************************************************
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com)
changed: [dhcp46-116.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com)
changed: [dhcp46-184.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com)
changed: [dhcp47-2.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com)

TASK [Deploy the pubkey ~/root/.ssh/authorized_keys on all nodes] **********************************************************************************************************************
changed: [dhcp47-2.lab.eng.blr.redhat.com]
changed: [dhcp46-184.lab.eng.blr.redhat.com]
changed: [dhcp47-193.lab.eng.blr.redhat.com]
changed: [dhcp46-116.lab.eng.blr.redhat.com]

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp46-116.lab.eng.blr.redhat.com : ok=3    changed=3    unreachable=0    failed=0   
dhcp46-184.lab.eng.blr.redhat.com : ok=3    changed=3    unreachable=0    failed=0   
dhcp47-193.lab.eng.blr.redhat.com : ok=3    changed=1    unreachable=0    failed=0   
dhcp47-2.lab.eng.blr.redhat.com : ok=3    changed=3    unreachable=0    failed=0   


PLAY [master_node] *********************************************************************************************************************************************************************

TASK [Make the volume a gluster shared volume] *****************************************************************************************************************************************
changed: [dhcp47-193.lab.eng.blr.redhat.com]

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp47-193.lab.eng.blr.redhat.com : ok=1    changed=1    unreachable=0    failed=0   


PLAY [gluster_servers] *****************************************************************************************************************************************************************

TASK [Create a user hacluster in all the cluster nodes if not already present] *********************************************************************************************************
ok: [dhcp47-2.lab.eng.blr.redhat.com]
ok: [dhcp46-116.lab.eng.blr.redhat.com]
ok: [dhcp47-193.lab.eng.blr.redhat.com]
ok: [dhcp46-184.lab.eng.blr.redhat.com]

TASK [Set the hacluster user the same password on all machines] ************************************************************************************************************************
changed: [dhcp47-2.lab.eng.blr.redhat.com]
changed: [dhcp46-116.lab.eng.blr.redhat.com]
changed: [dhcp46-184.lab.eng.blr.redhat.com]
changed: [dhcp47-193.lab.eng.blr.redhat.com]

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp46-116.lab.eng.blr.redhat.com : ok=2    changed=1    unreachable=0    failed=0   
dhcp46-184.lab.eng.blr.redhat.com : ok=2    changed=1    unreachable=0    failed=0   
dhcp47-193.lab.eng.blr.redhat.com : ok=2    changed=1    unreachable=0    failed=0   
dhcp47-2.lab.eng.blr.redhat.com : ok=2    changed=1    unreachable=0    failed=0   


PLAY [gluster_servers] *****************************************************************************************************************************************************************

TASK [Pcs cluster authenticate the hacluster users in all the nodes] *******************************************************************************************************************
changed: [dhcp47-2.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com)
changed: [dhcp46-116.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com)
changed: [dhcp46-184.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com)
changed: [dhcp47-193.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com)
changed: [dhcp47-2.lab.eng.blr.redhat.com] => (item=dhcp46-116.lab.eng.blr.redhat.com)
changed: [dhcp46-116.lab.eng.blr.redhat.com] => (item=dhcp46-116.lab.eng.blr.redhat.com)
changed: [dhcp46-184.lab.eng.blr.redhat.com] => (item=dhcp46-116.lab.eng.blr.redhat.com)
changed: [dhcp47-193.lab.eng.blr.redhat.com] => (item=dhcp46-116.lab.eng.blr.redhat.com)
changed: [dhcp47-2.lab.eng.blr.redhat.com] => (item=dhcp46-184.lab.eng.blr.redhat.com)
changed: [dhcp46-116.lab.eng.blr.redhat.com] => (item=dhcp46-184.lab.eng.blr.redhat.com)
changed: [dhcp46-184.lab.eng.blr.redhat.com] => (item=dhcp46-184.lab.eng.blr.redhat.com)
changed: [dhcp47-193.lab.eng.blr.redhat.com] => (item=dhcp46-184.lab.eng.blr.redhat.com)
changed: [dhcp47-2.lab.eng.blr.redhat.com] => (item=dhcp47-2.lab.eng.blr.redhat.com)
changed: [dhcp46-116.lab.eng.blr.redhat.com] => (item=dhcp47-2.lab.eng.blr.redhat.com)
changed: [dhcp47-193.lab.eng.blr.redhat.com] => (item=dhcp47-2.lab.eng.blr.redhat.com)
changed: [dhcp46-184.lab.eng.blr.redhat.com] => (item=dhcp47-2.lab.eng.blr.redhat.com)

TASK [Pause for a few seconds] *********************************************************************************************************************************************************
Pausing for 3 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
ok: [dhcp47-193.lab.eng.blr.redhat.com]

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp46-116.lab.eng.blr.redhat.com : ok=1    changed=1    unreachable=0    failed=0   
dhcp46-184.lab.eng.blr.redhat.com : ok=1    changed=1    unreachable=0    failed=0   
dhcp47-193.lab.eng.blr.redhat.com : ok=2    changed=1    unreachable=0    failed=0   
dhcp47-2.lab.eng.blr.redhat.com : ok=1    changed=1    unreachable=0    failed=0   


PLAY [master_node] *********************************************************************************************************************************************************************

TASK [Create configuration directory for nfs-ganesha] **********************************************************************************************************************************
changed: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [Copy ganesha.conf to config directory on shared volume] **************************************************************************************************************************
changed: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [Creating ganesha-ha.conf file] ***************************************************************************************************************************************************
changed: [dhcp47-193.lab.eng.blr.redhat.com]

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp47-193.lab.eng.blr.redhat.com : ok=3    changed=3    unreachable=0    failed=0   


PLAY [gluster_servers] *****************************************************************************************************************************************************************

TASK [Define service port] *************************************************************************************************************************************************************
changed: [dhcp46-116.lab.eng.blr.redhat.com]
changed: [dhcp47-2.lab.eng.blr.redhat.com]
changed: [dhcp47-193.lab.eng.blr.redhat.com]
changed: [dhcp46-184.lab.eng.blr.redhat.com]

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp46-116.lab.eng.blr.redhat.com : ok=1    changed=1    unreachable=0    failed=0   
dhcp46-184.lab.eng.blr.redhat.com : ok=1    changed=1    unreachable=0    failed=0   
dhcp47-193.lab.eng.blr.redhat.com : ok=1    changed=1    unreachable=0    failed=0   
dhcp47-2.lab.eng.blr.redhat.com : ok=1    changed=1    unreachable=0    failed=0   


PLAY [master_node] *********************************************************************************************************************************************************************

TASK [Enable nfs-ganesha] **************************************************************************************************************************************************************
fatal: [dhcp47-193.lab.eng.blr.redhat.com]: FAILED! => {"changed": true, "cmd": "gluster nfs-ganesha enable --mode=script", "delta": "0:00:10.702735", "end": "2018-03-26 12:42:44.520213", "failed_when_result": true, "msg": "non-zero return code", "rc": 1, "start": "2018-03-26 12:42:33.817478", "stderr": "nfs-ganesha: failed: NFS-Ganesha failed to start.Please see log file for details", "stderr_lines": ["nfs-ganesha: failed: NFS-Ganesha failed to start.Please see log file for details"], "stdout": "This will take a few minutes to complete. Please wait ..", "stdout_lines": ["This will take a few minutes to complete. Please wait .."]}
...ignoring

TASK [Pause for 30 seconds (takes a while to enable NFS Ganesha)] **********************************************************************************************************************
Pausing for 30 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
ok: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [Check NFS Ganesha status] ********************************************************************************************************************************************************
changed: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [Report NFS Ganesha status] *******************************************************************************************************************************************************
skipping: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [Report NFS Ganesha status (If any errors)] ***************************************************************************************************************************************
ok: [dhcp47-193.lab.eng.blr.redhat.com] => {
    "msg": "Error: cluster is not currently running on this node"
}

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp47-193.lab.eng.blr.redhat.com : ok=4    changed=2    unreachable=0    failed=0   

[root@dhcp47-193 gdeploy]# vi ganesha_setup.conf 
[root@dhcp47-193 gdeploy]# gdeploy  -c ganesha_setup.conf 

PLAY [gluster_servers] *****************************************************************************************************************************************************************

TASK [Start firewalld if not already started] ******************************************************************************************************************************************
fatal: [dhcp47-2.labb.eng.blr.redhat.com]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true}
ok: [dhcp46-116.lab.eng.blr.redhat.com]
ok: [dhcp46-184.lab.eng.blr.redhat.com]
ok: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [Add/Delete services to firewalld rules] ******************************************************************************************************************************************
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=glusterfs)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=glusterfs)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=glusterfs)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=nlm)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=nlm)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=nlm)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=nfs)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=nfs)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=nfs)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=rpc-bind)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=rpc-bind)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=rpc-bind)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=high-availability)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=high-availability)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=high-availability)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=mountd)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=mountd)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=mountd)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=rquota)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=rquota)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=rquota)

TASK [Reloads the firewall] ************************************************************************************************************************************************************
changed: [dhcp46-184.lab.eng.blr.redhat.com]
changed: [dhcp46-116.lab.eng.blr.redhat.com]
changed: [dhcp47-193.lab.eng.blr.redhat.com]
    to retry, use: --limit @/tmp/tmp0hDpxC/firewalld-service-op.retry

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp46-116.lab.eng.blr.redhat.com : ok=3    changed=1    unreachable=0    failed=0   
dhcp46-184.lab.eng.blr.redhat.com : ok=3    changed=1    unreachable=0    failed=0   
dhcp47-193.lab.eng.blr.redhat.com : ok=3    changed=1    unreachable=0    failed=0   
dhcp47-2.labb.eng.blr.redhat.com : ok=0    changed=0    unreachable=1    failed=0   

Ignoring errors...

PLAY [gluster_servers] *****************************************************************************************************************************************************************

TASK [Start firewalld if not already started] ******************************************************************************************************************************************
fatal: [dhcp47-2.labb.eng.blr.redhat.com]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true}
ok: [dhcp47-193.lab.eng.blr.redhat.com]
ok: [dhcp46-184.lab.eng.blr.redhat.com]
ok: [dhcp46-116.lab.eng.blr.redhat.com]

TASK [Open/Close firewalld ports] ******************************************************************************************************************************************************
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=111/tcp)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=111/tcp)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=111/tcp)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=2049/tcp)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=2049/tcp)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=2049/tcp)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=54321/tcp)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=54321/tcp)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=54321/tcp)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=5900/tcp)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=5900/tcp)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=5900/tcp)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=5900-6923/tcp)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=5900-6923/tcp)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=5900-6923/tcp)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=5666/tcp)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=5666/tcp)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=5666/tcp)
ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=16514/tcp)
ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=16514/tcp)
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=16514/tcp)

TASK [Reloads the firewall] ************************************************************************************************************************************************************
changed: [dhcp46-116.lab.eng.blr.redhat.com]
changed: [dhcp46-184.lab.eng.blr.redhat.com]
changed: [dhcp47-193.lab.eng.blr.redhat.com]
    to retry, use: --limit @/tmp/tmp0hDpxC/firewalld-ports-op.retry

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp46-116.lab.eng.blr.redhat.com : ok=3    changed=1    unreachable=0    failed=0   
dhcp46-184.lab.eng.blr.redhat.com : ok=3    changed=1    unreachable=0    failed=0   
dhcp47-193.lab.eng.blr.redhat.com : ok=3    changed=1    unreachable=0    failed=0   
dhcp47-2.labb.eng.blr.redhat.com : ok=0    changed=0    unreachable=1    failed=0   

Ignoring errors...

PLAY [cluster_nodes] *******************************************************************************************************************************************************************

TASK [Check if nfs-ganesha is installed] ***********************************************************************************************************************************************
fatal: [dhcp47-2.labb.eng.blr.redhat.com]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true}
 [WARNING]: Consider using yum, dnf or zypper module rather than running rpm

changed: [dhcp46-116.lab.eng.blr.redhat.com]
changed: [dhcp47-193.lab.eng.blr.redhat.com]
changed: [dhcp46-184.lab.eng.blr.redhat.com]

TASK [fail] ****************************************************************************************************************************************************************************
skipping: [dhcp47-193.lab.eng.blr.redhat.com]
skipping: [dhcp46-116.lab.eng.blr.redhat.com]
skipping: [dhcp46-184.lab.eng.blr.redhat.com]

TASK [Check if corosync is installed] **************************************************************************************************************************************************
changed: [dhcp47-193.lab.eng.blr.redhat.com]
changed: [dhcp46-116.lab.eng.blr.redhat.com]
changed: [dhcp46-184.lab.eng.blr.redhat.com]

TASK [fail] ****************************************************************************************************************************************************************************
skipping: [dhcp47-193.lab.eng.blr.redhat.com]
skipping: [dhcp46-116.lab.eng.blr.redhat.com]
skipping: [dhcp46-184.lab.eng.blr.redhat.com]

TASK [Check if pacemaker is installed] *************************************************************************************************************************************************
changed: [dhcp46-116.lab.eng.blr.redhat.com]
changed: [dhcp47-193.lab.eng.blr.redhat.com]
changed: [dhcp46-184.lab.eng.blr.redhat.com]

TASK [fail] ****************************************************************************************************************************************************************************
skipping: [dhcp47-193.lab.eng.blr.redhat.com]
skipping: [dhcp46-116.lab.eng.blr.redhat.com]
skipping: [dhcp46-184.lab.eng.blr.redhat.com]

TASK [Check if libntirpc is installed] *************************************************************************************************************************************************
changed: [dhcp46-116.lab.eng.blr.redhat.com]
changed: [dhcp47-193.lab.eng.blr.redhat.com]
changed: [dhcp46-184.lab.eng.blr.redhat.com]

TASK [fail] ****************************************************************************************************************************************************************************
skipping: [dhcp47-193.lab.eng.blr.redhat.com]
skipping: [dhcp46-116.lab.eng.blr.redhat.com]
skipping: [dhcp46-184.lab.eng.blr.redhat.com]

TASK [Check if pcs is installed] *******************************************************************************************************************************************************
changed: [dhcp46-116.lab.eng.blr.redhat.com]
changed: [dhcp47-193.lab.eng.blr.redhat.com]
changed: [dhcp46-184.lab.eng.blr.redhat.com]

TASK [fail] ****************************************************************************************************************************************************************************
skipping: [dhcp47-193.lab.eng.blr.redhat.com]
skipping: [dhcp46-116.lab.eng.blr.redhat.com]
skipping: [dhcp46-184.lab.eng.blr.redhat.com]
    to retry, use: --limit @/tmp/tmp0hDpxC/check_package_installs_ganesha.retry

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp46-116.lab.eng.blr.redhat.com : ok=5    changed=5    unreachable=0    failed=0   
dhcp46-184.lab.eng.blr.redhat.com : ok=5    changed=5    unreachable=0    failed=0   
dhcp47-193.lab.eng.blr.redhat.com : ok=5    changed=5    unreachable=0    failed=0   
dhcp47-2.labb.eng.blr.redhat.com : ok=0    changed=0    unreachable=1    failed=0   

Ignoring errors...

PLAY [cluster_nodes] *******************************************************************************************************************************************************************

TASK [Stop kernel NFS] *****************************************************************************************************************************************************************
fatal: [dhcp47-2.labb.eng.blr.redhat.com]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true}
ok: [dhcp46-116.lab.eng.blr.redhat.com]
ok: [dhcp46-184.lab.eng.blr.redhat.com]
ok: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [Stop network manager service] ****************************************************************************************************************************************************
ok: [dhcp46-116.lab.eng.blr.redhat.com]
ok: [dhcp47-193.lab.eng.blr.redhat.com]
ok: [dhcp46-184.lab.eng.blr.redhat.com]

TASK [Disable network manager service] *************************************************************************************************************************************************
ok: [dhcp46-116.lab.eng.blr.redhat.com]
ok: [dhcp47-193.lab.eng.blr.redhat.com]
ok: [dhcp46-184.lab.eng.blr.redhat.com]

TASK [Start network service] ***********************************************************************************************************************************************************
ok: [dhcp46-116.lab.eng.blr.redhat.com]
ok: [dhcp47-193.lab.eng.blr.redhat.com]
ok: [dhcp46-184.lab.eng.blr.redhat.com]

TASK [Enable network service] **********************************************************************************************************************************************************
ok: [dhcp46-116.lab.eng.blr.redhat.com]
ok: [dhcp46-184.lab.eng.blr.redhat.com]
ok: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [Start pcsd service] **************************************************************************************************************************************************************
ok: [dhcp46-116.lab.eng.blr.redhat.com]
ok: [dhcp46-184.lab.eng.blr.redhat.com]
ok: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [Enable pcsd service] *************************************************************************************************************************************************************
ok: [dhcp46-116.lab.eng.blr.redhat.com]
ok: [dhcp47-193.lab.eng.blr.redhat.com]
ok: [dhcp46-184.lab.eng.blr.redhat.com]

TASK [Enable pacemaker service] ********************************************************************************************************************************************************
changed: [dhcp46-116.lab.eng.blr.redhat.com]
changed: [dhcp47-193.lab.eng.blr.redhat.com]
changed: [dhcp46-184.lab.eng.blr.redhat.com]

TASK [GlusterFS Share Storage service] *************************************************************************************************************************************************
ok: [dhcp46-116.lab.eng.blr.redhat.com]
ok: [dhcp47-193.lab.eng.blr.redhat.com]
ok: [dhcp46-184.lab.eng.blr.redhat.com]

TASK [Enable nfs-ganesha service] ******************************************************************************************************************************************************
ok: [dhcp46-116.lab.eng.blr.redhat.com]
ok: [dhcp47-193.lab.eng.blr.redhat.com]
ok: [dhcp46-184.lab.eng.blr.redhat.com]
    to retry, use: --limit @/tmp/tmp0hDpxC/configure-services.retry

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp46-116.lab.eng.blr.redhat.com : ok=10   changed=1    unreachable=0    failed=0   
dhcp46-184.lab.eng.blr.redhat.com : ok=10   changed=1    unreachable=0    failed=0   
dhcp47-193.lab.eng.blr.redhat.com : ok=10   changed=1    unreachable=0    failed=0   
dhcp47-2.labb.eng.blr.redhat.com : ok=0    changed=0    unreachable=1    failed=0   

Ignoring errors...

PLAY [master_node] *********************************************************************************************************************************************************************

TASK [Generate ssh key in one of the nodes in HA cluster] ******************************************************************************************************************************
changed: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [Copy the public key to the local] ************************************************************************************************************************************************
changed: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [Copy the private key to the local] ***********************************************************************************************************************************************
changed: [dhcp47-193.lab.eng.blr.redhat.com]

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp47-193.lab.eng.blr.redhat.com : ok=3    changed=3    unreachable=0    failed=0   


PLAY [gluster_servers] *****************************************************************************************************************************************************************

TASK [Copy the public key to remote nodes] *********************************************************************************************************************************************
failed: [dhcp47-2.labb.eng.blr.redhat.com] (item=dhcp47-193.lab.eng.blr.redhat.com) => {"item": "dhcp47-193.lab.eng.blr.redhat.com", "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true}
fatal: [dhcp47-2.labb.eng.blr.redhat.com]: UNREACHABLE! => {"changed": false, "msg": "All items completed", "results": [{"_ansible_ignore_errors": null, "_ansible_item_result": true, "item": "dhcp47-193.lab.eng.blr.redhat.com", "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true}]}
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com)
changed: [dhcp46-116.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com)
changed: [dhcp46-184.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com)

TASK [Copy the private key to remote node] *********************************************************************************************************************************************
ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com)
changed: [dhcp46-116.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com)
changed: [dhcp46-184.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com)

TASK [Deploy the pubkey ~/root/.ssh/authorized_keys on all nodes] **********************************************************************************************************************
changed: [dhcp46-116.lab.eng.blr.redhat.com]
changed: [dhcp47-193.lab.eng.blr.redhat.com]
changed: [dhcp46-184.lab.eng.blr.redhat.com]
    to retry, use: --limit @/tmp/tmp0hDpxC/copy-ssh-key.retry

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp46-116.lab.eng.blr.redhat.com : ok=3    changed=3    unreachable=0    failed=0   
dhcp46-184.lab.eng.blr.redhat.com : ok=3    changed=3    unreachable=0    failed=0   
dhcp47-193.lab.eng.blr.redhat.com : ok=3    changed=1    unreachable=0    failed=0   
dhcp47-2.labb.eng.blr.redhat.com : ok=0    changed=0    unreachable=1    failed=0   

Ignoring errors...

PLAY [master_node] *********************************************************************************************************************************************************************

TASK [Make the volume a gluster shared volume] *****************************************************************************************************************************************
changed: [dhcp47-193.lab.eng.blr.redhat.com]

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp47-193.lab.eng.blr.redhat.com : ok=1    changed=1    unreachable=0    failed=0   


PLAY [gluster_servers] *****************************************************************************************************************************************************************

TASK [Create a user hacluster in all the cluster nodes if not already present] *********************************************************************************************************
fatal: [dhcp47-2.labb.eng.blr.redhat.com]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true}
ok: [dhcp46-184.lab.eng.blr.redhat.com]
ok: [dhcp46-116.lab.eng.blr.redhat.com]
ok: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [Set the hacluster user the same password on all machines] ************************************************************************************************************************
changed: [dhcp46-116.lab.eng.blr.redhat.com]
changed: [dhcp47-193.lab.eng.blr.redhat.com]
changed: [dhcp46-184.lab.eng.blr.redhat.com]
    to retry, use: --limit @/tmp/tmp0hDpxC/set-pcs-auth-passwd.retry

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp46-116.lab.eng.blr.redhat.com : ok=2    changed=1    unreachable=0    failed=0   
dhcp46-184.lab.eng.blr.redhat.com : ok=2    changed=1    unreachable=0    failed=0   
dhcp47-193.lab.eng.blr.redhat.com : ok=2    changed=1    unreachable=0    failed=0   
dhcp47-2.labb.eng.blr.redhat.com : ok=0    changed=0    unreachable=1    failed=0   

Ignoring errors...

PLAY [gluster_servers] *****************************************************************************************************************************************************************

TASK [Pcs cluster authenticate the hacluster users in all the nodes] *******************************************************************************************************************
failed: [dhcp47-2.labb.eng.blr.redhat.com] (item=dhcp47-193.lab.eng.blr.redhat.com) => {"item": "dhcp47-193.lab.eng.blr.redhat.com", "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true}
failed: [dhcp47-2.labb.eng.blr.redhat.com] (item=dhcp46-116.lab.eng.blr.redhat.com) => {"item": "dhcp46-116.lab.eng.blr.redhat.com", "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true}
failed: [dhcp47-2.labb.eng.blr.redhat.com] (item=dhcp46-184.lab.eng.blr.redhat.com) => {"item": "dhcp46-184.lab.eng.blr.redhat.com", "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true}
failed: [dhcp47-2.labb.eng.blr.redhat.com] (item=dhcp47-2.labb.eng.blr.redhat.com) => {"item": "dhcp47-2.labb.eng.blr.redhat.com", "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true}
fatal: [dhcp47-2.labb.eng.blr.redhat.com]: UNREACHABLE! => {"changed": false, "msg": "All items completed", "results": [{"_ansible_ignore_errors": null, "_ansible_item_result": true, "item": "dhcp47-193.lab.eng.blr.redhat.com", "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true}, {"_ansible_ignore_errors": null, "_ansible_item_result": true, "item": "dhcp46-116.lab.eng.blr.redhat.com", "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true}, {"_ansible_ignore_errors": null, "_ansible_item_result": true, "item": "dhcp46-184.lab.eng.blr.redhat.com", "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true}, {"_ansible_ignore_errors": null, "_ansible_item_result": true, "item": "dhcp47-2.labb.eng.blr.redhat.com", "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true}]}
changed: [dhcp46-116.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com)
changed: [dhcp46-184.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com)
changed: [dhcp47-193.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com)
changed: [dhcp46-116.lab.eng.blr.redhat.com] => (item=dhcp46-116.lab.eng.blr.redhat.com)
changed: [dhcp46-184.lab.eng.blr.redhat.com] => (item=dhcp46-116.lab.eng.blr.redhat.com)
changed: [dhcp47-193.lab.eng.blr.redhat.com] => (item=dhcp46-116.lab.eng.blr.redhat.com)
changed: [dhcp46-116.lab.eng.blr.redhat.com] => (item=dhcp46-184.lab.eng.blr.redhat.com)
changed: [dhcp46-184.lab.eng.blr.redhat.com] => (item=dhcp46-184.lab.eng.blr.redhat.com)
changed: [dhcp47-193.lab.eng.blr.redhat.com] => (item=dhcp46-184.lab.eng.blr.redhat.com)
failed: [dhcp46-116.lab.eng.blr.redhat.com] (item=dhcp47-2.labb.eng.blr.redhat.com) => {"changed": true, "cmd": "pcs cluster auth -u hacluster -p hacluster dhcp47-2.labb.eng.blr.redhat.com", "delta": "0:00:00.827062", "end": "2018-03-26 12:55:42.810885", "item": "dhcp47-2.labb.eng.blr.redhat.com", "msg": "non-zero return code", "rc": 1, "start": "2018-03-26 12:55:41.983823", "stderr": "Error: Unable to communicate with dhcp47-2.labb.eng.blr.redhat.com", "stderr_lines": ["Error: Unable to communicate with dhcp47-2.labb.eng.blr.redhat.com"], "stdout": "", "stdout_lines": []}
failed: [dhcp46-184.lab.eng.blr.redhat.com] (item=dhcp47-2.labb.eng.blr.redhat.com) => {"changed": true, "cmd": "pcs cluster auth -u hacluster -p hacluster dhcp47-2.labb.eng.blr.redhat.com", "delta": "0:00:00.934702", "end": "2018-03-26 12:55:43.317563", "item": "dhcp47-2.labb.eng.blr.redhat.com", "msg": "non-zero return code", "rc": 1, "start": "2018-03-26 12:55:42.382861", "stderr": "Error: Unable to communicate with dhcp47-2.labb.eng.blr.redhat.com", "stderr_lines": ["Error: Unable to communicate with dhcp47-2.labb.eng.blr.redhat.com"], "stdout": "", "stdout_lines": []}
failed: [dhcp47-193.lab.eng.blr.redhat.com] (item=dhcp47-2.labb.eng.blr.redhat.com) => {"changed": true, "cmd": "pcs cluster auth -u hacluster -p hacluster dhcp47-2.labb.eng.blr.redhat.com", "delta": "0:00:01.003482", "end": "2018-03-26 12:55:43.387661", "item": "dhcp47-2.labb.eng.blr.redhat.com", "msg": "non-zero return code", "rc": 1, "start": "2018-03-26 12:55:42.384179", "stderr": "Error: Unable to communicate with dhcp47-2.labb.eng.blr.redhat.com", "stderr_lines": ["Error: Unable to communicate with dhcp47-2.labb.eng.blr.redhat.com"], "stdout": "", "stdout_lines": []}
    to retry, use: --limit @/tmp/tmp0hDpxC/pcs-authentication.retry

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp46-116.lab.eng.blr.redhat.com : ok=0    changed=0    unreachable=0    failed=1   
dhcp46-184.lab.eng.blr.redhat.com : ok=0    changed=0    unreachable=0    failed=1   
dhcp47-193.lab.eng.blr.redhat.com : ok=0    changed=0    unreachable=0    failed=1   
dhcp47-2.labb.eng.blr.redhat.com : ok=0    changed=0    unreachable=1    failed=0   

Ignoring errors...

PLAY [master_node] *********************************************************************************************************************************************************************

TASK [Create configuration directory for nfs-ganesha] **********************************************************************************************************************************
changed: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [Copy ganesha.conf to config directory on shared volume] **************************************************************************************************************************
changed: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [Creating ganesha-ha.conf file] ***************************************************************************************************************************************************
changed: [dhcp47-193.lab.eng.blr.redhat.com]

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp47-193.lab.eng.blr.redhat.com : ok=3    changed=3    unreachable=0    failed=0   


PLAY [gluster_servers] *****************************************************************************************************************************************************************

TASK [Define service port] *************************************************************************************************************************************************************
fatal: [dhcp47-2.labb.eng.blr.redhat.com]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true}
changed: [dhcp46-116.lab.eng.blr.redhat.com]
changed: [dhcp47-193.lab.eng.blr.redhat.com]
changed: [dhcp46-184.lab.eng.blr.redhat.com]
    to retry, use: --limit @/tmp/tmp0hDpxC/define_service_ports.retry

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp46-116.lab.eng.blr.redhat.com : ok=1    changed=1    unreachable=0    failed=0   
dhcp46-184.lab.eng.blr.redhat.com : ok=1    changed=1    unreachable=0    failed=0   
dhcp47-193.lab.eng.blr.redhat.com : ok=1    changed=1    unreachable=0    failed=0   
dhcp47-2.labb.eng.blr.redhat.com : ok=0    changed=0    unreachable=1    failed=0   

Ignoring errors...

PLAY [master_node] *********************************************************************************************************************************************************************

TASK [Enable nfs-ganesha] **************************************************************************************************************************************************************
fatal: [dhcp47-193.lab.eng.blr.redhat.com]: FAILED! => {"changed": true, "cmd": "gluster nfs-ganesha enable --mode=script", "delta": "0:00:49.946201", "end": "2018-03-26 12:56:44.035225", "failed_when_result": true, "msg": "non-zero return code", "rc": 1, "start": "2018-03-26 12:55:54.089024", "stderr": "nfs-ganesha: failed: Failed to set up HA config for NFS-Ganesha. Please check the log file for details", "stderr_lines": ["nfs-ganesha: failed: Failed to set up HA config for NFS-Ganesha. Please check the log file for details"], "stdout": "This will take a few minutes to complete. Please wait ..", "stdout_lines": ["This will take a few minutes to complete. Please wait .."]}
...ignoring

TASK [Pause for 30 seconds (takes a while to enable NFS Ganesha)] **********************************************************************************************************************
Pausing for 30 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
ok: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [Check NFS Ganesha status] ********************************************************************************************************************************************************
changed: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [Report NFS Ganesha status] *******************************************************************************************************************************************************
skipping: [dhcp47-193.lab.eng.blr.redhat.com]

TASK [Report NFS Ganesha status (If any errors)] ***************************************************************************************************************************************
ok: [dhcp47-193.lab.eng.blr.redhat.com] => {
    "msg": "Error: cluster is not currently running on this node"
}

PLAY RECAP *****************************************************************************************************************************************************************************
dhcp47-193.lab.eng.blr.redhat.com : ok=4    changed=2    unreachable=0    failed=0   


=========================================



ganesha-ha.conf


[root@dhcp46-116 ganesha]# cat /var/run/gluster/shared_storage/nfs-ganesha/ganesha-ha.conf
# Provide a unique name for the cluster.
HA_NAME="ganesha-ha-360"

# The subset of nodes of the Gluster Trusted Storage Pool that forms the ganesha
# HA cluster. Hostname should specified, IP addresses are not allowed.
# Maximum number of 16 nodes are supported.

HA_CLUSTER_NODES="dhcp47-193.lab.eng.blr.redhat.com,dhcp46-116.lab.eng.blr.redhat.com,dhcp46-184.lab.eng.blr.redhat.com,dhcp47-2.labb.eng.blr.redhat.com"

# Virtual IPs of each of the nodes specified above.
VIP_dhcp47-193.lab.eng.blr.redhat.com="10.70.46.51"
VIP_dhcp46-116.lab.eng.blr.redhat.com="10.70.46.54"
VIP_dhcp46-184.lab.eng.blr.redhat.com="10.70.46.55"
VIP_dhcp47-2.labb.eng.blr.redhat.com="10.70.46.56"


==========================================



# service nfs-ganesha status
Redirecting to /bin/systemctl status nfs-ganesha.service
● nfs-ganesha.service - NFS-Ganesha file server
   Loaded: loaded (/usr/lib/systemd/system/nfs-ganesha.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2018-03-26 12:56:16 IST; 1h 50min ago
     Docs: http://github.com/nfs-ganesha/nfs-ganesha/wiki
  Process: 4001 ExecStartPost=/bin/bash -c /usr/bin/sleep 2 && /bin/dbus-send --system   --dest=org.ganesha.nfsd --type=method_call /org/ganesha/nfsd/admin  org.ganesha.nfsd.admin.init_fds_limit (code=exited, status=0/SUCCESS)
  Process: 3997 ExecStartPost=/bin/bash -c prlimit --pid $MAINPID --nofile=$NOFILE:$NOFILE (code=exited, status=0/SUCCESS)
  Process: 3995 ExecStart=/bin/bash -c ${NUMACTL} ${NUMAOPTS} /usr/bin/ganesha.nfsd ${OPTIONS} ${EPOCH} (code=exited, status=0/SUCCESS)
 Main PID: 3996 (ganesha.nfsd)
    Tasks: 272
   CGroup: /system.slice/nfs-ganesha.service
           └─3996 /usr/bin/ganesha.nfsd -L /var/log/ganesha/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT -E 6537151427721232384

Mar 26 12:56:14 dhcp46-116.lab.eng.blr.redhat.com systemd[1]: Starting NFS-Ganesha file server...
Mar 26 12:56:16 dhcp46-116.lab.eng.blr.redhat.com systemd[1]: Started NFS-Ganesha file server.



Since the nfs-ganesha service is observed been running post setup creation fails on 2 out of 4 nodes and even symlinks are not cleaned.Based on these 2 observation moving this BZ to Assigned state.

Comment 18 Manisha Saini 2018-04-05 14:30:18 UTC
Verified this BZ on 

# rpm -qa | grep ganesha
glusterfs-ganesha-3.12.2-7.el7rhgs.x86_64
nfs-ganesha-2.5.5-3.el7rhgs.x86_64
nfs-ganesha-gluster-2.5.5-3.el7rhgs.x86_64

Intentionally tried failing the "gluster nfs-ganesha enable" command.The ganesha service was not found running.Works as expected.
Moving this BZ to verified state.

Have raised a separate Bug[1] for the use case in comment #13.This needs to be fixed in Gdeploy.

[1]https://bugzilla.redhat.com/show_bug.cgi?id=1563923

Comment 24 errata-xmlrpc 2018-09-04 06:26:56 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2607


Note You need to log in before you can comment on or make changes to this bug.