Description of problem: As per the new implementation for 3.1 that we are propsoing is HA by default for nfs-ganesha server. The problem is that there have been cases when pcs cluster setup has failed in some scenarios but the point to note is that nfs-ganesha process has spawned on the nodes, whereas nfs-ganesha should also not spawn in case pcs cluster setup has failed. Otherwise we are not providing HA as default with the setup that we are doing One of the example scenario is BZ 1225507, there may be others. Version-Release number of selected component (if applicable): glusterfs-3.7.0-2.el6rhs.x86_64 nfs-ganesha-2.2.0-0.el6.x86_64 How reproducible: several times Expected results: We need to have a check that decides if the pcs cluster is up only then nfs-ganesha should be spawned. Additional info:
Hi, Please provide the doc text for this known issue in the Doc Text field.
Doc text is edited. Please sign off to be included in Known Issues.
Doc text looks good to me.
Upstream reference : https://review.gluster.org/#/c/16823/ Downstream reference : https://code.engineering.redhat.com/gerrit/104144
Tested this with # rpm -qa | grep ganesha glusterfs-ganesha-3.12.2-5.el7rhgs.x86_64 nfs-ganesha-2.5.5-3.el7rhgs.x86_64 nfs-ganesha-gluster-2.5.5-3.el7rhgs.x86_64 To test the fix,I intentionally gave wrong hostname (ivalid domain) so that ganesha cluster creation fails.But even though ganesha enable command is failed,Ganesha service was running on 2 out of 4 nodes.Even symlinks are not been cleaned on these 2 nodes post setup failure. ======================================== ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=nlm) ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=nlm) ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=nfs) ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=nfs) ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=nfs) ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=nfs) ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=rpc-bind) ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=rpc-bind) ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=rpc-bind) ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=rpc-bind) ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=high-availability) ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=high-availability) ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=high-availability) ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=high-availability) ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=mountd) ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=mountd) ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=mountd) ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=mountd) ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=rquota) ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=rquota) ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=rquota) ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=rquota) TASK [Reloads the firewall] ************************************************************************************************************************************************************ changed: [dhcp47-2.lab.eng.blr.redhat.com] changed: [dhcp46-116.lab.eng.blr.redhat.com] changed: [dhcp46-184.lab.eng.blr.redhat.com] changed: [dhcp47-193.lab.eng.blr.redhat.com] PLAY RECAP ***************************************************************************************************************************************************************************** dhcp46-116.lab.eng.blr.redhat.com : ok=3 changed=1 unreachable=0 failed=0 dhcp46-184.lab.eng.blr.redhat.com : ok=3 changed=1 unreachable=0 failed=0 dhcp47-193.lab.eng.blr.redhat.com : ok=3 changed=1 unreachable=0 failed=0 dhcp47-2.lab.eng.blr.redhat.com : ok=3 changed=1 unreachable=0 failed=0 PLAY [gluster_servers] ***************************************************************************************************************************************************************** TASK [Start firewalld if not already started] ****************************************************************************************************************************************** ok: [dhcp47-2.lab.eng.blr.redhat.com] ok: [dhcp46-116.lab.eng.blr.redhat.com] ok: [dhcp47-193.lab.eng.blr.redhat.com] ok: [dhcp46-184.lab.eng.blr.redhat.com] TASK [Open/Close firewalld ports] ****************************************************************************************************************************************************** ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=111/tcp) ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=111/tcp) ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=111/tcp) ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=111/tcp) ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=2049/tcp) ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=2049/tcp) ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=2049/tcp) ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=2049/tcp) ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=54321/tcp) ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=54321/tcp) ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=54321/tcp) ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=54321/tcp) ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=5900/tcp) ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=5900/tcp) ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=5900/tcp) ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=5900/tcp) ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=5900-6923/tcp) ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=5900-6923/tcp) ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=5900-6923/tcp) ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=5900-6923/tcp) ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=5666/tcp) ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=5666/tcp) ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=5666/tcp) ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=5666/tcp) ok: [dhcp47-2.lab.eng.blr.redhat.com] => (item=16514/tcp) ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=16514/tcp) ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=16514/tcp) ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=16514/tcp) TASK [Reloads the firewall] ************************************************************************************************************************************************************ changed: [dhcp47-2.lab.eng.blr.redhat.com] changed: [dhcp46-184.lab.eng.blr.redhat.com] changed: [dhcp46-116.lab.eng.blr.redhat.com] changed: [dhcp47-193.lab.eng.blr.redhat.com] PLAY RECAP ***************************************************************************************************************************************************************************** dhcp46-116.lab.eng.blr.redhat.com : ok=3 changed=1 unreachable=0 failed=0 dhcp46-184.lab.eng.blr.redhat.com : ok=3 changed=1 unreachable=0 failed=0 dhcp47-193.lab.eng.blr.redhat.com : ok=3 changed=1 unreachable=0 failed=0 dhcp47-2.lab.eng.blr.redhat.com : ok=3 changed=1 unreachable=0 failed=0 PLAY [cluster_nodes] ******************************************************************************************************************************************************************* TASK [Check if nfs-ganesha is installed] *********************************************************************************************************************************************** [WARNING]: Consider using yum, dnf or zypper module rather than running rpm changed: [dhcp46-184.lab.eng.blr.redhat.com] changed: [dhcp46-116.lab.eng.blr.redhat.com] changed: [dhcp47-2.lab.eng.blr.redhat.com] changed: [dhcp47-193.lab.eng.blr.redhat.com] TASK [fail] **************************************************************************************************************************************************************************** skipping: [dhcp47-193.lab.eng.blr.redhat.com] skipping: [dhcp46-116.lab.eng.blr.redhat.com] skipping: [dhcp46-184.lab.eng.blr.redhat.com] skipping: [dhcp47-2.lab.eng.blr.redhat.com] TASK [Check if corosync is installed] ************************************************************************************************************************************************** changed: [dhcp46-184.lab.eng.blr.redhat.com] changed: [dhcp47-193.lab.eng.blr.redhat.com] changed: [dhcp46-116.lab.eng.blr.redhat.com] changed: [dhcp47-2.lab.eng.blr.redhat.com] TASK [fail] **************************************************************************************************************************************************************************** skipping: [dhcp47-193.lab.eng.blr.redhat.com] skipping: [dhcp46-116.lab.eng.blr.redhat.com] skipping: [dhcp46-184.lab.eng.blr.redhat.com] skipping: [dhcp47-2.lab.eng.blr.redhat.com] TASK [Check if pacemaker is installed] ************************************************************************************************************************************************* changed: [dhcp46-116.lab.eng.blr.redhat.com] changed: [dhcp47-2.lab.eng.blr.redhat.com] changed: [dhcp47-193.lab.eng.blr.redhat.com] changed: [dhcp46-184.lab.eng.blr.redhat.com] TASK [fail] **************************************************************************************************************************************************************************** skipping: [dhcp47-193.lab.eng.blr.redhat.com] skipping: [dhcp46-116.lab.eng.blr.redhat.com] skipping: [dhcp46-184.lab.eng.blr.redhat.com] skipping: [dhcp47-2.lab.eng.blr.redhat.com] TASK [Check if libntirpc is installed] ************************************************************************************************************************************************* changed: [dhcp46-116.lab.eng.blr.redhat.com] changed: [dhcp46-184.lab.eng.blr.redhat.com] changed: [dhcp47-193.lab.eng.blr.redhat.com] changed: [dhcp47-2.lab.eng.blr.redhat.com] TASK [fail] **************************************************************************************************************************************************************************** skipping: [dhcp47-193.lab.eng.blr.redhat.com] skipping: [dhcp46-116.lab.eng.blr.redhat.com] skipping: [dhcp46-184.lab.eng.blr.redhat.com] skipping: [dhcp47-2.lab.eng.blr.redhat.com] TASK [Check if pcs is installed] ******************************************************************************************************************************************************* changed: [dhcp46-116.lab.eng.blr.redhat.com] changed: [dhcp47-193.lab.eng.blr.redhat.com] changed: [dhcp46-184.lab.eng.blr.redhat.com] changed: [dhcp47-2.lab.eng.blr.redhat.com] TASK [fail] **************************************************************************************************************************************************************************** skipping: [dhcp47-193.lab.eng.blr.redhat.com] skipping: [dhcp46-116.lab.eng.blr.redhat.com] skipping: [dhcp46-184.lab.eng.blr.redhat.com] skipping: [dhcp47-2.lab.eng.blr.redhat.com] PLAY RECAP ***************************************************************************************************************************************************************************** dhcp46-116.lab.eng.blr.redhat.com : ok=5 changed=5 unreachable=0 failed=0 dhcp46-184.lab.eng.blr.redhat.com : ok=5 changed=5 unreachable=0 failed=0 dhcp47-193.lab.eng.blr.redhat.com : ok=5 changed=5 unreachable=0 failed=0 dhcp47-2.lab.eng.blr.redhat.com : ok=5 changed=5 unreachable=0 failed=0 PLAY [cluster_nodes] ******************************************************************************************************************************************************************* TASK [Stop kernel NFS] ***************************************************************************************************************************************************************** ok: [dhcp47-2.lab.eng.blr.redhat.com] ok: [dhcp46-116.lab.eng.blr.redhat.com] ok: [dhcp46-184.lab.eng.blr.redhat.com] ok: [dhcp47-193.lab.eng.blr.redhat.com] TASK [Stop network manager service] **************************************************************************************************************************************************** ok: [dhcp46-184.lab.eng.blr.redhat.com] ok: [dhcp47-193.lab.eng.blr.redhat.com] ok: [dhcp46-116.lab.eng.blr.redhat.com] ok: [dhcp47-2.lab.eng.blr.redhat.com] TASK [Disable network manager service] ************************************************************************************************************************************************* ok: [dhcp47-193.lab.eng.blr.redhat.com] ok: [dhcp46-116.lab.eng.blr.redhat.com] ok: [dhcp47-2.lab.eng.blr.redhat.com] ok: [dhcp46-184.lab.eng.blr.redhat.com] TASK [Start network service] *********************************************************************************************************************************************************** ok: [dhcp47-2.lab.eng.blr.redhat.com] ok: [dhcp46-184.lab.eng.blr.redhat.com] ok: [dhcp46-116.lab.eng.blr.redhat.com] ok: [dhcp47-193.lab.eng.blr.redhat.com] TASK [Enable network service] ********************************************************************************************************************************************************** ok: [dhcp46-184.lab.eng.blr.redhat.com] ok: [dhcp47-193.lab.eng.blr.redhat.com] ok: [dhcp47-2.lab.eng.blr.redhat.com] ok: [dhcp46-116.lab.eng.blr.redhat.com] TASK [Start pcsd service] ************************************************************************************************************************************************************** ok: [dhcp47-193.lab.eng.blr.redhat.com] ok: [dhcp47-2.lab.eng.blr.redhat.com] ok: [dhcp46-116.lab.eng.blr.redhat.com] ok: [dhcp46-184.lab.eng.blr.redhat.com] TASK [Enable pcsd service] ************************************************************************************************************************************************************* ok: [dhcp46-116.lab.eng.blr.redhat.com] ok: [dhcp47-193.lab.eng.blr.redhat.com] ok: [dhcp46-184.lab.eng.blr.redhat.com] ok: [dhcp47-2.lab.eng.blr.redhat.com] TASK [Enable pacemaker service] ******************************************************************************************************************************************************** changed: [dhcp47-193.lab.eng.blr.redhat.com] changed: [dhcp46-116.lab.eng.blr.redhat.com] changed: [dhcp47-2.lab.eng.blr.redhat.com] changed: [dhcp46-184.lab.eng.blr.redhat.com] TASK [GlusterFS Share Storage service] ************************************************************************************************************************************************* ok: [dhcp46-116.lab.eng.blr.redhat.com] ok: [dhcp47-193.lab.eng.blr.redhat.com] ok: [dhcp47-2.lab.eng.blr.redhat.com] ok: [dhcp46-184.lab.eng.blr.redhat.com] TASK [Enable nfs-ganesha service] ****************************************************************************************************************************************************** ok: [dhcp46-116.lab.eng.blr.redhat.com] ok: [dhcp47-193.lab.eng.blr.redhat.com] ok: [dhcp47-2.lab.eng.blr.redhat.com] ok: [dhcp46-184.lab.eng.blr.redhat.com] PLAY RECAP ***************************************************************************************************************************************************************************** dhcp46-116.lab.eng.blr.redhat.com : ok=10 changed=1 unreachable=0 failed=0 dhcp46-184.lab.eng.blr.redhat.com : ok=10 changed=1 unreachable=0 failed=0 dhcp47-193.lab.eng.blr.redhat.com : ok=10 changed=1 unreachable=0 failed=0 dhcp47-2.lab.eng.blr.redhat.com : ok=10 changed=1 unreachable=0 failed=0 PLAY [master_node] ********************************************************************************************************************************************************************* TASK [Generate ssh key in one of the nodes in HA cluster] ****************************************************************************************************************************** changed: [dhcp47-193.lab.eng.blr.redhat.com] TASK [Copy the public key to the local] ************************************************************************************************************************************************ changed: [dhcp47-193.lab.eng.blr.redhat.com] TASK [Copy the private key to the local] *********************************************************************************************************************************************** changed: [dhcp47-193.lab.eng.blr.redhat.com] PLAY RECAP ***************************************************************************************************************************************************************************** dhcp47-193.lab.eng.blr.redhat.com : ok=3 changed=3 unreachable=0 failed=0 PLAY [gluster_servers] ***************************************************************************************************************************************************************** TASK [Copy the public key to remote nodes] ********************************************************************************************************************************************* ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com) changed: [dhcp46-184.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com) changed: [dhcp47-2.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com) changed: [dhcp46-116.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com) TASK [Copy the private key to remote node] ********************************************************************************************************************************************* ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com) changed: [dhcp46-116.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com) changed: [dhcp46-184.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com) changed: [dhcp47-2.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com) TASK [Deploy the pubkey ~/root/.ssh/authorized_keys on all nodes] ********************************************************************************************************************** changed: [dhcp47-2.lab.eng.blr.redhat.com] changed: [dhcp46-184.lab.eng.blr.redhat.com] changed: [dhcp47-193.lab.eng.blr.redhat.com] changed: [dhcp46-116.lab.eng.blr.redhat.com] PLAY RECAP ***************************************************************************************************************************************************************************** dhcp46-116.lab.eng.blr.redhat.com : ok=3 changed=3 unreachable=0 failed=0 dhcp46-184.lab.eng.blr.redhat.com : ok=3 changed=3 unreachable=0 failed=0 dhcp47-193.lab.eng.blr.redhat.com : ok=3 changed=1 unreachable=0 failed=0 dhcp47-2.lab.eng.blr.redhat.com : ok=3 changed=3 unreachable=0 failed=0 PLAY [master_node] ********************************************************************************************************************************************************************* TASK [Make the volume a gluster shared volume] ***************************************************************************************************************************************** changed: [dhcp47-193.lab.eng.blr.redhat.com] PLAY RECAP ***************************************************************************************************************************************************************************** dhcp47-193.lab.eng.blr.redhat.com : ok=1 changed=1 unreachable=0 failed=0 PLAY [gluster_servers] ***************************************************************************************************************************************************************** TASK [Create a user hacluster in all the cluster nodes if not already present] ********************************************************************************************************* ok: [dhcp47-2.lab.eng.blr.redhat.com] ok: [dhcp46-116.lab.eng.blr.redhat.com] ok: [dhcp47-193.lab.eng.blr.redhat.com] ok: [dhcp46-184.lab.eng.blr.redhat.com] TASK [Set the hacluster user the same password on all machines] ************************************************************************************************************************ changed: [dhcp47-2.lab.eng.blr.redhat.com] changed: [dhcp46-116.lab.eng.blr.redhat.com] changed: [dhcp46-184.lab.eng.blr.redhat.com] changed: [dhcp47-193.lab.eng.blr.redhat.com] PLAY RECAP ***************************************************************************************************************************************************************************** dhcp46-116.lab.eng.blr.redhat.com : ok=2 changed=1 unreachable=0 failed=0 dhcp46-184.lab.eng.blr.redhat.com : ok=2 changed=1 unreachable=0 failed=0 dhcp47-193.lab.eng.blr.redhat.com : ok=2 changed=1 unreachable=0 failed=0 dhcp47-2.lab.eng.blr.redhat.com : ok=2 changed=1 unreachable=0 failed=0 PLAY [gluster_servers] ***************************************************************************************************************************************************************** TASK [Pcs cluster authenticate the hacluster users in all the nodes] ******************************************************************************************************************* changed: [dhcp47-2.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com) changed: [dhcp46-116.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com) changed: [dhcp46-184.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com) changed: [dhcp47-193.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com) changed: [dhcp47-2.lab.eng.blr.redhat.com] => (item=dhcp46-116.lab.eng.blr.redhat.com) changed: [dhcp46-116.lab.eng.blr.redhat.com] => (item=dhcp46-116.lab.eng.blr.redhat.com) changed: [dhcp46-184.lab.eng.blr.redhat.com] => (item=dhcp46-116.lab.eng.blr.redhat.com) changed: [dhcp47-193.lab.eng.blr.redhat.com] => (item=dhcp46-116.lab.eng.blr.redhat.com) changed: [dhcp47-2.lab.eng.blr.redhat.com] => (item=dhcp46-184.lab.eng.blr.redhat.com) changed: [dhcp46-116.lab.eng.blr.redhat.com] => (item=dhcp46-184.lab.eng.blr.redhat.com) changed: [dhcp46-184.lab.eng.blr.redhat.com] => (item=dhcp46-184.lab.eng.blr.redhat.com) changed: [dhcp47-193.lab.eng.blr.redhat.com] => (item=dhcp46-184.lab.eng.blr.redhat.com) changed: [dhcp47-2.lab.eng.blr.redhat.com] => (item=dhcp47-2.lab.eng.blr.redhat.com) changed: [dhcp46-116.lab.eng.blr.redhat.com] => (item=dhcp47-2.lab.eng.blr.redhat.com) changed: [dhcp47-193.lab.eng.blr.redhat.com] => (item=dhcp47-2.lab.eng.blr.redhat.com) changed: [dhcp46-184.lab.eng.blr.redhat.com] => (item=dhcp47-2.lab.eng.blr.redhat.com) TASK [Pause for a few seconds] ********************************************************************************************************************************************************* Pausing for 3 seconds (ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort) ok: [dhcp47-193.lab.eng.blr.redhat.com] PLAY RECAP ***************************************************************************************************************************************************************************** dhcp46-116.lab.eng.blr.redhat.com : ok=1 changed=1 unreachable=0 failed=0 dhcp46-184.lab.eng.blr.redhat.com : ok=1 changed=1 unreachable=0 failed=0 dhcp47-193.lab.eng.blr.redhat.com : ok=2 changed=1 unreachable=0 failed=0 dhcp47-2.lab.eng.blr.redhat.com : ok=1 changed=1 unreachable=0 failed=0 PLAY [master_node] ********************************************************************************************************************************************************************* TASK [Create configuration directory for nfs-ganesha] ********************************************************************************************************************************** changed: [dhcp47-193.lab.eng.blr.redhat.com] TASK [Copy ganesha.conf to config directory on shared volume] ************************************************************************************************************************** changed: [dhcp47-193.lab.eng.blr.redhat.com] TASK [Creating ganesha-ha.conf file] *************************************************************************************************************************************************** changed: [dhcp47-193.lab.eng.blr.redhat.com] PLAY RECAP ***************************************************************************************************************************************************************************** dhcp47-193.lab.eng.blr.redhat.com : ok=3 changed=3 unreachable=0 failed=0 PLAY [gluster_servers] ***************************************************************************************************************************************************************** TASK [Define service port] ************************************************************************************************************************************************************* changed: [dhcp46-116.lab.eng.blr.redhat.com] changed: [dhcp47-2.lab.eng.blr.redhat.com] changed: [dhcp47-193.lab.eng.blr.redhat.com] changed: [dhcp46-184.lab.eng.blr.redhat.com] PLAY RECAP ***************************************************************************************************************************************************************************** dhcp46-116.lab.eng.blr.redhat.com : ok=1 changed=1 unreachable=0 failed=0 dhcp46-184.lab.eng.blr.redhat.com : ok=1 changed=1 unreachable=0 failed=0 dhcp47-193.lab.eng.blr.redhat.com : ok=1 changed=1 unreachable=0 failed=0 dhcp47-2.lab.eng.blr.redhat.com : ok=1 changed=1 unreachable=0 failed=0 PLAY [master_node] ********************************************************************************************************************************************************************* TASK [Enable nfs-ganesha] ************************************************************************************************************************************************************** fatal: [dhcp47-193.lab.eng.blr.redhat.com]: FAILED! => {"changed": true, "cmd": "gluster nfs-ganesha enable --mode=script", "delta": "0:00:10.702735", "end": "2018-03-26 12:42:44.520213", "failed_when_result": true, "msg": "non-zero return code", "rc": 1, "start": "2018-03-26 12:42:33.817478", "stderr": "nfs-ganesha: failed: NFS-Ganesha failed to start.Please see log file for details", "stderr_lines": ["nfs-ganesha: failed: NFS-Ganesha failed to start.Please see log file for details"], "stdout": "This will take a few minutes to complete. Please wait ..", "stdout_lines": ["This will take a few minutes to complete. Please wait .."]} ...ignoring TASK [Pause for 30 seconds (takes a while to enable NFS Ganesha)] ********************************************************************************************************************** Pausing for 30 seconds (ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort) ok: [dhcp47-193.lab.eng.blr.redhat.com] TASK [Check NFS Ganesha status] ******************************************************************************************************************************************************** changed: [dhcp47-193.lab.eng.blr.redhat.com] TASK [Report NFS Ganesha status] ******************************************************************************************************************************************************* skipping: [dhcp47-193.lab.eng.blr.redhat.com] TASK [Report NFS Ganesha status (If any errors)] *************************************************************************************************************************************** ok: [dhcp47-193.lab.eng.blr.redhat.com] => { "msg": "Error: cluster is not currently running on this node" } PLAY RECAP ***************************************************************************************************************************************************************************** dhcp47-193.lab.eng.blr.redhat.com : ok=4 changed=2 unreachable=0 failed=0 [root@dhcp47-193 gdeploy]# vi ganesha_setup.conf [root@dhcp47-193 gdeploy]# gdeploy -c ganesha_setup.conf PLAY [gluster_servers] ***************************************************************************************************************************************************************** TASK [Start firewalld if not already started] ****************************************************************************************************************************************** fatal: [dhcp47-2.labb.eng.blr.redhat.com]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true} ok: [dhcp46-116.lab.eng.blr.redhat.com] ok: [dhcp46-184.lab.eng.blr.redhat.com] ok: [dhcp47-193.lab.eng.blr.redhat.com] TASK [Add/Delete services to firewalld rules] ****************************************************************************************************************************************** ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=glusterfs) ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=glusterfs) ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=glusterfs) ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=nlm) ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=nlm) ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=nlm) ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=nfs) ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=nfs) ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=nfs) ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=rpc-bind) ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=rpc-bind) ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=rpc-bind) ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=high-availability) ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=high-availability) ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=high-availability) ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=mountd) ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=mountd) ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=mountd) ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=rquota) ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=rquota) ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=rquota) TASK [Reloads the firewall] ************************************************************************************************************************************************************ changed: [dhcp46-184.lab.eng.blr.redhat.com] changed: [dhcp46-116.lab.eng.blr.redhat.com] changed: [dhcp47-193.lab.eng.blr.redhat.com] to retry, use: --limit @/tmp/tmp0hDpxC/firewalld-service-op.retry PLAY RECAP ***************************************************************************************************************************************************************************** dhcp46-116.lab.eng.blr.redhat.com : ok=3 changed=1 unreachable=0 failed=0 dhcp46-184.lab.eng.blr.redhat.com : ok=3 changed=1 unreachable=0 failed=0 dhcp47-193.lab.eng.blr.redhat.com : ok=3 changed=1 unreachable=0 failed=0 dhcp47-2.labb.eng.blr.redhat.com : ok=0 changed=0 unreachable=1 failed=0 Ignoring errors... PLAY [gluster_servers] ***************************************************************************************************************************************************************** TASK [Start firewalld if not already started] ****************************************************************************************************************************************** fatal: [dhcp47-2.labb.eng.blr.redhat.com]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true} ok: [dhcp47-193.lab.eng.blr.redhat.com] ok: [dhcp46-184.lab.eng.blr.redhat.com] ok: [dhcp46-116.lab.eng.blr.redhat.com] TASK [Open/Close firewalld ports] ****************************************************************************************************************************************************** ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=111/tcp) ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=111/tcp) ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=111/tcp) ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=2049/tcp) ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=2049/tcp) ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=2049/tcp) ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=54321/tcp) ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=54321/tcp) ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=54321/tcp) ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=5900/tcp) ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=5900/tcp) ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=5900/tcp) ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=5900-6923/tcp) ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=5900-6923/tcp) ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=5900-6923/tcp) ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=5666/tcp) ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=5666/tcp) ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=5666/tcp) ok: [dhcp46-116.lab.eng.blr.redhat.com] => (item=16514/tcp) ok: [dhcp46-184.lab.eng.blr.redhat.com] => (item=16514/tcp) ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=16514/tcp) TASK [Reloads the firewall] ************************************************************************************************************************************************************ changed: [dhcp46-116.lab.eng.blr.redhat.com] changed: [dhcp46-184.lab.eng.blr.redhat.com] changed: [dhcp47-193.lab.eng.blr.redhat.com] to retry, use: --limit @/tmp/tmp0hDpxC/firewalld-ports-op.retry PLAY RECAP ***************************************************************************************************************************************************************************** dhcp46-116.lab.eng.blr.redhat.com : ok=3 changed=1 unreachable=0 failed=0 dhcp46-184.lab.eng.blr.redhat.com : ok=3 changed=1 unreachable=0 failed=0 dhcp47-193.lab.eng.blr.redhat.com : ok=3 changed=1 unreachable=0 failed=0 dhcp47-2.labb.eng.blr.redhat.com : ok=0 changed=0 unreachable=1 failed=0 Ignoring errors... PLAY [cluster_nodes] ******************************************************************************************************************************************************************* TASK [Check if nfs-ganesha is installed] *********************************************************************************************************************************************** fatal: [dhcp47-2.labb.eng.blr.redhat.com]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true} [WARNING]: Consider using yum, dnf or zypper module rather than running rpm changed: [dhcp46-116.lab.eng.blr.redhat.com] changed: [dhcp47-193.lab.eng.blr.redhat.com] changed: [dhcp46-184.lab.eng.blr.redhat.com] TASK [fail] **************************************************************************************************************************************************************************** skipping: [dhcp47-193.lab.eng.blr.redhat.com] skipping: [dhcp46-116.lab.eng.blr.redhat.com] skipping: [dhcp46-184.lab.eng.blr.redhat.com] TASK [Check if corosync is installed] ************************************************************************************************************************************************** changed: [dhcp47-193.lab.eng.blr.redhat.com] changed: [dhcp46-116.lab.eng.blr.redhat.com] changed: [dhcp46-184.lab.eng.blr.redhat.com] TASK [fail] **************************************************************************************************************************************************************************** skipping: [dhcp47-193.lab.eng.blr.redhat.com] skipping: [dhcp46-116.lab.eng.blr.redhat.com] skipping: [dhcp46-184.lab.eng.blr.redhat.com] TASK [Check if pacemaker is installed] ************************************************************************************************************************************************* changed: [dhcp46-116.lab.eng.blr.redhat.com] changed: [dhcp47-193.lab.eng.blr.redhat.com] changed: [dhcp46-184.lab.eng.blr.redhat.com] TASK [fail] **************************************************************************************************************************************************************************** skipping: [dhcp47-193.lab.eng.blr.redhat.com] skipping: [dhcp46-116.lab.eng.blr.redhat.com] skipping: [dhcp46-184.lab.eng.blr.redhat.com] TASK [Check if libntirpc is installed] ************************************************************************************************************************************************* changed: [dhcp46-116.lab.eng.blr.redhat.com] changed: [dhcp47-193.lab.eng.blr.redhat.com] changed: [dhcp46-184.lab.eng.blr.redhat.com] TASK [fail] **************************************************************************************************************************************************************************** skipping: [dhcp47-193.lab.eng.blr.redhat.com] skipping: [dhcp46-116.lab.eng.blr.redhat.com] skipping: [dhcp46-184.lab.eng.blr.redhat.com] TASK [Check if pcs is installed] ******************************************************************************************************************************************************* changed: [dhcp46-116.lab.eng.blr.redhat.com] changed: [dhcp47-193.lab.eng.blr.redhat.com] changed: [dhcp46-184.lab.eng.blr.redhat.com] TASK [fail] **************************************************************************************************************************************************************************** skipping: [dhcp47-193.lab.eng.blr.redhat.com] skipping: [dhcp46-116.lab.eng.blr.redhat.com] skipping: [dhcp46-184.lab.eng.blr.redhat.com] to retry, use: --limit @/tmp/tmp0hDpxC/check_package_installs_ganesha.retry PLAY RECAP ***************************************************************************************************************************************************************************** dhcp46-116.lab.eng.blr.redhat.com : ok=5 changed=5 unreachable=0 failed=0 dhcp46-184.lab.eng.blr.redhat.com : ok=5 changed=5 unreachable=0 failed=0 dhcp47-193.lab.eng.blr.redhat.com : ok=5 changed=5 unreachable=0 failed=0 dhcp47-2.labb.eng.blr.redhat.com : ok=0 changed=0 unreachable=1 failed=0 Ignoring errors... PLAY [cluster_nodes] ******************************************************************************************************************************************************************* TASK [Stop kernel NFS] ***************************************************************************************************************************************************************** fatal: [dhcp47-2.labb.eng.blr.redhat.com]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true} ok: [dhcp46-116.lab.eng.blr.redhat.com] ok: [dhcp46-184.lab.eng.blr.redhat.com] ok: [dhcp47-193.lab.eng.blr.redhat.com] TASK [Stop network manager service] **************************************************************************************************************************************************** ok: [dhcp46-116.lab.eng.blr.redhat.com] ok: [dhcp47-193.lab.eng.blr.redhat.com] ok: [dhcp46-184.lab.eng.blr.redhat.com] TASK [Disable network manager service] ************************************************************************************************************************************************* ok: [dhcp46-116.lab.eng.blr.redhat.com] ok: [dhcp47-193.lab.eng.blr.redhat.com] ok: [dhcp46-184.lab.eng.blr.redhat.com] TASK [Start network service] *********************************************************************************************************************************************************** ok: [dhcp46-116.lab.eng.blr.redhat.com] ok: [dhcp47-193.lab.eng.blr.redhat.com] ok: [dhcp46-184.lab.eng.blr.redhat.com] TASK [Enable network service] ********************************************************************************************************************************************************** ok: [dhcp46-116.lab.eng.blr.redhat.com] ok: [dhcp46-184.lab.eng.blr.redhat.com] ok: [dhcp47-193.lab.eng.blr.redhat.com] TASK [Start pcsd service] ************************************************************************************************************************************************************** ok: [dhcp46-116.lab.eng.blr.redhat.com] ok: [dhcp46-184.lab.eng.blr.redhat.com] ok: [dhcp47-193.lab.eng.blr.redhat.com] TASK [Enable pcsd service] ************************************************************************************************************************************************************* ok: [dhcp46-116.lab.eng.blr.redhat.com] ok: [dhcp47-193.lab.eng.blr.redhat.com] ok: [dhcp46-184.lab.eng.blr.redhat.com] TASK [Enable pacemaker service] ******************************************************************************************************************************************************** changed: [dhcp46-116.lab.eng.blr.redhat.com] changed: [dhcp47-193.lab.eng.blr.redhat.com] changed: [dhcp46-184.lab.eng.blr.redhat.com] TASK [GlusterFS Share Storage service] ************************************************************************************************************************************************* ok: [dhcp46-116.lab.eng.blr.redhat.com] ok: [dhcp47-193.lab.eng.blr.redhat.com] ok: [dhcp46-184.lab.eng.blr.redhat.com] TASK [Enable nfs-ganesha service] ****************************************************************************************************************************************************** ok: [dhcp46-116.lab.eng.blr.redhat.com] ok: [dhcp47-193.lab.eng.blr.redhat.com] ok: [dhcp46-184.lab.eng.blr.redhat.com] to retry, use: --limit @/tmp/tmp0hDpxC/configure-services.retry PLAY RECAP ***************************************************************************************************************************************************************************** dhcp46-116.lab.eng.blr.redhat.com : ok=10 changed=1 unreachable=0 failed=0 dhcp46-184.lab.eng.blr.redhat.com : ok=10 changed=1 unreachable=0 failed=0 dhcp47-193.lab.eng.blr.redhat.com : ok=10 changed=1 unreachable=0 failed=0 dhcp47-2.labb.eng.blr.redhat.com : ok=0 changed=0 unreachable=1 failed=0 Ignoring errors... PLAY [master_node] ********************************************************************************************************************************************************************* TASK [Generate ssh key in one of the nodes in HA cluster] ****************************************************************************************************************************** changed: [dhcp47-193.lab.eng.blr.redhat.com] TASK [Copy the public key to the local] ************************************************************************************************************************************************ changed: [dhcp47-193.lab.eng.blr.redhat.com] TASK [Copy the private key to the local] *********************************************************************************************************************************************** changed: [dhcp47-193.lab.eng.blr.redhat.com] PLAY RECAP ***************************************************************************************************************************************************************************** dhcp47-193.lab.eng.blr.redhat.com : ok=3 changed=3 unreachable=0 failed=0 PLAY [gluster_servers] ***************************************************************************************************************************************************************** TASK [Copy the public key to remote nodes] ********************************************************************************************************************************************* failed: [dhcp47-2.labb.eng.blr.redhat.com] (item=dhcp47-193.lab.eng.blr.redhat.com) => {"item": "dhcp47-193.lab.eng.blr.redhat.com", "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true} fatal: [dhcp47-2.labb.eng.blr.redhat.com]: UNREACHABLE! => {"changed": false, "msg": "All items completed", "results": [{"_ansible_ignore_errors": null, "_ansible_item_result": true, "item": "dhcp47-193.lab.eng.blr.redhat.com", "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true}]} ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com) changed: [dhcp46-116.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com) changed: [dhcp46-184.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com) TASK [Copy the private key to remote node] ********************************************************************************************************************************************* ok: [dhcp47-193.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com) changed: [dhcp46-116.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com) changed: [dhcp46-184.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com) TASK [Deploy the pubkey ~/root/.ssh/authorized_keys on all nodes] ********************************************************************************************************************** changed: [dhcp46-116.lab.eng.blr.redhat.com] changed: [dhcp47-193.lab.eng.blr.redhat.com] changed: [dhcp46-184.lab.eng.blr.redhat.com] to retry, use: --limit @/tmp/tmp0hDpxC/copy-ssh-key.retry PLAY RECAP ***************************************************************************************************************************************************************************** dhcp46-116.lab.eng.blr.redhat.com : ok=3 changed=3 unreachable=0 failed=0 dhcp46-184.lab.eng.blr.redhat.com : ok=3 changed=3 unreachable=0 failed=0 dhcp47-193.lab.eng.blr.redhat.com : ok=3 changed=1 unreachable=0 failed=0 dhcp47-2.labb.eng.blr.redhat.com : ok=0 changed=0 unreachable=1 failed=0 Ignoring errors... PLAY [master_node] ********************************************************************************************************************************************************************* TASK [Make the volume a gluster shared volume] ***************************************************************************************************************************************** changed: [dhcp47-193.lab.eng.blr.redhat.com] PLAY RECAP ***************************************************************************************************************************************************************************** dhcp47-193.lab.eng.blr.redhat.com : ok=1 changed=1 unreachable=0 failed=0 PLAY [gluster_servers] ***************************************************************************************************************************************************************** TASK [Create a user hacluster in all the cluster nodes if not already present] ********************************************************************************************************* fatal: [dhcp47-2.labb.eng.blr.redhat.com]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true} ok: [dhcp46-184.lab.eng.blr.redhat.com] ok: [dhcp46-116.lab.eng.blr.redhat.com] ok: [dhcp47-193.lab.eng.blr.redhat.com] TASK [Set the hacluster user the same password on all machines] ************************************************************************************************************************ changed: [dhcp46-116.lab.eng.blr.redhat.com] changed: [dhcp47-193.lab.eng.blr.redhat.com] changed: [dhcp46-184.lab.eng.blr.redhat.com] to retry, use: --limit @/tmp/tmp0hDpxC/set-pcs-auth-passwd.retry PLAY RECAP ***************************************************************************************************************************************************************************** dhcp46-116.lab.eng.blr.redhat.com : ok=2 changed=1 unreachable=0 failed=0 dhcp46-184.lab.eng.blr.redhat.com : ok=2 changed=1 unreachable=0 failed=0 dhcp47-193.lab.eng.blr.redhat.com : ok=2 changed=1 unreachable=0 failed=0 dhcp47-2.labb.eng.blr.redhat.com : ok=0 changed=0 unreachable=1 failed=0 Ignoring errors... PLAY [gluster_servers] ***************************************************************************************************************************************************************** TASK [Pcs cluster authenticate the hacluster users in all the nodes] ******************************************************************************************************************* failed: [dhcp47-2.labb.eng.blr.redhat.com] (item=dhcp47-193.lab.eng.blr.redhat.com) => {"item": "dhcp47-193.lab.eng.blr.redhat.com", "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true} failed: [dhcp47-2.labb.eng.blr.redhat.com] (item=dhcp46-116.lab.eng.blr.redhat.com) => {"item": "dhcp46-116.lab.eng.blr.redhat.com", "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true} failed: [dhcp47-2.labb.eng.blr.redhat.com] (item=dhcp46-184.lab.eng.blr.redhat.com) => {"item": "dhcp46-184.lab.eng.blr.redhat.com", "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true} failed: [dhcp47-2.labb.eng.blr.redhat.com] (item=dhcp47-2.labb.eng.blr.redhat.com) => {"item": "dhcp47-2.labb.eng.blr.redhat.com", "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true} fatal: [dhcp47-2.labb.eng.blr.redhat.com]: UNREACHABLE! => {"changed": false, "msg": "All items completed", "results": [{"_ansible_ignore_errors": null, "_ansible_item_result": true, "item": "dhcp47-193.lab.eng.blr.redhat.com", "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true}, {"_ansible_ignore_errors": null, "_ansible_item_result": true, "item": "dhcp46-116.lab.eng.blr.redhat.com", "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true}, {"_ansible_ignore_errors": null, "_ansible_item_result": true, "item": "dhcp46-184.lab.eng.blr.redhat.com", "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true}, {"_ansible_ignore_errors": null, "_ansible_item_result": true, "item": "dhcp47-2.labb.eng.blr.redhat.com", "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true}]} changed: [dhcp46-116.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com) changed: [dhcp46-184.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com) changed: [dhcp47-193.lab.eng.blr.redhat.com] => (item=dhcp47-193.lab.eng.blr.redhat.com) changed: [dhcp46-116.lab.eng.blr.redhat.com] => (item=dhcp46-116.lab.eng.blr.redhat.com) changed: [dhcp46-184.lab.eng.blr.redhat.com] => (item=dhcp46-116.lab.eng.blr.redhat.com) changed: [dhcp47-193.lab.eng.blr.redhat.com] => (item=dhcp46-116.lab.eng.blr.redhat.com) changed: [dhcp46-116.lab.eng.blr.redhat.com] => (item=dhcp46-184.lab.eng.blr.redhat.com) changed: [dhcp46-184.lab.eng.blr.redhat.com] => (item=dhcp46-184.lab.eng.blr.redhat.com) changed: [dhcp47-193.lab.eng.blr.redhat.com] => (item=dhcp46-184.lab.eng.blr.redhat.com) failed: [dhcp46-116.lab.eng.blr.redhat.com] (item=dhcp47-2.labb.eng.blr.redhat.com) => {"changed": true, "cmd": "pcs cluster auth -u hacluster -p hacluster dhcp47-2.labb.eng.blr.redhat.com", "delta": "0:00:00.827062", "end": "2018-03-26 12:55:42.810885", "item": "dhcp47-2.labb.eng.blr.redhat.com", "msg": "non-zero return code", "rc": 1, "start": "2018-03-26 12:55:41.983823", "stderr": "Error: Unable to communicate with dhcp47-2.labb.eng.blr.redhat.com", "stderr_lines": ["Error: Unable to communicate with dhcp47-2.labb.eng.blr.redhat.com"], "stdout": "", "stdout_lines": []} failed: [dhcp46-184.lab.eng.blr.redhat.com] (item=dhcp47-2.labb.eng.blr.redhat.com) => {"changed": true, "cmd": "pcs cluster auth -u hacluster -p hacluster dhcp47-2.labb.eng.blr.redhat.com", "delta": "0:00:00.934702", "end": "2018-03-26 12:55:43.317563", "item": "dhcp47-2.labb.eng.blr.redhat.com", "msg": "non-zero return code", "rc": 1, "start": "2018-03-26 12:55:42.382861", "stderr": "Error: Unable to communicate with dhcp47-2.labb.eng.blr.redhat.com", "stderr_lines": ["Error: Unable to communicate with dhcp47-2.labb.eng.blr.redhat.com"], "stdout": "", "stdout_lines": []} failed: [dhcp47-193.lab.eng.blr.redhat.com] (item=dhcp47-2.labb.eng.blr.redhat.com) => {"changed": true, "cmd": "pcs cluster auth -u hacluster -p hacluster dhcp47-2.labb.eng.blr.redhat.com", "delta": "0:00:01.003482", "end": "2018-03-26 12:55:43.387661", "item": "dhcp47-2.labb.eng.blr.redhat.com", "msg": "non-zero return code", "rc": 1, "start": "2018-03-26 12:55:42.384179", "stderr": "Error: Unable to communicate with dhcp47-2.labb.eng.blr.redhat.com", "stderr_lines": ["Error: Unable to communicate with dhcp47-2.labb.eng.blr.redhat.com"], "stdout": "", "stdout_lines": []} to retry, use: --limit @/tmp/tmp0hDpxC/pcs-authentication.retry PLAY RECAP ***************************************************************************************************************************************************************************** dhcp46-116.lab.eng.blr.redhat.com : ok=0 changed=0 unreachable=0 failed=1 dhcp46-184.lab.eng.blr.redhat.com : ok=0 changed=0 unreachable=0 failed=1 dhcp47-193.lab.eng.blr.redhat.com : ok=0 changed=0 unreachable=0 failed=1 dhcp47-2.labb.eng.blr.redhat.com : ok=0 changed=0 unreachable=1 failed=0 Ignoring errors... PLAY [master_node] ********************************************************************************************************************************************************************* TASK [Create configuration directory for nfs-ganesha] ********************************************************************************************************************************** changed: [dhcp47-193.lab.eng.blr.redhat.com] TASK [Copy ganesha.conf to config directory on shared volume] ************************************************************************************************************************** changed: [dhcp47-193.lab.eng.blr.redhat.com] TASK [Creating ganesha-ha.conf file] *************************************************************************************************************************************************** changed: [dhcp47-193.lab.eng.blr.redhat.com] PLAY RECAP ***************************************************************************************************************************************************************************** dhcp47-193.lab.eng.blr.redhat.com : ok=3 changed=3 unreachable=0 failed=0 PLAY [gluster_servers] ***************************************************************************************************************************************************************** TASK [Define service port] ************************************************************************************************************************************************************* fatal: [dhcp47-2.labb.eng.blr.redhat.com]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname dhcp47-2.labb.eng.blr.redhat.com: Name or service not known\r\n", "unreachable": true} changed: [dhcp46-116.lab.eng.blr.redhat.com] changed: [dhcp47-193.lab.eng.blr.redhat.com] changed: [dhcp46-184.lab.eng.blr.redhat.com] to retry, use: --limit @/tmp/tmp0hDpxC/define_service_ports.retry PLAY RECAP ***************************************************************************************************************************************************************************** dhcp46-116.lab.eng.blr.redhat.com : ok=1 changed=1 unreachable=0 failed=0 dhcp46-184.lab.eng.blr.redhat.com : ok=1 changed=1 unreachable=0 failed=0 dhcp47-193.lab.eng.blr.redhat.com : ok=1 changed=1 unreachable=0 failed=0 dhcp47-2.labb.eng.blr.redhat.com : ok=0 changed=0 unreachable=1 failed=0 Ignoring errors... PLAY [master_node] ********************************************************************************************************************************************************************* TASK [Enable nfs-ganesha] ************************************************************************************************************************************************************** fatal: [dhcp47-193.lab.eng.blr.redhat.com]: FAILED! => {"changed": true, "cmd": "gluster nfs-ganesha enable --mode=script", "delta": "0:00:49.946201", "end": "2018-03-26 12:56:44.035225", "failed_when_result": true, "msg": "non-zero return code", "rc": 1, "start": "2018-03-26 12:55:54.089024", "stderr": "nfs-ganesha: failed: Failed to set up HA config for NFS-Ganesha. Please check the log file for details", "stderr_lines": ["nfs-ganesha: failed: Failed to set up HA config for NFS-Ganesha. Please check the log file for details"], "stdout": "This will take a few minutes to complete. Please wait ..", "stdout_lines": ["This will take a few minutes to complete. Please wait .."]} ...ignoring TASK [Pause for 30 seconds (takes a while to enable NFS Ganesha)] ********************************************************************************************************************** Pausing for 30 seconds (ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort) ok: [dhcp47-193.lab.eng.blr.redhat.com] TASK [Check NFS Ganesha status] ******************************************************************************************************************************************************** changed: [dhcp47-193.lab.eng.blr.redhat.com] TASK [Report NFS Ganesha status] ******************************************************************************************************************************************************* skipping: [dhcp47-193.lab.eng.blr.redhat.com] TASK [Report NFS Ganesha status (If any errors)] *************************************************************************************************************************************** ok: [dhcp47-193.lab.eng.blr.redhat.com] => { "msg": "Error: cluster is not currently running on this node" } PLAY RECAP ***************************************************************************************************************************************************************************** dhcp47-193.lab.eng.blr.redhat.com : ok=4 changed=2 unreachable=0 failed=0 ========================================= ganesha-ha.conf [root@dhcp46-116 ganesha]# cat /var/run/gluster/shared_storage/nfs-ganesha/ganesha-ha.conf # Provide a unique name for the cluster. HA_NAME="ganesha-ha-360" # The subset of nodes of the Gluster Trusted Storage Pool that forms the ganesha # HA cluster. Hostname should specified, IP addresses are not allowed. # Maximum number of 16 nodes are supported. HA_CLUSTER_NODES="dhcp47-193.lab.eng.blr.redhat.com,dhcp46-116.lab.eng.blr.redhat.com,dhcp46-184.lab.eng.blr.redhat.com,dhcp47-2.labb.eng.blr.redhat.com" # Virtual IPs of each of the nodes specified above. VIP_dhcp47-193.lab.eng.blr.redhat.com="10.70.46.51" VIP_dhcp46-116.lab.eng.blr.redhat.com="10.70.46.54" VIP_dhcp46-184.lab.eng.blr.redhat.com="10.70.46.55" VIP_dhcp47-2.labb.eng.blr.redhat.com="10.70.46.56" ========================================== # service nfs-ganesha status Redirecting to /bin/systemctl status nfs-ganesha.service ● nfs-ganesha.service - NFS-Ganesha file server Loaded: loaded (/usr/lib/systemd/system/nfs-ganesha.service; enabled; vendor preset: disabled) Active: active (running) since Mon 2018-03-26 12:56:16 IST; 1h 50min ago Docs: http://github.com/nfs-ganesha/nfs-ganesha/wiki Process: 4001 ExecStartPost=/bin/bash -c /usr/bin/sleep 2 && /bin/dbus-send --system --dest=org.ganesha.nfsd --type=method_call /org/ganesha/nfsd/admin org.ganesha.nfsd.admin.init_fds_limit (code=exited, status=0/SUCCESS) Process: 3997 ExecStartPost=/bin/bash -c prlimit --pid $MAINPID --nofile=$NOFILE:$NOFILE (code=exited, status=0/SUCCESS) Process: 3995 ExecStart=/bin/bash -c ${NUMACTL} ${NUMAOPTS} /usr/bin/ganesha.nfsd ${OPTIONS} ${EPOCH} (code=exited, status=0/SUCCESS) Main PID: 3996 (ganesha.nfsd) Tasks: 272 CGroup: /system.slice/nfs-ganesha.service └─3996 /usr/bin/ganesha.nfsd -L /var/log/ganesha/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT -E 6537151427721232384 Mar 26 12:56:14 dhcp46-116.lab.eng.blr.redhat.com systemd[1]: Starting NFS-Ganesha file server... Mar 26 12:56:16 dhcp46-116.lab.eng.blr.redhat.com systemd[1]: Started NFS-Ganesha file server. Since the nfs-ganesha service is observed been running post setup creation fails on 2 out of 4 nodes and even symlinks are not cleaned.Based on these 2 observation moving this BZ to Assigned state.
Verified this BZ on # rpm -qa | grep ganesha glusterfs-ganesha-3.12.2-7.el7rhgs.x86_64 nfs-ganesha-2.5.5-3.el7rhgs.x86_64 nfs-ganesha-gluster-2.5.5-3.el7rhgs.x86_64 Intentionally tried failing the "gluster nfs-ganesha enable" command.The ganesha service was not found running.Works as expected. Moving this BZ to verified state. Have raised a separate Bug[1] for the use case in comment #13.This needs to be fixed in Gdeploy. [1]https://bugzilla.redhat.com/show_bug.cgi?id=1563923
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:2607