Bug 1478897

Summary:	[Ganesha] : Unable to bring up a Ganesha HA cluster on SELinux disabled machines on latest gluster bits.
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Ambarish <asoman>
Component:	common-ha	Assignee:	Kaleb KEITHLEY <kkeithle>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Manisha Saini <msaini>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	rhgs-3.3	CC:	amukherj, bturner, dang, ffilz, jthottan, jupittma, kkeithle, mbenjamin, msaini, rhinduja, rhs-bugs, sheggodu, skoduri, storage-qa-internal
Target Milestone:	---
Target Release:	RHGS 3.4.0
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	glusterfs-3.12.2-2	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1490909 (view as bug list)		Environment:
Last Closed:	2018-09-06 04:18:16 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1490909
Bug Blocks:	1503134

Description Ambarish 2017-08-07 13:02:18 UTC

Description of problem:
-----------------------

I can't bring up a Ganesha cluster on 3.8.4-38 :

<snip>

[root@gqas009 /]# gluster nfs-ganesha enable
Enabling NFS-Ganesha requires Gluster-NFS to be disabled across the trusted pool. Do you still want to continue?
 (y/n) y
This will take a few minutes to complete. Please wait ..
nfs-ganesha: failed: creation of symlink ganesha.conf in /etc/ganesha failed

<snip>


[root@gqas009 /]# sestatus 
SELinux status:                 disabled
[root@gqas009 /]# 



It's a regression introduced b/w gluster 37 and 38 build.I can enable Ganesha on 3.8.4-37.


It's coz of the recent fixes that wnet into fixing https://bugzilla.redhat.com/show_bug.cgi?id=1466144 (semanage commands inside ganesha-ha.sh).


I tried setting up manually,it errors out with an OSError on the semanage commands:

<snip>

[root@gqas009 ganesha]# ganesha-ha.sh --setup  /var/run/gluster/shared_storage/nfs-ganesha/
-bash: ganesha-ha.sh: command not found
[root@gqas009 ganesha]# 
[root@gqas009 ganesha]# ./ganesha-ha.sh --setup  /var/run/gluster/shared_storage/nfs-ganesha/
SELinux:  Could not downgrade policy file /etc/selinux/targeted/policy/policy.30, searching for an older version.
SELinux:  Could not open policy file <= /etc/selinux/targeted/policy/policy.30:  No such file or directory
/sbin/load_policy:  Can't load policy:  No such file or directory
libsemanage.semanage_reload_policy: load_policy returned error code 2. (No such file or directory).
SELinux:  Could not downgrade policy file /etc/selinux/targeted/policy/policy.30, searching for an older version.
SELinux:  Could not open policy file <= /etc/selinux/targeted/policy/policy.30:  No such file or directory
/sbin/load_policy:  Can't load policy:  No such file or directory
libsemanage.semanage_reload_policy: load_policy returned error code 2. (No such file or directory).
OSError: No such file or directory
gqas016.sbu.lab.eng.bos.redhat.com: Already authorized
gqas009.sbu.lab.eng.bos.redhat.com: Already authorized
Destroying cluster on nodes: gqas009.sbu.lab.eng.bos.redhat.com, gqas016.sbu.lab.eng.bos.redhat.com...
gqas016.sbu.lab.eng.bos.redhat.com: Stopping Cluster (pacemaker)...
gqas009.sbu.lab.eng.bos.redhat.com: Stopping Cluster (pacemaker)...
gqas016.sbu.lab.eng.bos.redhat.com: Successfully destroyed cluster
gqas009.sbu.lab.eng.bos.redhat.com: Successfully destroyed cluster

Sending 'pacemaker_remote authkey' to 'gqas009.sbu.lab.eng.bos.redhat.com', 'gqas016.sbu.lab.eng.bos.redhat.com'
gqas009.sbu.lab.eng.bos.redhat.com: successful distribution of the file 'pacemaker_remote authkey'
gqas016.sbu.lab.eng.bos.redhat.com: successful distribution of the file 'pacemaker_remote authkey'
Sending cluster config files to the nodes...
gqas009.sbu.lab.eng.bos.redhat.com: Succeeded
gqas016.sbu.lab.eng.bos.redhat.com: Succeeded

Synchronizing pcsd certificates on nodes gqas009.sbu.lab.eng.bos.redhat.com, gqas016.sbu.lab.eng.bos.redhat.com...
gqas016.sbu.lab.eng.bos.redhat.com: Success
gqas009.sbu.lab.eng.bos.redhat.com: Success
Restarting pcsd on the nodes in order to reload the certificates...
gqas016.sbu.lab.eng.bos.redhat.com: Success
gqas009.sbu.lab.eng.bos.redhat.com: Success
gqas016.sbu.lab.eng.bos.redhat.com: Starting Cluster...
gqas009.sbu.lab.eng.bos.redhat.com: Starting Cluster...



Adding nfs-grace-clone gqas009.sbu.lab.eng.bos.redhat.com-cluster_ip-1 (kind: Mandatory) (Options: first-action=start then-action=start)
Adding nfs-grace-clone gqas016.sbu.lab.eng.bos.redhat.com-cluster_ip-1 (kind: Mandatory) (Options: first-action=start then-action=start)
CIB updated
Warning: Identity file /var/lib/glusterd/nfs/secret.pem not accessible: No such file or directory.
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
Warning: Identity file /var/lib/glusterd/nfs/secret.pem not accessible: No such file or directory.
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
SELinux:  Could not downgrade policy file /etc/selinux/targeted/policy/policy.30, searching for an older version.
SELinux:  Could not open policy file <= /etc/selinux/targeted/policy/policy.30:  No such file or directory
/sbin/load_policy:  Can't load policy:  No such file or directory
libsemanage.semanage_reload_policy: load_policy returned error code 2. (No such file or directory).
SELinux:  Could not downgrade policy file /etc/selinux/targeted/policy/policy.30, searching for an older version.
SELinux:  Could not open policy file <= /etc/selinux/targeted/policy/policy.30:  No such file or directory
/sbin/load_policy:  Can't load policy:  No such file or directory
libsemanage.semanage_reload_policy: load_policy returned error code 2. (No such file or directory).
OSError: No such file or directory
[root@gqas009 ganesha]# 
</snip>
 

Version-Release number of selected component (if applicable):
-------------------------------------------------------------

3.8.4-38

How reproducible:
-----------------

All the time,on selinux disabled machines.

Comment 8 Jiffin 2017-09-14 11:37:14 UTC

Patch posted upstream for review https://review.gluster.org/#/c/18264/

Comment 10 Justin Pittman 2018-03-04 05:05:24 UTC

I tested with SELinux set to permissive, not disabled, but got the same error message on gluster v3.8 and a workaround.

Workaround: moved the conf files out from under /var/run/gluster/shared_storage (the Install doc for Ganesha on Gluster says to "copy" these conf files from /etc/ganesha to /var/run/gluster/shared_storage, so they exist in both locations).   Is this a valid workaround?

[jp@host3 ~]$ sudo gluster --version
glusterfs 3.8.4 built on Jan  4 2018 03:26:44
[jp@host3 ~]$ sudo gluster nfs-ganesha enable
Enabling NFS-Ganesha requires Gluster-NFS to be disabled across the trusted pool. Do you still want to continue?
 (y/n) y
This will take a few minutes to complete. Please wait ..
nfs-ganesha: failed: creation of symlink ganesha.conf in /etc/ganesha failed
[jp@host3 ~]$ sudo setenforce permissive
[jp@host3 ~]$ sudo mv /var/run/gluster/shared_storage/nfs-ganesha/ganesha.conf /etc/ganesha/ganesha.conf
[jp@host3 ~]$ sudo mv /var/run/gluster/shared_storage/nfs-ganesha/ganesha-ha.conf /etc/ganesha/ganesha-ha.conf
[jp@host3 ~]$ sudo gluster nfs-ganesha enable
Enabling NFS-Ganesha requires Gluster-NFS to be disabled across the trusted pool. Do you still want to continue?
 (y/n) y
This will take a few minutes to complete. Please wait ..
nfs-ganesha : success

Comment 11 Manisha Saini 2018-04-21 11:24:09 UTC

Verified this with-

# rpm -qa | grep ganesha
glusterfs-ganesha-3.12.2-7.el7rhgs.x86_64
nfs-ganesha-2.5.5-5.el7rhgs.x86_64
nfs-ganesha-debuginfo-2.5.5-5.el7rhgs.x86_64
nfs-ganesha-gluster-2.5.5-5.el7rhgs.x86_64


[root@dhcp37-121 ganesha]# getenforce
Disabled

-----------------------
# pcs status
Cluster name: ganesha-ha-360
Stack: corosync
Current DC: dhcp37-136.lab.eng.blr.redhat.com (version 1.1.18-11.el7-2b07d5c5a9) - partition with quorum
Last updated: Sat Apr 21 06:21:40 2018
Last change: Sat Apr 21 16:48:07 2018 by root via cibadmin on dhcp37-121.lab.eng.blr.redhat.com

4 nodes configured
24 resources configured

Online: [ dhcp37-103.lab.eng.blr.redhat.com dhcp37-121.lab.eng.blr.redhat.com dhcp37-136.lab.eng.blr.redhat.com dhcp37-218.lab.eng.blr.redhat.com ]

Full list of resources:

 Clone Set: nfs_setup-clone [nfs_setup]
     Started: [ dhcp37-103.lab.eng.blr.redhat.com dhcp37-121.lab.eng.blr.redhat.com dhcp37-136.lab.eng.blr.redhat.com dhcp37-218.lab.eng.blr.redhat.com ]
 Clone Set: nfs-mon-clone [nfs-mon]
     Started: [ dhcp37-103.lab.eng.blr.redhat.com dhcp37-121.lab.eng.blr.redhat.com dhcp37-136.lab.eng.blr.redhat.com dhcp37-218.lab.eng.blr.redhat.com ]
 Clone Set: nfs-grace-clone [nfs-grace]
     Started: [ dhcp37-103.lab.eng.blr.redhat.com dhcp37-121.lab.eng.blr.redhat.com dhcp37-136.lab.eng.blr.redhat.com dhcp37-218.lab.eng.blr.redhat.com ]
 Resource Group: dhcp37-121.lab.eng.blr.redhat.com-group
     dhcp37-121.lab.eng.blr.redhat.com-nfs_block	(ocf::heartbeat:portblock):	Started dhcp37-121.lab.eng.blr.redhat.com
     dhcp37-121.lab.eng.blr.redhat.com-cluster_ip-1	(ocf::heartbeat:IPaddr):	Started dhcp37-121.lab.eng.blr.redhat.com
     dhcp37-121.lab.eng.blr.redhat.com-nfs_unblock	(ocf::heartbeat:portblock):	Started dhcp37-121.lab.eng.blr.redhat.com
 Resource Group: dhcp37-103.lab.eng.blr.redhat.com-group
     dhcp37-103.lab.eng.blr.redhat.com-nfs_block	(ocf::heartbeat:portblock):	Started dhcp37-103.lab.eng.blr.redhat.com
     dhcp37-103.lab.eng.blr.redhat.com-cluster_ip-1	(ocf::heartbeat:IPaddr):	Started dhcp37-103.lab.eng.blr.redhat.com
     dhcp37-103.lab.eng.blr.redhat.com-nfs_unblock	(ocf::heartbeat:portblock):	Started dhcp37-103.lab.eng.blr.redhat.com
 Resource Group: dhcp37-218.lab.eng.blr.redhat.com-group
     dhcp37-218.lab.eng.blr.redhat.com-nfs_block	(ocf::heartbeat:portblock):	Started dhcp37-218.lab.eng.blr.redhat.com
     dhcp37-218.lab.eng.blr.redhat.com-cluster_ip-1	(ocf::heartbeat:IPaddr):	Started dhcp37-218.lab.eng.blr.redhat.com
     dhcp37-218.lab.eng.blr.redhat.com-nfs_unblock	(ocf::heartbeat:portblock):	Started dhcp37-218.lab.eng.blr.redhat.com
 Resource Group: dhcp37-136.lab.eng.blr.redhat.com-group
     dhcp37-136.lab.eng.blr.redhat.com-nfs_block	(ocf::heartbeat:portblock):	Started dhcp37-136.lab.eng.blr.redhat.com
     dhcp37-136.lab.eng.blr.redhat.com-cluster_ip-1	(ocf::heartbeat:IPaddr):	Started dhcp37-136.lab.eng.blr.redhat.com
     dhcp37-136.lab.eng.blr.redhat.com-nfs_unblock	(ocf::heartbeat:portblock):	Started dhcp37-136.lab.eng.blr.redhat.com

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled
-----------------------


Gdeploy output--

----------------------


TASK [Enable nfs-ganesha] *****************************************************************************************************************************************************************************************
changed: [dhcp37-121.lab.eng.blr.redhat.com]

TASK [Pause for 30 seconds (takes a while to enable NFS Ganesha)] *************************************************************************************************************************************************
Pausing for 30 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
ok: [dhcp37-121.lab.eng.blr.redhat.com]

TASK [Check NFS Ganesha status] ***********************************************************************************************************************************************************************************
changed: [dhcp37-121.lab.eng.blr.redhat.com]

TASK [Report NFS Ganesha status] **********************************************************************************************************************************************************************************
ok: [dhcp37-121.lab.eng.blr.redhat.com] => {
    "msg": "Cluster HA Status: HEALTHY"
}

TASK [Report NFS Ganesha status (If any errors)] ******************************************************************************************************************************************************************
skipping: [dhcp37-121.lab.eng.blr.redhat.com]

PLAY RECAP ********************************************************************************************************************************************************************************************************
dhcp37-121.lab.eng.blr.redhat.com : ok=4    changed=2    unreachable=0    failed=0   

------------------------

Moving this BZ to verified state.