Fedora Account System
Red Hat Associate
Red Hat Customer
Description of problem: ----------------------- I can't bring up a Ganesha cluster on 3.8.4-38 : <snip> [root@gqas009 /]# gluster nfs-ganesha enable Enabling NFS-Ganesha requires Gluster-NFS to be disabled across the trusted pool. Do you still want to continue? (y/n) y This will take a few minutes to complete. Please wait .. nfs-ganesha: failed: creation of symlink ganesha.conf in /etc/ganesha failed <snip> [root@gqas009 /]# sestatus SELinux status: disabled [root@gqas009 /]# It's a regression introduced b/w gluster 37 and 38 build.I can enable Ganesha on 3.8.4-37. It's coz of the recent fixes that wnet into fixing https://bugzilla.redhat.com/show_bug.cgi?id=1466144 (semanage commands inside ganesha-ha.sh). I tried setting up manually,it errors out with an OSError on the semanage commands: <snip> [root@gqas009 ganesha]# ganesha-ha.sh --setup /var/run/gluster/shared_storage/nfs-ganesha/ -bash: ganesha-ha.sh: command not found [root@gqas009 ganesha]# [root@gqas009 ganesha]# ./ganesha-ha.sh --setup /var/run/gluster/shared_storage/nfs-ganesha/ SELinux: Could not downgrade policy file /etc/selinux/targeted/policy/policy.30, searching for an older version. SELinux: Could not open policy file <= /etc/selinux/targeted/policy/policy.30: No such file or directory /sbin/load_policy: Can't load policy: No such file or directory libsemanage.semanage_reload_policy: load_policy returned error code 2. (No such file or directory). SELinux: Could not downgrade policy file /etc/selinux/targeted/policy/policy.30, searching for an older version. SELinux: Could not open policy file <= /etc/selinux/targeted/policy/policy.30: No such file or directory /sbin/load_policy: Can't load policy: No such file or directory libsemanage.semanage_reload_policy: load_policy returned error code 2. (No such file or directory). OSError: No such file or directory gqas016.sbu.lab.eng.bos.redhat.com: Already authorized gqas009.sbu.lab.eng.bos.redhat.com: Already authorized Destroying cluster on nodes: gqas009.sbu.lab.eng.bos.redhat.com, gqas016.sbu.lab.eng.bos.redhat.com... gqas016.sbu.lab.eng.bos.redhat.com: Stopping Cluster (pacemaker)... gqas009.sbu.lab.eng.bos.redhat.com: Stopping Cluster (pacemaker)... gqas016.sbu.lab.eng.bos.redhat.com: Successfully destroyed cluster gqas009.sbu.lab.eng.bos.redhat.com: Successfully destroyed cluster Sending 'pacemaker_remote authkey' to 'gqas009.sbu.lab.eng.bos.redhat.com', 'gqas016.sbu.lab.eng.bos.redhat.com' gqas009.sbu.lab.eng.bos.redhat.com: successful distribution of the file 'pacemaker_remote authkey' gqas016.sbu.lab.eng.bos.redhat.com: successful distribution of the file 'pacemaker_remote authkey' Sending cluster config files to the nodes... gqas009.sbu.lab.eng.bos.redhat.com: Succeeded gqas016.sbu.lab.eng.bos.redhat.com: Succeeded Synchronizing pcsd certificates on nodes gqas009.sbu.lab.eng.bos.redhat.com, gqas016.sbu.lab.eng.bos.redhat.com... gqas016.sbu.lab.eng.bos.redhat.com: Success gqas009.sbu.lab.eng.bos.redhat.com: Success Restarting pcsd on the nodes in order to reload the certificates... gqas016.sbu.lab.eng.bos.redhat.com: Success gqas009.sbu.lab.eng.bos.redhat.com: Success gqas016.sbu.lab.eng.bos.redhat.com: Starting Cluster... gqas009.sbu.lab.eng.bos.redhat.com: Starting Cluster... Adding nfs-grace-clone gqas009.sbu.lab.eng.bos.redhat.com-cluster_ip-1 (kind: Mandatory) (Options: first-action=start then-action=start) Adding nfs-grace-clone gqas016.sbu.lab.eng.bos.redhat.com-cluster_ip-1 (kind: Mandatory) (Options: first-action=start then-action=start) CIB updated Warning: Identity file /var/lib/glusterd/nfs/secret.pem not accessible: No such file or directory. Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). Warning: Identity file /var/lib/glusterd/nfs/secret.pem not accessible: No such file or directory. Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). SELinux: Could not downgrade policy file /etc/selinux/targeted/policy/policy.30, searching for an older version. SELinux: Could not open policy file <= /etc/selinux/targeted/policy/policy.30: No such file or directory /sbin/load_policy: Can't load policy: No such file or directory libsemanage.semanage_reload_policy: load_policy returned error code 2. (No such file or directory). SELinux: Could not downgrade policy file /etc/selinux/targeted/policy/policy.30, searching for an older version. SELinux: Could not open policy file <= /etc/selinux/targeted/policy/policy.30: No such file or directory /sbin/load_policy: Can't load policy: No such file or directory libsemanage.semanage_reload_policy: load_policy returned error code 2. (No such file or directory). OSError: No such file or directory [root@gqas009 ganesha]# </snip> Version-Release number of selected component (if applicable): ------------------------------------------------------------- 3.8.4-38 How reproducible: ----------------- All the time,on selinux disabled machines.
Patch posted upstream for review https://review.gluster.org/#/c/18264/
I tested with SELinux set to permissive, not disabled, but got the same error message on gluster v3.8 and a workaround. Workaround: moved the conf files out from under /var/run/gluster/shared_storage (the Install doc for Ganesha on Gluster says to "copy" these conf files from /etc/ganesha to /var/run/gluster/shared_storage, so they exist in both locations). Is this a valid workaround? [jp@host3 ~]$ sudo gluster --version glusterfs 3.8.4 built on Jan 4 2018 03:26:44 [jp@host3 ~]$ sudo gluster nfs-ganesha enable Enabling NFS-Ganesha requires Gluster-NFS to be disabled across the trusted pool. Do you still want to continue? (y/n) y This will take a few minutes to complete. Please wait .. nfs-ganesha: failed: creation of symlink ganesha.conf in /etc/ganesha failed [jp@host3 ~]$ sudo setenforce permissive [jp@host3 ~]$ sudo mv /var/run/gluster/shared_storage/nfs-ganesha/ganesha.conf /etc/ganesha/ganesha.conf [jp@host3 ~]$ sudo mv /var/run/gluster/shared_storage/nfs-ganesha/ganesha-ha.conf /etc/ganesha/ganesha-ha.conf [jp@host3 ~]$ sudo gluster nfs-ganesha enable Enabling NFS-Ganesha requires Gluster-NFS to be disabled across the trusted pool. Do you still want to continue? (y/n) y This will take a few minutes to complete. Please wait .. nfs-ganesha : success
Verified this with- # rpm -qa | grep ganesha glusterfs-ganesha-3.12.2-7.el7rhgs.x86_64 nfs-ganesha-2.5.5-5.el7rhgs.x86_64 nfs-ganesha-debuginfo-2.5.5-5.el7rhgs.x86_64 nfs-ganesha-gluster-2.5.5-5.el7rhgs.x86_64 [root@dhcp37-121 ganesha]# getenforce Disabled ----------------------- # pcs status Cluster name: ganesha-ha-360 Stack: corosync Current DC: dhcp37-136.lab.eng.blr.redhat.com (version 1.1.18-11.el7-2b07d5c5a9) - partition with quorum Last updated: Sat Apr 21 06:21:40 2018 Last change: Sat Apr 21 16:48:07 2018 by root via cibadmin on dhcp37-121.lab.eng.blr.redhat.com 4 nodes configured 24 resources configured Online: [ dhcp37-103.lab.eng.blr.redhat.com dhcp37-121.lab.eng.blr.redhat.com dhcp37-136.lab.eng.blr.redhat.com dhcp37-218.lab.eng.blr.redhat.com ] Full list of resources: Clone Set: nfs_setup-clone [nfs_setup] Started: [ dhcp37-103.lab.eng.blr.redhat.com dhcp37-121.lab.eng.blr.redhat.com dhcp37-136.lab.eng.blr.redhat.com dhcp37-218.lab.eng.blr.redhat.com ] Clone Set: nfs-mon-clone [nfs-mon] Started: [ dhcp37-103.lab.eng.blr.redhat.com dhcp37-121.lab.eng.blr.redhat.com dhcp37-136.lab.eng.blr.redhat.com dhcp37-218.lab.eng.blr.redhat.com ] Clone Set: nfs-grace-clone [nfs-grace] Started: [ dhcp37-103.lab.eng.blr.redhat.com dhcp37-121.lab.eng.blr.redhat.com dhcp37-136.lab.eng.blr.redhat.com dhcp37-218.lab.eng.blr.redhat.com ] Resource Group: dhcp37-121.lab.eng.blr.redhat.com-group dhcp37-121.lab.eng.blr.redhat.com-nfs_block (ocf::heartbeat:portblock): Started dhcp37-121.lab.eng.blr.redhat.com dhcp37-121.lab.eng.blr.redhat.com-cluster_ip-1 (ocf::heartbeat:IPaddr): Started dhcp37-121.lab.eng.blr.redhat.com dhcp37-121.lab.eng.blr.redhat.com-nfs_unblock (ocf::heartbeat:portblock): Started dhcp37-121.lab.eng.blr.redhat.com Resource Group: dhcp37-103.lab.eng.blr.redhat.com-group dhcp37-103.lab.eng.blr.redhat.com-nfs_block (ocf::heartbeat:portblock): Started dhcp37-103.lab.eng.blr.redhat.com dhcp37-103.lab.eng.blr.redhat.com-cluster_ip-1 (ocf::heartbeat:IPaddr): Started dhcp37-103.lab.eng.blr.redhat.com dhcp37-103.lab.eng.blr.redhat.com-nfs_unblock (ocf::heartbeat:portblock): Started dhcp37-103.lab.eng.blr.redhat.com Resource Group: dhcp37-218.lab.eng.blr.redhat.com-group dhcp37-218.lab.eng.blr.redhat.com-nfs_block (ocf::heartbeat:portblock): Started dhcp37-218.lab.eng.blr.redhat.com dhcp37-218.lab.eng.blr.redhat.com-cluster_ip-1 (ocf::heartbeat:IPaddr): Started dhcp37-218.lab.eng.blr.redhat.com dhcp37-218.lab.eng.blr.redhat.com-nfs_unblock (ocf::heartbeat:portblock): Started dhcp37-218.lab.eng.blr.redhat.com Resource Group: dhcp37-136.lab.eng.blr.redhat.com-group dhcp37-136.lab.eng.blr.redhat.com-nfs_block (ocf::heartbeat:portblock): Started dhcp37-136.lab.eng.blr.redhat.com dhcp37-136.lab.eng.blr.redhat.com-cluster_ip-1 (ocf::heartbeat:IPaddr): Started dhcp37-136.lab.eng.blr.redhat.com dhcp37-136.lab.eng.blr.redhat.com-nfs_unblock (ocf::heartbeat:portblock): Started dhcp37-136.lab.eng.blr.redhat.com Daemon Status: corosync: active/disabled pacemaker: active/disabled pcsd: active/enabled ----------------------- Gdeploy output-- ---------------------- TASK [Enable nfs-ganesha] ***************************************************************************************************************************************************************************************** changed: [dhcp37-121.lab.eng.blr.redhat.com] TASK [Pause for 30 seconds (takes a while to enable NFS Ganesha)] ************************************************************************************************************************************************* Pausing for 30 seconds (ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort) ok: [dhcp37-121.lab.eng.blr.redhat.com] TASK [Check NFS Ganesha status] *********************************************************************************************************************************************************************************** changed: [dhcp37-121.lab.eng.blr.redhat.com] TASK [Report NFS Ganesha status] ********************************************************************************************************************************************************************************** ok: [dhcp37-121.lab.eng.blr.redhat.com] => { "msg": "Cluster HA Status: HEALTHY" } TASK [Report NFS Ganesha status (If any errors)] ****************************************************************************************************************************************************************** skipping: [dhcp37-121.lab.eng.blr.redhat.com] PLAY RECAP ******************************************************************************************************************************************************************************************************** dhcp37-121.lab.eng.blr.redhat.com : ok=4 changed=2 unreachable=0 failed=0 ------------------------ Moving this BZ to verified state.