Bug 1469561 - [GANESHA] Upgrade nfs-ganesha from 3.2 to 3.2 async is breaking due to selinux boolean ganesha_use_fusefs in off state
[GANESHA] Upgrade nfs-ganesha from 3.2 to 3.2 async is breaking due to selinu...
Status: CLOSED WONTFIX
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: nfs-ganesha (Show other bugs)
3.2
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Kaleb KEITHLEY
Manisha Saini
: ZStream
Depends On:
Blocks: 1470136 1470040
  Show dependency treegraph
 
Reported: 2017-07-11 09:51 EDT by Manisha Saini
Modified: 2017-07-14 00:12 EDT (History)
11 users (show)

See Also:
Fixed In Version:
Doc Type: Release Note
Doc Text:
Story Points: ---
Clone Of:
: 1470040 1470136 (view as bug list)
Environment:
Last Closed: 2017-07-14 00:12:28 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Manisha Saini 2017-07-11 09:51:18 EDT
Description of problem:
Upgrade nfs-ganesha from 3.2 to 3.2 async is breaking due to selinux boolean ganesha_use_fusefs in off state

Version-Release number of selected component (if applicable):

# rpm -qa | grep ganesha
nfs-ganesha-debuginfo-2.4.1-11.el7rhgs.x86_64
nfs-ganesha-2.4.1-11.el7rhgs.x86_64
nfs-ganesha-gluster-2.4.1-11.el7rhgs.x86_64
glusterfs-ganesha-3.8.4-18.5.el7rhgs.x86_64


How reproducible:
1/1

Steps to Reproduce:
1.Create 4 node ganesha cluster up and running and upgrade it to latest async glusterfs-ganesha-3.8.4-18.5.el7rhgs.x86_64 following ganesha upgrade steps in offline mode.
After upgrade,before enabling nfs-ganesha,set gluster_use_execmem boolean to on

# setsebool -P gluster_use_execmem  on
[root@dhcp37-58 nfs-ganesha]# semanage boolean -l | grep  gluster_use_execmem
gluster_use_execmem            (on   ,   on)  Allow gluster to use execmem





Actual results:
ganesha service fails to come up and after upgrade ganesha_use_fusefs  boolean was in off state.

[root@dhcp37-58 ~]# semanage boolean -l | grep ganesha
ganesha_use_fusefs             (off  ,  off)  Allow ganesha to use fusefs


Expected results:
Ganesha upgrade should not break

Additional info:



================

#  ausearch -m avc -m user_avc -m selinux_err -i -ts recent | grep ganesha
type=SYSCALL msg=audit(07/11/2017 18:13:12.309:6326) : arch=x86_64 syscall=open success=no exit=EACCES(Permission denied) a0=0x7f57d4f9b490 a1=O_RDONLY a2=0x1b6 a3=0x24 items=0 ppid=1 pid=2852 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=ganesha.nfsd exe=/usr/bin/ganesha.nfsd subj=system_u:system_r:ganesha_t:s0 key=(null) 
type=AVC msg=audit(07/11/2017 18:13:12.309:6326) : avc:  denied  { search } for  pid=2852 comm=ganesha.nfsd name=/ dev="fuse" ino=1 scontext=system_u:system_r:ganesha_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=dir 
==============
# tailf /var/log/ganesha.log
11/07/2017 17:26:19 : epoch 42250000 : dhcp37-58.lab.eng.blr.redhat.com : ganesha.nfsd-7322[Admin] destroy_fsals :FSAL :EVENT :Exports for FSAL GLUSTER shut down
11/07/2017 17:26:19 : epoch 42250000 : dhcp37-58.lab.eng.blr.redhat.com : ganesha.nfsd-7322[Admin] destroy_fsals :FSAL :CRIT :Extra references (5) hanging around to FSAL GLUSTER
11/07/2017 17:26:19 : epoch 42250000 : dhcp37-58.lab.eng.blr.redhat.com : ganesha.nfsd-7322[Admin] destroy_fsals :FSAL :EVENT :Unloading FSAL GLUSTER
11/07/2017 17:26:19 : epoch 42250000 : dhcp37-58.lab.eng.blr.redhat.com : ganesha.nfsd-7322[Admin] destroy_fsals :FSAL :EVENT :FSAL GLUSTER unloaded
11/07/2017 17:26:19 : epoch 42250000 : dhcp37-58.lab.eng.blr.redhat.com : ganesha.nfsd-7322[Admin] do_shutdown :MAIN :EVENT :FSAL system destroyed.
11/07/2017 17:26:19 : epoch 42250000 : dhcp37-58.lab.eng.blr.redhat.com : ganesha.nfsd-7322[main] nfs_start :MAIN :EVENT :NFS EXIT: regular exit
11/07/2017 18:13:12 : epoch 42250000 : dhcp37-58.lab.eng.blr.redhat.com : ganesha.nfsd-2851[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version /builddir/build/BUILD/nfs-ganesha-2.4.1/src, built at Jun  7 2017 01:38:50 on 
11/07/2017 18:13:12 : epoch 42250000 : dhcp37-58.lab.eng.blr.redhat.com : ganesha.nfsd-2852[main] main :NFS STARTUP :CRIT :Error (token scan) while parsing (/etc/ganesha/ganesha.conf)
11/07/2017 18:13:12 : epoch 42250000 : dhcp37-58.lab.eng.blr.redhat.com : ganesha.nfsd-2852[main] config_errs_to_log :CONFIG :CRIT :Config File (<unknown file>:0): new file (/etc/ganesha/ganesha.conf) open error (Permission denied), ignored
11/07/2017 18:13:12 : epoch 42250000 : dhcp37-58.lab.eng.blr.redhat.com : ganesha.nfsd-2852[main] main :NFS STARTUP :FATAL :Fatal errors.  Server exiting...



Meanwhile I will check for freash installation of 3.2 async with RHEL 7.4 and will update the bug
Comment 2 Manisha Saini 2017-07-11 12:08:59 EDT
Fresh setup installation with 3.2 async works fine

ganesha_use_fusefs  boolean is set to on while doing gluster nfs-ganesha enable.


# semanage boolean -l | grep ganesha
ganesha_use_fusefs             (on   ,   on)  Allow ganesha to use fusefs


Issue is only observed in upgrade path
Comment 3 Kaleb KEITHLEY 2017-07-11 14:40:15 EDT
This is what I think is happening during an update:

According to the yum logs the system first had selinux-policy-targeted-3.13.1-102.el7_3.16. There is no ganesha_use_fusefs in this package.

Then the system was updated to RHEL-7.4. glusterfs-ganesha was updated at 18:00:12. Then at 18:00:37 selinux-policy-targeted was updated to 3.13.1-166.el7. This has ganesha_use_fusefs.

ganesha_use_fusefs still wasn't available when glusterfs-ganesha was updated so the semanage command (silently) failed.

rpm only allows a 'Requires: selinux-policy-targeted >= NV'. I.e. NV = 3.13.1. It doesn't allow a 'Requires: selinux-policy-targeted >= NVR'. I.e. NVR = 3.13.1-166.

Thus, for the purposes of upgrading, 3.13.1-102.el7_3.16 satisfies the Requires: but doesn't have the necessary ganesha_fuse_fusefs for the %post to work.

Of course on a fresh install you will get the correct version of selinux-policy-targeted and everything works as expected.

Off the top of my head the only way to force selinux-policy-targeted to be updated before glusterfs-ganesha is to explicitly update it first, before applying the rest of the update. IOW this has to be prominently documented in the Release Notes.
Comment 4 Kaleb KEITHLEY 2017-07-11 18:34:59 EDT
Perhaps Lukas can suggest some trick to make selinux-policy-targeted update before glusterfs-ganesha?
Comment 6 Manisha Saini 2017-07-12 07:07:29 EDT
Kaleb,

Yum update will pull all the packages all at once,ganesha and selinux packages.
We cannot only update the selinux package first followed by ganesha package.

However after upgrading both selinux and ganesha packages,we can document to enable this boolean manually before doing gluster nfs-ganesha enable.
I will verify this steps manually too following the upgrade path.

Need your opinion on this...
Comment 7 Kaleb KEITHLEY 2017-07-12 07:31:28 EDT
I believe Lukas' suggestion of using the %trigger to run the semanage command after selinux-policy-targeted is updated will fix this.
Comment 10 Manisha Saini 2017-07-12 08:48:08 EDT
Verified the path from 3.2 to 3.2 async.
After upgrade from 3.2 to 3.2 async, setting boolean ganesha_use_fusefs  to ON manually before enabling ganesha,works fine.

Raising the documentation bug for 3.2 async to set this boolean on manually after upgrade
Comment 11 Manisha Saini 2017-07-12 08:58:10 EDT
Raised the documentation bug for the same for 3.2 async- 
https://bugzilla.redhat.com/show_bug.cgi?id=1470146

Note You need to log in before you can comment on or make changes to this bug.