Bug 1566483

Summary: [Ganesha] Observing "Failed to open v4 recovery dir" message in ganesha.log when ganesha cluster goes to 90 seconds of GRACE period and selinux is in ENFORCING mode
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Manisha Saini <msaini>
Component: nfs-ganeshaAssignee: Kaleb KEITHLEY <kkeithle>
Status: CLOSED WONTFIX QA Contact: Manisha Saini <msaini>
Severity: low Docs Contact:
Priority: unspecified    
Version: rhgs-3.4CC: amukherj, dang, ffilz, grajoria, jthottan, msaini, pasik, rhs-bugs, storage-qa-internal
Target Milestone: ---Keywords: ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1569344 (view as bug list) Environment:
Last Closed: 2018-11-19 10:36:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1569344    

Description Manisha Saini 2018-04-12 12:14:46 UTC 
Description of problem:

Observing ""Failed to open v4 recovery dir (/var/lib/nfs/ganesha//v4recov), errno=13" " message in ganesha.log when ganesha cluster goes to 90 seconds of GRACE period and selinux is in ENFORCING mode


Version-Release number of selected component (if applicable):


selinux-policy-3.13.1-192.el7_5.3.noarch

# rpm -qa | grep ganesha
nfs-ganesha-gluster-2.5.5-4.el7rhgs.x86_64
glusterfs-ganesha-3.12.2-7.el7rhgs.x86_64
nfs-ganesha-2.5.5-4.el7rhgs.x86_64


How reproducible:
100%


Steps to Reproduce:
1.Create 4 node ganesha cluster.Sel Selinux to ENFORCING node in all the 4 nodes
2.Reboot any one node.


Actual results:
Post reboot,when ganesha cluster goes to 90 seconds to  GRACE period,observing "Failed to open v4 recovery dir (/var/lib/nfs/ganesha//v4recov), errno=13" messages in ganesha.log 


=============

12/04/2018 17:34:55 : epoch 9e940000 : dhcp46-184.lab.eng.blr.redhat.com : ganesha.nfsd-23203[dbus_heartbeat] nfs4_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90
12/04/2018 17:34:55 : epoch 9e940000 : dhcp46-184.lab.eng.blr.redhat.com : ganesha.nfsd-23203[dbus_heartbeat] nfs4_start_grace :STATE :EVENT :NFS Server recovery event 5 nodeid -1 ip 
12/04/2018 17:34:55 : epoch 9e940000 : dhcp46-184.lab.eng.blr.redhat.com : ganesha.nfsd-23203[dbus_heartbeat] fs_read_recov_clids_takeover :CLIENT ID :EVENT :Recovery for nodeid -1 dir (/var/lib/nfs/ganesha//v4recov)
12/04/2018 17:34:55 : epoch 9e940000 : dhcp46-184.lab.eng.blr.redhat.com : ganesha.nfsd-23203[dbus_heartbeat] fs_read_recov_clids_impl :CLIENT ID :EVENT :Failed to open v4 recovery dir (/var/lib/nfs/ganesha//v4recov), errno=13
12/04/2018 17:34:55 : epoch 9e940000 : dhcp46-184.lab.eng.blr.redhat.com : ganesha.nfsd-23203[dbus_heartbeat] fs_read_recov_clids_takeover :CLIENT ID :EVENT :Failed to read v4 recovery dir (/var/lib/nfs/ganesha//v4recov)
12/04/2018 17:36:28 : epoch 9e940000 : dhcp46-184.lab.eng.blr.redhat.com : ganesha.nfsd-23203[reaper] nfs_in_grace :STATE :EVENT :NFS Server Now NOT IN GRACE

==========================




Expected results:
There should not be any failures when ganesha nodes goes to 90 seconds of GRACE period 


Additional info:

These messages are not been observed when selinux is set to PERMISSIVE mode.
Comment 2 Frank Filz 2018-04-12 12:59:23 UTC 
What is the ownership of the recovery directory? This sounds like an issue with ownership of the directory and maybe some SELinux stuff.
Comment 4 Daniel Gryniewicz 2018-04-12 14:35:41 UTC 
Can you include ls -Z output?
Comment 6 Daniel Gryniewicz 2018-04-12 17:33:39 UTC 
Okay, so I'm pretty sure that when SELinux is enforcing, the files need to be ganesha_t not fusefs_t.