Bug 1398513 - [GSS] nfs-ganesha export failing each time trying to export it using gluster command or dbus_send.sh command
Summary: [GSS] nfs-ganesha export failing each time trying to export it using gluster ...
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: nfs-ganesha
Version: rhgs-3.1
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: ---
: RHGS 3.2.0
Assignee: Kaleb KEITHLEY
QA Contact: surabhi
URL:
Whiteboard:
Depends On:
Blocks: 1369781
TreeView+ depends on / blocked
 
Reported: 2016-11-25 04:36 UTC by Bipin Kunal
Modified: 2020-01-17 16:15 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-02-09 12:41:28 UTC
Target Upstream Version:

Attachments (Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

Description Bipin Kunal 2016-11-25 04:36:15 UTC 
Description of problem:
nfs-ganesha export failing each time trying to export it using gluster command or dbus_send.sh command


Version-Release number of selected component (if applicable):
root@ens-gfs-03 ganesha]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.2 (Maipo)

[root@ens-gfs-03 ganesha]# cat /etc/redhat-storage-release 
Red Hat Gluster Storage Server 3.1 Update 3


The ganesha.nfsd failing with "lookup_export failed with Export id not found", the ganesha.log showing below error messages when ran with NIV_DEBUG option

24/11/2016 13:34:59 : epoch c6270000 : <server> : ganesha.nfsd-5249[work-11] cache_inode_close :INODE :CRIT :FSAL_close failed, returning 37(CACHE_INODE_SERVERFAULT) for entry 0x7f154815bab0
24/11/2016 13:34:59 : epoch c6270000 : <server> : ganesha.nfsd-5249[work-11] cache_inode_lru_clean :INODE LRU :CRIT :Error closing file in cleanup: 37.
24/11/2016 13:35:01 : epoch c6270000 : <server> : ganesha.nfsd-5249[work-4] cache_inode_lookup_impl :INODE :EVENT :FSAL returned STALE from a lookup.
24/11/2016 18:53:53 : epoch c6270000 : <server> : ganesha.nfsd-5249[dbus_heartbeat] glusterfs_create_export :FSAL :EVENT :Volume ens_volume01 exported at : '/'
24/11/2016 18:54:02 : epoch c6270000 : <server> : ganesha.nfsd-5249[work-12] nfs3_Errno_verbose :NFS3 :CRIT :Error CACHE_INODE_IO_ERROR in nfs3_lookup converted to NFS3ERR_IO but was set non-retryable
24/11/2016 18:54:02 : epoch c6270000 : <server> : ganesha.nfsd-5249[work-3] nfs3_Errno_verbose :NFS3 :CRIT :Error CACHE_INODE_IO_ERROR in nfs3_lookup converted to NFS3ERR_IO but was set non-retryable
24/11/2016 18:54:02 : epoch c6270000 : <server> : ganesha.nfsd-5249[work-15] nfs3_Errno_verbose :NFS3 :CRIT :Error CACHE_INODE_IO_ERROR in nfs3_lookup converted to NFS3ERR_IO but was set non-retryable
24/11/2016 18:54:02 : epoch c6270000 : <server> : ganesha.nfsd-5249[work-9] nfs3_Errno_verbose :NFS3 :CRIT :Error CACHE_INODE_IO_ERROR in nfs3_lookup converted to NFS3ERR_IO but was set non-retryable
24/11/2016 18:54:03 : epoch c6270000 : <server> : ganesha.nfsd-5249[work-12] nfs3_Errno_verbose :NFS3 :CRIT :Error CACHE_INODE_IO_ERROR in nfs3_lookup converted to NFS3ERR_IO but was set non-retryable
24/11/2016 18:54:09 : epoch c6270000 : <server> : ganesha.nfsd-5249[work-8] cache_inode_lookup_impl :INODE :EVENT :FSAL returned STALE from a lookup.
24/11/2016 20:00:11 : epoch c6270000 : <server> : ganesha.nfsd-5249[dbus_heartbeat] dbus_message_entrypoint :DBUS :MAJ :Method (RemoveExport) on (org.ganesha.nfsd.exportmgr) failed: name = (org.freedesktop.DBus.Error.InvalidArgs), message = (lookup_export failed with Export id not found)
Comment 2 Bipin Kunal 2016-11-25 04:37:52 UTC 
 Actions Performed:
         - ens_volume01 gluster volume is a tier volume and "gluster status" showing tier migration is in progress.
         - The nfs-ganesha export goes offline as soon as we export the nfs-ganesha by "gluster volume set ens_volume01 ganesha.enable on" command
         - the ganesha.log file showing the NFS lookup failure due to inode caching
              Error CACHE_INODE_IO_ERROR in nfs3_lookup converted to NFS3ERR_IO but was set non-retryable 
         - the ganesha-gfapi.log showing "remote operation failed" and split-brain messages
         - pcs status looks good
         - ganesha-ha.sh --status looks good
         - we tried to export the gluster volume through nfs-ganesha again using "/usr/libexec/ganesha/dbus-send.sh /etc/ganesha on ens_volume01" within few seconds it again went to offline
         - "showmount -e localhost" showing no entry for ens_volume01
         - Restarted ens_volume01 volume
         - reexported volume, but still went offline
         - Executed "gluster volume set ens_volume01 ganesha.enable off"
         - Executed "gluster nfs-ganesha disable"
         - Executed "gluster volume set all cluster.enable-shared-storage disable"
         - Executed "gluster volume set all cluster.enable-shared-storage enable"
         - Executed "gluster nfs-ganesha enable"
         - Executed "gluster volume set ens_volume01 ganesha.enable on"
         - No chance in export behavior for ens_volume01
         - killed ganesha.nfsd process from ens-gfs-03 volume and started in debug mode using NIV_FULL_DEBUG
         - Executed "/usr/libexec/ganesha/dbus-send.sh /etc/ganesha on ens_volume01" command to reexport volume
         - Uploaded the latest debug ganesha.log and ganesha-gfapi.log for in detailed analysis
Comment 10 Bipin Kunal 2017-02-09 12:09:35 UTC 
Sure Soumya, Please close the Bug.
Comment 11 Soumya Koduri 2017-02-09 12:41:28 UTC 
Thanks Bipin.
Note You need to log in before you can comment on or make changes to this bug.