Bug 1398513

Summary: [GSS] nfs-ganesha export failing each time trying to export it using gluster command or dbus_send.sh command
Product: Red Hat Gluster Storage Reporter: Bipin Kunal <bkunal>
Component: nfs-ganeshaAssignee: Kaleb KEITHLEY <kkeithle>
Status: CLOSED WORKSFORME QA Contact: surabhi <sbhaloth>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: rhgs-3.1CC: amukherj, bkunal, rhs-bugs, rnalakka, skoduri, storage-qa-internal
Target Milestone: ---   
Target Release: RHGS 3.2.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-02-09 12:41:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1369781    

Description Bipin Kunal 2016-11-25 04:36:15 UTC 
Description of problem:
nfs-ganesha export failing each time trying to export it using gluster command or dbus_send.sh command


Version-Release number of selected component (if applicable):
root@ens-gfs-03 ganesha]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.2 (Maipo)

[root@ens-gfs-03 ganesha]# cat /etc/redhat-storage-release 
Red Hat Gluster Storage Server 3.1 Update 3


The ganesha.nfsd failing with "lookup_export failed with Export id not found", the ganesha.log showing below error messages when ran with NIV_DEBUG option

24/11/2016 13:34:59 : epoch c6270000 : <server> : ganesha.nfsd-5249[work-11] cache_inode_close :INODE :CRIT :FSAL_close failed, returning 37(CACHE_INODE_SERVERFAULT) for entry 0x7f154815bab0
24/11/2016 13:34:59 : epoch c6270000 : <server> : ganesha.nfsd-5249[work-11] cache_inode_lru_clean :INODE LRU :CRIT :Error closing file in cleanup: 37.
24/11/2016 13:35:01 : epoch c6270000 : <server> : ganesha.nfsd-5249[work-4] cache_inode_lookup_impl :INODE :EVENT :FSAL returned STALE from a lookup.
24/11/2016 18:53:53 : epoch c6270000 : <server> : ganesha.nfsd-5249[dbus_heartbeat] glusterfs_create_export :FSAL :EVENT :Volume ens_volume01 exported at : '/'
24/11/2016 18:54:02 : epoch c6270000 : <server> : ganesha.nfsd-5249[work-12] nfs3_Errno_verbose :NFS3 :CRIT :Error CACHE_INODE_IO_ERROR in nfs3_lookup converted to NFS3ERR_IO but was set non-retryable
24/11/2016 18:54:02 : epoch c6270000 : <server> : ganesha.nfsd-5249[work-3] nfs3_Errno_verbose :NFS3 :CRIT :Error CACHE_INODE_IO_ERROR in nfs3_lookup converted to NFS3ERR_IO but was set non-retryable
24/11/2016 18:54:02 : epoch c6270000 : <server> : ganesha.nfsd-5249[work-15] nfs3_Errno_verbose :NFS3 :CRIT :Error CACHE_INODE_IO_ERROR in nfs3_lookup converted to NFS3ERR_IO but was set non-retryable
24/11/2016 18:54:02 : epoch c6270000 : <server> : ganesha.nfsd-5249[work-9] nfs3_Errno_verbose :NFS3 :CRIT :Error CACHE_INODE_IO_ERROR in nfs3_lookup converted to NFS3ERR_IO but was set non-retryable
24/11/2016 18:54:03 : epoch c6270000 : <server> : ganesha.nfsd-5249[work-12] nfs3_Errno_verbose :NFS3 :CRIT :Error CACHE_INODE_IO_ERROR in nfs3_lookup converted to NFS3ERR_IO but was set non-retryable
24/11/2016 18:54:09 : epoch c6270000 : <server> : ganesha.nfsd-5249[work-8] cache_inode_lookup_impl :INODE :EVENT :FSAL returned STALE from a lookup.
24/11/2016 20:00:11 : epoch c6270000 : <server> : ganesha.nfsd-5249[dbus_heartbeat] dbus_message_entrypoint :DBUS :MAJ :Method (RemoveExport) on (org.ganesha.nfsd.exportmgr) failed: name = (org.freedesktop.DBus.Error.InvalidArgs), message = (lookup_export failed with Export id not found)
Comment 2 Bipin Kunal 2016-11-25 04:37:52 UTC 
 Actions Performed:
         - ens_volume01 gluster volume is a tier volume and "gluster status" showing tier migration is in progress.
         - The nfs-ganesha export goes offline as soon as we export the nfs-ganesha by "gluster volume set ens_volume01 ganesha.enable on" command
         - the ganesha.log file showing the NFS lookup failure due to inode caching
              Error CACHE_INODE_IO_ERROR in nfs3_lookup converted to NFS3ERR_IO but was set non-retryable 
         - the ganesha-gfapi.log showing "remote operation failed" and split-brain messages
         - pcs status looks good
         - ganesha-ha.sh --status looks good
         - we tried to export the gluster volume through nfs-ganesha again using "/usr/libexec/ganesha/dbus-send.sh /etc/ganesha on ens_volume01" within few seconds it again went to offline
         - "showmount -e localhost" showing no entry for ens_volume01
         - Restarted ens_volume01 volume
         - reexported volume, but still went offline
         - Executed "gluster volume set ens_volume01 ganesha.enable off"
         - Executed "gluster nfs-ganesha disable"
         - Executed "gluster volume set all cluster.enable-shared-storage disable"
         - Executed "gluster volume set all cluster.enable-shared-storage enable"
         - Executed "gluster nfs-ganesha enable"
         - Executed "gluster volume set ens_volume01 ganesha.enable on"
         - No chance in export behavior for ens_volume01
         - killed ganesha.nfsd process from ens-gfs-03 volume and started in debug mode using NIV_FULL_DEBUG
         - Executed "/usr/libexec/ganesha/dbus-send.sh /etc/ganesha on ens_volume01" command to reexport volume
         - Uploaded the latest debug ganesha.log and ganesha-gfapi.log for in detailed analysis
Comment 10 Bipin Kunal 2017-02-09 12:09:35 UTC 
Sure Soumya, Please close the Bug.
Comment 11 Soumya Koduri 2017-02-09 12:41:28 UTC 
Thanks Bipin.