Bug 1463711

Summary: [GANESHA] Volume having ganesha enable on it fails to get re-export on rebooted node
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Manisha Saini <msaini>
Component: nfs-ganeshaAssignee: Daniel Gryniewicz <dang>
Status: CLOSED ERRATA QA Contact: Manisha Saini <msaini>
Severity: unspecified Docs Contact:
Priority: urgent    
Version: rhgs-3.3CC: amukherj, dang, jthottan, kkeithle, rhinduja, rhs-bugs, skoduri, storage-qa-internal
Target Milestone: ---   
Target Release: RHGS 3.3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: 3.3.0-devel-freeze-exception
Fixed In Version: nfs-ganesha-2.4.4-10 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-09-21 04:47:57 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1417151    

Description Manisha Saini 2017-06-21 14:35:42 UTC 
Description of problem:
Volume having ganesha enable on it fails to get re-export on rebooted node.

Version-Release number of selected component (if applicable):
# rpm -qa | grep ganesha
nfs-ganesha-2.4.4-8.el7rhgs.x86_64
glusterfs-ganesha-3.8.4-25.el7rhgs.x86_64
nfs-ganesha-debuginfo-2.4.4-8.el7rhgs.x86_64
nfs-ganesha-gluster-2.4.4-8.el7rhgs.x86_64


How reproducible:
2/2

Steps to Reproduce:
1.Create a 4 node ganesha cluster
2.Create 3 volume.Enable ganesha on it.(Export the volume one by one)
3.Stop the 1st two volume.
4.Reboot 1 of the node
5.When the node comes up,Volume does not get re exported on rebooted node

Actual results:
3rd Volume should get re exported on the rebooted node 

Expected results:
Volume failed to export

Additional info:


ganesha.log
============================

21/06/2017 19:51:50 : epoch 4a170000 : dhcp42-114.lab.eng.blr.redhat.com : ganesha.nfsd-3283[main] init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper.
21/06/2017 19:51:50 : epoch 4a170000 : dhcp42-114.lab.eng.blr.redhat.com : ganesha.nfsd-3283[main] init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized.
21/06/2017 19:51:50 : epoch 4a170000 : dhcp42-114.lab.eng.blr.redhat.com : ganesha.nfsd-3283[main] glusterfs_create_export :FSAL :EVENT :Volume ganeshavol1 exported at : '/'
21/06/2017 19:51:54 : epoch 4a170000 : dhcp42-114.lab.eng.blr.redhat.com : ganesha.nfsd-3283[main] glusterfs_get_fs :FSAL :CRIT :Unable to initialize volume. Volume: ganeshavol1
21/06/2017 19:51:55 : epoch 4a170000 : dhcp42-114.lab.eng.blr.redhat.com : ganesha.nfsd-3283[main] mdcache_fsal_create_export :FSAL :MAJ :Failed to call create_export on underlying FSAL GLUSTER
21/06/2017 19:51:55 : epoch 4a170000 : dhcp42-114.lab.eng.blr.redhat.com : ganesha.nfsd-3283[main] fsal_cfg_commit :CONFIG :CRIT :Could not create export for (/ganeshavol1) to (/ganeshavol1)
21/06/2017 19:51:55 : epoch 4a170000 : dhcp42-114.lab.eng.blr.redhat.com : ganesha.nfsd-3283[main] glusterfs_create_export :FSAL :EVENT :Volume ganeshavol3 exported at : '/'
21/06/2017 19:51:59 : epoch 4a170000 : dhcp42-114.lab.eng.blr.redhat.com : ganesha.nfsd-3283[main] glusterfs_get_fs :FSAL :CRIT :Unable to initialize volume. Volume: ganeshavol3
21/06/2017 19:52:00 : epoch 4a170000 : dhcp42-114.lab.eng.blr.redhat.com : ganesha.nfsd-3283[main] mdcache_fsal_create_export :FSAL :MAJ :Failed to call create_export on underlying FSAL GLUSTER
21/06/2017 19:52:00 : epoch 4a170000 : dhcp42-114.lab.eng.blr.redhat.com : ganesha.nfsd-3283[main] fsal_cfg_commit :CONFIG :CRIT :Could not create export for (/ganeshavol3) to (/ganeshavol3)
21/06/2017 19:52:00 : epoch 4a170000 : dhcp42-114.lab.eng.blr.redhat.com : ganesha.nfsd-3283[main] glusterfs_create_export :FSAL :EVENT :Volume ganeshavol2 exported at : '/'
21/06/2017 19:52:05 : epoch 4a170000 : dhcp42-114.lab.eng.blr.redhat.com : ganesha.nfsd-3283[main] fsal_attach_export :CONFIG :CRIT :Attaching export with out holding a reference!. hdl= = 0x0x7f9bf8f683d0
21/06/2017 19:52:05 : epoch 4a170000 : dhcp42-114.lab.eng.blr.redhat.com : ganesha.nfsd-3283[main] glusterfs_create_export :FSAL :CRIT :Unable to attach export. Export: /ganeshavol2
21/06/2017 19:52:05 : epoch 4a170000 : dhcp42-114.lab.eng.blr.redhat.com : ganesha.nfsd-3283[main] mdcache_fsal_create_export :FSAL :MAJ :Failed to call create_export on underlying FSAL GLUSTER
21/06/2017 19:52:05 : epoch 4a170000 : dhcp42-114.lab.eng.blr.redhat.com : ganesha.nfsd-3283[main] fsal_cfg_commit :CONFIG :CRIT :Could not create export for (/ganeshavol2) to (/ganeshavol2)
21/06/2017 19:52:05 : epoch 4a170000 : dhcp42-114.lab.eng.blr.redhat.com : ganesha.nfsd-3283[main] main :NFS STARTUP :WARN :No export entries found in configuration file !!!
21/06/2017 19:52:05 : epoch 4a170000 : dhcp42-114.lab.eng.blr.redhat.com : ganesha.nfsd-3283[main] config_errs_to_log :CONFIG :CRIT :Config File (/var/run/gluster/shared_storage/nfs-ganesha/exports/export.ganeshavol1.conf:7): 1 validation errors in block FSAL
21/06/2017 19:52:05 : epoch 4a170000 : dhcp42-114.lab.eng.blr.redhat.com : ganesha.nfsd-3283[main] config_errs_to_log :CONFIG :CRIT :Config File (/var/run/gluster/shared_storage/nfs-ganesha/exports/export.ganeshavol1.conf:7): Errors processing block (FSAL)
21/06/2017 19:52:05 : epoch 4a170000 : dhcp42-114.lab.eng.blr.redhat.com : ganesha.nfsd-3283[main] config_errs_to_log :CONFIG :CRIT :Config File (/var/run/gluster/shared_storage/nfs-ganesha/exports/export.ganeshavol1.conf:4): 1 validation errors in block EXPORT
21/06/2017 19:52:05 : epoch 4a170000 : dhcp42-114.lab.eng.blr.redhat.com : ganesha.nfsd-3283[main] config_errs_to_log :CONFIG :CRIT :Config File (/var/run/gluster/shared_storage/nfs-ganesha/exports/export.ganeshavol1.conf:4): Errors processing block (EXPORT)
21/06/2017 19:52:05 : epoch 4a170000 : dhcp42-114.lab.eng.blr.redhat.com : ganesha.nfsd-3283[main] config_errs_to_log :CONFIG :CRIT :Config File (/var/run/gluster/shared_storage/nfs-ganesha/exports/export.ganeshavol3.conf:7): 1 validation errors in block FSAL
21/06/2017 19:52:05 : epoch 4a170000 : dhcp42-114.lab.eng.blr.redhat.com : ganesha.nfsd-3283[main] config_errs_to_log :CONFIG :CRIT :Config File (/var/run/gluster/shared_storage/nfs-ganesha/exports/export.ganeshavol3.conf:7): Errors processing block (FSAL)
21/06/2017 19:52:05 : epoch 4a170000 : dhcp42-114.lab.eng.blr.redhat.com : ganesha.nfsd-3283[main] config_errs_to_log :CONFIG :CRIT :Config File (/var/run/gluster/shared_storage/nfs-ganesha/exports/export.ganeshavol3.conf:4): 1 validation errors in block EXPORT
===================================


==================

Order in which volume were exported-

}
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.ganeshavol1.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.ganeshavol3.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.ganeshavol2.conf"
========================

Rebooted Node-
[root@dhcp42-114 ~]# showmount -e
Export list for dhcp42-114.lab.eng.blr.redhat.com:


One of the other node-
[root@dhcp42-107 ganesha]# showmount -e
Export list for dhcp42-107.lab.eng.blr.redhat.com:
/ganeshavol2 (everyone)




Will attach the sosreports shortly
Comment 2 Daniel Gryniewicz 2017-06-21 14:51:57 UTC 
This is fixed in 2.5.x already, and a backport is going into 2.4.x as I type this.  It's a simple 3-line fix.

I believe this should be included if at all possible.  It seems a problem that a client will probably hit at some point (an export for a volume that's not ready).  And, the problem is severe:  The only possible way out of this situation is to restart Ganesha, and it has the potential to corrupt memory or crash.
Comment 6 Manisha Saini 2017-06-29 13:40:56 UTC 
Verified this bug on 

# rpm -qa | grep ganesha
nfs-ganesha-gluster-2.4.4-10.el7rhgs.x86_64
nfs-ganesha-debuginfo-2.4.4-10.el7rhgs.x86_64
glusterfs-ganesha-3.8.4-31.el7rhgs.x86_64
nfs-ganesha-2.4.4-10.el7rhgs.x86_64


Volume get exported on rebooted node.
Hence moving this to verified state.
Comment 8 errata-xmlrpc 2017-09-21 04:47:57 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:2779