Description of problem: Volume having ganesha enable on it fails to get re-export on rebooted node. Version-Release number of selected component (if applicable): # rpm -qa | grep ganesha nfs-ganesha-2.4.4-8.el7rhgs.x86_64 glusterfs-ganesha-3.8.4-25.el7rhgs.x86_64 nfs-ganesha-debuginfo-2.4.4-8.el7rhgs.x86_64 nfs-ganesha-gluster-2.4.4-8.el7rhgs.x86_64 How reproducible: 2/2 Steps to Reproduce: 1.Create a 4 node ganesha cluster 2.Create 3 volume.Enable ganesha on it.(Export the volume one by one) 3.Stop the 1st two volume. 4.Reboot 1 of the node 5.When the node comes up,Volume does not get re exported on rebooted node Actual results: 3rd Volume should get re exported on the rebooted node Expected results: Volume failed to export Additional info: ganesha.log ============================ 21/06/2017 19:51:50 : epoch 4a170000 : dhcp42-114.lab.eng.blr.redhat.com : ganesha.nfsd-3283[main] init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper. 21/06/2017 19:51:50 : epoch 4a170000 : dhcp42-114.lab.eng.blr.redhat.com : ganesha.nfsd-3283[main] init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized. 21/06/2017 19:51:50 : epoch 4a170000 : dhcp42-114.lab.eng.blr.redhat.com : ganesha.nfsd-3283[main] glusterfs_create_export :FSAL :EVENT :Volume ganeshavol1 exported at : '/' 21/06/2017 19:51:54 : epoch 4a170000 : dhcp42-114.lab.eng.blr.redhat.com : ganesha.nfsd-3283[main] glusterfs_get_fs :FSAL :CRIT :Unable to initialize volume. Volume: ganeshavol1 21/06/2017 19:51:55 : epoch 4a170000 : dhcp42-114.lab.eng.blr.redhat.com : ganesha.nfsd-3283[main] mdcache_fsal_create_export :FSAL :MAJ :Failed to call create_export on underlying FSAL GLUSTER 21/06/2017 19:51:55 : epoch 4a170000 : dhcp42-114.lab.eng.blr.redhat.com : ganesha.nfsd-3283[main] fsal_cfg_commit :CONFIG :CRIT :Could not create export for (/ganeshavol1) to (/ganeshavol1) 21/06/2017 19:51:55 : epoch 4a170000 : dhcp42-114.lab.eng.blr.redhat.com : ganesha.nfsd-3283[main] glusterfs_create_export :FSAL :EVENT :Volume ganeshavol3 exported at : '/' 21/06/2017 19:51:59 : epoch 4a170000 : dhcp42-114.lab.eng.blr.redhat.com : ganesha.nfsd-3283[main] glusterfs_get_fs :FSAL :CRIT :Unable to initialize volume. Volume: ganeshavol3 21/06/2017 19:52:00 : epoch 4a170000 : dhcp42-114.lab.eng.blr.redhat.com : ganesha.nfsd-3283[main] mdcache_fsal_create_export :FSAL :MAJ :Failed to call create_export on underlying FSAL GLUSTER 21/06/2017 19:52:00 : epoch 4a170000 : dhcp42-114.lab.eng.blr.redhat.com : ganesha.nfsd-3283[main] fsal_cfg_commit :CONFIG :CRIT :Could not create export for (/ganeshavol3) to (/ganeshavol3) 21/06/2017 19:52:00 : epoch 4a170000 : dhcp42-114.lab.eng.blr.redhat.com : ganesha.nfsd-3283[main] glusterfs_create_export :FSAL :EVENT :Volume ganeshavol2 exported at : '/' 21/06/2017 19:52:05 : epoch 4a170000 : dhcp42-114.lab.eng.blr.redhat.com : ganesha.nfsd-3283[main] fsal_attach_export :CONFIG :CRIT :Attaching export with out holding a reference!. hdl= = 0x0x7f9bf8f683d0 21/06/2017 19:52:05 : epoch 4a170000 : dhcp42-114.lab.eng.blr.redhat.com : ganesha.nfsd-3283[main] glusterfs_create_export :FSAL :CRIT :Unable to attach export. Export: /ganeshavol2 21/06/2017 19:52:05 : epoch 4a170000 : dhcp42-114.lab.eng.blr.redhat.com : ganesha.nfsd-3283[main] mdcache_fsal_create_export :FSAL :MAJ :Failed to call create_export on underlying FSAL GLUSTER 21/06/2017 19:52:05 : epoch 4a170000 : dhcp42-114.lab.eng.blr.redhat.com : ganesha.nfsd-3283[main] fsal_cfg_commit :CONFIG :CRIT :Could not create export for (/ganeshavol2) to (/ganeshavol2) 21/06/2017 19:52:05 : epoch 4a170000 : dhcp42-114.lab.eng.blr.redhat.com : ganesha.nfsd-3283[main] main :NFS STARTUP :WARN :No export entries found in configuration file !!! 21/06/2017 19:52:05 : epoch 4a170000 : dhcp42-114.lab.eng.blr.redhat.com : ganesha.nfsd-3283[main] config_errs_to_log :CONFIG :CRIT :Config File (/var/run/gluster/shared_storage/nfs-ganesha/exports/export.ganeshavol1.conf:7): 1 validation errors in block FSAL 21/06/2017 19:52:05 : epoch 4a170000 : dhcp42-114.lab.eng.blr.redhat.com : ganesha.nfsd-3283[main] config_errs_to_log :CONFIG :CRIT :Config File (/var/run/gluster/shared_storage/nfs-ganesha/exports/export.ganeshavol1.conf:7): Errors processing block (FSAL) 21/06/2017 19:52:05 : epoch 4a170000 : dhcp42-114.lab.eng.blr.redhat.com : ganesha.nfsd-3283[main] config_errs_to_log :CONFIG :CRIT :Config File (/var/run/gluster/shared_storage/nfs-ganesha/exports/export.ganeshavol1.conf:4): 1 validation errors in block EXPORT 21/06/2017 19:52:05 : epoch 4a170000 : dhcp42-114.lab.eng.blr.redhat.com : ganesha.nfsd-3283[main] config_errs_to_log :CONFIG :CRIT :Config File (/var/run/gluster/shared_storage/nfs-ganesha/exports/export.ganeshavol1.conf:4): Errors processing block (EXPORT) 21/06/2017 19:52:05 : epoch 4a170000 : dhcp42-114.lab.eng.blr.redhat.com : ganesha.nfsd-3283[main] config_errs_to_log :CONFIG :CRIT :Config File (/var/run/gluster/shared_storage/nfs-ganesha/exports/export.ganeshavol3.conf:7): 1 validation errors in block FSAL 21/06/2017 19:52:05 : epoch 4a170000 : dhcp42-114.lab.eng.blr.redhat.com : ganesha.nfsd-3283[main] config_errs_to_log :CONFIG :CRIT :Config File (/var/run/gluster/shared_storage/nfs-ganesha/exports/export.ganeshavol3.conf:7): Errors processing block (FSAL) 21/06/2017 19:52:05 : epoch 4a170000 : dhcp42-114.lab.eng.blr.redhat.com : ganesha.nfsd-3283[main] config_errs_to_log :CONFIG :CRIT :Config File (/var/run/gluster/shared_storage/nfs-ganesha/exports/export.ganeshavol3.conf:4): 1 validation errors in block EXPORT =================================== ================== Order in which volume were exported- } %include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.ganeshavol1.conf" %include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.ganeshavol3.conf" %include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.ganeshavol2.conf" ======================== Rebooted Node- [root@dhcp42-114 ~]# showmount -e Export list for dhcp42-114.lab.eng.blr.redhat.com: One of the other node- [root@dhcp42-107 ganesha]# showmount -e Export list for dhcp42-107.lab.eng.blr.redhat.com: /ganeshavol2 (everyone) Will attach the sosreports shortly
This is fixed in 2.5.x already, and a backport is going into 2.4.x as I type this. It's a simple 3-line fix. I believe this should be included if at all possible. It seems a problem that a client will probably hit at some point (an export for a volume that's not ready). And, the problem is severe: The only possible way out of this situation is to restart Ganesha, and it has the potential to corrupt memory or crash.
Verified this bug on # rpm -qa | grep ganesha nfs-ganesha-gluster-2.4.4-10.el7rhgs.x86_64 nfs-ganesha-debuginfo-2.4.4-10.el7rhgs.x86_64 glusterfs-ganesha-3.8.4-31.el7rhgs.x86_64 nfs-ganesha-2.4.4-10.el7rhgs.x86_64 Volume get exported on rebooted node. Hence moving this to verified state.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:2779