Bug 1551186

Summary: [Ganesha] Duplicate volume export entries in ganesha.conf causing volume unexport to fail
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Manisha Saini <msaini>
Component: nfs-ganeshaAssignee: Jiffin <jthottan>
Status: CLOSED ERRATA QA Contact: Manisha Saini <msaini>
Severity: unspecified Docs Contact:
Priority: urgent    
Version: rhgs-3.4CC: amukherj, dang, dblack, ffilz, grajoria, jthottan, rhinduja, rhs-bugs, storage-qa-internal
Target Milestone: ---   
Target Release: RHGS 3.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-3.12.2-6 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1555195 (view as bug list) Environment:
Last Closed: 2018-09-04 06:44:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1555195    
Bug Blocks: 1503137, 1566569    

Description Manisha Saini 2018-03-03 06:47:50 UTC
Description of problem:

While exporting multiple volumes,duplicate volume entries are been observed in /etc/ganesha/ganesha.conf which leads to failure while unexporting volumes

Version-Release number of selected component (if applicable):

# rpm -qa | grep ganesha
nfs-ganesha-2.5.5-2.el7rhgs.x86_64
nfs-ganesha-gluster-2.5.5-2.el7rhgs.x86_64
glusterfs-ganesha-3.12.2-4.el7rhgs.x86_64


How reproducible:
3/3


Steps to Reproduce:

1.Create 4 node ganesha cluster
2.Create and start 25 volumes
3.Export all volumes via ganesha
for i in $(seq 1 25)
do
        gluster v set vol$i ganesha.enable on
        sleep 10
done
4.Unexport the ganesha volumes
for i in $(seq 1 25)
do
        gluster v set vol$i ganesha.enable off
        sleep 10
done

Actual results:
In /etc/ganesha/ganesha.conf ,duplicate volume export entries are observed causing volume unexport failed.

========
#/etc/ganesha/ganesha.conf


EXPORT_DEFAULTS {
	# To reflect nfsnobody
        Anonymous_uid = 65534;
        Anonymous_gid = 65534;
}
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol1.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol2.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol3.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol4.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol5.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol6.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol7.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol8.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol9.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol10.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol11.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol12.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol13.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol14.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol15.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol15.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol16.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol17.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol18.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol18.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol19.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol20.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol20.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol21.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol21.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol22.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol23.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol23.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol24.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol25.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol25.conf"



[root@dhcp37-218 exports]# ls
export.vol10.conf  export.vol14.conf  export.vol18.conf  export.vol21.conf  export.vol25.conf  export.vol5.conf  export.vol9.conf
export.vol11.conf  export.vol15.conf  export.vol19.conf  export.vol22.conf  export.vol2.conf   export.vol6.conf
export.vol12.conf  export.vol16.conf  export.vol1.conf   export.vol23.conf  export.vol3.conf   export.vol7.conf
export.vol13.conf  export.vol17.conf  export.vol20.conf  export.vol24.conf  export.vol4.conf   export.vol8.conf


# showmount -e
Export list for dhcp37-218.lab.eng.blr.redhat.com:
/vol1  (everyone)
/vol2  (everyone)
/vol3  (everyone)
/vol4  (everyone)
/vol5  (everyone)
/vol6  (everyone)
/vol7  (everyone)
/vol8  (everyone)
/vol9  (everyone)
/vol10 (everyone)
/vol11 (everyone)
/vol12 (everyone)
/vol13 (everyone)
/vol14 (everyone)
/vol15 (everyone)
/vol16 (everyone)
/vol17 (everyone)
/vol18 (everyone)
/vol19 (everyone)
/vol20 (everyone)
/vol21 (everyone)
/vol22 (everyone)
/vol23 (everyone)
/vol24 (everyone)
/vol25 (everyone)





#/var/log/ganesha/ganesha.log

03/03/2018 11:54:45 : epoch 10430000 : dhcp37-121.lab.eng.blr.redhat.com : ganesha.nfsd-6103[dbus_heartbeat] glusterfs_create_export :FSAL :EVENT :Volume vol1 exported at : '/'
03/03/2018 11:54:55 : epoch 10430000 : dhcp37-121.lab.eng.blr.redhat.com : ganesha.nfsd-6103[dbus_heartbeat] glusterfs_create_export :FSAL :EVENT :Volume vol2 exported at : '/'
03/03/2018 11:55:06 : epoch 10430000 : dhcp37-121.lab.eng.blr.redhat.com : ganesha.nfsd-6103[dbus_heartbeat] glusterfs_create_export :FSAL :EVENT :Volume vol3 exported at : '/'
03/03/2018 11:55:17 : epoch 10430000 : dhcp37-121.lab.eng.blr.redhat.com : ganesha.nfsd-6103[dbus_heartbeat] glusterfs_create_export :FSAL :EVENT :Volume vol4 exported at : '/'
03/03/2018 11:55:27 : epoch 10430000 : dhcp37-121.lab.eng.blr.redhat.com : ganesha.nfsd-6103[dbus_heartbeat] glusterfs_create_export :FSAL :EVENT :Volume vol5 exported at : '/'
03/03/2018 11:55:38 : epoch 10430000 : dhcp37-121.lab.eng.blr.redhat.com : ganesha.nfsd-6103[dbus_heartbeat] glusterfs_create_export :FSAL :EVENT :Volume vol6 exported at : '/'
03/03/2018 11:55:49 : epoch 10430000 : dhcp37-121.lab.eng.blr.redhat.com : ganesha.nfsd-6103[dbus_heartbeat] glusterfs_create_export :FSAL :EVENT :Volume vol7 exported at : '/'
03/03/2018 11:55:59 : epoch 10430000 : dhcp37-121.lab.eng.blr.redhat.com : ganesha.nfsd-6103[dbus_heartbeat] glusterfs_create_export :FSAL :EVENT :Volume vol8 exported at : '/'
03/03/2018 11:56:10 : epoch 10430000 : dhcp37-121.lab.eng.blr.redhat.com : ganesha.nfsd-6103[dbus_heartbeat] glusterfs_create_export :FSAL :EVENT :Volume vol9 exported at : '/'
03/03/2018 11:56:21 : epoch 10430000 : dhcp37-121.lab.eng.blr.redhat.com : ganesha.nfsd-6103[dbus_heartbeat] glusterfs_create_export :FSAL :EVENT :Volume vol10 exported at : '/'
03/03/2018 11:56:31 : epoch 10430000 : dhcp37-121.lab.eng.blr.redhat.com : ganesha.nfsd-6103[dbus_heartbeat] glusterfs_create_export :FSAL :EVENT :Volume vol11 exported at : '/'
03/03/2018 11:56:42 : epoch 10430000 : dhcp37-121.lab.eng.blr.redhat.com : ganesha.nfsd-6103[dbus_heartbeat] glusterfs_create_export :FSAL :EVENT :Volume vol12 exported at : '/'
03/03/2018 11:56:53 : epoch 10430000 : dhcp37-121.lab.eng.blr.redhat.com : ganesha.nfsd-6103[dbus_heartbeat] glusterfs_create_export :FSAL :EVENT :Volume vol13 exported at : '/'
03/03/2018 11:57:03 : epoch 10430000 : dhcp37-121.lab.eng.blr.redhat.com : ganesha.nfsd-6103[dbus_heartbeat] glusterfs_create_export :FSAL :EVENT :Volume vol14 exported at : '/'
03/03/2018 11:57:14 : epoch 10430000 : dhcp37-121.lab.eng.blr.redhat.com : ganesha.nfsd-6103[dbus_heartbeat] glusterfs_create_export :FSAL :EVENT :Volume vol15 exported at : '/'
03/03/2018 11:57:25 : epoch 10430000 : dhcp37-121.lab.eng.blr.redhat.com : ganesha.nfsd-6103[dbus_heartbeat] glusterfs_create_export :FSAL :EVENT :Volume vol16 exported at : '/'
03/03/2018 11:57:35 : epoch 10430000 : dhcp37-121.lab.eng.blr.redhat.com : ganesha.nfsd-6103[dbus_heartbeat] glusterfs_create_export :FSAL :EVENT :Volume vol17 exported at : '/'
03/03/2018 11:57:46 : epoch 10430000 : dhcp37-121.lab.eng.blr.redhat.com : ganesha.nfsd-6103[dbus_heartbeat] glusterfs_create_export :FSAL :EVENT :Volume vol18 exported at : '/'
03/03/2018 11:57:57 : epoch 10430000 : dhcp37-121.lab.eng.blr.redhat.com : ganesha.nfsd-6103[dbus_heartbeat] glusterfs_create_export :FSAL :EVENT :Volume vol19 exported at : '/'
03/03/2018 11:58:08 : epoch 10430000 : dhcp37-121.lab.eng.blr.redhat.com : ganesha.nfsd-6103[dbus_heartbeat] glusterfs_create_export :FSAL :EVENT :Volume vol20 exported at : '/'
03/03/2018 11:58:19 : epoch 10430000 : dhcp37-121.lab.eng.blr.redhat.com : ganesha.nfsd-6103[dbus_heartbeat] glusterfs_create_export :FSAL :EVENT :Volume vol21 exported at : '/'
03/03/2018 11:58:30 : epoch 10430000 : dhcp37-121.lab.eng.blr.redhat.com : ganesha.nfsd-6103[dbus_heartbeat] glusterfs_create_export :FSAL :EVENT :Volume vol22 exported at : '/'
03/03/2018 11:58:40 : epoch 10430000 : dhcp37-121.lab.eng.blr.redhat.com : ganesha.nfsd-6103[dbus_heartbeat] glusterfs_create_export :FSAL :EVENT :Volume vol23 exported at : '/'
03/03/2018 11:58:51 : epoch 10430000 : dhcp37-121.lab.eng.blr.redhat.com : ganesha.nfsd-6103[dbus_heartbeat] glusterfs_create_export :FSAL :EVENT :Volume vol24 exported at : '/'
03/03/2018 11:59:02 : epoch 10430000 : dhcp37-121.lab.eng.blr.redhat.com : ganesha.nfsd-6103[dbus_heartbeat] glusterfs_create_export :FSAL :EVENT :Volume vol25 exported at : '/'


Volume unexport-

# sh ganesha_export.sh 
volume set: success
volume set: success
volume set: success
volume set: success
volume set: success
volume set: success
volume set: success
volume set: success
volume set: success
volume set: success
volume set: success
volume set: success
volume set: success
volume set: success
volume set: failed: Staging failed on dhcp37-103.lab.eng.blr.redhat.com. Error: Dynamic export addition/deletion failed. Please see log file for details
Staging failed on dhcp37-136.lab.eng.blr.redhat.com. Error: Dynamic export addition/deletion failed. Please see log file for details
volume set: success
volume set: success
volume set: failed: Staging failed on dhcp37-103.lab.eng.blr.redhat.com. Error: Dynamic export addition/deletion failed. Please see log file for details
Staging failed on dhcp37-136.lab.eng.blr.redhat.com. Error: Dynamic export addition/deletion failed. Please see log file for details
volume set: success
volume set: failed: Staging failed on dhcp37-103.lab.eng.blr.redhat.com. Error: Dynamic export addition/deletion failed. Please see log file for details
Staging failed on dhcp37-136.lab.eng.blr.redhat.com. Error: Dynamic export addition/deletion failed. Please see log file for details
volume set: failed: Staging failed on dhcp37-103.lab.eng.blr.redhat.com. Error: Dynamic export addition/deletion failed. Please see log file for details
Staging failed on dhcp37-136.lab.eng.blr.redhat.com. Error: Dynamic export addition/deletion failed. Please see log file for details
volume set: success
volume set: failed: Staging failed on dhcp37-103.lab.eng.blr.redhat.com. Error: Dynamic export addition/deletion failed. Please see log file for details
Staging failed on dhcp37-136.lab.eng.blr.redhat.com. Error: Dynamic export addition/deletion failed. Please see log file for details
volume set: success
volume set: failed: Staging failed on dhcp37-103.lab.eng.blr.redhat.com. Error: Dynamic export addition/deletion failed. Please see log file for details
Staging failed on dhcp37-136.lab.eng.blr.redhat.com. Error: Dynamic export addition/deletion failed. Please see log file for details



=======================


Expected results:

Volume export/unexport should work fine


Additional info:

Comment 10 errata-xmlrpc 2018-09-04 06:44:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2607