Bug 1555195

Summary:	[Ganesha] Duplicate volume export entries in ganesha.conf causing volume unexport to fail
Product:	[Community] GlusterFS	Reporter:	Jiffin <jthottan>
Component:	glusterd	Assignee:	Jiffin <jthottan>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	unspecified	Docs Contact:
Priority:	urgent
Version:	3.10	CC:	bugs, dang, ffilz, grajoria, jthottan, msaini, rhs-bugs, storage-qa-internal
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	glusterfs-3.10.12	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1551186	Environment:
Last Closed:	2018-05-07 15:05:04 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1551186

Description Jiffin 2018-03-14 06:40:50 UTC

+++ This bug was initially created as a clone of Bug #1551186 +++

Description of problem:

While exporting multiple volumes,duplicate volume entries are been observed in /etc/ganesha/ganesha.conf which leads to failure while unexporting volumes

Version-Release number of selected component (if applicable):

3.10


How reproducible:
3/3


Steps to Reproduce:

1.Create 4 node ganesha cluster
2.Create and start 25 volumes
3.Export all volumes via ganesha
for i in $(seq 1 25)
do
        gluster v set vol$i ganesha.enable on
        sleep 10
done
4.Unexport the ganesha volumes
for i in $(seq 1 25)
do
        gluster v set vol$i ganesha.enable off
        sleep 10
done

Actual results:
In /etc/ganesha/ganesha.conf ,duplicate volume export entries are observed causing volume unexport failed.

========
#/etc/ganesha/ganesha.conf


EXPORT_DEFAULTS {
	# To reflect nfsnobody
        Anonymous_uid = 65534;
        Anonymous_gid = 65534;
}
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol1.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol2.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol3.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol4.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol5.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol6.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol7.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol8.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol9.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol10.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol11.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol12.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol13.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol14.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol15.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol15.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol16.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol17.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol18.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol18.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol19.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol20.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol20.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol21.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol21.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol22.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol23.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol23.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol24.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol25.conf"
%include "/var/run/gluster/shared_storage/nfs-ganesha/exports/export.vol25.conf"



[root@dhcp37-218 exports]# ls
export.vol10.conf  export.vol14.conf  export.vol18.conf  export.vol21.conf  export.vol25.conf  export.vol5.conf  export.vol9.conf
export.vol11.conf  export.vol15.conf  export.vol19.conf  export.vol22.conf  export.vol2.conf   export.vol6.conf
export.vol12.conf  export.vol16.conf  export.vol1.conf   export.vol23.conf  export.vol3.conf   export.vol7.conf
export.vol13.conf  export.vol17.conf  export.vol20.conf  export.vol24.conf  export.vol4.conf   export.vol8.conf


# showmount -e
Export list for dhcp37-218.lab.eng.blr.redhat.com:
/vol1  (everyone)
/vol2  (everyone)
/vol3  (everyone)
/vol4  (everyone)
/vol5  (everyone)
/vol6  (everyone)
/vol7  (everyone)
/vol8  (everyone)
/vol9  (everyone)
/vol10 (everyone)
/vol11 (everyone)
/vol12 (everyone)
/vol13 (everyone)
/vol14 (everyone)
/vol15 (everyone)
/vol16 (everyone)
/vol17 (everyone)
/vol18 (everyone)
/vol19 (everyone)
/vol20 (everyone)
/vol21 (everyone)
/vol22 (everyone)
/vol23 (everyone)
/vol24 (everyone)
/vol25 (everyone)


Volume unexport-

# sh ganesha_export.sh 
volume set: success
volume set: success
volume set: success
volume set: success
volume set: success
volume set: success
volume set: success
volume set: success
volume set: success
volume set: success
volume set: success
volume set: success
volume set: success
volume set: success
volume set: failed: Staging failed on dhcp37-103.lab.eng.blr.redhat.com. Error: Dynamic export addition/deletion failed. Please see log file for details
Staging failed on dhcp37-136.lab.eng.blr.redhat.com. Error: Dynamic export addition/deletion failed. Please see log file for details
volume set: success
volume set: success
volume set: failed: Staging failed on dhcp37-103.lab.eng.blr.redhat.com. Error: Dynamic export addition/deletion failed. Please see log file for details
Staging failed on dhcp37-136.lab.eng.blr.redhat.com. Error: Dynamic export addition/deletion failed. Please see log file for details
volume set: success
volume set: failed: Staging failed on dhcp37-103.lab.eng.blr.redhat.com. Error: Dynamic export addition/deletion failed. Please see log file for details
Staging failed on dhcp37-136.lab.eng.blr.redhat.com. Error: Dynamic export addition/deletion failed. Please see log file for details
volume set: failed: Staging failed on dhcp37-103.lab.eng.blr.redhat.com. Error: Dynamic export addition/deletion failed. Please see log file for details
Staging failed on dhcp37-136.lab.eng.blr.redhat.com. Error: Dynamic export addition/deletion failed. Please see log file for details
volume set: success
volume set: failed: Staging failed on dhcp37-103.lab.eng.blr.redhat.com. Error: Dynamic export addition/deletion failed. Please see log file for details
Staging failed on dhcp37-136.lab.eng.blr.redhat.com. Error: Dynamic export addition/deletion failed. Please see log file for details
volume set: success
volume set: failed: Staging failed on dhcp37-103.lab.eng.blr.redhat.com. Error: Dynamic export addition/deletion failed. Please see log file for details
Staging failed on dhcp37-136.lab.eng.blr.redhat.com. Error: Dynamic export addition/deletion failed. Please see log file for details



=======================


Expected results:

Volume export/unexport should work fine


Additional info:

--- Additional comment from Jiffin on 2018-03-13 04:26:11 EDT ---

There is possibility of having multiple entries get created in ganesha.conf because create-ganesha-export.sh during volume set ganesha.enable command and can be executed from multiple nodes at the same time. Even though script checks for duplicate entry, it cannot trusted properly. So it is better to execute the script from only one node, so above scenarios can be avoided.

Second unexport failure due to duplicate entries I still need to debug that

Comment 1 Worker Ant 2018-03-14 06:42:27 UTC

REVIEW: https://review.gluster.org/19716 (glusterd/ganesha : create/remove export file only from the node which performs ganesha.enable) posted (#1) for review on release-3.10 by jiffin tony Thottan

Comment 2 Worker Ant 2018-03-16 13:36:35 UTC

COMMIT: https://review.gluster.org/19716 committed in release-3.10 by "Shyamsundar Ranganathan" <srangana> with a commit message- glusterd/ganesha : create/remove export file only from the node which performs ganesha.enable

As part of volume set ganesha.enable on the ganesha export configuration file will be created/removed
using "create-export-ganesha.sh". This performed from the nodes which are part of ganesha cluster.
But it is not need since the file is saved in shared storage and consumed by the nodes in the ganesha cluster.

Change-Id: I2583899972b47d451a90711940293004a3af4690
BUG: 1555195
Signed-off-by: Jiffin Tony Thottan <jthottan>

Comment 3 Shyamsundar 2018-05-07 15:05:04 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.10.12, please open a new bug report.

glusterfs-3.10.12 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/announce/2018-April/000095.html
[2] https://www.gluster.org/pipermail/gluster-users/