Bug 1355956

Summary:	RFE : move ganesha related configuration into shared storage
Product:	[Community] GlusterFS	Reporter:	Jiffin <jthottan>
Component:	ganesha-nfs	Assignee:	Jiffin <jthottan>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	mainline	CC:	bugs, jthottan, kkeithle, mzywusko, ndevos, skoduri
Target Milestone:	---	Keywords:	FutureFeature, Triaged
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	glusterfs-3.10.0	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1376874 (view as bug list)		Environment:
Last Closed:	2017-03-06 17:20:39 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1261866, 1376874

Description Jiffin 2016-07-13 07:02:51 UTC

Description of problem:
In the present design, we maintain a copy of the below database (required for NFS-Ganesha setup) on all the nodes of the Gluster Storage pool -
* /etc/ganesha/ganesha-ha.conf (with the details about the nodes forming NFS-Ganesha cluster and their VIPs to be configured)

* /etc/ganesha/ganesha.conf (contains configuration parameters and the volumes list to be exported, used by NFS-Ganesha process while starting)

* /etc/ganesha/exports/ganesha.$vol.conf (config file of a volume created when a volume is being exported by NFS-Ganesha)

* /etc/ganesha/exports/.export_added (each volume should have unique ExportID.  This file is to keep track of the last ExportID used)

Outstanding Issues:
Since each node shall have a local copy, we have few issues with respect to synchronizing them across the nodes. Few of them are outlined below: 
* When a node reboots, depending on the order the services ‘pacemaker’ and ‘glusterd’ starts, we may run into a  case, where in glusterd couldn’t sync the export files on this node.

* There could be case where in ‘.export_added’ could go out of sync across the nodes thus resulting in different ExportIDs for the same volume exported across different nfs-ganesha heads. This shall result in NFS mounts throwing “Stale File handle error” post failover 

* When a node is down while performing “refresh-config”, the export config of that volume doesn’t get synced when it comes back up.

* As one of the pre-requisites, ganesha-ha.conf has to be copied to all the nodes in the gluster cluster before enabling nfs-ganesha (setup).

* When any changes are made to main “/etc/ganesha.conf” file, it has to be manually copied to all the nodes in the cluster.
 
* When “refresh-config” is performed, the script syncs up the volume config  only across the NFS-Ganesha cluster but not to all the nodes in the Gluster storage pool

Version-Release number of selected component (if applicable):
mainline


Additional info:
The existing users shall be impacted while doing upgrade. We may need to provide scripts for the smooth upgrade to move the older config data to shared_storage and also few of the setup pre-requisites need to be removed/modified accordingly.

Comment 1 Vijay Bellur 2016-07-13 09:55:25 UTC

REVIEW: http://review.gluster.org/14906 (glusterd/ganesha : Move ganesha-ha.conf and ganesha.conf to shared storage) posted (#1) for review on master by jiffin tony Thottan (jthottan)

Comment 2 Vijay Bellur 2016-07-13 09:55:28 UTC

REVIEW: http://review.gluster.org/14907 (Revert "glusterd-ganesha : copy ganesha export configuration files during reboot") posted (#1) for review on master by jiffin tony Thottan (jthottan)

Comment 3 Vijay Bellur 2016-07-13 09:55:31 UTC

REVIEW: http://review.gluster.org/14908 (glusterd/ganesha : create export configuration file in shared storage) posted (#1) for review on master by jiffin tony Thottan (jthottan)

Comment 4 Vijay Bellur 2016-07-13 09:55:35 UTC

REVIEW: http://review.gluster.org/14909 (ganesha/scripts : Modifying ganesha-ha.sh for share storage related changes) posted (#1) for review on master by jiffin tony Thottan (jthottan)

Comment 5 Vijay Bellur 2016-07-15 09:24:44 UTC

REVIEW: http://review.gluster.org/14908 (glusterd/ganesha : create export configuration file in shared storage) posted (#2) for review on master by jiffin tony Thottan (jthottan)

Comment 6 Vijay Bellur 2016-07-15 09:24:47 UTC

REVIEW: http://review.gluster.org/14909 (ganesha/scripts : Modifying ganesha-ha.sh for share storage related changes) posted (#2) for review on master by jiffin tony Thottan (jthottan)

Comment 7 Vijay Bellur 2016-07-15 09:24:50 UTC

REVIEW: http://review.gluster.org/14906 (glusterd/ganesha : Move ganesha-ha.conf and ganesha.conf to shared storage) posted (#2) for review on master by jiffin tony Thottan (jthottan)

Comment 8 Vijay Bellur 2016-07-15 09:24:54 UTC

REVIEW: http://review.gluster.org/14907 (Revert "glusterd-ganesha : copy ganesha export configuration files during reboot") posted (#2) for review on master by jiffin tony Thottan (jthottan)

Comment 9 Vijay Bellur 2016-08-08 09:25:14 UTC

REVIEW: http://review.gluster.org/14908 (glusterd/ganesha : create export configuration file in shared storage) posted (#3) for review on master by jiffin tony Thottan (jthottan)

Comment 10 Vijay Bellur 2016-08-08 09:25:17 UTC

REVIEW: http://review.gluster.org/14909 (ganesha/scripts : Modifying ganesha-ha.sh for share storage related changes) posted (#3) for review on master by jiffin tony Thottan (jthottan)

Comment 11 Vijay Bellur 2016-08-08 09:25:22 UTC

REVIEW: http://review.gluster.org/14906 (glusterd/ganesha : Move ganesha-ha.conf and ganesha.conf to shared storage) posted (#3) for review on master by jiffin tony Thottan (jthottan)

Comment 12 Vijay Bellur 2016-08-08 09:25:26 UTC

REVIEW: http://review.gluster.org/14907 (Revert "glusterd-ganesha : copy ganesha export configuration files during reboot") posted (#3) for review on master by jiffin tony Thottan (jthottan)

Comment 13 Worker Ant 2016-08-25 08:37:43 UTC

REVIEW: http://review.gluster.org/14908 (glusterd/ganesha : create export configuration file in shared storage) posted (#4) for review on master by jiffin tony Thottan (jthottan)

Comment 14 Worker Ant 2016-08-25 08:37:47 UTC

REVIEW: http://review.gluster.org/14909 (ganesha/scripts : Modifying ganesha-ha.sh for share storage related changes) posted (#4) for review on master by jiffin tony Thottan (jthottan)

Comment 15 Worker Ant 2016-08-25 08:37:51 UTC

REVIEW: http://review.gluster.org/14906 (glusterd/ganesha : Move ganesha-ha.conf and ganesha.conf to shared storage) posted (#4) for review on master by jiffin tony Thottan (jthottan)

Comment 16 Worker Ant 2016-08-25 08:37:55 UTC

REVIEW: http://review.gluster.org/14907 (Revert "glusterd-ganesha : copy ganesha export configuration files during reboot") posted (#4) for review on master by jiffin tony Thottan (jthottan)

Comment 17 Worker Ant 2016-08-25 13:56:01 UTC

COMMIT: http://review.gluster.org/14906 committed in master by Kaleb KEITHLEY (kkeithle) 
------
commit 76726da0e86077a8f3a59c02a47fcf2e3994218f
Author: Jiffin Tony Thottan <jthottan>
Date:   Tue Jul 12 15:44:23 2016 +0530

    glusterd/ganesha : Move ganesha-ha.conf and ganesha.conf to shared storage
    
    Currently all the ganesha related configuration files(ganesha.conf,
    ganesha-ha.conf, export files, etc) is stored locally at /etc/ganesha
    on a every node in ganesha cluster. Usually we end up in two issues by
    doing so :
    * difficult in modifiying ganesha related conf file
    * diffciult to maintain consistency of conf file across ganesha cluster
    To tackle this, we plan to move all the ganesha configuration to shared
    storage. As a first step in this patch ganesha.conf and ganesha-ha.conf
    move to shared storage. Here actual ganesha.conf will resides in shared
    stoarge and symlinks will be created in /etc/ganesha when the option
    "gluster nfs-ganesha enable" is executed and remove those during the
    "disable" part.
    
    Modified prerequisites to done before running globaloption:
    	* enable shared storage
    	* create nfs-ganesha folder in shared storage
    	* create ganesha.conf and ganesha-ha.conf in it
    
    More details can be found at http://review.gluster.org/#/c/15105/
    
    Change-Id: Ifabb6c5db50061f077a03932940190af74e2ca7f
    BUG: 1355956
    Signed-off-by: Jiffin Tony Thottan <jthottan>
    Reviewed-on: http://review.gluster.org/14906
    Reviewed-by: soumya k <skoduri>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    Smoke: Gluster Build System <jenkins.org>
    Reviewed-by: Kaleb KEITHLEY <kkeithle>
    CentOS-regression: Gluster Build System <jenkins.org>

Comment 18 Worker Ant 2016-08-26 11:46:10 UTC

COMMIT: http://review.gluster.org/14907 committed in master by Kaleb KEITHLEY (kkeithle) 
------
commit a769be2c0d4227fead7348e769bd1ae49e267971
Author: Jiffin Tony Thottan <jthottan>
Date:   Tue Jul 12 17:23:03 2016 +0530

    Revert "glusterd-ganesha : copy ganesha export configuration files during reboot"
    
    This reverts commit f71e2fa49af185779b9f43e146effd122d4e9da0.
    
    Reason:
    As part of sync up node reboot this patch copies ganesha export conf file
    from a source node. This change is no more require if the export files are
    available in shared storage.
    
    Change-Id: Id9c1ae78377bbd7d5d80aa1c14f534e30feaae97
    BUG: 1355956
    Signed-off-by: Jiffin Tony Thottan <jthottan>
    Reviewed-on: http://review.gluster.org/14907
    Reviewed-by: soumya k <skoduri>
    Smoke: Gluster Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    Reviewed-by: Kaleb KEITHLEY <kkeithle>

Comment 19 Worker Ant 2016-08-26 11:47:22 UTC

COMMIT: http://review.gluster.org/14909 committed in master by Kaleb KEITHLEY (kkeithle) 
------
commit 2f44a979132a43c1cf12ed2b9e9e94dbe1311c9e
Author: Jiffin Tony Thottan <jthottan>
Date:   Wed Jul 13 12:09:43 2016 +0530

    ganesha/scripts : Modifying ganesha-ha.sh for share storage related changes
    
    Currently the ganesha related configurations are "scp"ied for operations like
    add, delete, refresh-config in ganesha-ha.sh. This is no more required since
    all the conf files are available in shared storage and every node can directly
    access them from shared storage.
    
    More details can be found at http://review.gluster.org/#/c/15105/
    
    Change-Id: Ic025eb4dc246db61d6fbe969ca60214751fbf3ba
    BUG: 1355956
    Signed-off-by: Jiffin Tony Thottan <jthottan>
    Reviewed-on: http://review.gluster.org/14909
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: soumya k <skoduri>
    Smoke: Gluster Build System <jenkins.org>
    Reviewed-by: Kaleb KEITHLEY <kkeithle>

Comment 20 Worker Ant 2016-08-26 11:47:55 UTC

COMMIT: http://review.gluster.org/14908 committed in master by Kaleb KEITHLEY (kkeithle) 
------
commit 4786150ebe36154ec0762a951612f66f68a7e95c
Author: Jiffin Tony Thottan <jthottan>
Date:   Wed Jul 13 11:38:10 2016 +0530

    glusterd/ganesha : create export configuration file in shared storage
    
    This is the second patch which moves export related configuration for
    a volume into shared storage. The main change includes in scripts
    create-export-ganesha.sh, dbus-send.sh and the handling of volume set
    command "ganesha.enable". The manipulation of EXPORT_ID move from
    dbus-send.sh to create-export-ganesha.sh.
    
    In volume set handling following has performed
    
        stage                  |    commit
    ----------------------------------------------------------
    1.) gluster v set <volname> ganesha.enable on
    
     None			   |	create export file
    			   |    in node where cli executed,
    			   |	thne export volume via dbus
    
    2.) gluster v set <volname> ganesha.enable off
    
    unexport volume via dbus   |	remove export file from the
    			   |    shared storage
    -----------------------------------------------------------
    
    More details can be found at http://review.gluster.org/#/c/15105/
    
    Change-Id: Ia8b0e89bc8fff24b0bc5d20a538a89212894a8e4
    BUG: 1355956
    Signed-off-by: Jiffin Tony Thottan <jthottan>
    Reviewed-on: http://review.gluster.org/14908
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: soumya k <skoduri>
    Smoke: Gluster Build System <jenkins.org>
    Reviewed-by: Kaleb KEITHLEY <kkeithle>

Comment 21 Worker Ant 2016-08-30 13:11:32 UTC

REVIEW: http://review.gluster.org/15355 (ganesha/glustrd : Correct the path for ganesha conf dir in Makefile) posted (#1) for review on master by jiffin tony Thottan (jthottan)

Comment 22 Worker Ant 2016-09-16 05:45:31 UTC

REVIEW: http://review.gluster.org/15355 (ganesha/glusterd : Correct the path for ganesha conf dir in Makefile) posted (#2) for review on master by jiffin tony Thottan (jthottan)

Comment 23 Worker Ant 2016-09-16 14:22:54 UTC

COMMIT: http://review.gluster.org/15355 committed in master by Atin Mukherjee (amukherj) 
------
commit 7e18e16f9f62ed95acded0fb3f2a8784087f59c0
Author: Jiffin Tony Thottan <jthottan>
Date:   Tue Aug 30 18:36:51 2016 +0530

    ganesha/glusterd : Correct the path for ganesha conf dir in Makefile
    
    The value for ganesha conf dir in Makefile.am is mistakenly entered as
    following :
    DCONFDIR=\"/$(runstatedir)/gluster/shared_storage/nfs-ganesha\"
    
    Here value for runstatedir is seems to "NULL" which results wrong path
    for ganesha configuration directory
    
    Change-Id: I0b7ebd8e2503de0cb79b601553c4405d0d1fd711
    BUG: 1355956
    Signed-off-by: Jiffin Tony Thottan <jthottan>
    Reviewed-on: http://review.gluster.org/15355
    NetBSD-regression: NetBSD Build System <jenkins.org>
    Reviewed-by: soumya k <skoduri>
    Reviewed-by: Niels de Vos <ndevos>
    CentOS-regression: Gluster Build System <jenkins.org>
    Smoke: Gluster Build System <jenkins.org>

Comment 24 Shyamsundar 2017-03-06 17:20:39 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.10.0, please open a new bug report.

glusterfs-3.10.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/gluster-users/2017-February/030119.html
[2] https://www.gluster.org/pipermail/gluster-users/