Bug 1376874 - RFE : move ganesha related configuration into shared storage
Summary: RFE : move ganesha related configuration into shared storage
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: ganesha-nfs
Version: 3.9
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Jiffin
QA Contact:
URL:
Whiteboard:
Depends On: 1355956
Blocks: 1261866
TreeView+ depends on / blocked
 
Reported: 2016-09-16 17:41 UTC by Jiffin
Modified: 2016-12-06 06:00 UTC (History)
7 users (show)

Fixed In Version: glusterfs-3.9.0
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1355956
Environment:
Last Closed: 2016-12-06 06:00:45 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:


Attachments (Terms of Use)

Description Jiffin 2016-09-16 17:41:55 UTC
A bug fixes for above feature


+++ This bug was initially created as a clone of Bug #1355956 +++

Description of problem:
In the present design, we maintain a copy of the below database (required for NFS-Ganesha setup) on all the nodes of the Gluster Storage pool -
* /etc/ganesha/ganesha-ha.conf (with the details about the nodes forming NFS-Ganesha cluster and their VIPs to be configured)

* /etc/ganesha/ganesha.conf (contains configuration parameters and the volumes list to be exported, used by NFS-Ganesha process while starting)

* /etc/ganesha/exports/ganesha.$vol.conf (config file of a volume created when a volume is being exported by NFS-Ganesha)

* /etc/ganesha/exports/.export_added (each volume should have unique ExportID.  This file is to keep track of the last ExportID used)

Outstanding Issues:
Since each node shall have a local copy, we have few issues with respect to synchronizing them across the nodes. Few of them are outlined below: 
* When a node reboots, depending on the order the services ‘pacemaker’ and ‘glusterd’ starts, we may run into a  case, where in glusterd couldn’t sync the export files on this node.

* There could be case where in ‘.export_added’ could go out of sync across the nodes thus resulting in different ExportIDs for the same volume exported across different nfs-ganesha heads. This shall result in NFS mounts throwing “Stale File handle error” post failover 

* When a node is down while performing “refresh-config”, the export config of that volume doesn’t get synced when it comes back up.

* As one of the pre-requisites, ganesha-ha.conf has to be copied to all the nodes in the gluster cluster before enabling nfs-ganesha (setup).

* When any changes are made to main “/etc/ganesha.conf” file, it has to be manually copied to all the nodes in the cluster.
 
* When “refresh-config” is performed, the script syncs up the volume config  only across the NFS-Ganesha cluster but not to all the nodes in the Gluster storage pool

Version-Release number of selected component (if applicable):
mainline


Additional info:
The existing users shall be impacted while doing upgrade. We may need to provide scripts for the smooth upgrade to move the older config data to shared_storage and also few of the setup pre-requisites need to be removed/modified accordingly.

--- Additional comment from Vijay Bellur on 2016-07-13 05:55:25 EDT ---

REVIEW: http://review.gluster.org/14906 (glusterd/ganesha : Move ganesha-ha.conf and ganesha.conf to shared storage) posted (#1) for review on master by jiffin tony Thottan (jthottan@redhat.com)

--- Additional comment from Vijay Bellur on 2016-07-13 05:55:28 EDT ---

REVIEW: http://review.gluster.org/14907 (Revert "glusterd-ganesha : copy ganesha export configuration files during reboot") posted (#1) for review on master by jiffin tony Thottan (jthottan@redhat.com)

--- Additional comment from Vijay Bellur on 2016-07-13 05:55:31 EDT ---

REVIEW: http://review.gluster.org/14908 (glusterd/ganesha : create export configuration file in shared storage) posted (#1) for review on master by jiffin tony Thottan (jthottan@redhat.com)

--- Additional comment from Vijay Bellur on 2016-07-13 05:55:35 EDT ---

REVIEW: http://review.gluster.org/14909 (ganesha/scripts : Modifying ganesha-ha.sh for share storage related changes) posted (#1) for review on master by jiffin tony Thottan (jthottan@redhat.com)

--- Additional comment from Vijay Bellur on 2016-07-15 05:24:44 EDT ---

REVIEW: http://review.gluster.org/14908 (glusterd/ganesha : create export configuration file in shared storage) posted (#2) for review on master by jiffin tony Thottan (jthottan@redhat.com)

--- Additional comment from Vijay Bellur on 2016-07-15 05:24:47 EDT ---

REVIEW: http://review.gluster.org/14909 (ganesha/scripts : Modifying ganesha-ha.sh for share storage related changes) posted (#2) for review on master by jiffin tony Thottan (jthottan@redhat.com)

--- Additional comment from Vijay Bellur on 2016-07-15 05:24:50 EDT ---

REVIEW: http://review.gluster.org/14906 (glusterd/ganesha : Move ganesha-ha.conf and ganesha.conf to shared storage) posted (#2) for review on master by jiffin tony Thottan (jthottan@redhat.com)

--- Additional comment from Vijay Bellur on 2016-07-15 05:24:54 EDT ---

REVIEW: http://review.gluster.org/14907 (Revert "glusterd-ganesha : copy ganesha export configuration files during reboot") posted (#2) for review on master by jiffin tony Thottan (jthottan@redhat.com)

--- Additional comment from Vijay Bellur on 2016-08-08 05:25:14 EDT ---

REVIEW: http://review.gluster.org/14908 (glusterd/ganesha : create export configuration file in shared storage) posted (#3) for review on master by jiffin tony Thottan (jthottan@redhat.com)

--- Additional comment from Vijay Bellur on 2016-08-08 05:25:17 EDT ---

REVIEW: http://review.gluster.org/14909 (ganesha/scripts : Modifying ganesha-ha.sh for share storage related changes) posted (#3) for review on master by jiffin tony Thottan (jthottan@redhat.com)

--- Additional comment from Vijay Bellur on 2016-08-08 05:25:22 EDT ---

REVIEW: http://review.gluster.org/14906 (glusterd/ganesha : Move ganesha-ha.conf and ganesha.conf to shared storage) posted (#3) for review on master by jiffin tony Thottan (jthottan@redhat.com)

--- Additional comment from Vijay Bellur on 2016-08-08 05:25:26 EDT ---

REVIEW: http://review.gluster.org/14907 (Revert "glusterd-ganesha : copy ganesha export configuration files during reboot") posted (#3) for review on master by jiffin tony Thottan (jthottan@redhat.com)

--- Additional comment from Worker Ant on 2016-08-25 04:37:43 EDT ---

REVIEW: http://review.gluster.org/14908 (glusterd/ganesha : create export configuration file in shared storage) posted (#4) for review on master by jiffin tony Thottan (jthottan@redhat.com)

--- Additional comment from Worker Ant on 2016-08-25 04:37:47 EDT ---

REVIEW: http://review.gluster.org/14909 (ganesha/scripts : Modifying ganesha-ha.sh for share storage related changes) posted (#4) for review on master by jiffin tony Thottan (jthottan@redhat.com)

--- Additional comment from Worker Ant on 2016-08-25 04:37:51 EDT ---

REVIEW: http://review.gluster.org/14906 (glusterd/ganesha : Move ganesha-ha.conf and ganesha.conf to shared storage) posted (#4) for review on master by jiffin tony Thottan (jthottan@redhat.com)

--- Additional comment from Worker Ant on 2016-08-25 04:37:55 EDT ---

REVIEW: http://review.gluster.org/14907 (Revert "glusterd-ganesha : copy ganesha export configuration files during reboot") posted (#4) for review on master by jiffin tony Thottan (jthottan@redhat.com)

--- Additional comment from Worker Ant on 2016-08-25 09:56:01 EDT ---

COMMIT: http://review.gluster.org/14906 committed in master by Kaleb KEITHLEY (kkeithle@redhat.com) 
------
commit 76726da0e86077a8f3a59c02a47fcf2e3994218f
Author: Jiffin Tony Thottan <jthottan@redhat.com>
Date:   Tue Jul 12 15:44:23 2016 +0530

    glusterd/ganesha : Move ganesha-ha.conf and ganesha.conf to shared storage
    
    Currently all the ganesha related configuration files(ganesha.conf,
    ganesha-ha.conf, export files, etc) is stored locally at /etc/ganesha
    on a every node in ganesha cluster. Usually we end up in two issues by
    doing so :
    * difficult in modifiying ganesha related conf file
    * diffciult to maintain consistency of conf file across ganesha cluster
    To tackle this, we plan to move all the ganesha configuration to shared
    storage. As a first step in this patch ganesha.conf and ganesha-ha.conf
    move to shared storage. Here actual ganesha.conf will resides in shared
    stoarge and symlinks will be created in /etc/ganesha when the option
    "gluster nfs-ganesha enable" is executed and remove those during the
    "disable" part.
    
    Modified prerequisites to done before running globaloption:
    	* enable shared storage
    	* create nfs-ganesha folder in shared storage
    	* create ganesha.conf and ganesha-ha.conf in it
    
    More details can be found at http://review.gluster.org/#/c/15105/
    
    Change-Id: Ifabb6c5db50061f077a03932940190af74e2ca7f
    BUG: 1355956
    Signed-off-by: Jiffin Tony Thottan <jthottan@redhat.com>
    Reviewed-on: http://review.gluster.org/14906
    Reviewed-by: soumya k <skoduri@redhat.com>
    NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    Smoke: Gluster Build System <jenkins@build.gluster.org>
    Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
    CentOS-regression: Gluster Build System <jenkins@build.gluster.org>

--- Additional comment from Worker Ant on 2016-08-26 07:46:10 EDT ---

COMMIT: http://review.gluster.org/14907 committed in master by Kaleb KEITHLEY (kkeithle@redhat.com) 
------
commit a769be2c0d4227fead7348e769bd1ae49e267971
Author: Jiffin Tony Thottan <jthottan@redhat.com>
Date:   Tue Jul 12 17:23:03 2016 +0530

    Revert "glusterd-ganesha : copy ganesha export configuration files during reboot"
    
    This reverts commit f71e2fa49af185779b9f43e146effd122d4e9da0.
    
    Reason:
    As part of sync up node reboot this patch copies ganesha export conf file
    from a source node. This change is no more require if the export files are
    available in shared storage.
    
    Change-Id: Id9c1ae78377bbd7d5d80aa1c14f534e30feaae97
    BUG: 1355956
    Signed-off-by: Jiffin Tony Thottan <jthottan@redhat.com>
    Reviewed-on: http://review.gluster.org/14907
    Reviewed-by: soumya k <skoduri@redhat.com>
    Smoke: Gluster Build System <jenkins@build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
    NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>

--- Additional comment from Worker Ant on 2016-08-26 07:47:22 EDT ---

COMMIT: http://review.gluster.org/14909 committed in master by Kaleb KEITHLEY (kkeithle@redhat.com) 
------
commit 2f44a979132a43c1cf12ed2b9e9e94dbe1311c9e
Author: Jiffin Tony Thottan <jthottan@redhat.com>
Date:   Wed Jul 13 12:09:43 2016 +0530

    ganesha/scripts : Modifying ganesha-ha.sh for share storage related changes
    
    Currently the ganesha related configurations are "scp"ied for operations like
    add, delete, refresh-config in ganesha-ha.sh. This is no more required since
    all the conf files are available in shared storage and every node can directly
    access them from shared storage.
    
    More details can be found at http://review.gluster.org/#/c/15105/
    
    Change-Id: Ic025eb4dc246db61d6fbe969ca60214751fbf3ba
    BUG: 1355956
    Signed-off-by: Jiffin Tony Thottan <jthottan@redhat.com>
    Reviewed-on: http://review.gluster.org/14909
    NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
    Reviewed-by: soumya k <skoduri@redhat.com>
    Smoke: Gluster Build System <jenkins@build.gluster.org>
    Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>

--- Additional comment from Worker Ant on 2016-08-26 07:47:55 EDT ---

COMMIT: http://review.gluster.org/14908 committed in master by Kaleb KEITHLEY (kkeithle@redhat.com) 
------
commit 4786150ebe36154ec0762a951612f66f68a7e95c
Author: Jiffin Tony Thottan <jthottan@redhat.com>
Date:   Wed Jul 13 11:38:10 2016 +0530

    glusterd/ganesha : create export configuration file in shared storage
    
    This is the second patch which moves export related configuration for
    a volume into shared storage. The main change includes in scripts
    create-export-ganesha.sh, dbus-send.sh and the handling of volume set
    command "ganesha.enable". The manipulation of EXPORT_ID move from
    dbus-send.sh to create-export-ganesha.sh.
    
    In volume set handling following has performed
    
        stage                  |    commit
    ----------------------------------------------------------
    1.) gluster v set <volname> ganesha.enable on
    
     None			   |	create export file
    			   |    in node where cli executed,
    			   |	thne export volume via dbus
    
    2.) gluster v set <volname> ganesha.enable off
    
    unexport volume via dbus   |	remove export file from the
    			   |    shared storage
    -----------------------------------------------------------
    
    More details can be found at http://review.gluster.org/#/c/15105/
    
    Change-Id: Ia8b0e89bc8fff24b0bc5d20a538a89212894a8e4
    BUG: 1355956
    Signed-off-by: Jiffin Tony Thottan <jthottan@redhat.com>
    Reviewed-on: http://review.gluster.org/14908
    NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
    Reviewed-by: soumya k <skoduri@redhat.com>
    Smoke: Gluster Build System <jenkins@build.gluster.org>
    Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>

--- Additional comment from Worker Ant on 2016-08-30 09:11:32 EDT ---

REVIEW: http://review.gluster.org/15355 (ganesha/glustrd : Correct the path for ganesha conf dir in Makefile) posted (#1) for review on master by jiffin tony Thottan (jthottan@redhat.com)

--- Additional comment from Worker Ant on 2016-09-16 01:45:31 EDT ---

REVIEW: http://review.gluster.org/15355 (ganesha/glusterd : Correct the path for ganesha conf dir in Makefile) posted (#2) for review on master by jiffin tony Thottan (jthottan@redhat.com)

--- Additional comment from Worker Ant on 2016-09-16 10:22:54 EDT ---

COMMIT: http://review.gluster.org/15355 committed in master by Atin Mukherjee (amukherj@redhat.com) 
------
commit 7e18e16f9f62ed95acded0fb3f2a8784087f59c0
Author: Jiffin Tony Thottan <jthottan@redhat.com>
Date:   Tue Aug 30 18:36:51 2016 +0530

    ganesha/glusterd : Correct the path for ganesha conf dir in Makefile
    
    The value for ganesha conf dir in Makefile.am is mistakenly entered as
    following :
    DCONFDIR=\"/$(runstatedir)/gluster/shared_storage/nfs-ganesha\"
    
    Here value for runstatedir is seems to "NULL" which results wrong path
    for ganesha configuration directory
    
    Change-Id: I0b7ebd8e2503de0cb79b601553c4405d0d1fd711
    BUG: 1355956
    Signed-off-by: Jiffin Tony Thottan <jthottan@redhat.com>
    Reviewed-on: http://review.gluster.org/15355
    NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    Reviewed-by: soumya k <skoduri@redhat.com>
    Reviewed-by: Niels de Vos <ndevos@redhat.com>
    CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
    Smoke: Gluster Build System <jenkins@build.gluster.org>

Comment 1 Worker Ant 2016-09-16 17:47:57 UTC
REVIEW: http://review.gluster.org/15521 (ganesha/glusterd : Correct the path for ganesha conf dir in Makefile) posted (#1) for review on release-3.9 by jiffin tony Thottan (jthottan@redhat.com)

Comment 2 Worker Ant 2016-09-17 10:44:04 UTC
COMMIT: http://review.gluster.org/15521 committed in release-3.9 by Atin Mukherjee (amukherj@redhat.com) 
------
commit b7bb059865c43e3098bd5682aedd0b9b716d9d23
Author: Jiffin Tony Thottan <jthottan@redhat.com>
Date:   Tue Aug 30 18:36:51 2016 +0530

    ganesha/glusterd : Correct the path for ganesha conf dir in Makefile
    
    The value for ganesha conf dir in Makefile.am is mistakenly entered as
    following :
    DCONFDIR=\"/$(runstatedir)/gluster/shared_storage/nfs-ganesha\"
    
    Here value for runstatedir is seems to "NULL" which results wrong path
    for ganesha configuration directory
    
    Upstream reference :
    > Change-Id: I0b7ebd8e2503de0cb79b601553c4405d0d1fd711
    > BUG: 1355956
    > Signed-off-by: Jiffin Tony Thottan <jthottan@redhat.com>
    > Reviewed-on: http://review.gluster.org/15355
    > NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    > Reviewed-by: soumya k <skoduri@redhat.com>
    > Reviewed-by: Niels de Vos <ndevos@redhat.com>
    > CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
    > Smoke: Gluster Build System <jenkins@build.gluster.org>
    > (cherry picked from commit 7e18e16f9f62ed95acded0fb3f2a8784087f59c0)
    
    Change-Id: I0b7ebd8e2503de0cb79b601553c4405d0d1fd711
    BUG: 1376874
    Signed-off-by: Jiffin Tony Thottan <jthottan@redhat.com>
    Reviewed-on: http://review.gluster.org/15521
    Reviewed-by: soumya k <skoduri@redhat.com>
    Smoke: Gluster Build System <jenkins@build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
    NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    Reviewed-by: Atin Mukherjee <amukherj@redhat.com>

Comment 3 Aravinda VK 2016-10-27 05:22:09 UTC
glusterfs-3.9.0rc2 is released[1] and packages are available for different distributions[2] to test.

[1] http://www.gluster.org/pipermail/maintainers/2016-October/001601.html
[2] http://www.gluster.org/pipermail/maintainers/2016-October/001605.html and http://www.gluster.org/pipermail/maintainers/2016-October/001606.html

Comment 4 Aravinda VK 2016-12-06 06:00:45 UTC
Gluster 3.9 GA is released http://blog.gluster.org/2016/11/announcing-gluster-3-9/


Note You need to log in before you can comment on or make changes to this bug.