Bug 904065 - Feature: persistent caching of connected NFS-clients
Feature: persistent caching of connected NFS-clients
Status: CLOSED CURRENTRELEASE
Product: GlusterFS
Classification: Community
Component: nfs (Show other bugs)
pre-release
Unspecified Unspecified
medium Severity low
: ---
: ---
Assigned To: Niels de Vos
: FutureFeature, Patch
Depends On:
Blocks: 904074
  Show dependency treegraph
 
Reported: 2013-01-25 07:05 EST by Niels de Vos
Modified: 2014-04-17 07:40 EDT (History)
1 user (show)

See Also:
Fixed In Version: glusterfs-3.5.0
Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of:
: 904074 (view as bug list)
Environment:
Last Closed: 2014-04-17 07:40:30 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
systemtap script that can be used to verify the FILE* leaks (2.30 KB, text/plain)
2013-07-02 10:21 EDT, Niels de Vos
no flags Details

  None (edit)
Description Niels de Vos 2013-01-25 07:05:41 EST
Description of problem:
When a NFS-client mounts a volume, showmount lists the NFS-clients. However,
after a restart of the Gluster NFS-server, the NFS-clients are not listed
anymore. The NFS-clients keep functioning without any issue though.

Steps to Reproduce:
1. mount a volume through NFS
2. list the NFS-clients "showmount -a $SERVER"
3. service glusterd restart
4. list the NFS-clients "showmount -a $SERVER"

Actual results:
In step 4 the NFS-clients are not listed anymore.

Expected results:
The NFS-clients should be listed in step 4. This is how other NFS-servers
(like nfsd in the RHEL kernel) function.

Additional info:
The persistent logging of the RHEL NFS-server is done through the rpc.mountd
process. rpc.mountd saves the NFS-client and mountpoint in /var/lib/nfs/rwtab.
Comment 1 Niels de Vos 2013-01-25 07:38:50 EST
For this to work nicely, I'd like to introduce a rwtab similar to what rpc.mountd
uses. When the Gluster NFS-server starts, it should read this file, when an
NFS-client mounts a volume, this file should be extended, and upon unmounting the
file should be updated again.

This binds the rwtab-cache to a specific NFS-server. This may not be desirable 
when a Virtual IP-address is in use. When the IP-address is failed-over, the 
NFS-client will contact an other NFS-server than the one used for mounting. The
MOUNT protocol will not be used, and therefore the NFS-server is not notified
that there is a new client connecting. This is not a problem for the NFS-protocol
itself, but makes it difficult to give accurate results about which NFS-client is
using which NFS-server.

Possible solutions when using a Virtual IP-address:
a. leave it, this may not be a real concern
b. save the rwtab-cache on a Gluster volume and use the same cache on all 
   NFS-servers
c. save the rwtab-cache on a Gluster volume, but name the file depending on the
   IP-address used for mounting
d. something else?
Comment 2 Niels de Vos 2013-01-25 08:04:40 EST
Patch available for testing and review: http://review.gluster.org/4430
Comment 3 Niels de Vos 2013-01-25 09:44:24 EST
Thread for discussions/feedback:
- http://lists.nongnu.org/archive/html/gluster-devel/2013-01/msg00093.html
Comment 4 Niels de Vos 2013-06-05 09:48:48 EDT
Patches:

store: move glusterd_store functions from mgmt/glusterd to libglusterfs
- http://review.gluster.org/4676

store: Add (un)locking functionality
- http://review.gluster.org/4677

nfs: persistent caching of connected NFS-clients
- http://review.gluster.org/4430
Comment 5 Anand Avati 2013-06-20 09:49:33 EDT
COMMIT: http://review.gluster.org/4676 committed in master by Vijay Bellur (vbellur@redhat.com) 
------
commit bc27b7a9e44f2af647b87ab393b0fd1cacd211cf
Author: Niels de Vos <ndevos@redhat.com>
Date:   Fri Mar 15 10:26:53 2013 +0100

    store: move glusterd_store functions from mgmt/glusterd to libglusterfs
    
    Making the glusterd_store_* functions re-usable will help with future
    changes that need to read/write lists of items.
    
    BUG: 904065
    Change-Id: I99fb8eced76d12d5a254567eccff9790b43d8da3
    Signed-off-by: Niels de Vos <ndevos@redhat.com>
    Reviewed-on: http://review.gluster.org/4676
    Reviewed-by: Krishnan Parthasarathi <kparthas@redhat.com>
    Tested-by: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Vijay Bellur <vbellur@redhat.com>
Comment 6 Anand Avati 2013-06-20 09:58:49 EDT
REVIEW: http://review.gluster.org/5243 (store: Fix resource leaks in gf_store_iter_* functions) posted (#1) for review on master by Krishnan Parthasarathi (kparthas@redhat.com)
Comment 7 Anand Avati 2013-06-21 09:17:26 EDT
REVIEW: http://review.gluster.org/4677 (store: Add (un)locking functionality) posted (#7) for review on master by Niels de Vos (ndevos@redhat.com)
Comment 8 Anand Avati 2013-06-21 09:23:56 EDT
REVIEW: http://review.gluster.org/4430 (nfs: persistent caching of connected NFS-clients) posted (#10) for review on master by Niels de Vos (ndevos@redhat.com)
Comment 9 Anand Avati 2013-07-02 03:48:09 EDT
REVIEW: http://review.gluster.org/5243 (store: Fix resource leaks in gf_store_iter_* functions) posted (#2) for review on master by Krishnan Parthasarathi (kparthas@redhat.com)
Comment 10 Niels de Vos 2013-07-02 10:21:34 EDT
Created attachment 767776 [details]
systemtap script that can be used to verify the FILE* leaks

This systemtap script contains a little description in the header. When
running this script, all fdopen() calls should have one or more fclose()
calls. The example below shows correct results gathered by setting a volume
option with http://review.gluster.org/5243 #2.

...
open for file: /var/lib/glusterd/vols/ufo/rbstate.tmp
open returned: 7
open for file: <unknown>
open returned: 0
_IO_new_fdopen return for fd: 10, 0x7fb9c4007010
_IO_new_fdopen return for fd: 10, 0x7fb9c4007010
_IO_new_fclose: 0x7fb9c4007010 (fd=10)
_IO_new_fclose: 0x7fb9c4007010 (fd=10)
close: 10
open for file: /var/lib/glusterd/vols/ufo
open returned: 10
close: 10
close: 7
open for file: /var/lib/glusterd/vols/ufo/node_state.info.tmp
open returned: 7
open for file: <unknown>
open returned: 0
_IO_new_fdopen return for fd: 10, 0x7fb9c4007010
_IO_new_fdopen return for fd: 10, 0x7fb9c4007010
_IO_new_fclose: 0x7fb9c4007010 (fd=10)
_IO_new_fclose: 0x7fb9c4007010 (fd=10)
close: 10
...
Comment 11 Anand Avati 2013-07-02 13:26:28 EDT
COMMIT: http://review.gluster.org/5243 committed in master by Vijay Bellur (vbellur@redhat.com) 
------
commit 979a17d49a8dc9a19d9f3a466c137d5cf2c79a07
Author: Krishnan Parthasarathi <kparthas@redhat.com>
Date:   Wed Jun 19 16:07:25 2013 +0530

    store: Fix resource leaks in gf_store_iter_* functions
    
    Also, removed (redundant) member fd from gf_store_iter_t
    
    Change-Id: I40f0469997f77fa2f578a5495ca4ce285f1a59f2
    BUG: 904065
    Signed-off-by: Krishnan Parthasarathi <kparthas@redhat.com>
    Reviewed-on: http://review.gluster.org/5243
    Reviewed-by: Niels de Vos <ndevos@redhat.com>
    Tested-by: Niels de Vos <ndevos@redhat.com>
    Tested-by: Gluster Build System <jenkins@build.gluster.com>
Comment 12 Anand Avati 2013-07-03 04:31:47 EDT
REVIEW: http://review.gluster.org/5279 (store: move glusterd_store functions from mgmt/glusterd to libglusterfs) posted (#1) for review on release-3.4 by Krishnan Parthasarathi (kparthas@redhat.com)
Comment 13 Anand Avati 2013-07-04 00:28:21 EDT
COMMIT: http://review.gluster.org/5279 committed in release-3.4 by Vijay Bellur (vbellur@redhat.com) 
------
commit f173fb79f9f0bb518226f057c6c71284ae203092
Author: Krishnan Parthasarathi <kparthas@redhat.com>
Date:   Wed Jul 3 13:45:29 2013 +0530

    store: move glusterd_store functions from mgmt/glusterd to libglusterfs
    
            Backport of http://review.gluster.org/4676 and
                        http://review.gluster.org/5243
    
    Making the glusterd_store_* functions re-usable will help with future
    changes that need to read/write lists of items.
    
    BUG: 904065
    Change-Id: I99fb8eced76d12d5a254567eccff9790b43d8da3
    Original-author: Niels de Vos <ndevos@redhat.com>
    Signed-off-by: Krishnan Parthasarathi <kparthas@redhat.com>
    Reviewed-on: http://review.gluster.org/5279
    Reviewed-by: Niels de Vos <ndevos@redhat.com>
    Tested-by: Gluster Build System <jenkins@build.gluster.com>
Comment 14 Anand Avati 2013-07-31 05:35:13 EDT
REVIEW: http://review.gluster.org/4677 (store: Add (un)locking functionality) posted (#8) for review on master by Niels de Vos (ndevos@redhat.com)
Comment 15 Anand Avati 2013-07-31 05:35:42 EDT
REVIEW: http://review.gluster.org/4430 (nfs: persistent caching of connected NFS-clients) posted (#11) for review on master by Niels de Vos (ndevos@redhat.com)
Comment 16 Anand Avati 2013-08-02 05:14:48 EDT
COMMIT: http://review.gluster.org/4677 committed in master by Vijay Bellur (vbellur@redhat.com) 
------
commit c95db3046c672473611d9ac0ab6cd93bd8211347
Author: Niels de Vos <ndevos@redhat.com>
Date:   Fri Mar 15 10:34:45 2013 +0100

    store: Add (un)locking functionality
    
    Some configuration/cache files (like the NFS rmtab) can be stored on
    a GlusterFS volume and be used by multiple storage servers. This
    requires suitable locking for the gf_store_handle_t structure. Introduce
    gf_store_lock() and gf_store_unlock() for this purpose. The
    gf_store_locked_local() function can be used to check if the
    gf_store_handle_t has been locked by the current process.
    
    This change also includes an unrelated correction where a FILE* was
    getting leaked. Krishnan Parthasarathi identified this while reviewing
    the new locking functionality.
    
    Change-Id: I431b7510801841d4bad64480b4bb99d87e2ad347
    BUG: 904065
    Signed-off-by: Niels de Vos <ndevos@redhat.com>
    Reviewed-on: http://review.gluster.org/4677
    Reviewed-by: Rajesh Joseph <rjoseph@redhat.com>
    Tested-by: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Vijay Bellur <vbellur@redhat.com>
Comment 17 Anand Avati 2013-08-07 10:09:21 EDT
REVIEW: http://review.gluster.org/4430 (nfs: persistent caching of connected NFS-clients) posted (#12) for review on master by Niels de Vos (ndevos@redhat.com)
Comment 18 Anand Avati 2013-08-09 08:32:33 EDT
REVIEW: http://review.gluster.org/4430 (nfs: persistent caching of connected NFS-clients) posted (#13) for review on master by Niels de Vos (ndevos@redhat.com)
Comment 19 Anand Avati 2013-08-20 10:09:28 EDT
REVIEW: http://review.gluster.org/4430 (nfs: persistent caching of connected NFS-clients) posted (#14) for review on master by Niels de Vos (ndevos@redhat.com)
Comment 20 Anand Avati 2013-08-24 08:13:33 EDT
REVIEW: http://review.gluster.org/4430 (nfs: persistent caching of connected NFS-clients) posted (#15) for review on master by Niels de Vos (ndevos@redhat.com)
Comment 21 Anand Avati 2013-08-27 08:39:21 EDT
REVIEW: http://review.gluster.org/4430 (nfs: persistent caching of connected NFS-clients) posted (#16) for review on master by Niels de Vos (ndevos@redhat.com)
Comment 22 Anand Avati 2013-08-28 09:55:25 EDT
COMMIT: http://review.gluster.org/4430 committed in master by Anand Avati (avati@redhat.com) 
------
commit bbefeffafe9a2a5ba493e4bc0c9c9480d577e881
Author: Niels de Vos <ndevos@redhat.com>
Date:   Fri Aug 9 14:17:33 2013 +0200

    nfs: persistent caching of connected NFS-clients
    
    Introduce /var/lib/glusterfs/nfs/rmtab to contain a list of NFS-clients
    which have a volume mounted. The volume option 'nfs.mount-rmtab' can be
    set to an alternative filename. When the file is located on shared
    storage, multiple gNFS servers can use the same file to present a single
    NFS-server.
    
    This cache is read when a system administrator calls 'showmount -a' and
    updated when an NFS-client calls MNT or UMNT from the MOUNT protocol.
    
    Usage:
    - create a volume for storing the shared rmtab file
    - mount the volume on all storage servers, at the same location
    - make sure that the volume is mounted at boot (add to /etc/fstab)
    - place the rmtab file on the volume:
       # gluster volume set <VOLUME> nfs.mount-rmtab <MOUNTPOINT>/<FILENAME>
    - any subsequent mount requests will add an entry to this file
    - 'showmount -a' requests will return the NFS-clients using the cluster
    
    Note:
    The NFS-server does currently not support reconfigure(). When a
    configuration option is set/changed, the NFS-server glusterfs process
    gets restarted. This causes the active NFS-clients to be forgotten (the
    entries are saved in the old rmtab, but we do not have a reference to
    that file any more, so we can't re-add them). Therefor a re-mount done
    by the NFS-clients is needed before they get listed in the rmtab again.
    
    Change-Id: I58f47135d60ad112849d647bea4e1129683dd2b3
    BUG: 904065
    Signed-off-by: Niels de Vos <ndevos@redhat.com>
    Reviewed-on: http://review.gluster.org/4430
    Tested-by: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Harshavardhana <harsha@harshavardhana.net>
    Tested-by: Harshavardhana <harsha@harshavardhana.net>
    Reviewed-by: Rajesh Joseph <rjoseph@redhat.com>
Comment 23 Niels de Vos 2014-04-17 07:40:30 EDT
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.5.0, please reopen this bug report.

glusterfs-3.5.0 has been announced on the Gluster Developers mailinglist [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/6137
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Note You need to log in before you can comment on or make changes to this bug.