Description of problem: When a NFS-client mounts a volume, showmount lists the NFS-clients. However, after a restart of the Gluster NFS-server, the NFS-clients are not listed anymore. The NFS-clients keep functioning without any issue though. Steps to Reproduce: 1. mount a volume through NFS 2. list the NFS-clients "showmount -a $SERVER" 3. service glusterd restart 4. list the NFS-clients "showmount -a $SERVER" Actual results: In step 4 the NFS-clients are not listed anymore. Expected results: The NFS-clients should be listed in step 4. This is how other NFS-servers (like nfsd in the RHEL kernel) function. Additional info: The persistent logging of the RHEL NFS-server is done through the rpc.mountd process. rpc.mountd saves the NFS-client and mountpoint in /var/lib/nfs/rwtab.
For this to work nicely, I'd like to introduce a rwtab similar to what rpc.mountd uses. When the Gluster NFS-server starts, it should read this file, when an NFS-client mounts a volume, this file should be extended, and upon unmounting the file should be updated again. This binds the rwtab-cache to a specific NFS-server. This may not be desirable when a Virtual IP-address is in use. When the IP-address is failed-over, the NFS-client will contact an other NFS-server than the one used for mounting. The MOUNT protocol will not be used, and therefore the NFS-server is not notified that there is a new client connecting. This is not a problem for the NFS-protocol itself, but makes it difficult to give accurate results about which NFS-client is using which NFS-server. Possible solutions when using a Virtual IP-address: a. leave it, this may not be a real concern b. save the rwtab-cache on a Gluster volume and use the same cache on all NFS-servers c. save the rwtab-cache on a Gluster volume, but name the file depending on the IP-address used for mounting d. something else?
Patch available for testing and review: http://review.gluster.org/4430
Thread for discussions/feedback: - http://lists.nongnu.org/archive/html/gluster-devel/2013-01/msg00093.html
Patches: store: move glusterd_store functions from mgmt/glusterd to libglusterfs - http://review.gluster.org/4676 store: Add (un)locking functionality - http://review.gluster.org/4677 nfs: persistent caching of connected NFS-clients - http://review.gluster.org/4430
COMMIT: http://review.gluster.org/4676 committed in master by Vijay Bellur (vbellur) ------ commit bc27b7a9e44f2af647b87ab393b0fd1cacd211cf Author: Niels de Vos <ndevos> Date: Fri Mar 15 10:26:53 2013 +0100 store: move glusterd_store functions from mgmt/glusterd to libglusterfs Making the glusterd_store_* functions re-usable will help with future changes that need to read/write lists of items. BUG: 904065 Change-Id: I99fb8eced76d12d5a254567eccff9790b43d8da3 Signed-off-by: Niels de Vos <ndevos> Reviewed-on: http://review.gluster.org/4676 Reviewed-by: Krishnan Parthasarathi <kparthas> Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Vijay Bellur <vbellur>
REVIEW: http://review.gluster.org/5243 (store: Fix resource leaks in gf_store_iter_* functions) posted (#1) for review on master by Krishnan Parthasarathi (kparthas)
REVIEW: http://review.gluster.org/4677 (store: Add (un)locking functionality) posted (#7) for review on master by Niels de Vos (ndevos)
REVIEW: http://review.gluster.org/4430 (nfs: persistent caching of connected NFS-clients) posted (#10) for review on master by Niels de Vos (ndevos)
REVIEW: http://review.gluster.org/5243 (store: Fix resource leaks in gf_store_iter_* functions) posted (#2) for review on master by Krishnan Parthasarathi (kparthas)
Created attachment 767776 [details] systemtap script that can be used to verify the FILE* leaks This systemtap script contains a little description in the header. When running this script, all fdopen() calls should have one or more fclose() calls. The example below shows correct results gathered by setting a volume option with http://review.gluster.org/5243 #2. ... open for file: /var/lib/glusterd/vols/ufo/rbstate.tmp open returned: 7 open for file: <unknown> open returned: 0 _IO_new_fdopen return for fd: 10, 0x7fb9c4007010 _IO_new_fdopen return for fd: 10, 0x7fb9c4007010 _IO_new_fclose: 0x7fb9c4007010 (fd=10) _IO_new_fclose: 0x7fb9c4007010 (fd=10) close: 10 open for file: /var/lib/glusterd/vols/ufo open returned: 10 close: 10 close: 7 open for file: /var/lib/glusterd/vols/ufo/node_state.info.tmp open returned: 7 open for file: <unknown> open returned: 0 _IO_new_fdopen return for fd: 10, 0x7fb9c4007010 _IO_new_fdopen return for fd: 10, 0x7fb9c4007010 _IO_new_fclose: 0x7fb9c4007010 (fd=10) _IO_new_fclose: 0x7fb9c4007010 (fd=10) close: 10 ...
COMMIT: http://review.gluster.org/5243 committed in master by Vijay Bellur (vbellur) ------ commit 979a17d49a8dc9a19d9f3a466c137d5cf2c79a07 Author: Krishnan Parthasarathi <kparthas> Date: Wed Jun 19 16:07:25 2013 +0530 store: Fix resource leaks in gf_store_iter_* functions Also, removed (redundant) member fd from gf_store_iter_t Change-Id: I40f0469997f77fa2f578a5495ca4ce285f1a59f2 BUG: 904065 Signed-off-by: Krishnan Parthasarathi <kparthas> Reviewed-on: http://review.gluster.org/5243 Reviewed-by: Niels de Vos <ndevos> Tested-by: Niels de Vos <ndevos> Tested-by: Gluster Build System <jenkins.com>
REVIEW: http://review.gluster.org/5279 (store: move glusterd_store functions from mgmt/glusterd to libglusterfs) posted (#1) for review on release-3.4 by Krishnan Parthasarathi (kparthas)
COMMIT: http://review.gluster.org/5279 committed in release-3.4 by Vijay Bellur (vbellur) ------ commit f173fb79f9f0bb518226f057c6c71284ae203092 Author: Krishnan Parthasarathi <kparthas> Date: Wed Jul 3 13:45:29 2013 +0530 store: move glusterd_store functions from mgmt/glusterd to libglusterfs Backport of http://review.gluster.org/4676 and http://review.gluster.org/5243 Making the glusterd_store_* functions re-usable will help with future changes that need to read/write lists of items. BUG: 904065 Change-Id: I99fb8eced76d12d5a254567eccff9790b43d8da3 Original-author: Niels de Vos <ndevos> Signed-off-by: Krishnan Parthasarathi <kparthas> Reviewed-on: http://review.gluster.org/5279 Reviewed-by: Niels de Vos <ndevos> Tested-by: Gluster Build System <jenkins.com>
REVIEW: http://review.gluster.org/4677 (store: Add (un)locking functionality) posted (#8) for review on master by Niels de Vos (ndevos)
REVIEW: http://review.gluster.org/4430 (nfs: persistent caching of connected NFS-clients) posted (#11) for review on master by Niels de Vos (ndevos)
COMMIT: http://review.gluster.org/4677 committed in master by Vijay Bellur (vbellur) ------ commit c95db3046c672473611d9ac0ab6cd93bd8211347 Author: Niels de Vos <ndevos> Date: Fri Mar 15 10:34:45 2013 +0100 store: Add (un)locking functionality Some configuration/cache files (like the NFS rmtab) can be stored on a GlusterFS volume and be used by multiple storage servers. This requires suitable locking for the gf_store_handle_t structure. Introduce gf_store_lock() and gf_store_unlock() for this purpose. The gf_store_locked_local() function can be used to check if the gf_store_handle_t has been locked by the current process. This change also includes an unrelated correction where a FILE* was getting leaked. Krishnan Parthasarathi identified this while reviewing the new locking functionality. Change-Id: I431b7510801841d4bad64480b4bb99d87e2ad347 BUG: 904065 Signed-off-by: Niels de Vos <ndevos> Reviewed-on: http://review.gluster.org/4677 Reviewed-by: Rajesh Joseph <rjoseph> Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Vijay Bellur <vbellur>
REVIEW: http://review.gluster.org/4430 (nfs: persistent caching of connected NFS-clients) posted (#12) for review on master by Niels de Vos (ndevos)
REVIEW: http://review.gluster.org/4430 (nfs: persistent caching of connected NFS-clients) posted (#13) for review on master by Niels de Vos (ndevos)
REVIEW: http://review.gluster.org/4430 (nfs: persistent caching of connected NFS-clients) posted (#14) for review on master by Niels de Vos (ndevos)
REVIEW: http://review.gluster.org/4430 (nfs: persistent caching of connected NFS-clients) posted (#15) for review on master by Niels de Vos (ndevos)
REVIEW: http://review.gluster.org/4430 (nfs: persistent caching of connected NFS-clients) posted (#16) for review on master by Niels de Vos (ndevos)
COMMIT: http://review.gluster.org/4430 committed in master by Anand Avati (avati) ------ commit bbefeffafe9a2a5ba493e4bc0c9c9480d577e881 Author: Niels de Vos <ndevos> Date: Fri Aug 9 14:17:33 2013 +0200 nfs: persistent caching of connected NFS-clients Introduce /var/lib/glusterfs/nfs/rmtab to contain a list of NFS-clients which have a volume mounted. The volume option 'nfs.mount-rmtab' can be set to an alternative filename. When the file is located on shared storage, multiple gNFS servers can use the same file to present a single NFS-server. This cache is read when a system administrator calls 'showmount -a' and updated when an NFS-client calls MNT or UMNT from the MOUNT protocol. Usage: - create a volume for storing the shared rmtab file - mount the volume on all storage servers, at the same location - make sure that the volume is mounted at boot (add to /etc/fstab) - place the rmtab file on the volume: # gluster volume set <VOLUME> nfs.mount-rmtab <MOUNTPOINT>/<FILENAME> - any subsequent mount requests will add an entry to this file - 'showmount -a' requests will return the NFS-clients using the cluster Note: The NFS-server does currently not support reconfigure(). When a configuration option is set/changed, the NFS-server glusterfs process gets restarted. This causes the active NFS-clients to be forgotten (the entries are saved in the old rmtab, but we do not have a reference to that file any more, so we can't re-add them). Therefor a re-mount done by the NFS-clients is needed before they get listed in the rmtab again. Change-Id: I58f47135d60ad112849d647bea4e1129683dd2b3 BUG: 904065 Signed-off-by: Niels de Vos <ndevos> Reviewed-on: http://review.gluster.org/4430 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Harshavardhana <harsha> Tested-by: Harshavardhana <harsha> Reviewed-by: Rajesh Joseph <rjoseph>
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.5.0, please reopen this bug report. glusterfs-3.5.0 has been announced on the Gluster Developers mailinglist [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/6137 [2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user