+++ This bug was initially created as a clone of Bug #1166862 +++ Description of problem: This feature: http://review.gluster.org/#/c/4430/ Create bottleneck when several clients are accessing a nfs volume. On our setup: Gluster 3.5.2 under centos7. Hardware: dual Xeon® CPU E5-2640 64GB RAM SSD for rootfs 10Gb NIC Context: Around 700 nfs clients for small file or vm images. Version-Release number of selected component (if applicable): 3.5.2 How reproducible: Always as long as you have enough NFS clients Steps to Reproduce: 1. Create a volume accessible through gluster nfs 2. Make it accessible for 700 clients 3. See how it hangs Actual results: NFS client got intermittent hang (every minutes / for 10s each time). Even an “rpcinfo -t server nfs 3" will hang. Gluster nfs process literally eat the CPU of the server Expected results: No hanging Additional info: The cause: the rmtab file located in /var/lib/glusterd/nfs/ is flushed from memory to /var/lib/glusterd/nfs/rmtab.tmp. During this time, NFS server hang literraly. Workaroud: Move the file to memory for faster I/O using this option: set nfs.mount-rmtab: /dev/shm/glusterfs.rmtab Result: We still have some hang but for ~300ms now, the load average of the server is WAY better. Personal thought: This feature is not usable and should be disabled by default. You can find attached load average and Disk usage before and after using SHM for rmtab.
REVIEW: http://review.gluster.org/9223 (nfs: make it possible to disable nfs.mount-rmtab) posted (#2) for review on master by Niels de Vos (ndevos)
REVIEW: http://review.gluster.org/9223 (nfs: make it possible to disable nfs.mount-rmtab) posted (#4) for review on master by Niels de Vos (ndevos)
REVIEW: http://review.gluster.org/9223 (nfs: make it possible to disable nfs.mount-rmtab) posted (#5) for review on master by Niels de Vos (ndevos)
REVIEW: http://review.gluster.org/9223 (nfs: make it possible to disable nfs.mount-rmtab) posted (#6) for review on master by Niels de Vos (ndevos)
COMMIT: http://review.gluster.org/9223 committed in master by Niels de Vos (ndevos) ------ commit 331ef6e1a86bfc0a93f8a9dec6ad35c417873849 Author: Niels de Vos <ndevos> Date: Tue Dec 2 10:54:53 2014 +0100 nfs: make it possible to disable nfs.mount-rmtab When there are many NFS-clients doing very often mount/unmount actions, the updating of the 'rmtab' can become a bottleneck and cause delays. In these situations, the output of 'showmount' may be less important than the responsiveness of the (un)mounting. By setting 'nfs.mount-rmtab' to the value "/-", the cache file is not updated anymore, and the entries are only kept in memory. BUG: 1169317 Change-Id: I40c4d8d754932f86fb2b1b2588843390464c773d Reported-by: Cyril Peponnet <cyril> Signed-off-by: Niels de Vos <ndevos> Reviewed-on: http://review.gluster.org/9223 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: soumya k <skoduri> Reviewed-by: jiffin tony Thottan <jthottan> Reviewed-by: Kaleb KEITHLEY <kkeithle>
Fix for this BZ is already present in a GlusterFS release. You can find clone of this BZ, fixed in a GlusterFS release and closed. Hence closing this mainline BZ as well.
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.0, please open a new bug report. glusterfs-3.8.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://blog.gluster.org/2016/06/glusterfs-3-8-released/ [2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user