Bug 1169317

Summary:	rmtab file is a bottleneck when lot of clients are accessing a volume through NFS
Product:	[Community] GlusterFS	Reporter:	Niels de Vos <ndevos>
Component:	nfs	Assignee:	Niels de Vos <ndevos>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	mainline	CC:	bugs, gluster-bugs
Target Milestone:	---	Keywords:	Reopened, Triaged
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	glusterfs-3.8rc2	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:	1166862
Clones:	1215385 (view as bug list)		Environment:
Last Closed:	2016-06-16 12:40:24 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1166862, 1169320, 1215385, 1234096

Description Niels de Vos 2014-12-01 10:31:31 UTC

+++ This bug was initially created as a clone of Bug #1166862 +++

Description of problem:

This feature: http://review.gluster.org/#/c/4430/

Create bottleneck when several clients are accessing a nfs volume.

On our setup:

Gluster 3.5.2 under centos7.

Hardware:

	dual Xeon® CPU E5-2640
	64GB RAM
	SSD for rootfs
	10Gb NIC

Context:

	Around 700 nfs clients for small file or vm images.


Version-Release number of selected component (if applicable):

3.5.2

How reproducible:

Always as long as you have enough NFS clients

Steps to Reproduce:
1. Create a volume accessible through gluster nfs
2. Make it accessible for 700 clients
3. See how it hangs

Actual results:

NFS client got intermittent hang (every minutes / for 10s each time). Even an “rpcinfo -t server nfs 3" will hang.

Gluster nfs process literally eat the CPU of the server

Expected results:

No hanging

Additional info:

The cause:

the rmtab file located in /var/lib/glusterd/nfs/ is flushed from memory to  /var/lib/glusterd/nfs/rmtab.tmp. During this time, NFS server hang literraly.

Workaroud:

Move the file to memory for faster I/O using this option:

set nfs.mount-rmtab: /dev/shm/glusterfs.rmtab

Result:

We still have some hang but for ~300ms now, the load average of the server is WAY better.

Personal thought:

This feature is not usable and should be disabled by default.

You can find attached load average and Disk usage before and after using SHM for rmtab.

Comment 1 Anand Avati 2014-12-01 10:32:14 UTC

REVIEW: http://review.gluster.org/9223 (nfs: make it possible to disable nfs.mount-rmtab) posted (#2) for review on master by Niels de Vos (ndevos)

Comment 2 Anand Avati 2014-12-01 14:18:03 UTC

REVIEW: http://review.gluster.org/9223 (nfs: make it possible to disable nfs.mount-rmtab) posted (#4) for review on master by Niels de Vos (ndevos)

Comment 3 Anand Avati 2014-12-02 11:14:50 UTC

REVIEW: http://review.gluster.org/9223 (nfs: make it possible to disable nfs.mount-rmtab) posted (#5) for review on master by Niels de Vos (ndevos)

Comment 4 Anand Avati 2014-12-05 21:30:27 UTC

REVIEW: http://review.gluster.org/9223 (nfs: make it possible to disable nfs.mount-rmtab) posted (#6) for review on master by Niels de Vos (ndevos)

Comment 5 Anand Avati 2015-04-26 08:40:25 UTC

COMMIT: http://review.gluster.org/9223 committed in master by Niels de Vos (ndevos) 
------
commit 331ef6e1a86bfc0a93f8a9dec6ad35c417873849
Author: Niels de Vos <ndevos>
Date:   Tue Dec 2 10:54:53 2014 +0100

    nfs: make it possible to disable nfs.mount-rmtab
    
    When there are many NFS-clients doing very often mount/unmount actions,
    the updating of the 'rmtab' can become a bottleneck and cause delays. In
    these situations, the output of 'showmount' may be less important than
    the responsiveness of the (un)mounting.
    
    By setting 'nfs.mount-rmtab' to the value "/-", the cache file is not
    updated anymore, and the entries are only kept in memory.
    
    BUG: 1169317
    Change-Id: I40c4d8d754932f86fb2b1b2588843390464c773d
    Reported-by: Cyril Peponnet <cyril>
    Signed-off-by: Niels de Vos <ndevos>
    Reviewed-on: http://review.gluster.org/9223
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: soumya k <skoduri>
    Reviewed-by: jiffin tony Thottan <jthottan>
    Reviewed-by: Kaleb KEITHLEY <kkeithle>

Comment 6 Nagaprasad Sathyanarayana 2015-10-25 15:15:45 UTC

Fix for this BZ is already present in a GlusterFS release. You can find clone of this BZ, fixed in a GlusterFS release and closed. Hence closing this mainline BZ as well.

Comment 7 Niels de Vos 2016-06-16 12:40:24 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.0, please open a new bug report.

glusterfs-3.8.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://blog.gluster.org/2016/06/glusterfs-3-8-released/
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user