Bug 765422 (GLUSTER-3690)

Summary: Gluster reserving thousands of FDs for one directory
Product: [Community] GlusterFS Reporter: Anthony Delviscio <adelviscio>
Component: nfsAssignee: Vinayaga Raman <vraman>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.1.6CC: gluster-bugs, rwheeler
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-08-17 07:58:27 EDT Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
Description Flags
split nfs log
split nfs log
split nfs log 3/10
split nfs log 4/10
split nfs log 5/10
split nfs log 6/10
split nfs log 7/10
split nfs log 8/10
split nfs log 9/10
split nfs log 10/10 none

Description Anthony Delviscio 2011-10-03 19:14:42 EDT
GlusterFS allocates thousands of FDs for one particular directory.  This FD allocation is present on all nodes that serve the GlusterFS volume.  Over time, GlusterFS FD allocation will increase from ~5k to 50k+ FDs in the span of 10-20 minutes.

http://pastie.org/2633989 contains some information that was gathered for submission to the IRC channel.

It was suggested that I file a bug report.

If specific log files are required, let me know and I will attach them.
Thank you
Comment 1 Krishna Srinivas 2011-10-04 04:20:51 EDT
Can you attach the entire nfs log file? Did any of the servers go down and come back up? Are you using 3.1.6?
Comment 2 Anthony Delviscio 2011-10-04 06:23:03 EDT
(In reply to comment #1)
> Can you attach the entire nfs log file? Did any of the servers go down and come
> back up? Are you using 3.1.6?

The servers did not go down or reboot but certain daemons like SSHD were not able to open file handles for incoming connections.  Other processes on the machines were also adversely affected.

The nfs.log is too big and cannot be attached (8MB gzipped).  Can I email it to you instead?

We are using 3.1.6.

Thank you
Comment 3 Anthony Delviscio 2011-10-05 00:08:36 EDT
Created attachment 684

file one of ten
Comment 4 Anthony Delviscio 2011-10-05 00:09:05 EDT
Created attachment 685

file 2 of 10
Comment 5 Anthony Delviscio 2011-10-05 00:09:42 EDT
Created attachment 686
Comment 6 Anthony Delviscio 2011-10-05 00:10:08 EDT
Created attachment 687
Comment 7 Anthony Delviscio 2011-10-05 00:10:27 EDT
Created attachment 688
Comment 8 Anthony Delviscio 2011-10-05 00:11:08 EDT
Created attachment 689
Comment 9 Anthony Delviscio 2011-10-05 00:11:33 EDT
Created attachment 690
Comment 10 Anthony Delviscio 2011-10-05 00:12:07 EDT
Created attachment 691
Comment 11 Anthony Delviscio 2011-10-05 00:12:40 EDT
Created attachment 692
Comment 12 Anthony Delviscio 2011-10-05 00:13:07 EDT
Created attachment 693
Comment 13 Krishna Srinivas 2012-08-14 08:13:39 EDT
Hi Anthony, can you reproduce the issue with the latest release (3.3)?
Comment 14 Krishna Srinivas 2012-08-17 07:58:27 EDT
Anthony, you should not see this issue in 3.3 as we use "anonymous" stateless FDs and hence open-fd leak problem should not be seen at all. Please re-open it if you still see it.