Bug 765422 (GLUSTER-3690)

Summary: Gluster reserving thousands of FDs for one directory
Product: [Community] GlusterFS Reporter: Anthony Delviscio <adelviscio>
Component: nfsAssignee: Vinayaga Raman <vraman>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: medium Docs Contact:
Priority: medium    
Version: 3.1.6CC: gluster-bugs, rwheeler
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-08-17 11:58:27 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
split nfs log
none
split nfs log
none
split nfs log 3/10
none
split nfs log 4/10
none
split nfs log 5/10
none
split nfs log 6/10
none
split nfs log 7/10
none
split nfs log 8/10
none
split nfs log 9/10
none
split nfs log 10/10 none

Description Anthony Delviscio 2011-10-03 23:14:42 UTC 
GlusterFS allocates thousands of FDs for one particular directory.  This FD allocation is present on all nodes that serve the GlusterFS volume.  Over time, GlusterFS FD allocation will increase from ~5k to 50k+ FDs in the span of 10-20 minutes.

http://pastie.org/2633989 contains some information that was gathered for submission to the IRC channel.

It was suggested that I file a bug report.

If specific log files are required, let me know and I will attach them.
Thank you
Comment 1 Krishna Srinivas 2011-10-04 08:20:51 UTC 
Can you attach the entire nfs log file? Did any of the servers go down and come back up? Are you using 3.1.6?
Comment 2 Anthony Delviscio 2011-10-04 10:23:03 UTC 
(In reply to comment #1)
> Can you attach the entire nfs log file? Did any of the servers go down and come
> back up? Are you using 3.1.6?

The servers did not go down or reboot but certain daemons like SSHD were not able to open file handles for incoming connections.  Other processes on the machines were also adversely affected.

The nfs.log is too big and cannot be attached (8MB gzipped).  Can I email it to you instead?

We are using 3.1.6.

Thank you
Comment 3 Anthony Delviscio 2011-10-05 04:08:36 UTC 
Created attachment 684


file one of ten
Comment 4 Anthony Delviscio 2011-10-05 04:09:05 UTC 
Created attachment 685


file 2 of 10
Comment 5 Anthony Delviscio 2011-10-05 04:09:42 UTC 
Created attachment 686
Comment 6 Anthony Delviscio 2011-10-05 04:10:08 UTC 
Created attachment 687
Comment 7 Anthony Delviscio 2011-10-05 04:10:27 UTC 
Created attachment 688
Comment 8 Anthony Delviscio 2011-10-05 04:11:08 UTC 
Created attachment 689
Comment 9 Anthony Delviscio 2011-10-05 04:11:33 UTC 
Created attachment 690
Comment 10 Anthony Delviscio 2011-10-05 04:12:07 UTC 
Created attachment 691
Comment 11 Anthony Delviscio 2011-10-05 04:12:40 UTC 
Created attachment 692
Comment 12 Anthony Delviscio 2011-10-05 04:13:07 UTC 
Created attachment 693
Comment 13 Krishna Srinivas 2012-08-14 12:13:39 UTC 
Hi Anthony, can you reproduce the issue with the latest release (3.3)?
Comment 14 Krishna Srinivas 2012-08-17 11:58:27 UTC 
Anthony, you should not see this issue in 3.3 as we use "anonymous" stateless FDs and hence open-fd leak problem should not be seen at all. Please re-open it if you still see it.