Bug 765422 (GLUSTER-3690) - Gluster reserving thousands of FDs for one directory
Summary: Gluster reserving thousands of FDs for one directory
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: GLUSTER-3690
Product: GlusterFS
Classification: Community
Component: nfs
Version: 3.1.6
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: ---
Assignee: Vinayaga Raman
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2011-10-03 23:14 UTC by Anthony Delviscio
Modified: 2014-03-31 01:28 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2012-08-17 11:58:27 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:

Attachments (Terms of Use)
split nfs log (879.23 KB, application/gzip)
2011-10-05 04:08 UTC, Anthony Delviscio 		no flags Details
split nfs log (872.87 KB, application/gzip)
2011-10-05 04:09 UTC, Anthony Delviscio 		no flags Details
split nfs log 3/10 (869.05 KB, application/gzip)
2011-10-05 04:09 UTC, Anthony Delviscio 		no flags Details
split nfs log 4/10 (873.80 KB, application/gzip)
2011-10-05 04:10 UTC, Anthony Delviscio 		no flags Details
split nfs log 5/10 (871.21 KB, application/gzip)
2011-10-05 04:10 UTC, Anthony Delviscio 		no flags Details
split nfs log 6/10 (886.16 KB, application/gzip)
2011-10-05 04:11 UTC, Anthony Delviscio 		no flags Details
split nfs log 7/10 (722.92 KB, application/gzip)
2011-10-05 04:11 UTC, Anthony Delviscio 		no flags Details
split nfs log 8/10 (719.87 KB, application/gzip)
2011-10-05 04:12 UTC, Anthony Delviscio 		no flags Details
split nfs log 9/10 (723.74 KB, application/gzip)
2011-10-05 04:12 UTC, Anthony Delviscio 		no flags Details
split nfs log 10/10 (740.12 KB, application/gzip)
2011-10-05 04:13 UTC, Anthony Delviscio 		no flags Details
View All Add an attachment (proposed patch, testcase, etc.)

Description Anthony Delviscio 2011-10-03 23:14:42 UTC 
GlusterFS allocates thousands of FDs for one particular directory.  This FD allocation is present on all nodes that serve the GlusterFS volume.  Over time, GlusterFS FD allocation will increase from ~5k to 50k+ FDs in the span of 10-20 minutes.

http://pastie.org/2633989 contains some information that was gathered for submission to the IRC channel.

It was suggested that I file a bug report.

If specific log files are required, let me know and I will attach them.
Thank you
Comment 1 Krishna Srinivas 2011-10-04 08:20:51 UTC 
Can you attach the entire nfs log file? Did any of the servers go down and come back up? Are you using 3.1.6?
Comment 2 Anthony Delviscio 2011-10-04 10:23:03 UTC 
(In reply to comment #1)
> Can you attach the entire nfs log file? Did any of the servers go down and come
> back up? Are you using 3.1.6?

The servers did not go down or reboot but certain daemons like SSHD were not able to open file handles for incoming connections.  Other processes on the machines were also adversely affected.

The nfs.log is too big and cannot be attached (8MB gzipped).  Can I email it to you instead?

We are using 3.1.6.

Thank you
Comment 3 Anthony Delviscio 2011-10-05 04:08:36 UTC 
Created attachment 684


file one of ten
Comment 4 Anthony Delviscio 2011-10-05 04:09:05 UTC 
Created attachment 685


file 2 of 10
Comment 5 Anthony Delviscio 2011-10-05 04:09:42 UTC 
Created attachment 686
Comment 6 Anthony Delviscio 2011-10-05 04:10:08 UTC 
Created attachment 687
Comment 7 Anthony Delviscio 2011-10-05 04:10:27 UTC 
Created attachment 688
Comment 8 Anthony Delviscio 2011-10-05 04:11:08 UTC 
Created attachment 689
Comment 9 Anthony Delviscio 2011-10-05 04:11:33 UTC 
Created attachment 690
Comment 10 Anthony Delviscio 2011-10-05 04:12:07 UTC 
Created attachment 691
Comment 11 Anthony Delviscio 2011-10-05 04:12:40 UTC 
Created attachment 692
Comment 12 Anthony Delviscio 2011-10-05 04:13:07 UTC 
Created attachment 693
Comment 13 Krishna Srinivas 2012-08-14 12:13:39 UTC 
Hi Anthony, can you reproduce the issue with the latest release (3.3)?
Comment 14 Krishna Srinivas 2012-08-17 11:58:27 UTC 
Anthony, you should not see this issue in 3.3 as we use "anonymous" stateless FDs and hence open-fd leak problem should not be seen at all. Please re-open it if you still see it.
Note You need to log in before you can comment on or make changes to this bug.