Bug 765422 - (GLUSTER-3690) Gluster reserving thousands of FDs for one directory
Gluster reserving thousands of FDs for one directory
Status: CLOSED CURRENTRELEASE
Product: GlusterFS
Classification: Community
Component: nfs (Show other bugs)
3.1.6
x86_64 Linux
medium Severity medium
: ---
: ---
Assigned To: Vinayaga Raman
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2011-10-03 19:14 EDT by Anthony Delviscio
Modified: 2014-03-30 21:28 EDT (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-08-17 07:58:27 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
split nfs log (879.23 KB, application/gzip)
2011-10-05 00:08 EDT, Anthony Delviscio
no flags Details
split nfs log (872.87 KB, application/gzip)
2011-10-05 00:09 EDT, Anthony Delviscio
no flags Details
split nfs log 3/10 (869.05 KB, application/gzip)
2011-10-05 00:09 EDT, Anthony Delviscio
no flags Details
split nfs log 4/10 (873.80 KB, application/gzip)
2011-10-05 00:10 EDT, Anthony Delviscio
no flags Details
split nfs log 5/10 (871.21 KB, application/gzip)
2011-10-05 00:10 EDT, Anthony Delviscio
no flags Details
split nfs log 6/10 (886.16 KB, application/gzip)
2011-10-05 00:11 EDT, Anthony Delviscio
no flags Details
split nfs log 7/10 (722.92 KB, application/gzip)
2011-10-05 00:11 EDT, Anthony Delviscio
no flags Details
split nfs log 8/10 (719.87 KB, application/gzip)
2011-10-05 00:12 EDT, Anthony Delviscio
no flags Details
split nfs log 9/10 (723.74 KB, application/gzip)
2011-10-05 00:12 EDT, Anthony Delviscio
no flags Details
split nfs log 10/10 (740.12 KB, application/gzip)
2011-10-05 00:13 EDT, Anthony Delviscio
no flags Details

  None (edit)
Description Anthony Delviscio 2011-10-03 19:14:42 EDT
GlusterFS allocates thousands of FDs for one particular directory.  This FD allocation is present on all nodes that serve the GlusterFS volume.  Over time, GlusterFS FD allocation will increase from ~5k to 50k+ FDs in the span of 10-20 minutes.

http://pastie.org/2633989 contains some information that was gathered for submission to the IRC channel.

It was suggested that I file a bug report.

If specific log files are required, let me know and I will attach them.
Thank you
Comment 1 Krishna Srinivas 2011-10-04 04:20:51 EDT
Can you attach the entire nfs log file? Did any of the servers go down and come back up? Are you using 3.1.6?
Comment 2 Anthony Delviscio 2011-10-04 06:23:03 EDT
(In reply to comment #1)
> Can you attach the entire nfs log file? Did any of the servers go down and come
> back up? Are you using 3.1.6?

The servers did not go down or reboot but certain daemons like SSHD were not able to open file handles for incoming connections.  Other processes on the machines were also adversely affected.

The nfs.log is too big and cannot be attached (8MB gzipped).  Can I email it to you instead?

We are using 3.1.6.

Thank you
Comment 3 Anthony Delviscio 2011-10-05 00:08:36 EDT
Created attachment 684


file one of ten
Comment 4 Anthony Delviscio 2011-10-05 00:09:05 EDT
Created attachment 685


file 2 of 10
Comment 5 Anthony Delviscio 2011-10-05 00:09:42 EDT
Created attachment 686
Comment 6 Anthony Delviscio 2011-10-05 00:10:08 EDT
Created attachment 687
Comment 7 Anthony Delviscio 2011-10-05 00:10:27 EDT
Created attachment 688
Comment 8 Anthony Delviscio 2011-10-05 00:11:08 EDT
Created attachment 689
Comment 9 Anthony Delviscio 2011-10-05 00:11:33 EDT
Created attachment 690
Comment 10 Anthony Delviscio 2011-10-05 00:12:07 EDT
Created attachment 691
Comment 11 Anthony Delviscio 2011-10-05 00:12:40 EDT
Created attachment 692
Comment 12 Anthony Delviscio 2011-10-05 00:13:07 EDT
Created attachment 693
Comment 13 Krishna Srinivas 2012-08-14 08:13:39 EDT
Hi Anthony, can you reproduce the issue with the latest release (3.3)?
Comment 14 Krishna Srinivas 2012-08-17 07:58:27 EDT
Anthony, you should not see this issue in 3.3 as we use "anonymous" stateless FDs and hence open-fd leak problem should not be seen at all. Please re-open it if you still see it.

Note You need to log in before you can comment on or make changes to this bug.