Bug 178266

Summary: VFS: file-max limit 50905 reached
Product: [Fedora] Fedora Reporter: Ed Swierk <eswierk>
Component: kernelAssignee: Dave Jones <davej>
Status: CLOSED INSUFFICIENT_DATA QA Contact: Brian Brock <bbrock>
Severity: high Docs Contact:
Priority: medium    
Version: 4CC: pfrields, wtogami
Target Milestone: ---   
Target Release: ---   
Hardware: i386   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2006-05-05 12:59:20 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Ed Swierk 2006-01-18 22:34:29 UTC
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8) Gecko/20051201 Fedora/1.5-1.1.fc4.nr Firefox/1.5

Description of problem:
When I run make on a fairly large C++ program, I get an error "Too many open files in system", accompanied by the kernel message "VFS: file-max limit 50905 reached".

The error occurs at somewhat random points during the build, but usually occurs during linking, which presumably causes many library files to be opened simultaneously.

I can't imagine that ld actually opens anywhere near 50000 files simultaneously, however.  I even tried increasing the maximum file descriptors by running "echo 100000 >/proc/sys/fs/file-max".  The build runs a bit longer, but I soon get the same error.  This indicates to me that the kernel is somehow leaking fds internally.


Version-Release number of selected component (if applicable):
kernel-2.6.14-1.1656_FC4

How reproducible:
Always

Steps to Reproduce:
The problem is easy for me to reproduce, but unfortunately I cannot distribute the program I am attempting to build.  I will try to reproduce the problem by building some open-source software package, and will update this bug with more details.


Additional info:

The problem occurs every time on kernel-2.6.14-1.1656_FC4 and kernel-2.6.14-1.1644_FC4.

It does not occur at all on kernel-2.6.13-1.1532_FC4 or kernel-2.6.11-1.1369_FC4.

I am running Fedora Core 4 within qemu-0.7.2 configured with 512 MB of memory.  (The host machine is a 3 GHz dual-core Pentium D with 2 GB of RAM, also running FC4.)

Comment 1 Ed Swierk 2006-01-19 00:27:52 UTC
The problem also occurs on kernel-2.6.15-1.1824_FC4 (testing).


Comment 2 Ed Swierk 2006-01-19 03:13:09 UTC
This thread on the linux-kernel mailing list seems relevant:
http://groups.google.com/group/fa.linux.kernel/browse_thread/thread/43eef3f1d89a4ef2/

My vague understanding of this thread: It seems that the way the kernel frees up
closed file descriptors was changed in 2.6.14.  It now uses a mechanism called
an RCU callback, which ksoftirqd runs every so often, and can process a certain
number of tasks each time it runs, limited by a configurable parameter
(rcupdate.maxbatch).  If for whatever reason the callback does not run
frequently enough, closed-but-not-freed file descriptors start to pile up until
the maximum is reached.  (This situation seems quite possible given that
ksoftirqd runs at nice +19.)

My guess is that this change has made the kernel much more sensitive to the
weird realtime clock behavior that is common when running on a virtual machine
like qemu, where the frequency of timer interrupts might be off by a factor of
10 depending on how heavily loaded the host machine is.

Obviously I don't understand all the issues here, but perusing the linux-kernel
thread doesn't inspire confidence that every corner case has been accounted for.


Comment 3 Ed Swierk 2006-01-19 03:21:20 UTC
I can reproduce this problem by building the Boost library
(http://prdownloads.sourceforge.net/boost/boost_1_33_1.tar.bz2):

$ tar jxvf boost_1_32_0.tar.bz2
$ cd boost_1_32_0/tools/build/jam_src
$ ./build.sh
$ cd ../../..
$ tools/build/jam_src/bin.linuxx86/bjam

A few minutes into the build, I get the error:

/usr/include/features.h:314:25: error:
/usr/lib/gcc/i386-redhat-linux/4.0.2/../../../../include/c++/4.0.2/sys/cdefs.h:
Too many open files in system

with the following in dmesg:

VFS: file-max limit 50905 reached


Comment 4 Dave Jones 2006-02-03 05:42:12 UTC
This is a mass-update to all currently open kernel bugs.

A new kernel update has been released (Version: 2.6.15-1.1830_FC4)
based upon a new upstream kernel release.

Please retest against this new kernel, as a large number of patches
go into each upstream release, possibly including changes that
may address this problem.

This bug has been placed in NEEDINFO_REPORTER state.
Due to the large volume of inactive bugs in bugzilla, if this bug is
still in this state in two weeks time, it will be closed.

Should this bug still be relevant after this period, the reporter
can reopen the bug at any time. Any other users on the Cc: list
of this bug can request that the bug be reopened by adding a
comment to the bug.

If this bug is a problem preventing you from installing the
release this version is filed against, please see bug 169613.

Thank you.


Comment 5 John Thacker 2006-05-05 12:59:20 UTC
Closing due to lack of response.