Bug 178266 - VFS: file-max limit 50905 reached
Summary: VFS: file-max limit 50905 reached
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: Fedora
Classification: Fedora
Component: kernel
Version: 4
Hardware: i386
OS: Linux
medium
high
Target Milestone: ---
Assignee: Dave Jones
QA Contact: Brian Brock
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2006-01-18 22:34 UTC by Ed Swierk
Modified: 2015-01-04 22:24 UTC (History)
2 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2006-05-05 12:59:20 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)

Description Ed Swierk 2006-01-18 22:34:29 UTC
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8) Gecko/20051201 Fedora/1.5-1.1.fc4.nr Firefox/1.5

Description of problem:
When I run make on a fairly large C++ program, I get an error "Too many open files in system", accompanied by the kernel message "VFS: file-max limit 50905 reached".

The error occurs at somewhat random points during the build, but usually occurs during linking, which presumably causes many library files to be opened simultaneously.

I can't imagine that ld actually opens anywhere near 50000 files simultaneously, however.  I even tried increasing the maximum file descriptors by running "echo 100000 >/proc/sys/fs/file-max".  The build runs a bit longer, but I soon get the same error.  This indicates to me that the kernel is somehow leaking fds internally.


Version-Release number of selected component (if applicable):
kernel-2.6.14-1.1656_FC4

How reproducible:
Always

Steps to Reproduce:
The problem is easy for me to reproduce, but unfortunately I cannot distribute the program I am attempting to build.  I will try to reproduce the problem by building some open-source software package, and will update this bug with more details.


Additional info:

The problem occurs every time on kernel-2.6.14-1.1656_FC4 and kernel-2.6.14-1.1644_FC4.

It does not occur at all on kernel-2.6.13-1.1532_FC4 or kernel-2.6.11-1.1369_FC4.

I am running Fedora Core 4 within qemu-0.7.2 configured with 512 MB of memory.  (The host machine is a 3 GHz dual-core Pentium D with 2 GB of RAM, also running FC4.)

Comment 1 Ed Swierk 2006-01-19 00:27:52 UTC
The problem also occurs on kernel-2.6.15-1.1824_FC4 (testing).


Comment 2 Ed Swierk 2006-01-19 03:13:09 UTC
This thread on the linux-kernel mailing list seems relevant:
http://groups.google.com/group/fa.linux.kernel/browse_thread/thread/43eef3f1d89a4ef2/

My vague understanding of this thread: It seems that the way the kernel frees up
closed file descriptors was changed in 2.6.14.  It now uses a mechanism called
an RCU callback, which ksoftirqd runs every so often, and can process a certain
number of tasks each time it runs, limited by a configurable parameter
(rcupdate.maxbatch).  If for whatever reason the callback does not run
frequently enough, closed-but-not-freed file descriptors start to pile up until
the maximum is reached.  (This situation seems quite possible given that
ksoftirqd runs at nice +19.)

My guess is that this change has made the kernel much more sensitive to the
weird realtime clock behavior that is common when running on a virtual machine
like qemu, where the frequency of timer interrupts might be off by a factor of
10 depending on how heavily loaded the host machine is.

Obviously I don't understand all the issues here, but perusing the linux-kernel
thread doesn't inspire confidence that every corner case has been accounted for.


Comment 3 Ed Swierk 2006-01-19 03:21:20 UTC
I can reproduce this problem by building the Boost library
(http://prdownloads.sourceforge.net/boost/boost_1_33_1.tar.bz2):

$ tar jxvf boost_1_32_0.tar.bz2
$ cd boost_1_32_0/tools/build/jam_src
$ ./build.sh
$ cd ../../..
$ tools/build/jam_src/bin.linuxx86/bjam

A few minutes into the build, I get the error:

/usr/include/features.h:314:25: error:
/usr/lib/gcc/i386-redhat-linux/4.0.2/../../../../include/c++/4.0.2/sys/cdefs.h:
Too many open files in system

with the following in dmesg:

VFS: file-max limit 50905 reached


Comment 4 Dave Jones 2006-02-03 05:42:12 UTC
This is a mass-update to all currently open kernel bugs.

A new kernel update has been released (Version: 2.6.15-1.1830_FC4)
based upon a new upstream kernel release.

Please retest against this new kernel, as a large number of patches
go into each upstream release, possibly including changes that
may address this problem.

This bug has been placed in NEEDINFO_REPORTER state.
Due to the large volume of inactive bugs in bugzilla, if this bug is
still in this state in two weeks time, it will be closed.

Should this bug still be relevant after this period, the reporter
can reopen the bug at any time. Any other users on the Cc: list
of this bug can request that the bug be reopened by adding a
comment to the bug.

If this bug is a problem preventing you from installing the
release this version is filed against, please see bug 169613.

Thank you.


Comment 5 John Thacker 2006-05-05 12:59:20 UTC
Closing due to lack of response.


Note You need to log in before you can comment on or make changes to this bug.