This service will be undergoing maintenance at 00:00 UTC, 2016-08-01. It is expected to last about 1 hours

Bug 667539

Summary: "Too many open files" error when opening terminal
Product: [Fedora] Fedora Reporter: Roland McGrath <roland>
Component: gnome-terminalAssignee: Behdad Esfahbod <behdad>
Status: CLOSED WONTFIX QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: low    
Version: 14CC: behdad, gholms, jpazdziora, kas, purpleidea, redhat, ron, rvandolson, tom, trevor, vasmith, vdm-photo, zejdl
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-08-16 13:18:17 EDT Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:

Description Roland McGrath 2011-01-05 17:25:33 EST
Description of problem:
"Failed to create pipe for communicating with child process (Too many open files)" message in red when trying to open a new terminal window or tab.

Version-Release number of selected component (if applicable):
gnome-terminal-2.32.0-1.fc14.x86_64
vte-0.26.1-1.fc14.x86_64

How reproducible:
99%

Steps to Reproduce:
1.Have several terminal windows/tabs open in a session.
2.Log out and back in so the session's terminals restart.
3.One terminal's command is an "ssh" with automatic respawn, so early in the session it can fail repeatedly while NetworkManager settles down.
  
Actual results:
Aforementioned error.

Expected results:
No error.

Additional info:

It's my guess that the leak comes from a frequent respawn when the ssh command fails.

Looking at /proc/PID/fd for the gnome-terminal process shows there are indeed 1024 fd's open already, explaining why it hits the rlimit.  Most of them are to /dev/ptmx or /dev/pts/N, up to very high N (485), though only up to /dev/pts/5 actually has any live processes on it.
Comment 1 Tom Hughes 2011-01-28 04:50:32 EST
The problem is that gnome-terminal is not closing pseudo terminals when a terminal window is closed so that eventually it runs out of file descriptors.

I started a new gnome-terminal instance, then ran lsof to look for descriptors open on /dev/pt* and found:

gnome-ter 7853  thh   24u   CHR                5,2      0t0      5656 /dev/ptmx
gnome-ter 7853  thh   25u   CHR              136,0      0t0         3 /dev/pts/0

So just one as expected. Then I opened a second terminal:

gnome-ter 7853  thh   24u   CHR                5,2      0t0      5656 /dev/ptmx
gnome-ter 7853  thh   25u   CHR              136,0      0t0         3 /dev/pts/0
gnome-ter 7853  thh   29w   CHR                5,2      0t0      5656 /dev/ptmx
gnome-ter 7853  thh   32w   CHR              136,2      0t0         5 /dev/pts/2

So a second one has appeared, now if I close that then /dev/pts/2 and it's associated master should go away, but in fact after closing it I see:

gnome-ter 7853  thh   24u   CHR                5,2      0t0      5656 /dev/ptmx
gnome-ter 7853  thh   25u   CHR              136,0      0t0         3 /dev/pts/0
gnome-ter 7853  thh   29w   CHR                5,2      0t0      5656 /dev/ptmx
gnome-ter 7853  thh   32w   CHR              136,2      0t0         5 /dev/pts/2


So it still has the second pseudo terminal open.
Comment 2 Babar 2011-03-24 10:43:56 EDT
I tried to dig into this bug a bit more, because it's annoying my users a lot. After a few days of usage, gnome-terminal runs out of file-descriptors and goes crazy, taking 100% CPU and killing refresh on its screen and some others.

Only solution: Alt+F2 xterm, and kill it.

My guess is that in terminal-screen.c you save the PTY file descriptor in priv->pty_fd for every new connection, but you never close that, even in the finalize.

I haven't tried to patch in a close, nor can I explain the /dev/pmtx fd leak too, but it would be great if this could be fixed, as gnome-terminal is only usable for a few days for "normal" users.
Comment 3 Trevor Cordes 2011-03-29 01:23:16 EDT
"Me too".  I'm not doing anything weird (no often-dying ssh's), I just use a lot of windows and never reboot/logoff.  The limit appears to be 1024, probably the system FD limit.

This will be a major pain, killing and restarting/reorganizing all my hundreds of terminal windows.

Anyhow, this seems reported/solved in Ubuntu bugzilla, so perhaps we can swipe a patch from there or update from upstream:

https://bugs.launchpad.net/ubuntu/+source/vte/+bug/663364
Comment 4 Trevor Cordes 2011-03-29 01:40:18 EDT
Hmm, I started closing gnome-terminals at random, and after closing about 20 I noticed I could start opening new ones again, without the error.

So something can free up the fd's, though I'm not sure of what triggers it.  It doesn't seem to be something simple like closing a high number FD cleans up all deleted lower # FD's down to the next open one.

Anyhow, closing 20 terminals reduced the ll /proc/XXX/fd |wc from 1026 to 850.  Closing another 20 and I'm down to 750.

So there is a workaround, albeit annoying, without killing all gnome-terminals.

Just to be clear, I have maybe 100-200 terminals open max at any given time, never an actual 1024 open, which of course would trigger the same symptom.
Comment 5 Behdad Esfahbod 2011-03-29 09:29:22 EDT
Ok, now I'm better understanding this.  So it's not actually a leak.  Filed it here:

  https://bugzilla.gnome.org/show_bug.cgi?id=646098

Lets follow discussion upstream.

Normally, vte uses two fds.  One to the /dev/pts/X, another to /dev/ptmx.  Not sure if we need two.  Will investigate.

Then the scrollback buffer stuff uses three to store three different streams (text stream, attributes stream, and line-breaks stream).  And unless you are using unlimited scrollback, those streams will allocate another fd each to implement a ring.

So, each widget can use up to 8 fds.  With a 1024 per-process limit, we run out between 120 and 130 tabs.  That's way too fast.

A workaround would be to use unlimited scrollback.   Just make sure you have room under /tmp.
Comment 6 Behdad Esfahbod 2011-03-29 09:34:10 EDT
Looks like my ubuntu machine has the limit set at 32k instead of fedora's 1024.  Shall we do that by default?
Comment 7 Behdad Esfahbod 2011-03-29 09:43:21 EDT
Looks like there are two issues:

  - Leak that is already fixed upstream:

    https://bugzilla.gnome.org/show_bug.cgi?id=632257

  - vte generally using too many fds, fedora having a low default limit on fds.  The vte part is:

    https://bugzilla.gnome.org/show_bug.cgi?id=646098

Actionable items:

  - Fedora should pickup the upstream patch to fix the leak

  - We'll work upstream to increase g-t's fd limit to hard limit, and generally reduce vte's fd usage

  - Fedora should consider lifting the soft limit from 1024 to something larger
Comment 8 Trevor Cordes 2011-04-25 19:37:15 EDT
I agree with all your suggestions, thanks for the good work.

I'm a bit confused as to why this bug never hit me in F12 though.  I haven't change my usage patterns at all since that time.  In fact, I used to have more windows open.  Perhaps one of your findings above didn't hold for earlier versions but now does in F14.

I'll try the unlimited scrollback to try to mitigate for now.

Thanks!
Comment 9 Behdad Esfahbod 2011-04-26 18:26:55 EDT
(In reply to comment #8)
> I agree with all your suggestions, thanks for the good work.
> 
> I'm a bit confused as to why this bug never hit me in F12 though.  I haven't
> change my usage patterns at all since that time.  In fact, I used to have more
> windows open.  Perhaps one of your findings above didn't hold for earlier
> versions but now does in F14.

I'm not sure.  I did rewrite the vte buffer to use files, but that was earlier than f12 I guess.

> I'll try the unlimited scrollback to try to mitigate for now.
> 
> Thanks!
Comment 10 vdm 2011-09-23 06:44:08 EDT
I periodically got this issue to... Never seen it on FC7, Fedora 8... Fedora 11, only in Fedora 14.
However, number of open tabs and termanals in my case is rather small: 2-3 windows with 5-6 tabs in each window. I wonder why it hits the soft limit with such terminal number.
Comment 11 purpleidea 2012-03-14 00:02:28 EDT
FWIW: I have this problem and I've been using "unlimited scrollback" since it became a feature...
I'm using fedora 14.
Comment 12 Fedora End Of Life 2012-08-16 13:18:20 EDT
This message is a notice that Fedora 14 is now at end of life. Fedora 
has stopped maintaining and issuing updates for Fedora 14. It is 
Fedora's policy to close all bug reports from releases that are no 
longer maintained.  At this time, all open bugs with a Fedora 'version'
of '14' have been closed as WONTFIX.

(Please note: Our normal process is to give advanced warning of this 
occurring, but we forgot to do that. A thousand apologies.)

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, feel free to reopen 
this bug and simply change the 'version' to a later Fedora version.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we were unable to fix it before Fedora 14 reached end of life. If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora, you are encouraged to click on 
"Clone This Bug" (top right of this page) and open it against that 
version of Fedora.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping