Bug 667539

The problem is that gnome-terminal is not closing pseudo terminals when a terminal window is closed so that eventually it runs out of file descriptors.

I started a new gnome-terminal instance, then ran lsof to look for descriptors open on /dev/pt* and found:

gnome-ter 7853  thh   24u   CHR                5,2      0t0      5656 /dev/ptmx
gnome-ter 7853  thh   25u   CHR              136,0      0t0         3 /dev/pts/0

So just one as expected. Then I opened a second terminal:

gnome-ter 7853  thh   24u   CHR                5,2      0t0      5656 /dev/ptmx
gnome-ter 7853  thh   25u   CHR              136,0      0t0         3 /dev/pts/0
gnome-ter 7853  thh   29w   CHR                5,2      0t0      5656 /dev/ptmx
gnome-ter 7853  thh   32w   CHR              136,2      0t0         5 /dev/pts/2

So a second one has appeared, now if I close that then /dev/pts/2 and it's associated master should go away, but in fact after closing it I see:

gnome-ter 7853  thh   24u   CHR                5,2      0t0      5656 /dev/ptmx
gnome-ter 7853  thh   25u   CHR              136,0      0t0         3 /dev/pts/0
gnome-ter 7853  thh   29w   CHR                5,2      0t0      5656 /dev/ptmx
gnome-ter 7853  thh   32w   CHR              136,2      0t0         5 /dev/pts/2


So it still has the second pseudo terminal open.

I tried to dig into this bug a bit more, because it's annoying my users a lot. After a few days of usage, gnome-terminal runs out of file-descriptors and goes crazy, taking 100% CPU and killing refresh on its screen and some others.

Only solution: Alt+F2 xterm, and kill it.

My guess is that in terminal-screen.c you save the PTY file descriptor in priv->pty_fd for every new connection, but you never close that, even in the finalize.

I haven't tried to patch in a close, nor can I explain the /dev/pmtx fd leak too, but it would be great if this could be fixed, as gnome-terminal is only usable for a few days for "normal" users.

"Me too".  I'm not doing anything weird (no often-dying ssh's), I just use a lot of windows and never reboot/logoff.  The limit appears to be 1024, probably the system FD limit.

This will be a major pain, killing and restarting/reorganizing all my hundreds of terminal windows.

Anyhow, this seems reported/solved in Ubuntu bugzilla, so perhaps we can swipe a patch from there or update from upstream:

https://bugs.launchpad.net/ubuntu/+source/vte/+bug/663364

Hmm, I started closing gnome-terminals at random, and after closing about 20 I noticed I could start opening new ones again, without the error.

So something can free up the fd's, though I'm not sure of what triggers it.  It doesn't seem to be something simple like closing a high number FD cleans up all deleted lower # FD's down to the next open one.

Anyhow, closing 20 terminals reduced the ll /proc/XXX/fd |wc from 1026 to 850.  Closing another 20 and I'm down to 750.

So there is a workaround, albeit annoying, without killing all gnome-terminals.

Just to be clear, I have maybe 100-200 terminals open max at any given time, never an actual 1024 open, which of course would trigger the same symptom.

Ok, now I'm better understanding this.  So it's not actually a leak.  Filed it here:

  https://bugzilla.gnome.org/show_bug.cgi?id=646098

Lets follow discussion upstream.

Normally, vte uses two fds.  One to the /dev/pts/X, another to /dev/ptmx.  Not sure if we need two.  Will investigate.

Then the scrollback buffer stuff uses three to store three different streams (text stream, attributes stream, and line-breaks stream).  And unless you are using unlimited scrollback, those streams will allocate another fd each to implement a ring.

So, each widget can use up to 8 fds.  With a 1024 per-process limit, we run out between 120 and 130 tabs.  That's way too fast.

A workaround would be to use unlimited scrollback.   Just make sure you have room under /tmp.

Looks like my ubuntu machine has the limit set at 32k instead of fedora's 1024.  Shall we do that by default?

Looks like there are two issues:

  - Leak that is already fixed upstream:

    https://bugzilla.gnome.org/show_bug.cgi?id=632257

  - vte generally using too many fds, fedora having a low default limit on fds.  The vte part is:

    https://bugzilla.gnome.org/show_bug.cgi?id=646098

Actionable items:

  - Fedora should pickup the upstream patch to fix the leak

  - We'll work upstream to increase g-t's fd limit to hard limit, and generally reduce vte's fd usage

  - Fedora should consider lifting the soft limit from 1024 to something larger

I agree with all your suggestions, thanks for the good work.

I'm a bit confused as to why this bug never hit me in F12 though.  I haven't change my usage patterns at all since that time.  In fact, I used to have more windows open.  Perhaps one of your findings above didn't hold for earlier versions but now does in F14.

I'll try the unlimited scrollback to try to mitigate for now.

Thanks!

(In reply to comment #8)
> I agree with all your suggestions, thanks for the good work.
> 
> I'm a bit confused as to why this bug never hit me in F12 though.  I haven't
> change my usage patterns at all since that time.  In fact, I used to have more
> windows open.  Perhaps one of your findings above didn't hold for earlier
> versions but now does in F14.

I'm not sure.  I did rewrite the vte buffer to use files, but that was earlier than f12 I guess.

> I'll try the unlimited scrollback to try to mitigate for now.
> 
> Thanks!

I periodically got this issue to... Never seen it on FC7, Fedora 8... Fedora 11, only in Fedora 14.
However, number of open tabs and termanals in my case is rather small: 2-3 windows with 5-6 tabs in each window. I wonder why it hits the soft limit with such terminal number.

FWIW: I have this problem and I've been using "unlimited scrollback" since it became a feature...
I'm using fedora 14.

This message is a notice that Fedora 14 is now at end of life. Fedora 
has stopped maintaining and issuing updates for Fedora 14. It is 
Fedora's policy to close all bug reports from releases that are no 
longer maintained.  At this time, all open bugs with a Fedora 'version'
of '14' have been closed as WONTFIX.

(Please note: Our normal process is to give advanced warning of this 
occurring, but we forgot to do that. A thousand apologies.)

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, feel free to reopen 
this bug and simply change the 'version' to a later Fedora version.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we were unable to fix it before Fedora 14 reached end of life. If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora, you are encouraged to click on 
"Clone This Bug" (top right of this page) and open it against that 
version of Fedora.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

This was actually fixed.

(In reply to Behdad Esfahbod from comment #7)
> Looks like there are two issues:
> 
>   - Leak that is already fixed upstream:
> 
>     https://bugzilla.gnome.org/show_bug.cgi?id=632257

This was fixed in the vte-0.27.1 timeframe.

>   - vte generally using too many fds, fedora having a low default limit on
> fds.  The vte part is:
> 
>     https://bugzilla.gnome.org/show_bug.cgi?id=646098

vte was fixed in the 0.34.9 timeframe, and since gnome-terminal-3.15.0 the soft fd limit was raised to match the higher hard limit.