Bug 495190 - .xsession-errors file will fill up HDD inordinately
.xsession-errors file will fill up HDD inordinately
Status: ASSIGNED
Product: Fedora
Classification: Fedora
Component: gdm (Show other bugs)
rawhide
All Linux
low Severity low
: ---
: ---
Assigned To: Ray Strode [halfline]
Fedora Extras Quality Assurance
: FutureFeature
Depends On:
Blocks: F12Target 781461 781462
  Show dependency treegraph
 
Reported: 2009-04-10 03:36 EDT by Victoria Earl
Modified: 2015-01-14 18:27 EST (History)
22 users (show)

See Also:
Fixed In Version:
Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of:
: 781461 (view as bug list)
Environment:
Last Closed:
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Victoria Earl 2009-04-10 03:36:51 EDT
Description of problem:
Given the right set of (not too hard to reproduce) circumstances, the .xsession-errors file can and will fill up hundreds of GiB until the HDD itself is full.

Version-Release number of selected component (if applicable):
Does not seem to be applicable.

How reproducible:
Moderately difficult to reproduce.  Essentially, you need an application that reliably generates one or more of the same error over and over into the .xsession-errors file until the user executes an action.  The faster it generates errors, the better.  I've had this happen about six times inadvertently, so it should not be too difficult.

Steps to Reproduce:
1. Get an application to spam errors into the .xsession-errors file (located in /home/username/)
2. Wait.
  
Actual results:
.xsession-errors will continue to expand until it literally fills the HDD.  This has happened up to 500GiB for me.  Often the original errors that were spammed into the file are not reported because I need to delete the file and restart X to get my system to do much of anything.

Expected results:
.xsession-errors, ideally, would be capped in some manner depending on the free space left in the drive.  Old messages would be cycled out once the limit is reached.

Additional info:
This has happened on both my i686 laptop and x86_64 PC.  I don't know what is with my penchant for applications spamming errors into the .xsession-errors file.
Comment 1 Matěj Cepl 2009-04-10 11:44:32 EDT
Well, assigning to developers, but I don't think it is so nice idea -- we should rather spent our time on fixing the application itself ;-).
Comment 2 Victoria Earl 2009-04-10 13:27:20 EDT
(In reply to comment #1)
> Well, assigning to developers, but I don't think it is so nice idea -- we
> should rather spent our time on fixing the application itself ;-).  

That would be nice, but fixing the applications take more time, the bugs are more difficult to retrieve because the errors file needs to be deleted, and neglecting to put in a fail safe means that we leave the problem open for the next application that decides to massively spam the .xsession-errors file.  I mentioned it in passing in the report, but this happened to me six separate times on two computers - and I am quite sure it is more than one application that did this since my two systems run fairly unique environments, different programs, etc.

I would love to be able to put in bug reports for the actual sources of the error spamming, but I have nowhere to move a multi-GiB file - and restarting with the file still on the system results in a system that hangs on boot.  :(
Comment 3 Adam Jackson 2009-04-13 18:14:24 EDT
This is not an X server bug.  It is, if anything, an xinit bug.  We should probably do something like 'exec grep . | logger -p user.warning' instead, except that /var/log/messages seems to have become root-only, boo.
Comment 4 Bill Nottingham 2009-04-14 11:34:58 EDT
syslog does some duplicate detection, but it's easy to defeat, just by logging alternating different messages, and /var/log will fill up just the same.

I don't think it's the responsibility of the  X session to police the logs in this way; it can never do a really good job.b
Comment 5 Ray Strode [halfline] 2009-04-14 11:37:00 EDT
For what it's worth, gdm used to force ~/.xsession-errors to a particular size, but we changed it to no longer do that:

http://mail.gnome.org/archives/gdm-list/2007-November/msg00017.html

See the above thread for details and the rationale.
Comment 6 Victoria Earl 2009-04-14 12:21:17 EDT
Thank you all for your feedback so far.  If needed, I am happy to change the component for the report.

Thanks for the link to the thread.  I read it, and while I think the patch back then was a good idea, it wasn't to allow .xsession-errors to grow inordinately for its own sake.  It looks like the way the capping was handled was breaking some important things, and the most elegant solution was to simply not cap .xsession-errors.  However, I think the rationale for choosing that solution may be outdated.  From the thread:

"- .xsession-errors is ideally empty all the time.  UI programs aren't
supposed to write to stdout/stderr, so when they do it's normally for
exceptional reasons."

"ideally" and "aren't supposed to" does not mean it doesn't happen.  In fact, my .xsession-errors is rarely empty, even when I'm not doing mean and nasty things to my system (e.g., moving several GB files off an NAS device).  Considering the topic is two years old, it's likely that this is more common now than it was before.

"- most files in a typical user's home directory are orders of
magnitude bigger than .xsession-errors"

Normally yes, but not when an application spams an error.  Even if it is "for exceptional reasons", if the application is ignored (i.e., the computer has been left to do a certain task overnight) or the error is otherwise allowed to go on, .xsession-errors becomes a monster.

"- given the above two things, it's very unlikely that that
.xsession-errors will ever hit the user limit"

The logic behind the lifting of the cap essentially assumes that the .xsession-errors file will be "very unlikely" to cause problems because of its size.  Unfortunately, my experience has shown that is no longer the case.

Additionally, it appears that the method for forcing the file to a particular size caused apps to die.  That is a clear reason for a patch, but it does not necessarily mean the .xsession-errors file can or should be left to grow unimpeded until it fills up the HDD.

If a complicated solution breaks things, perhaps a simpler solution would suffice, for example, replacing .xsession-errors with a new, blank file if it gets too large.  That seems like it would be a fairly non-breaking-things solution, but I'm not knowledgeable about OS programming.  All I'd really like is to find a good solution, so I'm not bothered if this is handled by something other than X session.
Comment 7 Ray Strode [halfline] 2009-04-14 15:17:56 EDT
We could potentially watch the file with gio and truncate it if it gets too large.
Comment 8 Adam Jackson 2009-07-06 10:46:02 EDT
Well, for xinit this is kind of a WONTFIX as it's kind of out of scope.  But gdm might want to do better.

Reassigning to gdm.
Comment 9 Alfredo Ferrari 2009-07-13 05:46:14 EDT
This problem is really a killer. It makes user sessions crash all the time for
exceeded quota. My experience (Fedora 10 fully up-to-date) is that .xsession-errors grows up of 300 MBytes per day and sometimes more. Most of the messages are obviously debugging stuff from some applications, completely irrelevant. is there a way to switch off logging at all?
Comment 10 Matěj Cepl 2011-01-10 08:07:50 EST
(In reply to comment #9)
> This problem is really a killer.

Well, kind of workaround for hopeless situation is cron job nuking ~/.xsession-errors file in regular intervals (or when the file is bigger then something).
Comment 11 Fedora Admin XMLRPC Client 2011-06-21 11:29:42 EDT
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.
Comment 12 Fedora Admin XMLRPC Client 2011-06-21 11:31:40 EDT
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.
Comment 13 Fedora Admin XMLRPC Client 2011-06-21 11:34:28 EDT
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.
Comment 14 Fedora Admin XMLRPC Client 2011-06-21 11:37:23 EDT
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.
Comment 15 Fedora Admin XMLRPC Client 2011-06-21 11:43:22 EDT
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.
Comment 16 Fedora Admin XMLRPC Client 2011-06-21 11:47:23 EDT
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.
Comment 17 Fedora Admin XMLRPC Client 2011-06-21 11:49:39 EDT
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.
Comment 18 Fedora Admin XMLRPC Client 2011-06-21 11:51:56 EDT
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.
Comment 19 Fedora Admin XMLRPC Client 2011-06-21 11:53:07 EDT
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.
Comment 20 Neil Bryant 2011-10-28 15:53:44 EDT
Got here because a friend asked about a 500GB file on his machine. I have a simple suggestion, if I understand the mechanics correctly.

As I understand previous comments, the file contains (at least partly) stdout and stderr chatter from gui apps.

Would it be possible to get gdm to redirect/suppress those separately? My file is mostly eaten up with what looks like stdout from tracker. My feeling is that if I could set the equivalent of:
  1>/dev/null 2>~.xsession-errors
that I could cut the file size by far more than half. If I could make the change via an entry into gconf or similar, so much the better.

That would at least make it easier to triage what needs attention in the log. And for users like my friend, it would probably shrink it to the point that an eventual restart happens before it gets too big.

I haven't a clue what the code looks like, so if this isn't feasible, etc, please disregard.
Comment 21 Matthew Woehlke 2011-11-21 00:27:38 EST
Please add some sort of size limit or rotation method that does not require restarting X (to *all* DM's, if that is where it needs to happen). Fixing apps is not a solution; it is too easy for one rogue app to do Bad Things to a system as things currently stand. (And some of us are not willing to accept "restart X " as a "solution".)

Right now, my .xsession-errors is 0 bytes, thanks to 'truncate' (which was able to recover the disk space, fortunately!) and killing the offending process (knotify spitting out two gstreamer errors at the rate of *tens of MiB per second*... one, ironically, about being out of disk space), but the FD offset (next time something gets written to it) is floating somewhere around 1.3 TiB. (Yes, this broken process filled my *2 TB* HD.)

Really, I think writing directly to a file is broken. This needs to go through some kind of logger process that can perform live rotation when the file hits a certain size threshold. (And should have some form of protection against a rogue process thrashing the disk and wreaking havoc on its lifespan. Ideally by tracking output from every process separately and just dropping logging from anything that exceeds a rate limit.)
Comment 22 Bill Nottingham 2011-11-21 15:39:54 EST
Moving to systemd - moving to systemd for the session would allow for a much saner place for solving this.
Comment 23 Ahmed S. Darwish 2012-01-06 06:54:56 EST
After about a week of normal Fedora 15 usage, .xsession-errors fills the entire partition, leading to random errors all over the place!

This should have a much higher priority than "low": an unsuspecting user would probably blame the entire distribution, stability-wise, and switch to a "more stable" distro. 

Thanks,
Comment 24 Michal Schmidt 2012-01-06 08:03:07 EST
(In reply to comment #22)

Yes, systemd's journal will resolve this nicely.
Until we get there, this works ;) :
ln -sf /dev/null ~/.xsession-errors
Comment 25 Matthew Woehlke 2012-01-06 14:33:34 EST
(In reply to comment #24)
> (In reply to comment #22)
> 
> Yes, systemd's journal will resolve this nicely.
> Until we get there, this works ;) :
> ln -sf /dev/null ~/.xsession-errors

Where would one place this command to execute at the appropriate time? Unless it is done after the old log is moved but before X is redirected, all you'll do is make it so you can't find the ginormous file without lsof.
Comment 26 Michal Schmidt 2012-01-09 08:46:16 EST
I created the symlink just once. After reboot, it's still a symlink. gdm did not overwrite it with a regular file.
Comment 27 John Kissane 2012-01-10 04:23:23 EST
(In reply to comment #24)
> (In reply to comment #22)
> 
> Yes, systemd's journal will resolve this nicely.
> Until we get there, this works ;) :
> ln -sf /dev/null ~/.xsession-errors

One of my colleagues ran into this issue as well with Fedora 14 where a 70Gb .xsession-errors along with a 40Gb .xession-errors.old file caused him to exceed his disk quota on an nfs volume. I'll get him to try this solution to see if it stops the problem happening again.
Comment 28 Michal Schmidt 2012-01-13 09:59:40 EST
Though systemd may one day become a part of the solution, this bug is not systemd's fault. systemd does not create the file and does not redirect the logs to it.

Reassigning back to gdm.

I would also suggest filing separate bugreports for individual programs that are known to produce excessive amounts of messages.
Comment 29 John Kissane 2012-01-17 06:43:51 EST
(In reply to comment #24)
> (In reply to comment #22)
> 
> Yes, systemd's journal will resolve this nicely.
> Until we get there, this works ;) :
> ln -sf /dev/null ~/.xsession-errors

We tried this fix from a terminal login but when logging back into to Gnome, it moved the link to .xession-errors.old & created a new .xession-errors file which is now growing incessantly again.

Just wondering if there are any other suggestions?
Comment 30 Chris Gilbert 2012-02-01 04:42:26 EST
Sadly, this is not just an issue on Fedora/Redhat, it probably affects all distros.  It's happening on Mint for me, and there are many reports about it in the Ubuntu forums.  It took me ages to figure out what kept eating up disk space, and then found a 36GB .xession-errors.old log, and a 20GB xession-errors.  This is on an SSD disk too, so it's quite annoying.

I have seen someone claim a 768GB log file from this issue too - the problem is you don't notice until your disk is full, and then knock on effects like losing preferences from network manager and gnome start happening.  

This seems to work for me:

echo > .xsession-errors
sudo chattr -i .xsession-errors

This was a suggestion by someone on the ubuntu forums.  It makes the file immutable, so it can't be changed at all.  It seems ok for me, I haven't tested it through a reboot yet though.

This does need to be fixed at an X level, not an app level.  Expecting all app maintainers to fix their apps is unrealistic, and it is catching many users out, myself included.
Comment 31 Jérémie Grauer 2012-05-10 04:09:43 EDT
Hello,

Like other users, I was a victim of the behavior of this file many time. This bug is live since such a long time it's a shame that no one has done anything to fix it...

Personally, I would like to have this file caped in size since the current behavior seems to create more problems than solution.

Anyway the priority of this bug should be changed to high to have any chance of being processed before 2020. Please.
Comment 32 Burnie West 2012-10-31 14:35:39 EDT
This problem is I think exacerbated by the new structure of the GNOME desktop.

There are tons of xsession-errors of the form:
** (zeitgeist-datahub:1954): WARNING **: recent-manager-provider.vala:133: Desktop file for "file:<some file name>".

Also, I have arranged to have focus move with my mouse from window to window.
It appears that each time the focus moves, xsession-errors adds about 30-40
lines of the above form.

It seems to me that mouse tracking is a window manager issue, not an app issue.
Comment 33 Alan M. Evans 2012-12-03 12:28:17 EST
Just replying to this to get me on the CC list and to keep it alive.

This needs to be fixed. My .xsession-errors on a fully updated F17 is 460GB. After a few dozen lines at the beginning, it's just "Back to top level." for the rest of the file -- 460GB worth! How am I supposed to even figure out what application is doing that?

Over the weekend, my /home part filled completely, and now my email program is freaking out. My guess is that at this point that I've irreparably lost a couple dozen emails, which the email program simply lists as "Unknown".

I'm not just trying to rant here, but stressing that this is important. I would file a bug against the offending program if I knew what that was, but "Back to top level." doesn't give me much of a hint.

Restarting X periodically is a terrible solution for me, since my development machine serves a dual role as a light-duty server, so restarting is not always convenient.
Comment 34 Michal Schmidt 2012-12-03 12:31:08 EST
(In reply to comment #33)
> I would file a bug against the offending program if I knew what that was, but
> "Back to top level." doesn't give me much of a hint.

Looks like emacs bug 860312.
Comment 35 Alan M. Evans 2012-12-03 13:20:56 EST
(In reply to comment #34)
 
> Looks like emacs bug 860312.

Thanks for that. Good find. I suppose that I should have tried searching bugzilla for "Back to top level."

Can't recall using emacs, but I do a lot on this machine, so anything's possible. I've started monitoring .xsession-errors in a terminal window permanently fixed to one of my workspaces, which is a pity -- I now have to devote part of my attention to making sure that my machine doesn't hose itself. And if it does then what recourse do I have but to log out and back in? As I've said, this is not always convenient for me, and it seems silly anyway.

I just want to encourage whoever this bug is assigned to that this really needs some solution. Today, emacs; what will it be tomorrow? As much as I agree that applications need to be fixed, it seems to me that chasing this problem from that angle is just playing whack-a-mole. The session logging needs to be hardened against this somehow. Just my opinion, of course.
Comment 36 Paul Raines 2012-12-06 15:22:27 EST
One can make it so ~/.xsession-errors is ignored and all errors dumped to /dev/null for all users of a system by simply doing this:

echo 'exec > /dev/null 2>&1' > /etc/X11/xinit/xinitrc.d/xses_err_to_null.sh
chmod 755 /etc/X11/xinit/xinitrc.d/xses_err_to_null.sh

Users will have to relogin for it to take affect and it doesn't stop xinit from still creating (overwriting) a zero-length ~/.xsession-errors file.

The nice thing about this is it survives updates to the xorg-x11 packages.

Note You need to log in before you can comment on or make changes to this bug.