Bug 495190

Well, assigning to developers, but I don't think it is so nice idea -- we should rather spent our time on fixing the application itself ;-).

(In reply to comment #1)
> Well, assigning to developers, but I don't think it is so nice idea -- we
> should rather spent our time on fixing the application itself ;-).  

That would be nice, but fixing the applications take more time, the bugs are more difficult to retrieve because the errors file needs to be deleted, and neglecting to put in a fail safe means that we leave the problem open for the next application that decides to massively spam the .xsession-errors file.  I mentioned it in passing in the report, but this happened to me six separate times on two computers - and I am quite sure it is more than one application that did this since my two systems run fairly unique environments, different programs, etc.

I would love to be able to put in bug reports for the actual sources of the error spamming, but I have nowhere to move a multi-GiB file - and restarting with the file still on the system results in a system that hangs on boot.  :(

This is not an X server bug.  It is, if anything, an xinit bug.  We should probably do something like 'exec grep . | logger -p user.warning' instead, except that /var/log/messages seems to have become root-only, boo.

syslog does some duplicate detection, but it's easy to defeat, just by logging alternating different messages, and /var/log will fill up just the same.

I don't think it's the responsibility of the  X session to police the logs in this way; it can never do a really good job.b

For what it's worth, gdm used to force ~/.xsession-errors to a particular size, but we changed it to no longer do that:

http://mail.gnome.org/archives/gdm-list/2007-November/msg00017.html

See the above thread for details and the rationale.

Thank you all for your feedback so far.  If needed, I am happy to change the component for the report.

Thanks for the link to the thread.  I read it, and while I think the patch back then was a good idea, it wasn't to allow .xsession-errors to grow inordinately for its own sake.  It looks like the way the capping was handled was breaking some important things, and the most elegant solution was to simply not cap .xsession-errors.  However, I think the rationale for choosing that solution may be outdated.  From the thread:

"- .xsession-errors is ideally empty all the time.  UI programs aren't
supposed to write to stdout/stderr, so when they do it's normally for
exceptional reasons."

"ideally" and "aren't supposed to" does not mean it doesn't happen.  In fact, my .xsession-errors is rarely empty, even when I'm not doing mean and nasty things to my system (e.g., moving several GB files off an NAS device).  Considering the topic is two years old, it's likely that this is more common now than it was before.

"- most files in a typical user's home directory are orders of
magnitude bigger than .xsession-errors"

Normally yes, but not when an application spams an error.  Even if it is "for exceptional reasons", if the application is ignored (i.e., the computer has been left to do a certain task overnight) or the error is otherwise allowed to go on, .xsession-errors becomes a monster.

"- given the above two things, it's very unlikely that that
.xsession-errors will ever hit the user limit"

The logic behind the lifting of the cap essentially assumes that the .xsession-errors file will be "very unlikely" to cause problems because of its size.  Unfortunately, my experience has shown that is no longer the case.

Additionally, it appears that the method for forcing the file to a particular size caused apps to die.  That is a clear reason for a patch, but it does not necessarily mean the .xsession-errors file can or should be left to grow unimpeded until it fills up the HDD.

If a complicated solution breaks things, perhaps a simpler solution would suffice, for example, replacing .xsession-errors with a new, blank file if it gets too large.  That seems like it would be a fairly non-breaking-things solution, but I'm not knowledgeable about OS programming.  All I'd really like is to find a good solution, so I'm not bothered if this is handled by something other than X session.

We could potentially watch the file with gio and truncate it if it gets too large.

Well, for xinit this is kind of a WONTFIX as it's kind of out of scope.  But gdm might want to do better.

Reassigning to gdm.

This problem is really a killer. It makes user sessions crash all the time for
exceeded quota. My experience (Fedora 10 fully up-to-date) is that .xsession-errors grows up of 300 MBytes per day and sometimes more. Most of the messages are obviously debugging stuff from some applications, completely irrelevant. is there a way to switch off logging at all?

(In reply to comment #9)
> This problem is really a killer.

Well, kind of workaround for hopeless situation is cron job nuking ~/.xsession-errors file in regular intervals (or when the file is bigger then something).

This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

Got here because a friend asked about a 500GB file on his machine. I have a simple suggestion, if I understand the mechanics correctly.

As I understand previous comments, the file contains (at least partly) stdout and stderr chatter from gui apps.

Would it be possible to get gdm to redirect/suppress those separately? My file is mostly eaten up with what looks like stdout from tracker. My feeling is that if I could set the equivalent of:
  1>/dev/null 2>~.xsession-errors
that I could cut the file size by far more than half. If I could make the change via an entry into gconf or similar, so much the better.

That would at least make it easier to triage what needs attention in the log. And for users like my friend, it would probably shrink it to the point that an eventual restart happens before it gets too big.

I haven't a clue what the code looks like, so if this isn't feasible, etc, please disregard.

Please add some sort of size limit or rotation method that does not require restarting X (to *all* DM's, if that is where it needs to happen). Fixing apps is not a solution; it is too easy for one rogue app to do Bad Things to a system as things currently stand. (And some of us are not willing to accept "restart X " as a "solution".)

Right now, my .xsession-errors is 0 bytes, thanks to 'truncate' (which was able to recover the disk space, fortunately!) and killing the offending process (knotify spitting out two gstreamer errors at the rate of *tens of MiB per second*... one, ironically, about being out of disk space), but the FD offset (next time something gets written to it) is floating somewhere around 1.3 TiB. (Yes, this broken process filled my *2 TB* HD.)

Really, I think writing directly to a file is broken. This needs to go through some kind of logger process that can perform live rotation when the file hits a certain size threshold. (And should have some form of protection against a rogue process thrashing the disk and wreaking havoc on its lifespan. Ideally by tracking output from every process separately and just dropping logging from anything that exceeds a rate limit.)

Moving to systemd - moving to systemd for the session would allow for a much saner place for solving this.

After about a week of normal Fedora 15 usage, .xsession-errors fills the entire partition, leading to random errors all over the place!

This should have a much higher priority than "low": an unsuspecting user would probably blame the entire distribution, stability-wise, and switch to a "more stable" distro. 

Thanks,

(In reply to comment #22)

Yes, systemd's journal will resolve this nicely.
Until we get there, this works ;) :
ln -sf /dev/null ~/.xsession-errors

(In reply to comment #24)
> (In reply to comment #22)
> 
> Yes, systemd's journal will resolve this nicely.
> Until we get there, this works ;) :
> ln -sf /dev/null ~/.xsession-errors

Where would one place this command to execute at the appropriate time? Unless it is done after the old log is moved but before X is redirected, all you'll do is make it so you can't find the ginormous file without lsof.

I created the symlink just once. After reboot, it's still a symlink. gdm did not overwrite it with a regular file.

(In reply to comment #24)
> (In reply to comment #22)
> 
> Yes, systemd's journal will resolve this nicely.
> Until we get there, this works ;) :
> ln -sf /dev/null ~/.xsession-errors

One of my colleagues ran into this issue as well with Fedora 14 where a 70Gb .xsession-errors along with a 40Gb .xession-errors.old file caused him to exceed his disk quota on an nfs volume. I'll get him to try this solution to see if it stops the problem happening again.

Though systemd may one day become a part of the solution, this bug is not systemd's fault. systemd does not create the file and does not redirect the logs to it.

Reassigning back to gdm.

I would also suggest filing separate bugreports for individual programs that are known to produce excessive amounts of messages.

(In reply to comment #24)
> (In reply to comment #22)
> 
> Yes, systemd's journal will resolve this nicely.
> Until we get there, this works ;) :
> ln -sf /dev/null ~/.xsession-errors

We tried this fix from a terminal login but when logging back into to Gnome, it moved the link to .xession-errors.old & created a new .xession-errors file which is now growing incessantly again.

Just wondering if there are any other suggestions?

Sadly, this is not just an issue on Fedora/Redhat, it probably affects all distros.  It's happening on Mint for me, and there are many reports about it in the Ubuntu forums.  It took me ages to figure out what kept eating up disk space, and then found a 36GB .xession-errors.old log, and a 20GB xession-errors.  This is on an SSD disk too, so it's quite annoying.

I have seen someone claim a 768GB log file from this issue too - the problem is you don't notice until your disk is full, and then knock on effects like losing preferences from network manager and gnome start happening.  

This seems to work for me:

echo > .xsession-errors
sudo chattr -i .xsession-errors

This was a suggestion by someone on the ubuntu forums.  It makes the file immutable, so it can't be changed at all.  It seems ok for me, I haven't tested it through a reboot yet though.

This does need to be fixed at an X level, not an app level.  Expecting all app maintainers to fix their apps is unrealistic, and it is catching many users out, myself included.

Hello,

Like other users, I was a victim of the behavior of this file many time. This bug is live since such a long time it's a shame that no one has done anything to fix it...

Personally, I would like to have this file caped in size since the current behavior seems to create more problems than solution.

Anyway the priority of this bug should be changed to high to have any chance of being processed before 2020. Please.

This problem is I think exacerbated by the new structure of the GNOME desktop.

There are tons of xsession-errors of the form:
** (zeitgeist-datahub:1954): WARNING **: recent-manager-provider.vala:133: Desktop file for "file:<some file name>".

Also, I have arranged to have focus move with my mouse from window to window.
It appears that each time the focus moves, xsession-errors adds about 30-40
lines of the above form.

It seems to me that mouse tracking is a window manager issue, not an app issue.

Just replying to this to get me on the CC list and to keep it alive.

This needs to be fixed. My .xsession-errors on a fully updated F17 is 460GB. After a few dozen lines at the beginning, it's just "Back to top level." for the rest of the file -- 460GB worth! How am I supposed to even figure out what application is doing that?

Over the weekend, my /home part filled completely, and now my email program is freaking out. My guess is that at this point that I've irreparably lost a couple dozen emails, which the email program simply lists as "Unknown".

I'm not just trying to rant here, but stressing that this is important. I would file a bug against the offending program if I knew what that was, but "Back to top level." doesn't give me much of a hint.

Restarting X periodically is a terrible solution for me, since my development machine serves a dual role as a light-duty server, so restarting is not always convenient.

(In reply to comment #33)
> I would file a bug against the offending program if I knew what that was, but
> "Back to top level." doesn't give me much of a hint.

Looks like emacs bug 860312.

(In reply to comment #34)
 
> Looks like emacs bug 860312.

Thanks for that. Good find. I suppose that I should have tried searching bugzilla for "Back to top level."

Can't recall using emacs, but I do a lot on this machine, so anything's possible. I've started monitoring .xsession-errors in a terminal window permanently fixed to one of my workspaces, which is a pity -- I now have to devote part of my attention to making sure that my machine doesn't hose itself. And if it does then what recourse do I have but to log out and back in? As I've said, this is not always convenient for me, and it seems silly anyway.

I just want to encourage whoever this bug is assigned to that this really needs some solution. Today, emacs; what will it be tomorrow? As much as I agree that applications need to be fixed, it seems to me that chasing this problem from that angle is just playing whack-a-mole. The session logging needs to be hardened against this somehow. Just my opinion, of course.

One can make it so ~/.xsession-errors is ignored and all errors dumped to /dev/null for all users of a system by simply doing this:

echo 'exec > /dev/null 2>&1' > /etc/X11/xinit/xinitrc.d/xses_err_to_null.sh
chmod 755 /etc/X11/xinit/xinitrc.d/xses_err_to_null.sh

Users will have to relogin for it to take affect and it doesn't stop xinit from still creating (overwriting) a zero-length ~/.xsession-errors file.

The nice thing about this is it survives updates to the xorg-x11 packages.

This is still a problem on Fedora 26. Has this really not been addressed since 2012? Can the severity be raised? This one problem is causing me a lot of issues as I need my servers to stay up 24/7.

I have done some research and have already changed the .xsession-errors file to be a symbolic link to /dev/null as suggested on a site I found. Didn't help as the space used just jumped up another 82%.

Also, the fact that deleting the file doesn't actually reclaim the space seems like another even more serious bug.

The above solution by Paul Raines also doesn't work for me as I cannot have my users log out. 

Let me know if I can be of any assistance fixing this. With the exception of this one (serious) issue Fedora 26 has been working extremely well for me. 

I am running Fedora 27 on another machine and I believe it also has this issue.

> Also, the fact that deleting the file doesn't actually reclaim the space seems like another even more serious bug.

This is a feature (not bug) of UNIX file systems; deleting (actually, unlinking) an in-use file does not actually erase the file, it just makes it inaccessible via the file system.

You might try truncating the file instead, though I don't know if this will give X fits... (Alternatively, writing zeros over the file in large, round block sizes may allow the FS to reclaim that space using sparse storage, and should be safe since the file's logical size doesn't change.)

> Let me know if I can be of any assistance fixing this.

You could try submitting a patch to X to implement log rotation for the file...

Hi Matthew,
  Sorry, I made some mistakes in my post. First, I meant to say the space used jumped up another 5G, /home is now at 82% used. Second, I am well aware of how UNIX file systems work. I should have said that the person(s) who designed this should have not left the file open, or assumed it would not get too large, etc. Bad choice of words on my part.
 I haven't rebooted yet as this is going to cause a lot of problems (long story). My question is, will a reboot reclaim the space or should I go into rescue mode to actually delete the file? Also, will making the "/dev/null" change to the Xsession file fix this for good? I suppose that would be an acceptable solution and I would add it to my configuration scripts before putting the machine(s) into production.

> will a reboot reclaim the space

If you've unlinked the file, killing any processes with open file handles to it (i.e. X) should reclaim the space. Since a reboot kills ALL processes, yes, that should do it :-), but just restarting the X session should be sufficient. You may also have some success using `lsof` to find which process has large files open.

I haven't tried any of the redirecting tricks, so I can't give a definitive answer there.

The whole concept of the .xsession-errors file needs to be rethought (and for that matter the Xorg.0.log and /var/log/gdm/*.log files).  It is "bad practice" to create un-rotateable log files on such long running processes.  

My latest struggle with these files deals with the autostarting of pulseaudio causing the logs to fill up the root disk on multiuser systems (doing VNC, X2Go, ... logins) as only one pulseaudio process will run at one time.  I had to move /usr/bin/pulseaudio to pulseaudio.bin and make a wrapper that does a 'sleep 60; exit' if DISPLAY is not :0

Xorg should get a new feature for rotating Xorg.0.log to use in logrotate
and redirects like the gdm log and xsession-errors should be rewritten to 
pipe through a logger utility with rotation ability

Something else I might try is 'fdswap' from 
https://www.redpill-linpro.com/sysadvent/2015/12/04/changing-a-process-file-descriptor-with-gdb.html

This problem has apparently been around for several years and so I don't understand why it hasn't been fixed yet. After giving these "solutions" some actual thought I don't see how doign an rm on the file and then creating a symbolic link to /dev/null or wherever had any chance of working without a reboot.  

After I do some testing I will try the approach of editing the Xsession file and redirecting the errors to /dev/null.