Bug 529573 - abrt should carefully watch for space used in cache directory
abrt should carefully watch for space used in cache directory
Status: CLOSED CURRENTRELEASE
Product: Fedora
Classification: Fedora
Component: abrt (Show other bugs)
12
All Linux
low Severity medium
: ---
: ---
Assigned To: Jiri Moskovcak
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2009-10-18 15:16 EDT by Alexey Torkhov
Modified: 2015-02-01 17:49 EST (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-12-15 05:51:50 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Alexey Torkhov 2009-10-18 15:16:07 EDT
Description of problem:
abrt should carefully watch for space that it used in cache directory. For example, for me it used 13G of 30G and filled root filesystem.
I’m suggesting to regularly run tmpwatch by cron to clean up backtraces that is older a day or so.
Also, perhaps, it should have some kind of internal protection for filling up disk. For example, if less then 5% or so left on filesystem, stop to gathering stacktraces.

Version-Release number of selected component (if applicable):
abrt-0.0.10-1.fc12.x86_64
Comment 1 Jiri Moskovcak 2009-10-19 07:18:07 EDT
Abrt watches for the space filled by backtraces, it's quota can be changed in config file /etc/abrt/abrt.conf. The default value is 1G.

Jirka
Comment 2 Alexey Torkhov 2009-10-19 07:20:43 EDT
But it used 13G in /var/cache/abrt for me - that's a bit bigger than 1G...
Comment 3 Jiri Moskovcak 2009-10-19 07:34:31 EDT
This shouldn't happen, I just tried it and the quota seems to be working fine, are you sure those 13G aren't leftovers from older version of abrt? What happens if something crashes? abrt-applet should warn you about the quota being exceeded.

Jirka
Comment 4 Alexey Torkhov 2009-10-19 09:43:57 EDT
(In reply to comment #3)
> This shouldn't happen, I just tried it and the quota seems to be working fine,
> are you sure those 13G aren't leftovers from older version of abrt?

On this machine it had only following versions installed:
abrt-0.0.8.5
abrt-0.0.9
abrt-0.0.10

> What happens if something crashes? abrt-applet should warn you about the
> quota being exceeded.

Yes, something (that wasn’t abrt) notified me that I have zero space left on root filesystem and I had to purge abrt cache to prevent data corruption or other nasty things. I’ll try to check how it handles quota later, when a bit more crashes will be gathered.
Comment 5 Denys Vlasenko 2009-10-19 12:40:38 EDT
This is a bit strange, since I added the code which repeatedly deletes directories until we are back under quota:

        std::string worst_dir;
        while (g_settings_nMaxCrashReportsSize > 0
         && GetDirSize(DEBUG_DUMPS_DIR, &worst_dir, name) / (1024*1024) >= g_settings_nMaxCrashReportsSize
         && worst_dir != ""
        ) {
            log("Size of '%s' >= %u MB, deleting '%s'", DEBUG_DUMPS_DIR, g_settings_nMaxCrashReportsSize, worst_dir.c_str());
            g_pCommLayer->QuotaExceed(_("Report size exceeded the quota. Please check system's MaxCrashReportsSize value in abrt.conf."));
            DeleteDebugDumpDir(std::string(DEBUG_DUMPS_DIR) + "/" + worst_dir);
            worst_dir = "";
        }

Can you try lowering MaxCrashReportsSize in abrt.conf to, say, 10,
then "killall -ABRT"-ing a few processes, and watch for

Size of '/var/cache/abrt' >= 10 MB, deleting '/var/cache/abrt/DIR'

messages in syslog? If you don't see them, and /var/cache/abrt happily grows over 10 MB, then let me know.
Comment 6 Bug Zapper 2009-11-16 08:48:45 EST
This bug appears to have been reported against 'rawhide' during the Fedora 12 development cycle.
Changing version to '12'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 7 Denys Vlasenko 2009-12-15 05:51:50 EST
Closing. Please reopen if the problem persists in abrt >= 1.0.1
Comment 8 Stefan Ring 2010-04-26 10:01:57 EDT
It just happened for me. I tried to start gthumb on Fedora 13 which immediately invoked abrt. Unfortunately, my root partition had only 1.8G free space which was quickly filled, leaving the file system completely full. Removing anything from /var/cache/abrt did not make any more space available. Unfortunately, there was a kernel thread running at 100% CPU (flush-8.0 or sth. like that), and the disk was idle. I tried "sync", but it hust hung indefinitely. Then I tried to emergency-remount RO, (Alt-SysRq-U) which hung as well. Only Alt-Sysrq-B got me out of this misery. Thunderbird (which was running at the time) lost my NNTP subscriptions. The file system is ext4.

I know this is not abrt's fault but still, it was very annoying.

I then symlinked /var/cache/abrt to a larger partition and started gthumb again. Of course, it resulted in an abrt invocation again and created a core dump of 8G (virt size shown by "top" was really that large). When it was done, a large array of desktop notification messages popped up saying that the abrt quota was exceeded, and apparently it deleted all old entries from /var/cache/abrt.

My MaxCrashReportsSize is set at 1000 (default value).


Actual results:

/var/cache/abrt filled despite MaxCrashReportsSize set much smaller.

Expected results:

It should not write a core dump 8G in size.

Additional info:
abrt-1.0.9-2.fc13


I cannot reopen the bug.
Comment 9 Stefan Ring 2010-04-26 10:06:01 EDT
(In reply to comment #8)
> and apparently it deleted all old entries from /var/cache/abrt.

This is actually reported in #578840.
Comment 10 Stefan Ring 2010-04-26 11:03:19 EDT
I also noticed that all _except_ the really large abrt reports where deleted, leaving the directory size well over quota.

Note You need to log in before you can comment on or make changes to this bug.