Bug 529573

Summary: abrt should carefully watch for space used in cache directory
Product: [Fedora] Fedora Reporter: Alexey Torkhov <atorkhov>
Component: abrtAssignee: Jiri Moskovcak <jmoskovc>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: low    
Version: 12CC: dfediuck, dvlasenk, iprikryl, jmoskovc, mnowak, npajkovs, stefanrin
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2009-12-15 10:51:50 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Alexey Torkhov 2009-10-18 19:16:07 UTC
Description of problem:
abrt should carefully watch for space that it used in cache directory. For example, for me it used 13G of 30G and filled root filesystem.
I’m suggesting to regularly run tmpwatch by cron to clean up backtraces that is older a day or so.
Also, perhaps, it should have some kind of internal protection for filling up disk. For example, if less then 5% or so left on filesystem, stop to gathering stacktraces.

Version-Release number of selected component (if applicable):
abrt-0.0.10-1.fc12.x86_64

Comment 1 Jiri Moskovcak 2009-10-19 11:18:07 UTC
Abrt watches for the space filled by backtraces, it's quota can be changed in config file /etc/abrt/abrt.conf. The default value is 1G.

Jirka

Comment 2 Alexey Torkhov 2009-10-19 11:20:43 UTC
But it used 13G in /var/cache/abrt for me - that's a bit bigger than 1G...

Comment 3 Jiri Moskovcak 2009-10-19 11:34:31 UTC
This shouldn't happen, I just tried it and the quota seems to be working fine, are you sure those 13G aren't leftovers from older version of abrt? What happens if something crashes? abrt-applet should warn you about the quota being exceeded.

Jirka

Comment 4 Alexey Torkhov 2009-10-19 13:43:57 UTC
(In reply to comment #3)
> This shouldn't happen, I just tried it and the quota seems to be working fine,
> are you sure those 13G aren't leftovers from older version of abrt?

On this machine it had only following versions installed:
abrt-0.0.8.5
abrt-0.0.9
abrt-0.0.10

> What happens if something crashes? abrt-applet should warn you about the
> quota being exceeded.

Yes, something (that wasn’t abrt) notified me that I have zero space left on root filesystem and I had to purge abrt cache to prevent data corruption or other nasty things. I’ll try to check how it handles quota later, when a bit more crashes will be gathered.

Comment 5 Denys Vlasenko 2009-10-19 16:40:38 UTC
This is a bit strange, since I added the code which repeatedly deletes directories until we are back under quota:

        std::string worst_dir;
        while (g_settings_nMaxCrashReportsSize > 0
         && GetDirSize(DEBUG_DUMPS_DIR, &worst_dir, name) / (1024*1024) >= g_settings_nMaxCrashReportsSize
         && worst_dir != ""
        ) {
            log("Size of '%s' >= %u MB, deleting '%s'", DEBUG_DUMPS_DIR, g_settings_nMaxCrashReportsSize, worst_dir.c_str());
            g_pCommLayer->QuotaExceed(_("Report size exceeded the quota. Please check system's MaxCrashReportsSize value in abrt.conf."));
            DeleteDebugDumpDir(std::string(DEBUG_DUMPS_DIR) + "/" + worst_dir);
            worst_dir = "";
        }

Can you try lowering MaxCrashReportsSize in abrt.conf to, say, 10,
then "killall -ABRT"-ing a few processes, and watch for

Size of '/var/cache/abrt' >= 10 MB, deleting '/var/cache/abrt/DIR'

messages in syslog? If you don't see them, and /var/cache/abrt happily grows over 10 MB, then let me know.

Comment 6 Bug Zapper 2009-11-16 13:48:45 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 12 development cycle.
Changing version to '12'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Comment 7 Denys Vlasenko 2009-12-15 10:51:50 UTC
Closing. Please reopen if the problem persists in abrt >= 1.0.1

Comment 8 Stefan Ring 2010-04-26 14:01:57 UTC
It just happened for me. I tried to start gthumb on Fedora 13 which immediately invoked abrt. Unfortunately, my root partition had only 1.8G free space which was quickly filled, leaving the file system completely full. Removing anything from /var/cache/abrt did not make any more space available. Unfortunately, there was a kernel thread running at 100% CPU (flush-8.0 or sth. like that), and the disk was idle. I tried "sync", but it hust hung indefinitely. Then I tried to emergency-remount RO, (Alt-SysRq-U) which hung as well. Only Alt-Sysrq-B got me out of this misery. Thunderbird (which was running at the time) lost my NNTP subscriptions. The file system is ext4.

I know this is not abrt's fault but still, it was very annoying.

I then symlinked /var/cache/abrt to a larger partition and started gthumb again. Of course, it resulted in an abrt invocation again and created a core dump of 8G (virt size shown by "top" was really that large). When it was done, a large array of desktop notification messages popped up saying that the abrt quota was exceeded, and apparently it deleted all old entries from /var/cache/abrt.

My MaxCrashReportsSize is set at 1000 (default value).


Actual results:

/var/cache/abrt filled despite MaxCrashReportsSize set much smaller.

Expected results:

It should not write a core dump 8G in size.

Additional info:
abrt-1.0.9-2.fc13


I cannot reopen the bug.

Comment 9 Stefan Ring 2010-04-26 14:06:01 UTC
(In reply to comment #8)
> and apparently it deleted all old entries from /var/cache/abrt.

This is actually reported in #578840.

Comment 10 Stefan Ring 2010-04-26 15:03:19 UTC
I also noticed that all _except_ the really large abrt reports where deleted, leaving the directory size well over quota.