Description of problem: Hello. I recently experienced an issue with no space left on the device caused by ext4 inode depletion. There were almost 100 000 directories starting with "oops-2013-01-20" created in /var/spool/abrt. I asked Jakub Filak to do the problem analysis directly on my system. He flooded abrtd with abrt-dump-oops executed 1000 times in a 'for' cycle. /var/log/messages show, that abrtd wasn't processing the directories at all during the flooding and until it logged the last oops directory creation. Version-Release number of selected component (if applicable): abrt-2.0.20-1.fc18.armv7hl How reproducible: always Steps to Reproduce: while : ; do cat backtrace | abrt-dump-oops -D; done
NOTE : I'm checking the CPU utilization and it seems that abrtd is eating the whole CPU during the detection of new 'oops' directories.
Created attachment 684372 [details] example backtrace
Created attachment 684373 [details] cat messages | grep abrt | grep "Jan 21 10" > /tmp/messages-grep-abrt.txt
The following two files demonstrate what happens in GUI (xfce) ... it becomes unusable: http://jcapik.fedorapeople.org/files/abrt/Bug_902398/20130124_001.jpg http://jcapik.fedorapeople.org/files/abrt/Bug_902398/20130124_002.mp4
example koopses: http://jcapik.fedorapeople.org/files/abrt/Bug_902398/oops.tgz
(In reply to comment #4) > The following two files demonstrate what happens in GUI (xfce) ... it > becomes unusable: > > http://jcapik.fedorapeople.org/files/abrt/Bug_902398/20130124_001.jpg > > http://jcapik.fedorapeople.org/files/abrt/Bug_902398/20130124_002.mp4 Thank you for taking the time to provide these additional files. You have correctly recognized that this bug may cause the flooding of your desktop by ABRT notifications but it is a different issue and it happens only in particular circumstances (e.g. when abrt can't record kernel's package version data). If you have time and want to help us, please file a new bugzilla ticket.
Well. I thought both issues could have a common solution. Something like interval based throttling could solve the problem. I don't know the internal abrt design well, but I believe that something like limiting the number of allowed abrt-dump-oops executions to 10 per minute could solve the problem. Maybe the throttling needs to be implemented in a different layer in order to block the screen flooding with notification. In that case abrtd would have enough time to process all the oops directories and to find duplicities. You can't expect users to report more than 10 issues produced at the same time, but you can expect, that users want to continue using their workstation and that's impossible, when abrt generates tons of notifications, eats the whole CPU and makes the computer slow as hell. In that case any tries to process as many backtraces as possible doesn't make any sense and you should ignore them because you can't reliably process them because of the lack of resources.
Looks like kernel is flooding the log with wrong, or just over-eager WARNINGs. It may be related to bug 888388. I watched your video and abrt behavior indeed looks bad. I checked abrt source which was current circa January 2013 and it was generating maximum 16 oops reports per second - which is about what we see with your video. Current code generates at most 6 oopses per second, and will pause for one additional second for every oops it sees above six. This should be better than January 2013 behavior, but maybe it's still too aggressive. How about we add an 1 second sleep after each problem dir creation (of those six we create per log scan)? This will effectively reduce reporting rate to 1 report/sec. > You can't expect users to report more than 10 issues produced at the same time, but you can expect, that users want to continue using their workstation and that's impossible, when abrt generates tons of notifications, eats the whole CPU and makes the computer slow as hell. I fully agree. I in fact was saying this all along: making abrt try to report every problem is counter-productive: "report flood" is much worse than missing some problems.
Hi Denys. I'm ok with that. One per second is an acceptable rate.
Patches are ready and are under review in abrt ML.
Fixed in abrt git: commit e45bd71678032333820d47d8f3730b33f2b7690b Author: Denys Vlasenko <dvlasenk> Date: Wed Aug 21 15:20:33 2013 +0200 abrt-dump-oops: add -t option which slows down problem creation. rhbz#902398. Use this option in watch logger services.
gnome-abrt-0.3.1-1.fc18,abrt-2.1.7-1.fc18,libreport-2.1.7-1.fc18,satyr-0.9-1.fc18 has been submitted as an update for Fedora 18. https://admin.fedoraproject.org/updates/gnome-abrt-0.3.1-1.fc18,abrt-2.1.7-1.fc18,libreport-2.1.7-1.fc18,satyr-0.9-1.fc18
Package gnome-abrt-0.3.1-1.fc18, abrt-2.1.7-1.fc18, libreport-2.1.7-1.fc18, satyr-0.9-1.fc18: * should fix your issue, * was pushed to the Fedora 18 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing gnome-abrt-0.3.1-1.fc18 abrt-2.1.7-1.fc18 libreport-2.1.7-1.fc18 satyr-0.9-1.fc18' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2013-16676/gnome-abrt-0.3.1-1.fc18,abrt-2.1.7-1.fc18,libreport-2.1.7-1.fc18,satyr-0.9-1.fc18 then log in and leave karma (feedback).
gnome-abrt-0.3.1-1.fc18, abrt-2.1.7-1.fc18, libreport-2.1.7-1.fc18, satyr-0.9-1.fc18 has been pushed to the Fedora 18 stable repository. If problems still persist, please make note of it in this bug report.