Bug 564451 - ABRT makes my computer unusable
Summary: ABRT makes my computer unusable
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: abrt
Version: 12
Hardware: All
OS: Linux
low
medium
Target Milestone: ---
Assignee: Denys Vlasenko
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2010-02-12 19:34 UTC by Bastien Nocera
Modified: 2010-06-28 17:09 UTC (History)
8 users (show)

Fixed In Version: abrt-1.1.1-1.fc12
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2010-06-02 18:05:28 UTC


Attachments (Terms of Use)
abrt-debuginfo-install with disk space checks (14.39 KB, application/octet-stream)
2010-05-07 16:57 UTC, Denys Vlasenko
no flags Details

Description Bastien Nocera 2010-02-12 19:34:45 UTC
abrt-1.0.3-1.fc12.x86_64
kernel-2.6.31.9-174.fc12.x86_64

To be fair, not really abrt's problem, but because it enables core dumping, the kernel will make my machine unusable, using most of the CPU and disk resources.

It might even eat RAM as it's swapping out thing.

This bug should probably be made a dependency of a bug against the kernel using too much resources (or not allowing resource control).

Comment 1 Denys Vlasenko 2010-04-02 16:40:36 UTC
Can you be more specific?

How many directories do you have in /var/cache/abrt (every one of those is a dump made by abrt)? Can you attach your /var/log/messages? Etc...

Comment 2 Marcin Rzeźnicki 2010-04-26 19:16:43 UTC
Good day to all
Let me provide additional comment on this bug, because I strongly feel that abrt makes my computer not usable too.
Description:
KDE widget which I've been using for tagging my files in Nepomuk crashed, bringing Nepomuk down with it. I thought I might easily report this bug, using abrt, to respective maintainers of appropriate packages so that they could investigate the issue and maybe fix it. It's easy with abrt, right? WRONG!!! Abrt is totally useless for the sole purpose of it humble existence and, thus, probably envious of other software which is, at least somewhat, useful. 
As I am running my Linux on a VM, I provide it with the accurate amount of disk space I need at the moment. When abrt started "reporting" my hard drive had approximately 1 GB of free space. TO REPORT THAT BUG IN A WIDGET ABRT STARTED DOWNLOADING ABOUT 400-500 MB of debuginfo packages. I've downloaded qt-debug, kdelibs-debug, kdeworkspace-debug, glibc-debug, krb5-debug(?), libxml2-debug, acl-debug. It totally clogged my HD. My mistake is that I did not pay attention to what this miserable code is doing and tried to work while leaving it to do its work in the background. Of course, working was not entirely possible because of the heavy spiky load on the whole machine. Nevertheless, last time I checked it almost finished - what was left was unpacking qt-debuginfo. In this precise moment of time my HD had around 150 MB of free disk space, which turned to 0 right before my eyes. ABRT did not stop though. Why should it? It was still trying to unpack qt with ox-like stubbornness. Few moments later KDE desktop crashed. What was left was this awful ABRT window mocking me saying: "Ooops, I could not unpack qt-debuginfo" The very last thing I saw before I restarted my machine was the message: "Ok, It did not work. But you know what? there are 33 debuginfos more to download. I am doing it right away".
After rebooting I deleted all reports - disk free: 1.8 GB (800 MB more than before deleting, mind you).
So, summing this nice story up: to report any bug with abrt you need to download half of the internet to your HD; you cannot control how much of your disk space it occupies with its bits (at least I do not know how); it does not pay respect to the environment it works in - it does not monitor how much disk space is left, or, if it does, it does it wrong - this software is then considered dangerous. I am really worried about the state of enterprise computing if this software works on RHEL.

Comment 3 Denys Vlasenko 2010-05-05 15:55:41 UTC
> you cannot control how much of your disk space it occupies with its bits (at least I do not know how)

/etc/abrt/plugins/CCpp.conf:

# Keep @@LOCALSTATEDIR@@/cache/abrt-di
# from growing out-of-bounds.
DebugInfoCacheMB = 4000

> When abrt started "reporting" my hard drive had approximately 1 GB of free space. TO REPORT THAT BUG IN A WIDGET ABRT STARTED DOWNLOADING ABOUT 400-500 MB of debuginfo packages. I've downloaded qt-debug, kdelibs-debug, kdeworkspace-debug, glibc-debug, krb5-debug(?), libxml2-debug, acl-debug. It totally clogged my HD.

Why that widget links against krb5 libs, I do not know. Why you whine about that fact in *this* bug, I do not know either.

If you have an idea how to reliably produce a usable backtrace without downloading debuginfos for all loaded libs, I am all ears.

You got so carried away with whining and ranting that you forgot to provide facts. For example - what abrt version was that? Do you actually want something to be improved or you want to rant?

Comment 4 Marcin Rzeźnicki 2010-05-05 16:14:17 UTC
(In reply to comment #3)
> > you cannot control how much of your disk space it occupies with its bits (at least I do not know how)
> 
> /etc/abrt/plugins/CCpp.conf:
> 
> # Keep @@LOCALSTATEDIR@@/cache/abrt-di
> # from growing out-of-bounds.
> DebugInfoCacheMB = 4000
> 

Thanks, why it didn't find its way to the GUI?

> > When abrt started "reporting" my hard drive had approximately 1 GB of free space. TO REPORT THAT BUG IN A WIDGET ABRT STARTED DOWNLOADING ABOUT 400-500 MB of debuginfo packages. I've downloaded qt-debug, kdelibs-debug, kdeworkspace-debug, glibc-debug, krb5-debug(?), libxml2-debug, acl-debug. It totally clogged my HD.
> 
> Why that widget links against krb5 libs, I do not know. Why you whine about
> that fact in *this* bug, I do not know either.
> 

Because that is outrageous to download all this stuff all the time. If anything in KDE breaks you end up with huge qt-debuginfo etc. This procedure you will have to repeat every month, or so, because of the library update. 

> If you have an idea how to reliably produce a usable backtrace without
> downloading debuginfos for all loaded libs, I am all ears.
> 

What about sending the report to the server which would produce backtrace?

> You got so carried away with whining and ranting that you forgot to provide
> facts. For example - what abrt version was that?

1.0.9

> Do you actually want something
> to be improved or you want to rant?    

Isn't that obvious what is to be improved?
1. Correct cancellation of reporting when there is not enough HD free space
2. Calculating the amount of space needed for downloading everything needed UPFRONT, and presenting that info to the user
3. Better handling of progress reporting so that user does not go insane over staring at rapidly sliding bar (there are more legitimate purposes than this, you get the idea)
4. rethinking whole idea of abrt workings. Maybe it would be better to SEND the report and produce backtrack somewhere else? What about giving the choice? What about, instead of downloading everything that particular component has been linked against, determining what is in the stack trace and downloading only relevant stuff? that would eliminate tons of excessive traffic for sure

I am looking forward to hearing of your ideas

Comment 5 Denys Vlasenko 2010-05-07 16:57:53 UTC
Created attachment 412401 [details]
abrt-debuginfo-install with disk space checks

Basically, added "abort_if_low_on_disk_space MBYTES" function and call it at the beginning and after each rpm download / unpack operation:

@@ -245,8 +266,10 @@ download_packages() {
            echo "Processing: $file" >>unpack.OUT
            rpm2cpio <"$file" >"unpacked.cpio" 2>>unpack.OUT || error_msg_and_die "Can't convert '$file' to cpio"
            $keep_tmp || rm "$file"
+           abort_if_low_on_disk_space 256
            cpio -id <"unpacked.cpio" >>unpack.OUT 2>&1 || error_msg_and_die "Can't unpack '$file' cpio archive"
            rm "unpacked.cpio"
+           abort_if_low_on_disk_space 256
            # Copy debuginfo files to cachedir
            if test x"$cachedir" != x"" && test -d "$cachedir"; then
                # For every needed debuginfo, check whether we have it
@@ -291,16 +314,14 @@ test -e "$tempdir" && error_msg_and_die.
 # Intentionally not using -p: we want to abort if tempdir exists
 mkdir -- "$tempdir" || exit 2
 cd "$tempdir" || exit 2
+
+
+abort_if_low_on_disk_space 1024

Comment 6 Marcin Rzeźnicki 2010-05-07 17:17:39 UTC
(In reply to comment #5)
> Created an attachment (id=412401) [details]
> abrt-debuginfo-install with disk space checks
> 

Thx for patch.
What about other, I think worthy, improvements? Progress report, download size estimate - is there something that can be done in this regard? 
Btw - do you think that my idea of generating reports not on "client" machine is feasible?

Comment 7 Denys Vlasenko 2010-05-07 17:21:10 UTC
(In reply to comment #4)
> > > When abrt started "reporting" my hard drive had approximately 1 GB of free space. TO REPORT THAT BUG IN A WIDGET ABRT STARTED DOWNLOADING ABOUT 400-500 MB of debuginfo packages. I've downloaded qt-debug, kdelibs-debug, kdeworkspace-debug, glibc-debug, krb5-debug(?), libxml2-debug, acl-debug. It totally clogged my HD.
> > 
> > Why that widget links against krb5 libs, I do not know. Why you whine about
> > that fact in *this* bug, I do not know either.
> 
> Because that is outrageous to download all this stuff all the time. If anything
> in KDE breaks you end up with huge qt-debuginfo etc. This procedure you will
> have to repeat every month, or so, because of the library update.
> 
> > If you have an idea how to reliably produce a usable backtrace without
> > downloading debuginfos for all loaded libs, I am all ears.
> 
> What about sending the report to the server which would produce backtrace?

This is a good idea. But for this, this server should exist somewhere. It means, someone needs to provide $$$ for server and for people who will maintain it. Thus far, such Fedora infrastructure-related questions were not discussed with people who may allocate funds.

However, there always be people who do not want to send out their coredumps for privacy reasons/beliefs, or because their company policy disallows it. Those users will want to generate backtraces locally. Therefore this mode of operation should still exist (and be debugged).

> Isn't that obvious what is to be improved?
> 1. Correct cancellation of reporting when there is not enough HD free space

Yes, good idea. Please drop attached abrt-debuginfo-install into /usr/bin, it should abort if you have less than 1 Gb of free space or if during debuginfo install it sees free space dropping below 256 Mb.

> 2. Calculating the amount of space needed for downloading everything needed
> UPFRONT, and presenting that info to the user

"yum provides" reports the sizes of packages. Can use "yum info"...

> 3. Better handling of progress reporting so that user does not go insane over
> staring at rapidly sliding bar (there are more legitimate purposes than this,
> you get the idea)

I was advocating removing the bar and showing the log instead... other did not agree.

> 4. rethinking whole idea of abrt workings. Maybe it would be better to SEND the
> report and produce backtrack somewhere else? What about giving the choice? What
> about, instead of downloading everything that particular component has been
> linked against, determining what is in the stack trace and downloading only
> relevant stuff? that would eliminate tons of excessive traffic for sure

Here is how backtrace of "sleep 999" killed with SIGABRT looks like before debuginfos are installed:

gdb -batch -ex 'file /bin/sleep' -ex 'core-file coredump' -ex 'thread apply all backtrace full' -ex 'info sharedlib'
Core was generated by `sleep 1260'.
Program terminated with signal 6, Aborted.
#0  0x00007fbf42779380 in __nanosleep_nocancel () from /lib64/libc.so.6

Thread 1 (Thread 30259):
#0  0x00007fbf42779380 in __nanosleep_nocancel () from /lib64/libc.so.6
No symbol table info available.
#1  0x00000000004037ab in ?? ()
No symbol table info available.
#2  0x000000000040321b in ?? ()
No symbol table info available.
#3  0x000000000040176c in ?? ()
No symbol table info available.
#4  0x00007fbf426f0b4d in __libc_start_main () from /lib64/libc.so.6
No symbol table info available.
#5  0x00000000004012f9 in iswprint ()
No symbol table info available.
#6  0x00007fff7f2ae968 in ?? ()
No symbol table info available.
#7  0x000000000000001c in ?? ()
No symbol table info available.
#8  0x0000000000000002 in ?? ()
No symbol table info available.
#9  0x00007fff7f2affe2 in ?? ()
No symbol table info available.
#10 0x00007fff7f2affe8 in ?? ()
No symbol table info available.
#11 0x0000000000000000 in ?? ()
No symbol table info available.
From                To                  Syms Read   Shared Object Library
0x00007fbf42a54190  0x00007fbf42a576f8  Yes (*)     /lib64/librt.so.1
0x00007fbf426f07c0  0x00007fbf427faf5c  Yes (*)     /lib64/libc.so.6
0x00007fbf424ba3e0  0x00007fbf424c5da8  Yes (*)     /lib64/libpthread.so.0
0x00007fbf42c5aaf0  0x00007fbf42c72f64  Yes (*)     /lib64/ld-linux-x86-64.so.2
(*): Shared library is missing debugging information.

How from this data can you determine which debuginfo is needed to convert, for example, 0x000000000040321b address to function name, "xnanosleep"?

Comment 8 Marcin Rzeźnicki 2010-05-07 18:05:29 UTC
(In reply to comment #7)

> > What about sending the report to the server which would produce backtrace?
> 
> This is a good idea. But for this, this server should exist somewhere. It
> means, someone needs to provide $$$ for server and for people who will maintain
> it. Thus far, such Fedora infrastructure-related questions were not discussed
> with people who may allocate funds.
> 

Maybe community could help. I, for example, own low-end CentOS server. If someone could make estimates of expected traffic, bandwidth needed then probably  it could be taken care of.


> However, there always be people who do not want to send out their coredumps for
> privacy reasons/beliefs, or because their company policy disallows it. Those
> users will want to generate backtraces locally. Therefore this mode of
> operation should still exist (and be debugged).
> 

Yes, sure, therefore I mentioned it as an option.

> > Isn't that obvious what is to be improved?
> > 1. Correct cancellation of reporting when there is not enough HD free space
> 
> Yes, good idea. Please drop attached abrt-debuginfo-install into /usr/bin, it
> should abort if you have less than 1 Gb of free space or if during debuginfo
> install it sees free space dropping below 256 Mb.
> 
> > 2. Calculating the amount of space needed for downloading everything needed
> > UPFRONT, and presenting that info to the user
> 
> "yum provides" reports the sizes of packages. Can use "yum info"...
> 

Great, thx again.

> > 3. Better handling of progress reporting so that user does not go insane over
> > staring at rapidly sliding bar (there are more legitimate purposes than this,
> > you get the idea)
> 
> I was advocating removing the bar and showing the log instead... other did not
> agree.
> 

Yes, I understand. anyway, if someone asked me, bar is good as long as it conveys useful information. Having bar just running wild in front of your eyes is neither useful nor pretty 

> > 4. rethinking whole idea of abrt workings. Maybe it would be better to SEND the
> > report and produce backtrack somewhere else? What about giving the choice? What
> > about, instead of downloading everything that particular component has been
> > linked against, determining what is in the stack trace and downloading only
> > relevant stuff? that would eliminate tons of excessive traffic for sure
> 
> Here is how backtrace of "sleep 999" killed with SIGABRT looks like before
> debuginfos are installed:

I am not expert on  x86 stacks - I was thinking that it could be determined. Looking at this example I believe that: if an adress is found in shared libraries table range then name of lib is known and probably some mapping exists between names of libs and package name (package manager knows what files are inside the package so that is, rudimentary but still, one way), otherwise I assume this must be function internal to executable, so it should be found in the faulty package debuginfo . If I am wrong then please correct me.

Comment 9 Bastien Nocera 2010-05-11 14:38:41 UTC
Seriously, could we get back to my original bug report?

The problem isn't that disk space is taken, it's that the core dumping exercise takes all the resources away from the machine.

An example would be a quite loaded machine, where a gig of Firefox memory would suddenly crash, and the I/O from the saving to disk would make the desktop unusable, due to the lack of interactivity.

Comment 10 Denys Vlasenko 2010-05-12 12:31:36 UTC
> The problem isn't that disk space is taken, it's that the core dumping exercise
takes all the resources away from the machine.

I committed a fix for bug 588945 "sparse core files performance hit". Now core files will be sparse. Usually they take ~2-3 times less disk space.

This fix will be in the next update (I guess it will be abrt-1.1.1).

Comment 11 Fedora Update System 2010-05-28 10:56:50 UTC
abrt-1.1.1-1.fc13 has been submitted as an update for Fedora 13.
http://admin.fedoraproject.org/updates/abrt-1.1.1-1.fc13

Comment 12 Fedora Update System 2010-05-28 12:45:39 UTC
abrt-1.1.1-1.fc12 has been submitted as an update for Fedora 12.
http://admin.fedoraproject.org/updates/abrt-1.1.1-1.fc12

Comment 13 Fedora Update System 2010-05-28 18:05:09 UTC
abrt-1.1.1-1.fc13 has been pushed to the Fedora 13 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update abrt'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/abrt-1.1.1-1.fc13

Comment 14 Fedora Update System 2010-05-31 18:15:42 UTC
abrt-1.1.1-1.fc12 has been pushed to the Fedora 12 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update abrt'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/abrt-1.1.1-1.fc12

Comment 15 Fedora Update System 2010-06-02 18:04:42 UTC
abrt-1.1.1-1.fc13 has been pushed to the Fedora 13 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 16 Fedora Update System 2010-06-28 17:09:14 UTC
abrt-1.1.1-1.fc12 has been pushed to the Fedora 12 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.