Bug 999030 - abrt-handle-event core dumps
abrt-handle-event core dumps
Status: CLOSED CURRENTRELEASE
Product: Fedora
Classification: Fedora
Component: satyr (Show other bugs)
19
Unspecified Unspecified
unspecified Severity low
: ---
: ---
Assigned To: Jakub Filak
Fedora Extras Quality Assurance
:
Depends On: 997076
Blocks:
  Show dependency treegraph
 
Reported: 2013-08-20 10:38 EDT by Henrique Martins
Modified: 2016-11-30 19:44 EST (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-01-02 17:41:12 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Henrique Martins 2013-08-20 10:38:41 EDT
Description of problem:
Found a coredump in /var/tmp/abrt called 
abrt-handle-event-coredump, which is kind of ironic.
Traceback point to libsatyr.

Version-Release number of selected component (if applicable):
abrt-2.1.6-3.fc19.x86_64
satyr-0.5-2.fc19.x86_64

How reproducible:
It either happened when soffice.bin crashed, or when running abrt-tui, but no idea how to reproduce it. 

Steps to Reproduce:
1.  See above

Actual results:
core dump from abrt itself

Expected results:
no core dump

Additional info:
gdb /usr/libexec/abrt-handle-event abrt-handle-event-coredump 
GNU gdb (GDB) Fedora (7.6-34.fc19)
..
Reading symbols from /usr/libexec/abrt-handle-event...Reading symbols from /usr/libexec/abrt-handle-event...(no debugging symbols found)...done.
(no debugging symbols found)...done.

warning: core file may not match specified executable file.
[New LWP 19905]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/libexec/abrt-handle-event -e post-create -- /var/tmp/abrt/ccpp-2013-08-19-'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007fc520162e14 in sr_thread_frame_count () from /lib64/libsatyr.so.1
Traceback (most recent call last):
  File "/usr/share/gdb/auto-load/usr/lib64/libgobject-2.0.so.0.3600.3-gdb.py", line 9, in <module>
    from gobject import register
  File "/usr/share/glib-2.0/gdb/gobject.py", line 3, in <module>
    import gdb.backtrace
ImportError: No module named backtrace
Traceback (most recent call last):
  File "/usr/share/gdb/auto-load/usr/lib64/libgobject-2.0.so.0.3600.3-gdb.py", line 9, in <module>
    from gobject import register
  File "/usr/share/glib-2.0/gdb/gobject.py", line 3, in <module>
    import gdb.backtrace
ImportError: No module named backtrace
Missing separate debuginfos, use: debuginfo-install abrt-2.1.6-3.fc19.x86_64
(gdb) t
[Current thread is 1 (Thread 0x7fc5213be800 (LWP 19905))]
(gdb) bt
#0  0x00007fc520162e14 in sr_thread_frame_count () from /lib64/libsatyr.so.1
#1  0x00007fc521401a1d in is_crash_a_dup ()
#2  0x00007fc52081e2fb in consume_event_command_output ()
   from /lib64/libreport.so.0
#3  0x00007fc52081e34b in run_event_on_dir_name () from /lib64/libreport.so.0
#4  0x00007fc521401544 in main ()
Comment 1 John Schmitt 2013-09-16 18:55:12 EDT
Is this a symptom of the same issue?

Sep 16 15:31:21 test-vm33 kernel: [ 2549.969725] abrt-handle-eve[8094]: segfault at 0 ip 00007fc43226fe14 sp 00007fffdfe20950 error 4 in libsatyr.so.1.0.0[7fc432216000+13e000]
Sep 16 15:31:21 test-vm33 abrt[8127]: Saved core dump of pid 8094 (/usr/libexec/abrt-handle-event) to /var/tmp/abrt/abrt-handle-event-coredump (1736704 bytes)
Sep 16 15:31:21 test-vm33 abrtd: 'post-create' on '/var/tmp/abrt/ccpp-2013-09-16-15:31:20-8084' killed by signal 11
Sep 16 15:31:21 test-vm33 abrtd: Deleting problem directory '/var/tmp/abrt/ccpp-2013-09-16-15:31:20-8084'
Sep 16 15:32:14 test-vm33 abrt[8342]: Saved core dump of pid 8088 (/usr/sbin/iftop) to /var/tmp/abrt/ccpp-2013-09-16-15:32:14-8088 (28192768 bytes)
Sep 16 15:32:14 test-vm33 abrtd: Directory 'ccpp-2013-09-16-15:32:14-8088' creation detected
Sep 16 15:32:14 test-vm33 abrtd: Can't load public GPG key /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora
Sep 16 15:32:14 test-vm33 abrtd: Generating core_backtrace
Sep 16 15:32:14 test-vm33 abrtd: Generating backtrace
Sep 16 15:32:15 test-vm33 kernel: [ 2604.731712] abrt-handle-eve[8343]: segfault at 0 ip 00007ffc758a4e14 sp 00007fff1d52ce10 error 4 in libsatyr.so.1.0.0[7ffc7584b000+13e000]
Sep 16 15:32:15 test-vm33 abrt[8376]: Saved core dump of pid 8343 (/usr/libexec/abrt-handle-event) to /var/tmp/abrt/abrt-handle-event-coredump (1736704 bytes)
Sep 16 15:32:15 test-vm33 abrtd: 'post-create' on '/var/tmp/abrt/ccpp-2013-09-16-15:32:14-8088' killed by signal 11
Sep 16 15:32:15 test-vm33 abrtd: Deleting problem directory '/var/tmp/abrt/ccpp-2013-09-16-15:32:14-8088'
Sep 16 15:48:54 test-vm33 abrtd: Directory 'ccpp-2013-09-16-15:48:54-10463' creation detected
Sep 16 15:48:54 test-vm33 abrt[10472]: Saved core dump of pid 10463 (/usr/sbin/iftop) to /var/tmp/abrt/ccpp-2013-09-16-15:48:54-10463 (28192768 bytes)
Sep 16 15:48:54 test-vm33 abrtd: Can't load public GPG key /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora
Sep 16 15:48:54 test-vm33 abrtd: Generating core_backtrace
Sep 16 15:48:54 test-vm33 abrtd: Generating backtrace
Sep 16 15:48:55 test-vm33 kernel: [ 3603.659137] abrt-handle-eve[10473]: segfault at 0 ip 00007fbeb53f2e14 sp 00007fff16f85310 error 4 in libsatyr.so.1.0.0[7fbeb5399000+13e000]
Sep 16 15:48:55 test-vm33 abrt[10506]: Saved core dump of pid 10473 (/usr/libexec/abrt-handle-event) to /var/tmp/abrt/abrt-handle-event-coredump (1736704 bytes)
Sep 16 15:48:55 test-vm33 abrtd: 'post-create' on '/var/tmp/abrt/ccpp-2013-09-16-15:48:54-10463' killed by signal 11
Sep 16 15:48:55 test-vm33 abrtd: Deleting problem directory '/var/tmp/abrt/ccpp-2013-09-16-15:48:54-10463'
Sep 16 15:49:20 test-vm33 abrtd: Directory 'ccpp-2013-09-16-15:49:20-10468' creation detected
Sep 16 15:49:20 test-vm33 abrt[10726]: Saved core dump of pid 10468 (/usr/sbin/iftop) to /var/tmp/abrt/ccpp-2013-09-16-15:49:20-10468 (28561408 bytes)
Sep 16 15:49:20 test-vm33 abrtd: Can't load public GPG key /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora
Sep 16 15:49:20 test-vm33 abrtd: Generating core_backtrace
Sep 16 15:49:20 test-vm33 abrtd: Generating backtrace
Sep 16 15:49:20 test-vm33 kernel: [ 3629.145571] abrt-handle-eve[10727]: segfault at 0 ip 00007fa91bb53e14 sp 00007fff07e36e60 error 4 in libsatyr.so.1.0.0[7fa91bafa000+13e000]
Sep 16 15:49:20 test-vm33 abrt[10760]: Saved core dump of pid 10727 (/usr/libexec/abrt-handle-event) to /var/tmp/abrt/abrt-handle-event-coredump (1736704 bytes)
Sep 16 15:49:20 test-vm33 abrtd: 'post-create' on '/var/tmp/abrt/ccpp-2013-09-16-15:49:20-10468' killed by signal 11
Sep 16 15:49:20 test-vm33 abrtd: Deleting problem directory '/var/tmp/abrt/ccpp-2013-09-16-15:49:20-10468'
Comment 2 Jiri Moskovcak 2013-09-17 03:11:32 EDT
(In reply to John Schmitt from comment #1)
> Is this a symptom of the same issue?
> 

- probably yes
Comment 3 Jakub Filak 2013-09-17 07:48:21 EDT
Packages abrt-2.1.7-1.fc19, libreport-2.1.7-1.fc19, satyr-0.7-1.fc19 should fix this issue. If you have time to update the packages and re-test, please do so and report the results here. You can obtain the updated packages by typing 'yum update abrt libreport satyr' or using the graphical updater, Software Update.
Comment 4 Dr.Mike 2014-12-13 22:49:49 EST
 /usr/bin/abrt-watch-log -F BUG: WARNING: at WARNING: CPU: INFO: possible recursive locking detected ernel BUG at list_del corruption list_add corruption do_IRQ: stack overflow: ear stack overflow (cur: eneral protection fault nable to handle kernel ouble fault: RTNL: assertion failed eek!

looks bad to me and it appears that abrt-handle-event has a serious memory leak in the last two weeks.  It spawns and consumes some 500Kb per second on my Satelltie P75-A7200 until it reaches several Gb's and then stabilizes.  On smaller machines it would probably dump core eventually, but what I see is much paging and slow performance.

I just updated to:
Linux tobor 3.17.6-200.fc20.x86_64 #1 SMP Mon Dec 8 15:21:05 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

I also see the following in my proc list:
  915 ?        Ss     0:02 /usr/sbin/abrtd -d -s
 2160 ?        S      0:00  \_ abrt-server -s
 2165 ?        D      0:06  |   \_ /usr/libexec/abrt-handle-event -i -e post-create -- /var/tmp/abrt/oops-2014-12-13-19:42:14-29304-0
 2312 ?        S      0:00  \_ abrt-server -s
 2314 ?        D      0:05  |   \_ /usr/libexec/abrt-handle-event -i -e post-create -- /var/tmp/abrt/oops-2014-12-13-19:45:52-30201-0
 2437 ?        S      0:00  \_ abrt-server -s
 2439 ?        D      0:05  |   \_ /usr/libexec/abrt-handle-event -i -e post-create -- /var/tmp/abrt/oops-2014-12-13-19:51:59-31680-0
 2719 ?        S      0:00  \_ abrt-server -s
 2722 ?        D      0:04  |   \_ /usr/libexec/abrt-handle-event -i -e post-create -- /var/tmp/abrt/oops-2014-12-13-20:01:48-1553-0
 2760 ?        S      0:00  \_ abrt-server -s
 2762 ?        D      0:04  |   \_ /usr/libexec/abrt-handle-event -i -e post-create -- /var/tmp/abrt/oops-2014-12-13-20:04:36-1575-0
 2899 ?        S      0:00  \_ abrt-server -s
 2901 ?        D      0:03  |   \_ /usr/libexec/abrt-handle-event -i -e post-create -- /var/tmp/abrt/oops-2014-12-13-20:16:07-1730-0
 3069 ?        S      0:00  \_ abrt-server -s
 3071 ?        D      0:03  |   \_ /usr/libexec/abrt-handle-event -i -e post-create -- /var/tmp/abrt/oops-2014-12-13-20:18:21-1806-0
 3404 ?        S      0:00  \_ abrt-server -s
 3406 ?        D      0:01  |   \_ /usr/libexec/abrt-handle-event -i -e post-create -- /var/tmp/abrt/oops-2014-12-13-20:29:21-1967-0
 3515 ?        S      0:00  \_ abrt-server -s
 3517 ?        D      0:01  |   \_ /usr/libexec/abrt-handle-event -i -e post-create -- /var/tmp/abrt/oops-2014-12-13-20:41:59-2116-0
 5763 ?        S      0:00  \_ abrt-server -s
 5765 ?        D      0:00      \_ /usr/libexec/abrt-handle-event -i -e post-create -- /var/tmp/abrt/oops-2014-12-13-20:49:46-2204-0
  916 ?        Ss     0:00 /usr/bin/abrt-watch-log -F Backtrace /var/log/Xorg.0.log -- /usr/bin/abrt-dump-xorg -xD
  927 ?        Ss     0:00 /usr/bin/abrt-watch-log -F BUG: WARNING: at WARNING: CPU: INFO: possible recursive locking detected ernel BUG at list_del corruption list_add corruption do_IRQ: stack overflow: ear stack overflow (cur: eneral protection fault nable to handle kernel ouble fault: RTNL: assertion failed eek!
Comment 5 Jakub Filak 2015-01-02 17:41:12 EST
(In reply to Dr.Mike from comment #4)

Thank you for your comment. If I got you right, you considers the line with abrt-watch-log as a product of a buggy code, but I can assure you that the line is correct. See [1] for more details.

In your case, abrt-sever spawned so many abrt-handle-event processes because abrt-watch-log processes detected that many kernel oopses in "/var/log/messages". This should be fixed in abrt-2.1.7 (see bug #902398). If you have time to update the package and re-test, please do so and report the results here.


1: https://github.com/abrt/abrt/blob/master/src/lib/kernel.c#L82
Comment 6 Dr.Mike 2015-01-03 21:16:39 EST
(In reply to Jakub Filak from comment #5)
> (In reply to Dr.Mike from comment #4)
> 
> Thank you for your comment. If I got you right, you considers the line with
> abrt-watch-log as a product of a buggy code, but I can assure you that the
> line is correct. See [1] for more details.
> 
> In your case, abrt-sever spawned so many abrt-handle-event processes because
> abrt-watch-log processes detected that many kernel oopses in
> "/var/log/messages". This should be fixed in abrt-2.1.7 (see bug #902398).
> If you have time to update the package and re-test, please do so and report
> the results here.
> 
> 
> 1: https://github.com/abrt/abrt/blob/master/src/lib/kernel.c#L82

I have been running abrt 2.2.2 and have similar results.  The main problem is that it causes large amounts of disk I/O that stalls my i7 quad x64 that is also running other procs on all 4 cores at 100% cpu (it actually has 8 logical cores, but I found that running more than 4 actually reduces total throughput).  The result is that even switching between Xterms can take minutes to respond.  My tasks are mainly cycle limited, but disk I/O causes other things like even running 'ls' to take a very long time.

It might help to know I run FC20 on and external USB3 500Gb pocket drive, which was fast enough until about a number of weeks ago when abrt got updated.  I have also found that if I kill all abrt procs then my system runs well again.

So for now I will run without abrt until I hear that it no longer causes this problem.

Thanks, Mike

Note You need to log in before you can comment on or make changes to this bug.