| Summary: | abrt-handle-event core dumps | ||
|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Henrique Martins <fedora> |
| Component: | satyr | Assignee: | Jakub Filak <jfilak> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
| Severity: | low | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 19 | CC: | abrt-devel-list, ars_1, dvlasenk, iprikryl, jberan, jfilak, marmalodak, mkb, mmilata, mtoman |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2015-01-02 22:41:12 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Bug Depends On: | 997076 | ||
| Bug Blocks: | |||
Is this a symptom of the same issue? Sep 16 15:31:21 test-vm33 kernel: [ 2549.969725] abrt-handle-eve[8094]: segfault at 0 ip 00007fc43226fe14 sp 00007fffdfe20950 error 4 in libsatyr.so.1.0.0[7fc432216000+13e000] Sep 16 15:31:21 test-vm33 abrt[8127]: Saved core dump of pid 8094 (/usr/libexec/abrt-handle-event) to /var/tmp/abrt/abrt-handle-event-coredump (1736704 bytes) Sep 16 15:31:21 test-vm33 abrtd: 'post-create' on '/var/tmp/abrt/ccpp-2013-09-16-15:31:20-8084' killed by signal 11 Sep 16 15:31:21 test-vm33 abrtd: Deleting problem directory '/var/tmp/abrt/ccpp-2013-09-16-15:31:20-8084' Sep 16 15:32:14 test-vm33 abrt[8342]: Saved core dump of pid 8088 (/usr/sbin/iftop) to /var/tmp/abrt/ccpp-2013-09-16-15:32:14-8088 (28192768 bytes) Sep 16 15:32:14 test-vm33 abrtd: Directory 'ccpp-2013-09-16-15:32:14-8088' creation detected Sep 16 15:32:14 test-vm33 abrtd: Can't load public GPG key /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora Sep 16 15:32:14 test-vm33 abrtd: Generating core_backtrace Sep 16 15:32:14 test-vm33 abrtd: Generating backtrace Sep 16 15:32:15 test-vm33 kernel: [ 2604.731712] abrt-handle-eve[8343]: segfault at 0 ip 00007ffc758a4e14 sp 00007fff1d52ce10 error 4 in libsatyr.so.1.0.0[7ffc7584b000+13e000] Sep 16 15:32:15 test-vm33 abrt[8376]: Saved core dump of pid 8343 (/usr/libexec/abrt-handle-event) to /var/tmp/abrt/abrt-handle-event-coredump (1736704 bytes) Sep 16 15:32:15 test-vm33 abrtd: 'post-create' on '/var/tmp/abrt/ccpp-2013-09-16-15:32:14-8088' killed by signal 11 Sep 16 15:32:15 test-vm33 abrtd: Deleting problem directory '/var/tmp/abrt/ccpp-2013-09-16-15:32:14-8088' Sep 16 15:48:54 test-vm33 abrtd: Directory 'ccpp-2013-09-16-15:48:54-10463' creation detected Sep 16 15:48:54 test-vm33 abrt[10472]: Saved core dump of pid 10463 (/usr/sbin/iftop) to /var/tmp/abrt/ccpp-2013-09-16-15:48:54-10463 (28192768 bytes) Sep 16 15:48:54 test-vm33 abrtd: Can't load public GPG key /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora Sep 16 15:48:54 test-vm33 abrtd: Generating core_backtrace Sep 16 15:48:54 test-vm33 abrtd: Generating backtrace Sep 16 15:48:55 test-vm33 kernel: [ 3603.659137] abrt-handle-eve[10473]: segfault at 0 ip 00007fbeb53f2e14 sp 00007fff16f85310 error 4 in libsatyr.so.1.0.0[7fbeb5399000+13e000] Sep 16 15:48:55 test-vm33 abrt[10506]: Saved core dump of pid 10473 (/usr/libexec/abrt-handle-event) to /var/tmp/abrt/abrt-handle-event-coredump (1736704 bytes) Sep 16 15:48:55 test-vm33 abrtd: 'post-create' on '/var/tmp/abrt/ccpp-2013-09-16-15:48:54-10463' killed by signal 11 Sep 16 15:48:55 test-vm33 abrtd: Deleting problem directory '/var/tmp/abrt/ccpp-2013-09-16-15:48:54-10463' Sep 16 15:49:20 test-vm33 abrtd: Directory 'ccpp-2013-09-16-15:49:20-10468' creation detected Sep 16 15:49:20 test-vm33 abrt[10726]: Saved core dump of pid 10468 (/usr/sbin/iftop) to /var/tmp/abrt/ccpp-2013-09-16-15:49:20-10468 (28561408 bytes) Sep 16 15:49:20 test-vm33 abrtd: Can't load public GPG key /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora Sep 16 15:49:20 test-vm33 abrtd: Generating core_backtrace Sep 16 15:49:20 test-vm33 abrtd: Generating backtrace Sep 16 15:49:20 test-vm33 kernel: [ 3629.145571] abrt-handle-eve[10727]: segfault at 0 ip 00007fa91bb53e14 sp 00007fff07e36e60 error 4 in libsatyr.so.1.0.0[7fa91bafa000+13e000] Sep 16 15:49:20 test-vm33 abrt[10760]: Saved core dump of pid 10727 (/usr/libexec/abrt-handle-event) to /var/tmp/abrt/abrt-handle-event-coredump (1736704 bytes) Sep 16 15:49:20 test-vm33 abrtd: 'post-create' on '/var/tmp/abrt/ccpp-2013-09-16-15:49:20-10468' killed by signal 11 Sep 16 15:49:20 test-vm33 abrtd: Deleting problem directory '/var/tmp/abrt/ccpp-2013-09-16-15:49:20-10468' (In reply to John Schmitt from comment #1) > Is this a symptom of the same issue? > - probably yes Packages abrt-2.1.7-1.fc19, libreport-2.1.7-1.fc19, satyr-0.7-1.fc19 should fix this issue. If you have time to update the packages and re-test, please do so and report the results here. You can obtain the updated packages by typing 'yum update abrt libreport satyr' or using the graphical updater, Software Update. /usr/bin/abrt-watch-log -F BUG: WARNING: at WARNING: CPU: INFO: possible recursive locking detected ernel BUG at list_del corruption list_add corruption do_IRQ: stack overflow: ear stack overflow (cur: eneral protection fault nable to handle kernel ouble fault: RTNL: assertion failed eek! looks bad to me and it appears that abrt-handle-event has a serious memory leak in the last two weeks. It spawns and consumes some 500Kb per second on my Satelltie P75-A7200 until it reaches several Gb's and then stabilizes. On smaller machines it would probably dump core eventually, but what I see is much paging and slow performance. I just updated to: Linux tobor 3.17.6-200.fc20.x86_64 #1 SMP Mon Dec 8 15:21:05 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux I also see the following in my proc list: 915 ? Ss 0:02 /usr/sbin/abrtd -d -s 2160 ? S 0:00 \_ abrt-server -s 2165 ? D 0:06 | \_ /usr/libexec/abrt-handle-event -i -e post-create -- /var/tmp/abrt/oops-2014-12-13-19:42:14-29304-0 2312 ? S 0:00 \_ abrt-server -s 2314 ? D 0:05 | \_ /usr/libexec/abrt-handle-event -i -e post-create -- /var/tmp/abrt/oops-2014-12-13-19:45:52-30201-0 2437 ? S 0:00 \_ abrt-server -s 2439 ? D 0:05 | \_ /usr/libexec/abrt-handle-event -i -e post-create -- /var/tmp/abrt/oops-2014-12-13-19:51:59-31680-0 2719 ? S 0:00 \_ abrt-server -s 2722 ? D 0:04 | \_ /usr/libexec/abrt-handle-event -i -e post-create -- /var/tmp/abrt/oops-2014-12-13-20:01:48-1553-0 2760 ? S 0:00 \_ abrt-server -s 2762 ? D 0:04 | \_ /usr/libexec/abrt-handle-event -i -e post-create -- /var/tmp/abrt/oops-2014-12-13-20:04:36-1575-0 2899 ? S 0:00 \_ abrt-server -s 2901 ? D 0:03 | \_ /usr/libexec/abrt-handle-event -i -e post-create -- /var/tmp/abrt/oops-2014-12-13-20:16:07-1730-0 3069 ? S 0:00 \_ abrt-server -s 3071 ? D 0:03 | \_ /usr/libexec/abrt-handle-event -i -e post-create -- /var/tmp/abrt/oops-2014-12-13-20:18:21-1806-0 3404 ? S 0:00 \_ abrt-server -s 3406 ? D 0:01 | \_ /usr/libexec/abrt-handle-event -i -e post-create -- /var/tmp/abrt/oops-2014-12-13-20:29:21-1967-0 3515 ? S 0:00 \_ abrt-server -s 3517 ? D 0:01 | \_ /usr/libexec/abrt-handle-event -i -e post-create -- /var/tmp/abrt/oops-2014-12-13-20:41:59-2116-0 5763 ? S 0:00 \_ abrt-server -s 5765 ? D 0:00 \_ /usr/libexec/abrt-handle-event -i -e post-create -- /var/tmp/abrt/oops-2014-12-13-20:49:46-2204-0 916 ? Ss 0:00 /usr/bin/abrt-watch-log -F Backtrace /var/log/Xorg.0.log -- /usr/bin/abrt-dump-xorg -xD 927 ? Ss 0:00 /usr/bin/abrt-watch-log -F BUG: WARNING: at WARNING: CPU: INFO: possible recursive locking detected ernel BUG at list_del corruption list_add corruption do_IRQ: stack overflow: ear stack overflow (cur: eneral protection fault nable to handle kernel ouble fault: RTNL: assertion failed eek! (In reply to Dr.Mike from comment #4) Thank you for your comment. If I got you right, you considers the line with abrt-watch-log as a product of a buggy code, but I can assure you that the line is correct. See [1] for more details. In your case, abrt-sever spawned so many abrt-handle-event processes because abrt-watch-log processes detected that many kernel oopses in "/var/log/messages". This should be fixed in abrt-2.1.7 (see bug #902398). If you have time to update the package and re-test, please do so and report the results here. 1: https://github.com/abrt/abrt/blob/master/src/lib/kernel.c#L82 (In reply to Jakub Filak from comment #5) > (In reply to Dr.Mike from comment #4) > > Thank you for your comment. If I got you right, you considers the line with > abrt-watch-log as a product of a buggy code, but I can assure you that the > line is correct. See [1] for more details. > > In your case, abrt-sever spawned so many abrt-handle-event processes because > abrt-watch-log processes detected that many kernel oopses in > "/var/log/messages". This should be fixed in abrt-2.1.7 (see bug #902398). > If you have time to update the package and re-test, please do so and report > the results here. > > > 1: https://github.com/abrt/abrt/blob/master/src/lib/kernel.c#L82 I have been running abrt 2.2.2 and have similar results. The main problem is that it causes large amounts of disk I/O that stalls my i7 quad x64 that is also running other procs on all 4 cores at 100% cpu (it actually has 8 logical cores, but I found that running more than 4 actually reduces total throughput). The result is that even switching between Xterms can take minutes to respond. My tasks are mainly cycle limited, but disk I/O causes other things like even running 'ls' to take a very long time. It might help to know I run FC20 on and external USB3 500Gb pocket drive, which was fast enough until about a number of weeks ago when abrt got updated. I have also found that if I kill all abrt procs then my system runs well again. So for now I will run without abrt until I hear that it no longer causes this problem. Thanks, Mike |
Description of problem: Found a coredump in /var/tmp/abrt called abrt-handle-event-coredump, which is kind of ironic. Traceback point to libsatyr. Version-Release number of selected component (if applicable): abrt-2.1.6-3.fc19.x86_64 satyr-0.5-2.fc19.x86_64 How reproducible: It either happened when soffice.bin crashed, or when running abrt-tui, but no idea how to reproduce it. Steps to Reproduce: 1. See above Actual results: core dump from abrt itself Expected results: no core dump Additional info: gdb /usr/libexec/abrt-handle-event abrt-handle-event-coredump GNU gdb (GDB) Fedora (7.6-34.fc19) .. Reading symbols from /usr/libexec/abrt-handle-event...Reading symbols from /usr/libexec/abrt-handle-event...(no debugging symbols found)...done. (no debugging symbols found)...done. warning: core file may not match specified executable file. [New LWP 19905] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/libexec/abrt-handle-event -e post-create -- /var/tmp/abrt/ccpp-2013-08-19-'. Program terminated with signal 11, Segmentation fault. #0 0x00007fc520162e14 in sr_thread_frame_count () from /lib64/libsatyr.so.1 Traceback (most recent call last): File "/usr/share/gdb/auto-load/usr/lib64/libgobject-2.0.so.0.3600.3-gdb.py", line 9, in <module> from gobject import register File "/usr/share/glib-2.0/gdb/gobject.py", line 3, in <module> import gdb.backtrace ImportError: No module named backtrace Traceback (most recent call last): File "/usr/share/gdb/auto-load/usr/lib64/libgobject-2.0.so.0.3600.3-gdb.py", line 9, in <module> from gobject import register File "/usr/share/glib-2.0/gdb/gobject.py", line 3, in <module> import gdb.backtrace ImportError: No module named backtrace Missing separate debuginfos, use: debuginfo-install abrt-2.1.6-3.fc19.x86_64 (gdb) t [Current thread is 1 (Thread 0x7fc5213be800 (LWP 19905))] (gdb) bt #0 0x00007fc520162e14 in sr_thread_frame_count () from /lib64/libsatyr.so.1 #1 0x00007fc521401a1d in is_crash_a_dup () #2 0x00007fc52081e2fb in consume_event_command_output () from /lib64/libreport.so.0 #3 0x00007fc52081e34b in run_event_on_dir_name () from /lib64/libreport.so.0 #4 0x00007fc521401544 in main ()