Bug 1252590 - abrt-hook-ccpp process hangs forever
abrt-hook-ccpp process hangs forever
Status: CLOSED NEXTRELEASE
Product: Fedora
Classification: Fedora
Component: abrt (Show other bugs)
23
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: abrt
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-08-11 16:10 EDT by Mikhail
Modified: 2015-08-31 23:36 EDT (History)
8 users (show)

See Also:
Fixed In Version: 2.6.2-3.fc23
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-08-31 23:36:41 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Mikhail 2015-08-11 16:10:21 EDT
Description of problem:
Bug 1243791 is fixed, but I see that process abrt-hook-ccpp still hangs forever.
And in sometime this results in a hang of other programs.


$ ps ax | grep abrt
  885 ?        Ssl    0:00 /usr/sbin/abrtd -d -s
  898 ?        Ss     0:00 /usr/bin/abrt-dump-journal-oops -fxtD
 3343 pts/2    Sl+    0:02 /usr/bin/python3 /usr/bin/gnome-abrt
 8585 tty2     Sl+    0:03 abrt-applet
 9652 ?        S      0:02 /usr/libexec/abrt-hook-ccpp 11 0 1 1000 1000 1439318045 chrome 6639 1
29110 pts/14   S+     0:00 grep --color=auto abrt


# gdb -pid=9652
GNU gdb (GDB) Fedora 7.9.90.20150717-7.fc23
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

* * *

0x00007f28de43c81a in __GI___waitpid (pid=pid@entry=1, stat_loc=stat_loc@entry=0x7ffeccfba6dc, options=options@entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
29	  return SYSCALL_CANCEL (wait4, pid, stat_loc, options, NULL);
(gdb) thread apply all bt

Thread 1 (Thread 0x7f28dfa58880 (LWP 9652)):
#0  0x00007f28de43c81a in __GI___waitpid (pid=pid@entry=1, stat_loc=stat_loc@entry=0x7ffeccfba6dc, options=options@entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
#1  0x00007f28deedab91 in sr_core_stacktrace_from_core_hook (tid=1, executable=0x55df85b21c10 "/opt/google/chrome-unstable/chrome", signum=11, error_msg=0x7ffeccfba848) at core_unwind_elfutils.c:264
#2  0x00007f28deed6e1b in sr_abrt_get_core_stacktrace_from_core_hook (thread_id=<optimized out>, executable=<optimized out>, signum=<optimized out>, error_message=<optimized out>) at abrt.c:175
#3  0x000055df857a5dc7 in create_core_backtrace (dd=0x55df85b1f7b0, signal_no=11, executable=0x55df85b21c10 "/opt/google/chrome-unstable/chrome", tid=1) at abrt-hook-ccpp.c:422
#4  main (argc=<optimized out>, argv=<optimized out>) at abrt-hook-ccpp.c:962
(gdb)
Comment 1 Mikhail 2015-08-11 16:40:34 EDT
yet another case:


$ ps ax | grep abrt
  885 ?        Ssl    0:00 /usr/sbin/abrtd -d -s
  898 ?        Ss     0:00 /usr/bin/abrt-dump-journal-oops -fxtD
 3343 pts/2    Sl+    0:02 /usr/bin/python3 /usr/bin/gnome-abrt
 3773 ?        S      0:14 /usr/libexec/abrt-hook-ccpp 11 0 25129 1000 1000 1439325188 ImageBridgeChil 25129 25227
 5144 pts/14   S+     0:00 grep --color=auto abrt
 8585 tty2     Sl+    0:03 abrt-applet
 9652 ?        S      0:02 /usr/libexec/abrt-hook-ccpp 11 0 1 1000 1000 1439318045 chrome 6639 1





# gdb -pid=3773
GNU gdb (GDB) Fedora 7.9.90.20150717-7.fc23
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>


* * *

0x00007f8bb8a13bf0 in __nanosleep_nocancel () at ../sysdeps/unix/syscall-template.S:84
84	T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
(gdb) thread apply all bt

Thread 1 (Thread 0x7f8bba02f880 (LWP 3773)):
#0  0x00007f8bb8a13bf0 in __nanosleep_nocancel () at ../sysdeps/unix/syscall-template.S:84
#1  0x00007f8bb8a458e4 in usleep (useconds=useconds@entry=500000) at ../sysdeps/posix/usleep.c:32
#2  0x00007f8bb92816a2 in dd_lock (dd=dd@entry=0x55acea6a6030, sleep_usec=sleep_usec@entry=500000, flags=flags@entry=11) at dump_dir.c:421
#3  0x00007f8bb92819a6 in dd_do_open (dd=0x55acea6a6030, dir=dir@entry=0x55acea6a6490 "/var/spool/abrt/ccpp-2015-08-11-23:34:05-6639.new", flags=flags@entry=11) at dump_dir.c:933
#4  0x00007f8bb92820c9 in dd_opendir (dir=dir@entry=0x55acea6a6490 "/var/spool/abrt/ccpp-2015-08-11-23:34:05-6639.new", flags=flags@entry=11) at dump_dir.c:1020
#5  0x00007f8bb927fdb2 in this_is_a_dd (dirname=0x55acea6a6490 "/var/spool/abrt/ccpp-2015-08-11-23:34:05-6639.new") at dirsize.c:63
#6  libreport_get_dirsize_find_largest_dir (pPath=pPath@entry=0x55acea6ca530 "/var/spool/abrt", worst_dir=worst_dir@entry=0x7ffe1e54d290, excluded=excluded@entry=0x7ffe1e54d470 "ccpp-2015-08-12-01:33:08-25129")
    at dirsize.c:115
#7  0x00007f8bb9cbb4c7 in abrt_trim_problem_dirs (dirname=0x55acea6ca530 "/var/spool/abrt", cap_size=1341128704, exclude_path=<optimized out>) at hooklib.c:71
#8  0x000055ace8f6ee1e in main (argc=<optimized out>, argv=<optimized out>) at abrt-hook-ccpp.c:1012
(gdb)
Comment 2 Mikhail 2015-08-12 04:36:11 EDT
and yet another:

$ ps ax | grep abrt
  885 ?        Ssl    0:00 /usr/sbin/abrtd -d -s
  898 ?        Ss     0:00 /usr/bin/abrt-dump-journal-oops -fxtD
 3773 ?        S      0:15 /usr/libexec/abrt-hook-ccpp 11 0 25129 1000 1000 1439325188 ImageBridgeChil 25129 25227
 6234 tty2     Sl+    0:02 abrt-applet
 6309 pts/2    Sl+    0:02 /usr/bin/python3 /usr/bin/gnome-abrt
 9652 ?        S      0:02 /usr/libexec/abrt-hook-ccpp 11 0 1 1000 1000 1439318045 chrome 6639 1
12118 ?        S      0:08 /usr/libexec/abrt-hook-ccpp 11 0 23590 1000 1000 1439326973 nautilus 23590 23590
12283 pts/2    Z+     0:00 [abrt-handle-eve] <defunct>
18363 ?        S      0:13 /usr/libexec/abrt-hook-ccpp 11 0 27173 1000 1000 1439366924 ImageBridgeChil 27173 27368
24101 pts/14   S+     0:00 grep --color=auto abrt


# gdb -pid=12118
GNU gdb (GDB) Fedora 7.9.90.20150717-7.fc23
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

* * *

0x00007fd62d7a4bf0 in __nanosleep_nocancel () at ../sysdeps/unix/syscall-template.S:84
84	T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
(gdb) thread apply all bt

Thread 1 (Thread 0x7fd62edc0880 (LWP 12118)):
#0  0x00007fd62d7a4bf0 in __nanosleep_nocancel () at ../sysdeps/unix/syscall-template.S:84
#1  0x00007fd62d7d68e4 in usleep (useconds=useconds@entry=500000) at ../sysdeps/posix/usleep.c:32
#2  0x00007fd62e0126a2 in dd_lock (dd=dd@entry=0x55ca63187a10, sleep_usec=sleep_usec@entry=500000, flags=flags@entry=11) at dump_dir.c:421
#3  0x00007fd62e0129a6 in dd_do_open (dd=0x55ca63187a10, dir=dir@entry=0x55ca6313c7f0 "/var/spool/abrt/ccpp-2015-08-11-23:34:05-6639.new", flags=flags@entry=11) at dump_dir.c:933
#4  0x00007fd62e0130c9 in dd_opendir (dir=dir@entry=0x55ca6313c7f0 "/var/spool/abrt/ccpp-2015-08-11-23:34:05-6639.new", flags=flags@entry=11) at dump_dir.c:1020
#5  0x00007fd62e010db2 in this_is_a_dd (dirname=0x55ca6313c7f0 "/var/spool/abrt/ccpp-2015-08-11-23:34:05-6639.new") at dirsize.c:63
#6  libreport_get_dirsize_find_largest_dir (pPath=pPath@entry=0x55ca63187810 "/var/spool/abrt", worst_dir=worst_dir@entry=0x7fffb5e21880, excluded=excluded@entry=0x7fffb5e21a60 "ccpp-2015-08-12-02:02:53-23590")
    at dirsize.c:115
#7  0x00007fd62ea4c4c7 in abrt_trim_problem_dirs (dirname=0x55ca63187810 "/var/spool/abrt", cap_size=1341128704, exclude_path=<optimized out>) at hooklib.c:71
#8  0x000055ca62d5be1e in main (argc=<optimized out>, argv=<optimized out>) at abrt-hook-ccpp.c:1012
(gdb)
Comment 3 Jakub Filak 2015-08-12 08:00:58 EDT
Thank you for the report! I want to reproduce the hang but killing chrome processes with -SEGV does not work. Which version of google-chrome do you have installed and how can I reproduce the crash of chrome?
Comment 4 Mikhail 2015-08-12 08:26:57 EDT
$ google-chrome --version
Google Chrome 46.0.2478.0 dev

I was hoping that would be enough backtrace because I'm afraid to repeat what I was doing, I would not be simple.


Why??? Looks here: https://drive.google.com/file/d/0B0nwzlfiB4aQbzg0SThVWERIWlE/view?usp=sharing (Please use google chrome for HD quality)

I don't know how calculate count of tabs.
But count of process: 
$ ps ax | grep chrome | wc -l
72
Comment 5 Jakub Filak 2015-08-12 11:37:59 EDT
google-chrome uses own PID namespace, thus its thread IDs starts with 1 and only processes in the same PID namespace can use these thread IDs to send signals to the threads. abrt-hook-ccpp is run in the system PID namespace, so it has to use global TID in the both ptrace() and waitpid() calls which means that the last argument of the template in the /proc/sys/kernel/core_pattern file must changed from '%i' to '%I'.

In your particular case, abrt-hook-ccpp wanted to ptrace TID 1 (PID 1).

#3 create_core_backtrace (dd=..., signal_no=11, executable=..., tid=1) at abrt-hook-ccpp.c:422
Comment 6 Mikhail 2015-08-12 17:19:27 EDT
All subsequent freezes occurred due to the fact that abrt-hook-ccpp hung in chrome.

# kill 9652

$ ps ax | grep abrt
  885 ?        Ssl    0:00 /usr/sbin/abrtd -d -s
  898 ?        Ss     0:01 /usr/bin/abrt-dump-journal-oops -fxtD
 6309 pts/2    Sl+    0:08 /usr/bin/python3 /usr/bin/gnome-abrt
 7053 tty2     Sl+    0:02 abrt-applet
 9929 pts/14   S+     0:00 grep --color=auto abrt
Comment 7 Jakub Filak 2015-08-13 06:15:57 EDT
Upstream commit https://github.com/abrt/abrt/commit/d4758cf4c0c153ee41ddca06dcb213b210f4dd8b fixes the bug.
Comment 8 Fedora Update System 2015-08-17 07:06:56 EDT
abrt-2.6.2-4.fc23,libreport-2.6.2-3.fc23,gnome-abrt-1.2.0-5.fc23 has been submitted as an update for Fedora 23.
https://admin.fedoraproject.org/updates/abrt-2.6.2-4.fc23,libreport-2.6.2-3.fc23,gnome-abrt-1.2.0-5.fc23
Comment 9 Fedora Update System 2015-08-19 03:55:50 EDT
Package abrt-2.6.2-4.fc23, libreport-2.6.2-3.fc23, gnome-abrt-1.2.0-5.fc23:
* should fix your issue,
* was pushed to the Fedora 23 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing abrt-2.6.2-4.fc23 libreport-2.6.2-3.fc23 gnome-abrt-1.2.0-5.fc23'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2015-13626/abrt-2.6.2-4.fc23,libreport-2.6.2-3.fc23,gnome-abrt-1.2.0-5.fc23
then log in and leave karma (feedback).
Comment 10 Fedora Update System 2015-08-31 23:36:38 EDT
abrt-2.6.2-4.fc23, gnome-abrt-1.2.0-5.fc23, libreport-2.6.2-3.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.