Bug 1243791

Summary: abrt-dbus hangs on locked directories
Product: [Fedora] Fedora Reporter: Mikhail <mikhail.v.gavrilov>
Component: abrtAssignee: abrt <abrt-devel-list>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 22CC: abrt-devel-list, dvlasenk, iprikryl, jfilak, mhabrnal, michal.toman, mikhail.v.gavrilov, mmilata
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: abrt-2.6.1-2.fc22 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-07-30 00:50:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
journalctl -b | grep abrt
none
abrt-dbus -t 1000 -vvv none

Description Mikhail 2015-07-16 10:11:20 UTC 
Description of problem:


$ gnome-abrt 
/usr/lib64/python3.4/site-packages/gi/overrides/__init__.py:313: Warning: The property GSettings:schema is deprecated and shouldn't be used anymore. It will be removed in a future version.
  return super_init_func(self, **new_kwargs)

(gnome-abrt:17850): Gtk-CRITICAL **: gtk_header_bar_pack: assertion 'gtk_widget_get_parent (widget) == NULL' failed
WARNING:root:Can't get list of problems from DBus service: org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
WARNING:root:Can't get list of problems from DBus service: org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
WARNING:root:Can't get list of problems from DBus service: org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.


$ rpm -qa | grep gnome-abrt
gnome-abrt-1.2.0-3.fc22.x86_64
Comment 1 Jakub Filak 2015-07-16 10:20:46 UTC 
Thank you for the report! Please provide the output of the following commands:

$ sudo journalctl -b | grep abrt-dbus
$ sudo ls /var/spool/abrt
Comment 2 Mikhail 2015-07-16 10:22:46 UTC 
[mikhail@localhost ~]$ sudo journalctl -b | grep abrt-dbus
Error was encountered while opening journal files: Invalid argument
[mikhail@localhost ~]$ sudo ls /var/spool/abrt
abrt-applet-coredump		    ccpp-2015-07-14-22:07:25-19002.new	ccpp-2015-07-15-03:58:49-5147.new   ccpp-2015-07-15-13:13:59-19737.new	ccpp-2015-07-15-17:24:24-14077.new  last-via-server
ccpp-2015-07-14-10:39:57-3951.new   ccpp-2015-07-14-22:24:16-24118.new	ccpp-2015-07-15-03:59:30-4853.new   ccpp-2015-07-15-13:35:24-21529.new	ccpp-2015-07-15-17:24:44-15856.new  oops-2015-07-14-07:09:13-804-0
ccpp-2015-07-14-10:40:29-4286.new   ccpp-2015-07-14-22:47:11-24639.new	ccpp-2015-07-15-04:00:02-3957.new   ccpp-2015-07-15-13:36:47-2550.new	ccpp-2015-07-16-00:03:11-29690.new  oops-2015-07-14-10:36:11-802-0
ccpp-2015-07-14-10:40:29-5464.new   ccpp-2015-07-15-00:22:04-27619.new	ccpp-2015-07-15-04:00:20-4946.new   ccpp-2015-07-15-13:38:17-22489.new	ccpp-2015-07-16-13:45:01-4145.new   oops-2015-07-14-10:50:24-802-0
ccpp-2015-07-14-10:40:33-3918.new   ccpp-2015-07-15-00:49:45-15008.new	ccpp-2015-07-15-04:09:48-8687.new   ccpp-2015-07-15-13:38:40-24003.new	ccpp-2015-07-16-13:55:54-5105.new   oops-2015-07-15-04:30:17-790-0
ccpp-2015-07-14-10:43:06-2882.new   ccpp-2015-07-15-00:55:16-2524	ccpp-2015-07-15-04:33:34-3033.new   ccpp-2015-07-15-13:56:35-25491.new	ccpp-2015-07-16-13:56:15-5525.new
ccpp-2015-07-14-10:46:52-2522	    ccpp-2015-07-15-02:05:01-15250.new	ccpp-2015-07-15-11:46:14-15630.new  ccpp-2015-07-15-14:26:52-27394.new	ccpp-2015-07-16-14:15:37-13893.new
ccpp-2015-07-14-11:02:04-11197.new  ccpp-2015-07-15-02:49:58-2318.new	ccpp-2015-07-15-13:11:50-15597.new  ccpp-2015-07-15-16:27:58-8693.new	last-ccpp
Comment 3 Jakub Filak 2015-07-16 11:10:17 UTC 
Thank you! There is no 'abrt-dbus-coredump' file in '/var/spool/abrt' directory, so we can eliminate a crash of abrt-dbus from possible root causes of your problem.

What puzzles me a lot is the error message printed out by journalctl. Can you please log as root and run 'journactl'. If journalctl works, please run "journalctl -b | grep abrt" and report the results here.
Comment 4 Mikhail 2015-07-16 11:14:02 UTC 
Created attachment 1052657 [details]
journalctl -b | grep abrt
Comment 5 Jakub Filak 2015-07-16 13:53:00 UTC 
(In reply to Mikhail from comment #4)
> Created attachment 1052657 [details]
> journalctl -b | grep abrt

Excellent work!

I have found three problems in the logs which I cannot reproduce on my box:

1. abrt-applet.desktop[2824]: (abrt:2824): GLib-CRITICAL **: g_strsplit: assertion 'string != NULL' failed

I think I can fix this one without additional information.


2. abrt-hook-ccpp[8278]: Failed to create core_backtrace: PTRACE_SEIZE (tid 1) failed: Operation not permitted

This is probably caused by SELinux. Please run "ausearch -ts today -m avc | grep -B 2 -A 2 abrt" under root and provide the results.


3. abrt-dbus did not even try to start.

Please, log as root, run the following commands and provide the output:

$ killall abrt-dbus
$ abrt-dbus -t 1000 -vvv
Open gnome-abrt while the above command is running.
Comment 6 Mikhail 2015-07-16 13:59:44 UTC 
[root@localhost ~]# ausearch -ts today -m avc | grep -B 2 -A 2 abrt
[root@localhost ~]# 

Nothing returns.


$ killall abrt-dbus
$ abrt-dbus -t 1000 -vvv


Please see attach file.


$ ps ax | grep 7983
 7983 ?        S      0:00 /usr/libexec/abrt-hook-ccpp 11 0 1 1000 1000 1437036954 opera-developer 5105 1
Comment 7 Mikhail 2015-07-16 14:00:33 UTC 
Created attachment 1052733 [details]
abrt-dbus -t 1000 -vvv
Comment 8 Jakub Filak 2015-07-16 14:17:24 UTC 
(In reply to Mikhail from comment #7)
> Created attachment 1052733 [details]
> abrt-dbus -t 1000 -vvv

Thank you!

abrt-dbus is not broken, it just hangs on a locked problem directory. abrt-hook-ccpp is generating backtrace from the dying process of opera-developer and it is so slow and abrt-dbus does not skip the locked problem directory, so the dbus connection from gnome-abrt to abrt-dbus time outs.

We have to shorten the time-out for waiting for locked problem directories in abrt-dbus.
Comment 9 Mikhail 2015-07-16 14:41:04 UTC 
$ ps ax | grep 7983
 7983 ?        S      0:00 /usr/libexec/abrt-hook-ccpp 11 0 1 1000 1000 1437036954 opera-developer 5105 1
25341 pts/16   S+     0:00 grep --color=auto 7983


this line not disappear even after I close opera.
Comment 10 Fedora Update System 2015-07-21 16:53:02 UTC 
abrt-2.6.1-2.fc22,libreport-2.6.2-1.fc22 has been submitted as an update for Fedora 22.
https://admin.fedoraproject.org/updates/abrt-2.6.1-2.fc22,libreport-2.6.2-1.fc22
Comment 11 Fedora Update System 2015-07-29 01:41:51 UTC 
Package abrt-2.6.1-2.fc22, libreport-2.6.2-1.fc22:
* should fix your issue,
* was pushed to the Fedora 22 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing abrt-2.6.1-2.fc22 libreport-2.6.2-1.fc22'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2015-11974/abrt-2.6.1-2.fc22,libreport-2.6.2-1.fc22
then log in and leave karma (feedback).
Comment 12 Fedora Update System 2015-07-30 00:50:10 UTC 
abrt-2.6.1-2.fc22, libreport-2.6.2-1.fc22 has been pushed to the Fedora 22 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 13 Mikhail 2015-08-07 21:55:05 UTC 
Although lock gnome-abrt fixed
I still see that the problem is not resolved with abrt-hook-ccpp and it firmly lock many processes. only helps kill -9

$ ps ax | grep abrt
  764 ?        Ssl    0:00 /usr/sbin/abrtd -d -s
  779 ?        Ss     0:07 /usr/bin/abrt-dump-journal-oops -fxtD
  782 ?        Ss     0:00 /usr/bin/abrt-watch-log -F Backtrace /var/log/Xorg.0.log -- /usr/bin/abrt-dump-xorg -xD
 6077 ?        S      0:05 /usr/libexec/abrt-hook-ccpp 5 0 2706 1000 1000 1438983402 Chrome_DBThread 2706 2731
 7257 ?        S      0:00 /usr/libexec/abrt-hook-ccpp 11 0 1 1000 1000 1438965355 opera-developer 3668 1
12008 ?        S      0:00 /usr/libexec/abrt-hook-ccpp 11 0 1 1000 1000 1438983944 opera-developer 10546 1
12120 ?        Sl     0:00 /usr/sbin/abrt-dbus -t133
12559 ?        S      0:00 /usr/libexec/abrt-hook-ccpp 11 0 1 1000 1000 1438983969 opera-developer 10486 1
12804 tty2     Sl+    0:00 abrt-applet
13169 pts/9    S+     0:00 grep --color=auto abrt
15300 ?        S      0:16 /usr/libexec/abrt-hook-ccpp 11 0 6028 1000 1000 1438967773 ImageBridgeChil 6028 6618
29821 pts/2    Sl+    0:02 /usr/bin/python3 /usr/bin/gnome-abrt
31883 ?        S      0:05 /usr/libexec/abrt-hook-ccpp 5 0 29344 1000 1000 1438982412 Chrome_DBThread 29344 29373
Comment 14 Mikhail 2015-08-07 21:58:43 UTC 
backtrace some hanged abrt-hook-ccpp process

(gdb) thread apply all bt

Thread 1 (Thread 0x7fef4df7b840 (LWP 7257)):
#0  0x00007fef4c92266a in __GI___waitpid (pid=pid@entry=1, stat_loc=stat_loc@entry=0x7ffe295f7a5c, options=options@entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:31
#1  0x00007fef4d3b9b91 in sr_core_stacktrace_from_core_hook (tid=1, executable=0x5603ba0eedb0 "/usr/lib64/opera-developer/opera-developer", signum=11, error_msg=0x7ffe295f7bc8) at core_unwind_elfutils.c:264
#2  0x00007fef4d3b5e1b in sr_abrt_get_core_stacktrace_from_core_hook (thread_id=<optimized out>, executable=<optimized out>, signum=<optimized out>, error_message=<optimized out>) at abrt.c:175
#3  0x00005603b98f9dc7 in main (dd=0x5603ba0ad180, signal_no=11, executable=0x5603ba0eedb0 "/usr/lib64/opera-developer/opera-developer", tid=1) at abrt-hook-ccpp.c:422
#4  0x00005603b98f9dc7 in main (argc=<optimized out>, argv=<optimized out>) at abrt-hook-ccpp.c:962
(gdb)
Comment 15 Mikhail 2015-08-07 22:01:43 UTC 
yet another backtrace:

Thread 1 (Thread 0x7fa28e23a840 (LWP 6077)):
#0  0x00007fa28cbe1a40 in __nanosleep_nocancel () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007fa28cc12794 in usleep (useconds=useconds@entry=500000) at ../sysdeps/unix/sysv/linux/usleep.c:32
#2  0x00007fa28d4496a2 in dd_lock (dd=dd@entry=0x5601dbd41cc0, sleep_usec=sleep_usec@entry=500000, flags=flags@entry=11) at dump_dir.c:421
#3  0x00007fa28d4499a6 in dd_do_open (dd=0x5601dbd41cc0, dir=dir@entry=0x5601dbd7bfd0 "/var/spool/abrt/ccpp-2015-08-07-21:35:55-3668.new", flags=flags@entry=11) at dump_dir.c:933
#4  0x00007fa28d44a0c9 in dd_opendir (dir=dir@entry=0x5601dbd7bfd0 "/var/spool/abrt/ccpp-2015-08-07-21:35:55-3668.new", flags=flags@entry=11) at dump_dir.c:1020
#5  0x00007fa28d447db2 in libreport_get_dirsize_find_largest_dir (dirname=0x5601dbd7bfd0 "/var/spool/abrt/ccpp-2015-08-07-21:35:55-3668.new") at dirsize.c:63
#6  0x00007fa28d447db2 in libreport_get_dirsize_find_largest_dir (pPath=pPath@entry=0x5601dbd7c080 "/var/spool/abrt", worst_dir=worst_dir@entry=0x7ffc4c4e86c0, excluded=excluded@entry=0x7ffc4c4e88a0 "ccpp-2015-08-08-02:36:43-2706") at dirsize.c:115
#7  0x00007fa28de754c7 in abrt_trim_problem_dirs (dirname=0x5601dbd7c080 "/var/spool/abrt", cap_size=1341128704, exclude_path=<optimized out>) at hooklib.c:71
#8  0x00005601da50fe1e in main (argc=<optimized out>, argv=<optimized out>) at abrt-hook-ccpp.c:1012
(gdb) 


and yet another:

Thread 1 (Thread 0x7f0f32b98840 (LWP 15300)):
#0  0x00007f0f3153fa40 in __nanosleep_nocancel () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007f0f31570794 in usleep (useconds=useconds@entry=500000) at ../sysdeps/unix/sysv/linux/usleep.c:32
#2  0x00007f0f31da76a2 in dd_lock (dd=dd@entry=0x55bade027680, sleep_usec=sleep_usec@entry=500000, flags=flags@entry=11) at dump_dir.c:421
#3  0x00007f0f31da79a6 in dd_do_open (dd=0x55bade027680, dir=dir@entry=0x55bade025e90 "/var/spool/abrt/ccpp-2015-08-07-21:35:55-3668.new", flags=flags@entry=11) at dump_dir.c:933
#4  0x00007f0f31da80c9 in dd_opendir (dir=dir@entry=0x55bade025e90 "/var/spool/abrt/ccpp-2015-08-07-21:35:55-3668.new", flags=flags@entry=11) at dump_dir.c:1020
#5  0x00007f0f31da5db2 in libreport_get_dirsize_find_largest_dir (dirname=0x55bade025e90 "/var/spool/abrt/ccpp-2015-08-07-21:35:55-3668.new") at dirsize.c:63
#6  0x00007f0f31da5db2 in libreport_get_dirsize_find_largest_dir (pPath=pPath@entry=0x55bade028720 "/var/spool/abrt", worst_dir=worst_dir@entry=0x7fffecee6760, excluded=excluded@entry=0x7fffecee6940 "ccpp-2015-08-07-22:16:13-6028") at dirsize.c:115
#7  0x00007f0f327d34c7 in abrt_trim_problem_dirs (dirname=0x55bade028720 "/var/spool/abrt", cap_size=1341128704, exclude_path=<optimized out>) at hooklib.c:71
#8  0x000055badd00ae1e in main (argc=<optimized out>, argv=<optimized out>) at abrt-hook-ccpp.c:1012