Bug 1186018 - updating packages hangs in %post scripts running dbus-send
Summary: updating packages hangs in %post scripts running dbus-send
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: systemd
Version: 21
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: systemd-maint
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-01-26 20:16 UTC by Dennis Gilmore
Modified: 2015-04-02 15:37 UTC (History)
21 users (show)

Fixed In Version: systemd-216-24.fc21
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-04-02 15:37:34 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Dennis Gilmore 2015-01-26 20:16:50 UTC
Description of problem:

using dnf to update my desktop from Fedora 21 to rawhide, the upgrade process stalled. when looking into why I noticed that some packages were running "dbus-send --system --type=method_call --dest=org.freedesktop.DBus / org.freedesktop.DBus.ReloadConfig" which hung.

In my case it was rtkit avahi and imsettings

Version-Release number of selected component (if applicable):
systemd-218-3.fc22.x86_64

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Adam Williamson 2015-01-26 20:19:44 UTC
I saw what sounds like exactly this when I updated my system from F21 to Rawhide just after F21 release; I don't recall the precise details but I definitely figured out something was stuck on a dbus operation. Unfortunately for me I was running the 'yum upgrade' in a GNOME terminal and I decided to restart dbus to fix it, with predictably explosive consequences. Took an afternoon to clean up, but I have only myself to blame for that one.

Obviously upgrades 'ought' to be done offline with fedup or at least with yum/dnf on a console or in screen, but this still seems like it's a bug that should be fixed. And it is *possible* this happens even in the fedup environment, I don't think we've checked yet.

Comment 2 Zbigniew Jędrzejewski-Szmek 2015-01-26 20:22:35 UTC
Dennis, can you attach journalctl output (for the whole boot preferably)?
It might be related to https://bugzilla.redhat.com/show_bug.cgi?id=1167044 or something similar.

Comment 3 Jaroslav Škarvada 2015-01-27 08:06:46 UTC
Hi, I was also affected by this problem during upgrade to f22 from f21.

In my case it was rtkit and later other packages with dbus/systemd services. I resolved the problem by killing dbus-send processes that were spawned during the upgrade and later also some systemctl try-restart processes. This way the upgrade finishes.

Then I did 'rpm -qVa', which reveals the following files were modified (I haven't touched them):
/etc/dbus-1/system.d/org.freedesktop.systemd1.conf
/etc/dbus-1/system.d/org.freedesktop.machine1.conf

I deleted them and reinstalled package which provides them (i.e. systemd). I also reinstalled dbus package and just to be safe initiated selinux relabel. The problem seems to go away after reboot.

Comment 4 Jan Synacek 2015-02-02 07:22:19 UTC
*** Bug 1187695 has been marked as a duplicate of this bug. ***

Comment 5 Lennart Poettering 2015-02-04 19:22:15 UTC
well, the command you pasted talks to dbus-daemon. I am not sure how systemd is supposed to be involved. What is dbus-daemon doing when you issue this an this hangs? Is it running? Is it processing the request? Is it hanging too? What does strace say?

REassigning to dbus, since this appears to be more of a dbus bug...

Comment 6 Christopher Meng 2015-02-18 00:12:35 UTC
I met this long time ago in the last year, I used htop and found that dbus was invoked but it stopped, then I killed the subprocess of yum which is a scriptlet calling dbus and the bug was solved.

Comment 7 Michal Schmidt 2015-02-19 18:04:57 UTC
I narrowed this down. There is a seemingly insignificant difference in dbus.socket between F21 and F22:

-ListenStream=/var/run/dbus/system_bus_socket
+ListenStream=/run/dbus/system_bus_socket

However when systemd is asked to reload its configuration ("systemctl daemon-reload", invoked from RPM scriptlets), this difference causes it to create a new listening socket, making the old one inaccessible.

Here's a much easier way to reproduce the problem than a full Fedora upgrade:

1. Check the initial conditions. /var/run/dbus/system_bus_socket exists and both systemd and dbus-daemon have it open:

[root@localhost ~]# ss -lxp | grep system_bus_socket                                                    
u_str  LISTEN     0      128    /var/run/dbus/system_bus_socket 17470                 * 0       users:(("dbus-daemon",pid=456,fd=3),("systemd",pid=1,fd=38))

[root@localhost ~]# stat /run/dbus/system_bus_socket                                                    
  File: ‘/run/dbus/system_bus_socket’
  Size: 0               Blocks: 0          IO Block: 4096   socket
Device: 12h/18d Inode: 17475       Links: 1
Access: (0666/srw-rw-rw-)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:system_dbusd_var_run_t:s0
Access: 2015-02-19 18:32:52.920000000 +0100
Modify: 2015-02-19 18:31:26.406000000 +0100
Change: 2015-02-19 18:31:26.406000000 +0100
 Birth: -

2. Do the key change the upgrade of the dbus-daemon package would do:

[root@localhost ~]# sed -i -e 's#/var/run#/run#' /lib/systemd/system/dbus.socket                        
[root@localhost ~]# systemctl daemon-reload

3. Check the current situation:

[root@localhost ~]# ss -lxp | grep system_bus_socket                                                    
u_str  LISTEN     0      128    /var/run/dbus/system_bus_socket 17470                 * 0       users:(("dbus-daemon",pid=456,fd=3))
u_str  LISTEN     0      128    /run/dbus/system_bus_socket 68770                 * 0       users:(("systemd",pid=1,fd=20))

[root@localhost ~]# stat /run/dbus/system_bus_socket                                                    
  File: ‘/run/dbus/system_bus_socket’
  Size: 0               Blocks: 0          IO Block: 4096   socket
Device: 12h/18d Inode: 68772       Links: 1
Access: (0666/srw-rw-rw-)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:system_dbusd_var_run_t:s0
Access: 2015-02-19 18:33:28.816000000 +0100
Modify: 2015-02-19 18:33:28.816000000 +0100
Change: 2015-02-19 18:33:28.816000000 +0100
 Birth: -

See that there are now two UNIX sockets. Only the one created by systemd during daemon-reload is now visible in the filesystem.

Why does systemd do that? Because when deserializing the socket unit's state, it compares the open socket's address with the configured address (comparing the strings exactly) and if it does not find a match, it closes the socket and later creates a new one (during coldplugging).
(See http://cgit.freedesktop.org/systemd/systemd/tree/src/core/socket.c?id=v219#n2150)

As a fix systemd could canonicalize the paths before comparing (not only for UNIX sockets, but for FIFOs and specials too).

Comment 9 Jaroslav Reznik 2015-03-03 17:09:20 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 22 development cycle.
Changing version to '22'.

More information and reason for this action is here:
https://fedoraproject.org/wiki/Fedora_Program_Management/HouseKeeping/Fedora22

Comment 10 Fedora Update System 2015-03-15 04:00:35 UTC
systemd-219-9.fc22 has been submitted as an update for Fedora 22.
https://admin.fedoraproject.org/updates/systemd-219-9.fc22

Comment 11 Fedora Update System 2015-03-16 01:40:50 UTC
Package systemd-219-9.fc22:
* should fix your issue,
* was pushed to the Fedora 22 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing systemd-219-9.fc22'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2015-3949/systemd-219-9.fc22
then log in and leave karma (feedback).

Comment 12 Fedora Update System 2015-03-21 05:02:34 UTC
systemd-219-9.fc22 has been pushed to the Fedora 22 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 13 Elad Alfassa 2015-03-23 12:24:08 UTC
This fix needs to be pushed into F21 so yum distro-sync from F21 to F22 will work (and not hang).

Reopening this bug.

Comment 14 Fedora Update System 2015-03-26 14:30:30 UTC
systemd-216-23.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/systemd-216-23.fc21

Comment 15 Fedora Update System 2015-03-30 07:04:01 UTC
Package systemd-216-24.fc21:
* should fix your issue,
* was pushed to the Fedora 21 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing systemd-216-24.fc21'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2015-4991/systemd-216-24.fc21
then log in and leave karma (feedback).

Comment 16 Fedora Update System 2015-04-02 15:37:34 UTC
systemd-216-24.fc21 has been pushed to the Fedora 21 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.