Proposed as a Blocker and Freeze Exception for 30-beta by Fedora user mikhail using the blocker tracking app because:

 Because this bug completely broke system

Created attachment 1508436 [details]
full system log

Created attachment 1508437 [details]
dnf log

Encountered the same problem. Read the notes on the build:

* Thu Nov 22 2018 David Herrmann <dh.herrmann> - 1:1.12.10-9
- Switch to dbus-broker as the default implementation

What the issue was dbus-broker.service was not enabled after upgrading, breaking the system.

Fix it by running:

systemctl enable dbus-broker.service
systemctl --global dbus-broker.service

1. Why downgrading to 1.12.10-8 version and removing dbus-broker couldn't help?

2. I tried on the broken system under chroot enter `systemctl enable dbus-broker.service` then reboot and it couldn't helps.

That doesn't sound right, anyway, as it only changed a default (i.e. the dependency of the 'dbus' package). dbus-daemon still exists and should still work.

Mikhail, did you run the enable with the --global flag too? Also if you are still downgraded to 1.12.10-8, upgrade back to 1.12.10-9. I did not downgrade and my machine became usable after running the two enable commands.

Phea, of course I enabled service 'dbus-broker.service' with the --global flag too and upgrade dbus to 1.12.10-9 version again.

# rpm -qa | grep dbus | sort
abrt-dbus-2.11.0-1.fc30.x86_64
dbus-1.12.10-9.fc30.x86_64
dbus-broker-16-4.fc30.x86_64
dbus-common-1.12.10-9.fc30.noarch
dbus-daemon-1.12.10-9.fc30.x86_64
dbus-glib-0.110-3.fc29.x86_64
dbus-libs-1.12.10-9.fc30.i686
dbus-libs-1.12.10-9.fc30.x86_64
dbusmenu-qt-0.9.3-0.18.20150604.fc29.x86_64
dbus-tools-1.12.10-9.fc30.x86_64
dbus-x11-1.12.10-9.fc30.x86_64
dleyna-connector-dbus-0.3.0-3.fc29.x86_64
libdbusmenu-16.04.0-8.fc29.i686
libdbusmenu-16.04.0-8.fc29.x86_64
libdbusmenu-gtk2-16.04.0-8.fc29.i686
libdbusmenu-gtk2-16.04.0-8.fc29.x86_64
libdbusmenu-gtk3-16.04.0-8.fc29.i686
libdbusmenu-gtk3-16.04.0-8.fc29.x86_64
python3-dbus-1.2.8-3.fc29.x86_64
python3-pydbus-0.6.0-7.fc29.noarch
python3-slip-dbus-0.6.4-13.fc30.noarch



[root@localhost ~]# systemctl enable dbus-broker.service
Created symlink /etc/systemd/system/dbus.service → /usr/lib/systemd/system/dbus-broker.service.
Created symlink /etc/systemd/system/messagebus.service → /usr/lib/systemd/system/dbus-broker.service.
[root@localhost ~]# systemctl --global enable dbus-broker.service
Created symlink /etc/systemd/user/dbus.service → /usr/lib/systemd/user/dbus-broker.service.

Created attachment 1508442 [details]
after enable "dbus-broker.service" system stuck here

Mikhail, try setting the boot environment multi-user.target and then running the commands.

Created attachment 1508443 [details]
after enable "dbus-broker.service" - full system log

Nov 25 23:20:41 localhost.localdomain systemd-coredump[1067]: Process 1065 (systemd) of user 0 dumped core.
                                                              
                                                              Stack trace of thread 1065:
                                                              #0  0x00007f5933d9883b kill (libc.so.6)
                                                              #1  0x00005581772d0eda n/a (systemd)
                                                              #2  0x00007f5933f39070 __restore_rt (libpthread.so.0)
                                                              #3  0x00007f5933d9853f raise (libc.so.6)
                                                              #4  0x00007f5933d82895 abort (libc.so.6)
                                                              #5  0x00007f593410ed5f n/a (libsystemd-shared-239.so)
                                                              #6  0x00007f59341e30c1 sd_bus_slot_unref (libsystemd-shared-239.so)
                                                              #7  0x00007f59341ee1fb n/a (libsystemd-shared-239.so)
                                                              #8  0x00007f59341ef0f2 bus_ensure_running (libsystemd-shared-239.so)
                                                              #9  0x00007f59341efd08 sd_bus_call (libsystemd-shared-239.so)
                                                              #10 0x00007f59341c0cfe sd_bus_call_method (libsystemd-shared-239.so)
                                                              #11 0x00007f59341bee5b sd_bus_list_names (libsystemd-shared-239.so)
                                                              #12 0x00005581772de5a6 n/a (systemd)
                                                              #13 0x00007f5934216db2 n/a (libsystemd-shared-239.so)
                                                              #14 0x00007f5934218c55 sd_event_dispatch (libsystemd-shared-239.so)
                                                              #15 0x00007f5934218de0 sd_event_run (libsystemd-shared-239.so)
                                                              #16 0x000055817731215c n/a (systemd)
                                                              #17 0x00005581772cc78b n/a (systemd)
                                                              #18 0x00007f5933d84413 __libc_start_main (libc.so.6)
                                                              #19 0x00005581772ce41e n/a (systemd)
Nov 25 23:20:41 localhost.localdomain udev-configure-printer[1037]: MFG:Hewlett-Packard MDL:HP LaserJet Professional M1132 MFP SERN:- serial:000000000QH707YTPR1a
Nov 25 23:20:41 localhost.localdomain systemd-logind[975]: malloc_consolidate(): invalid chunk size
Nov 25 23:20:41 localhost.localdomain audit[975]: ANOM_ABEND auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:systemd_logind_t:s0 pid=975 comm="systemd-logind" exe="/usr/lib/systemd/systemd-logind" sig=6 res=1
Nov 25 23:20:42 localhost.localdomain rngd[1013]: Enabling JITTER rng support
Nov 25 23:20:46 localhost.localdomain udev-configure-printer[1037]: failed to connect to CUPS server; giving up
Nov 25 23:21:00 localhost.localdomain kernel: fbcon: Taking over console
Nov 25 23:21:00 localhost.localdomain kernel: Console: switching to colour frame buffer device 240x67
Nov 25 23:22:11 localhost.localdomain avahi-daemon[1011]: dbus_bus_get_private(): Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
Nov 25 23:22:11 localhost.localdomain avahi-daemon[1011]: WARNING: Failed to contact D-Bus daemon.
Nov 25 23:22:11 localhost.localdomain avahi-daemon[1011]: avahi-daemon 0.7 exiting.
Nov 25 23:22:11 localhost.localdomain firewalld[1040]: ERROR: Exception DBusException: org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.

Created attachment 1508444 [details]
systemd backtrace

> Mikhail, try setting the boot environment multi-user.target and then running
the commands.

Phea, I correctly understood, you suggest to do under chroot:
# systemctl set-default multi-user.target
and then reboot.

If yes it not helps too.

Hi Mikhail,

Thanks for reporting the bug, and sorry for the inconvenience!

There are a few bugs exposed by this:

1) a packaging bug that appears to cause dbus-broker not to be enabled correctly.
2) an SELinux related bug causing systemd-machined to be refused ownership of a name it claims[0].
3) a bug in dbus-broker that causes SELinux denial to become a fatal error[1].
4) a bug in systemd that causes it to abort on some dbus error [2].

I have fixed 2) upstream [3], and will push a new package to rawhide ASAP.

As a temporary work-around, I believe you should be able to boot your machine in SELinux permissive mode (after enabling dbus-broker explicitly as suggested by others).

[0]: USER_AVC pid=1058 uid=81 auid=4294967295 ses=4294967295 subj=system_u:system_r:unconfined_service_t:s0 msg='avc:  denied  { acquire_svc } for  scontext=system_u:system_r:systemd_machined_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=dbus permissive=0
[1]: ERROR peer_request_name @ ../src/bus/peer.c +318: Return code 3
[2]: systemd[1]: Assertion 'slot->n_ref > 0' failed at ../src/libsystemd/sd-bus/bus-slot.c:198, function sd_bus_slot_unref(). Aborting.
[3]: https://github.com/bus1/dbus-broker/commit/a35b2c24e3e99c121132d0b201179618f9be3485

That was a bit fast, this is the commit that should fix the SELinux bug in dbus-broker upstream: https://github.com/bus1/dbus-broker/commit/11957a930725d6e3af059bb5a0d98b540ce6fd20

Tom, big thanks, switching SELinux to permissive mode helps.

But I am still don't understand why downgrading dbus package not returned my system to bootable state?
It's very important for me as Rawhide user, because always something may happen Iand I need 100% working tool for returning my system to bootable state.

Please don't reccomend me Silver Blue, because flatpack isolation has performance impact for me.

For example in games launched under flatpack steam client I have twice lower FPS than rpm version.

I can confirm everything that happened here.
It took me a while to find out the responsible for my startup errors.

To be able to startup again I had to :
- systemctl enable dbus-broker.service
- reboot enabling systemd.debug_shell
- setenforce 0 (in debug shell)
- login

Without the setenforce 0 to go back to permissive, I can log in but am directly logged back out without being able to do anything, so my current workaround to boot is to modify the grub command line to add the systemd.debug_shell argument.

Once I add this, I can Ctrl-Alt-F9 while on login screen (SDDM for me), and execute any command I want there, especially the setenforce before I log in

I can confirm everything that happened here.
It took me a while to find out the responsible for my startup errors.

To be able to startup again I had to :
- systemctl enable dbus-broker.service
- reboot enabling systemd.debug_shell
- setenforce 0 (in debug shell)
- login

Without the setenforce 0 to go back to permissive, I can log in but am directly logged back out without being able to do anything, so my current workaround to boot is to modify the grub command line to add the systemd.debug_shell argument.

Once I add this, I can Ctrl-Alt-F9 while on login screen (SDDM for me), and execute any command I want there, especially the setenforce before I log in

I now pushed dbus-broker-16-6.fc30 to rawhide which may fix the "dbus-broker not getting enabled on upgrade" issue. I'll try to reproduce tomorrow (unless anyone can confirm before then), and I'll also have a look at why reverting to dbus-daemon doesn't work as expected.

Forgot to add: 16-6 should also fix the SELinux issue in dbus-broker (so enforcing 0 won't be needed any longer at least).

dbus-broker-16-7 has been pushed to rawhide, which should fix the issue of dbus-broker not being enabled on install.

*** Bug 1653037 has been marked as a duplicate of this bug. ***

Tom: are you actually trying to switch existing installs from dbus-daemon to dbus-broker on upgrade using package scripts? This seems questionable, and was not covered in the scope of the Change:

https://fedoraproject.org/wiki/Changes/DbusBrokerAsTheDefaultDbusImplementation#Scope

so has not been reviewed and approved by FESCo as part of the Change.

There is actually a comment in the dbus-broker spec file %post section which seems to explicitly acknowledge that this should *not* happen:

"We only do this on fresh installs, not on updates (updates retain the users preferences)."

yet there is then a triggerpostun for dbus-daemon which does exactly what that comment says will not happen!

This Change is really proving to be a complete disaster in implementation. I would suggest FESCo should do a comprehensive review of this whole thing post-F30.

note on #c25 - per http://ftp.rpm.org/api/4.4.2.2/triggers.html , "%triggerpostun -- dbus-daemon" will trigger any time dbus-daemon is *updated*, not just when it is *removed*. As explained there, if the intent is to switch to dbus-broker automatically on final removal of dbus-daemon , the value of $2 should be tested.

IIUC, the current implementation of this will result in the preset calls running *every time the dbus-daemon package is updated*.

(In reply to Adam Williamson from comment #25)
> There is actually a comment in the dbus-broker spec file %post section which
> seems to explicitly acknowledge that this should *not* happen:
> 
> "We only do this on fresh installs, not on updates (updates retain the users
> preferences)."
> 
> yet there is then a triggerpostun for dbus-daemon which does exactly what
> that comment says will not happen!

Bugs notwithstanding (I think you are right in #26 and will address that in the morning), the intention behind the comment was that on first install of the package, dbus-broker is enabled (in place of dbus-daemon), however, no subsequent changes are applied on updates of the dbus-broker package.

(In reply to Adam Williamson from comment #24)
> Tom: are you actually trying to switch existing installs from dbus-daemon to
> dbus-broker on upgrade using package scripts? This seems questionable, and
> was not covered in the scope of the Change:
> 
> https://fedoraproject.org/wiki/Changes/
> DbusBrokerAsTheDefaultDbusImplementation#Scope
> 
> so has not been reviewed and approved by FESCo as part of the Change.

Our intention was to switch from dbus-broker to dbus-daemon by default in F30. Which to us meant switching on upgrades from F29 to F30 as well. We make it possible to opt-in to dbus-daemon still, but the default will be dbus-broker.

I see that the Change can be read in a different way, and that is regrettable, and it certainly was not our intention to be vague.

We gave some thought to how we could do this in the least intrusive way (which isn't all that easy with such a core part of the system as dbus), and we landed on making it possible to switch between implementations, but that the default (if you haven't configured this manually at all) would be for F30 systems to run dbus-broker. We had first thought that this could just be done by switching over the presets, but (as I'm sure you know) that's not how it works, so this was the next closest thing.

If we were to just switch over the presets, but not explicitly make the switch, then users upgrading from F29 to F30 would be stuck with dbus-daemon unless they manualy intervened, and manual intervention would mean calling the right systemctl commands, then rebooting before potentially uninstalling dbus-daemon (using the wrong systemctl invocation, or uninstalling dbus-daemon before rebooting would both be bad). Switching your bus implementation on a running system is (naturally, I suppose) quite fragile. We figured that that's ok in the rare case that somebody want to deviate from the default, but in the common case of people just wanting the default, it seemed like quite a tall order.

Note that before we refactored the dbus package to allow switching back and forth between the bus implementations on a running system (as opposed to only on offline upgrades or similar), the only way of making dbus-broker the default would be to switch everybody over on upgrade to F30 without any opt-out. That's where we started from, and that's why it hadn't occurred to us thus far that "switching the default" was ambiguous (so it's not that we tried to sneak anything in).

Do you propose us going to FESCo to clarify this again, or change the behaviour we are aiming for, or was your comment just for the sake of future reference?

I mainly want to flag up to FESCo (and people in general) that this change actually is happening, since to me it was not at all clear from the Change page and I suspect it wasn't clear to others either.

I'm not suggesting any particular actions for FESCo to take, I'm sure they're capable of deciding that for themselves.

(In reply to Tom Gundersen from comment #22)
> dbus-broker-16-7 has been pushed to rawhide, which should fix the issue of
> dbus-broker not being enabled on install.

Tom, today I updated fresh system to dbus-broker-16-7, but looks like you forget again enable dbus-broker.service, because fresh system broke again. And when I enter in chroot and typed `# systemctl enable dbus-broker.service` I see that new symlinks was created, but when I typed `# systemctl enable dbus-broker.service` I don;t see than new symlinks created. So I suppose you only enable `dbus-broker.service` globally which is insufficient for solve the problem.

Mikhail, what do you mean by "updated fresh system"? I am not able to reporduce the problem with a fresh install (containing dbus-broker-16-7):

```
# dnf -y --releasever=rawhide --installroot=/var/lib/machines/rawhide --disablerepo='*' --enablerepo=fedora --enablerepo=updates --nogpgcheck install systemd passwd dnf fedora-release
# systemctl --root /var/lib/machines/rawhide/ is-enabled dbus-broker
enabled
```

(In reply to Tom Gundersen from comment #31)
> Mikhail, what do you mean by "updated fresh system"? I am not able to
> reporduce the problem with a fresh install (containing dbus-broker-16-7):

I am means the system without dbus-broker package with dbus 1.12.10-8

Adam, the bug you pointed out in #26 should be fixed in 16-8. Thanks for spotting that!

Mikhail, ah I got it. I think that too should be fixed by 16-8. Thanks for testing!

*** Bug 1654342 has been marked as a duplicate of this bug. ***

*** Bug 1653371 has been marked as a duplicate of this bug. ***

*** Bug 1653493 has been marked as a duplicate of this bug. ***

*** Bug 1653692 has been marked as a duplicate of this bug. ***

I manually upgraded to dbus-broker 16-8 from koji and it didn't help. dbus-broker still shows disabled, and doesn't start manually.

Gwyn: i guess by that you mean that you *first* hit the 'bad' update, then you ran the 'fixed' update over that broken state and it didn't fix it up, right? I think maybe Tom means that with the newer packages, upgrading from F29 won't run into the bug any more - but perhaps if you already ran into the bug, you have to fix it up manually.

The automated upgrade tests are working at present, so I don't think this bug is still present at least in its initial form.

This came up in the blocker meeting today; it does seem pretty clear that the initial, clear-cut bug here which meant you never wound up with a running dbus after update and reboot is fixed. There may still be other similar / related issues, possibly, but it'd be best to file new bugs for those and consider them separately. So if anyone's still having trouble, please file a fresh bug. Thanks.

Also note that *if you did get the bad update initially*, further updates alone may not fix it. Just go ahead and fix it up manually (by enabling dbus-broker.service, probably). That's just a kinda expected bump in the road for Rawhide users at this point. If anyone still has issues with dbus not working properly on a *fresh upgrade from F29 or earlier to F30*, that is something that should be filed as a bug.

(In reply to Adam Williamson from comment #40)
> Gwyn: i guess by that you mean that you *first* hit the 'bad' update, then
> you ran the 'fixed' update over that broken state and it didn't fix it up,
> right? I think maybe Tom means that with the newer packages, upgrading from
> F29 won't run into the bug any more - but perhaps if you already ran into
> the bug, you have to fix it up manually.

Yes, exactly. Manual fix-up is regrettably necessary if you first hit the bad package. (Sorry for not following up on this, I somehow missed Gwyn's message).

Hi,

I just had the same problem, today (jan 29th, 2019).

I am not sure it is the same problem, exactly, but for your information,...

I use the regular fedora desktop 29 (I mean no special release).
Linux xxxx.mylocaldomain 4.20.4-200.fc29.x86_64 #1 SMP Wed Jan 23 16:11:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

I did an update with DNF (with GUI version), and at reboot, black screen, no reaction, nothing.
In the infos (F5) during boot :
"Failed to listen on D-Bus System Message Bus Socket."

in the journal :
"janv. 29 15:30:15 localhost.localdomain systemd[1]: Failed to listen on D-Bus System Message Bus Socket."

I also saw the error :
"Unable to get DBus system bus connection: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory"

many times, in the logs when booting (F5).
and in the journal :

janv. 29 14:47:10 localhost.localdomain libvirtd[963]: 2019-01-29 13:47:10.690+0000: 1267: warning : networkStateInitialize:769 : DBus not available, disabling firewalld support in bridge_network_driver: internal error: Unable to get DBus system bus connection: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory

When I checked (with ls), the "/run/dbus" folder was not there (does not exists).

*******************************
Whith dnf history :
dnf history info 143
I get this :

    Upgraded  curl-7.61.1-6.fc29.x86_64                                       @@System
    Upgrade   dbus-1:1.12.12-1.fc29.x86_64                                    @updates
    Upgraded  dbus-1:1.12.10-1.fc29.x86_64                                    @@System
    Upgrade   dbus-common-1:1.12.12-1.fc29.noarch                             @updates
    Upgraded  dbus-common-1:1.12.10-1.fc29.noarch                             @@System
    Upgrade   dbus-daemon-1:1.12.12-1.fc29.x86_64                             @updates
    Upgraded  dbus-daemon-1:1.12.10-1.fc29.x86_64                             @@System
    Upgrade   dbus-libs-1:1.12.12-1.fc29.i686                                 @updates
    Upgraded  dbus-libs-1:1.12.10-1.fc29.i686                                 @@System
    Upgrade   dbus-libs-1:1.12.12-1.fc29.x86_64                               @updates
    Upgraded  dbus-libs-1:1.12.10-1.fc29.x86_64                               @@System
    Upgrade   dbus-tools-1:1.12.12-1.fc29.x86_64                              @updates
    Upgraded  dbus-tools-1:1.12.10-1.fc29.x86_64                              @@System
    Upgrade   dbus-x11-1:1.12.12-1.fc29.x86_64                                @updates
    Upgraded  dbus-x11-1:1.12.10-1.fc29.x86_64                                @@System
    Upgrade   file-5.34-11.fc29.x86_64                                        @updates
    Upgraded  file-5.34-7.fc29.x86_64         

Note : the upgrade is from 1.12.10-1, to 1.12.12-1
 
*********************************
Workaround that worked :

After reading this post, to repair my system, I did :
"A simple workaround is to run `systemctl enable dbus-daemon`. You should only use `systemctl enable dbus-broker`, if you want dbus-broker over dbus-daemon."

I did only the `systemctl enable dbus-daemon`.

My system booted properly.

Folder /run/dbus is there :
ll -d /run/dbus
drwxr-xr-x. 2 root root 60 29 janv. 17:51 /run/dbus

**********************************
the problem is solved, manually,... but this is strange....

Antoine: can you see what happens if you do:

systemctl preset dbus-daemon.service
systemctl preset dbus.socket
systemctl is-enabled dbus.socket
systemctl is-enabled dbus-daemon.service

?

If it says they're disabled, that's the problem for you: somehow the 'preset' mechanism is disabling them. (If it does, re-enable them manually before you reboot).

Note: I and a colleague tested on our machines and the upgrade went OK, and the update also passed openQA testing, so it doesn't seem like it's completely broken; there is probably something odd that happened on your system somehow to cause this.

Ok, but do I try this when in Gnome, in a terminal ?
it will break Gnome, and the services that need DBus, no ?
or shall I try this in Fedora rescue mode ?
Want to make sure before to break my system,... I work on it every day...
Thanks for the answer.

Hello,
Ok, I gave it a try.

systemctl preset dbus-daemon.service
Failed to preset unit: File /etc/systemd/system/dbus.service already exists and is a symlink to /usr/lib/systemd/system/dbus-broker.service.

systemctl preset dbus.socket
(nothing in return)

systemctl is-enabled dbus.socket
enabled

systemctl is-enabled dbus-daemon.service
enabled

so it seems that every thing is going fine, now.

Note : since then, I am now with Fedora 30.
uname -osr
Linux 5.3.6-200.fc30.x86_64 GNU/Linux

I guess there is no bug anymore.

Thanks.