Bug 1576387 - Deadlock in nss-systemd and dbus-daemon during startup
Summary: Deadlock in nss-systemd and dbus-daemon during startup
Alias: None
Product: Fedora
Classification: Fedora
Component: selinux-policy
Version: 28
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
Assignee: Lukas Vrabec
QA Contact: Fedora Extras Quality Assurance
Depends On:
TreeView+ depends on / blocked
Reported: 2018-05-09 10:54 UTC by David Herrmann
Modified: 2018-05-26 20:44 UTC (History)
14 users (show)

Fixed In Version: selinux-policy-3.14.1-29.fc28
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2018-05-26 20:44:10 UTC
Type: Bug

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1565735 0 unspecified CLOSED dbus-broker Fails to Start After Upgrade to F28 (beta) 2021-02-22 00:41:40 UTC

Description David Herrmann 2018-05-09 10:54:25 UTC
If the dbus configuration files contain an unknown username, or a username resolved via other means than /etc/passwd, startup of dbus-daemon as system-bus will trigger a 90s timeout and block the entire boot sequence.

This is easy to reproduce. Simply add something like this to /usr/share/dbus-1/system.conf:

  <policy user="UnknownUserName">
    <allow send_destination="com.example.foobar" />

After reboot, startup of dbus.service will trigger a 90s timeout in systemd. During that timeout, either basic.target is kept pending, or systemd blocks entirely on an internal sd-bus call waiting for dbus-daemon to respond.

This bug is new in F28 and requires 'systemd' to be in 'passwd' in your nsswitch.conf. Furthermore, you must have SELinux enabled in **enforcing** mode!

This deadlock condition arises in the startup-code of dbus-daemon. When dbus-daemon reads its configuration, it resolves all user-names it finds via nss getpwnam(3). This will trigger nss modules. In the case of an unknown username, this will end up in libnss_systemd.so (nss-systemd) as of F28. This module tries to query the dynamic username entries of systemd. But it does so using dbus to talk to the systemd D-Bus interface. This will thus try to synchronously connect to the dbus socket, which cannot be served since it is currently blocking on the dbus-daemon configuration parser.

Eventually, after 90s the sd-bus call used in nss-systemd will error out and mark the username as non-existant.

Fortunately, the systemd developers are aware of this deadlock and made sure to shortcut the dbus call in nss-systemd in case the SYSTEMD_NSS_BYPASS_BUS environment variable is set to non-zero. In case of disabled SELinux (or permissive mode), this works splendidly. With SELinux in enforcing mode, though, this will environment-check will fail.

Under the hood, nss-systemd uses secure_getenv(3) to query the environment variable (since you don't want to trick setuid programs into skipping the passwd lookup and thus possibly failing to resolve uids). glibc uses the AT_SECURE auxiliary vector entry (see getauxval(3)) to trigger the secure_getenv(3) fallback. In case of enabled SELinux, AT_SECURE is true whenever context transition occurs. This means, dbus-daemon runs with AT_SECURE *enabled*, and thus secure_getenv(3) **ALWAYS** returns NULL.

There are several possible workarounds, none seem satisfactory. The least ugly workaround seems to add 'noatsecure' for dbus_t, similarly to NetworkManager [1].

Any ideas?

[1] https://github.com/fedora-selinux/selinux-policy-contrib/commit/61ebb1a659e4f2a9f1f7ad017b1b4e264593515a

Comment 1 Zbigniew Jędrzejewski-Szmek 2018-05-09 14:29:06 UTC
Adding 'noatsecure' seems to be a reasonable solution.

More generally, there is no reason for daemons started by the init system to not trust their environment. Systemd sets up the environment for daemons (environment variables, but also ulimits and other settings), and has complete control over what is started, so it must be ultimately trusted, and selinux should not interfere with that environment. On the other hand, when a daemon is started by the init system, a context transition is generally expected to occur. Thus, setting AT_SECURE for daemons started by the init system only breaks things without any security benefit.

Comment 2 David Rheinsberg 2018-05-16 08:24:53 UTC
How about adding 'noatsecure' to every transition from 'init_t'->'*'? Zbigniew already explained that every process should trust the environment if spawned by systemd, so I don't see how 'AT_SECURE' makes sense in those cases?

Is there a way in SELinux to set 'noatsecure' for anything spawned by 'init_t'?

Comment 4 Fedora Update System 2018-05-24 14:35:35 UTC
selinux-policy-3.14.1-29.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-a74875b364

Comment 5 Fedora Update System 2018-05-25 18:42:20 UTC
selinux-policy-3.14.1-29.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-a74875b364

Comment 6 Fedora Update System 2018-05-26 20:44:10 UTC
selinux-policy-3.14.1-29.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.