Description of problem: I run minimal systemd-based containers. The Dockerfile is FROM fedora:rawhide # RUN dnf upgrade -y ENV container docker VOLUME /tmp /run ENTRYPOINT /usr/sbin/init COPY container-minimal.target /usr/lib/systemd/system/ RUN systemctl set-default container-minimal.target RUN for i in basic.target sysinit.target network.service netconsole.service ; do rm -f /usr/lib/systemd/system/$i && ln -s /dev/null /usr/lib/systemd/system/$i ; done LABEL RUN "docker run -ti -v /sys/fs/cgroup:/sys/fs/cgroup:ro ${NAME}" and the container-minimal.target file is [Unit] Description=Minimal target for systemd-based containers DefaultDependencies=false AllowIsolate=yes Requires=systemd-tmpfiles-setup.service systemd-journald.service dbus.service After=systemd-tmpfiles-setup.service systemd-journald.service dbus.service With Fedora 23 base image, the line masking sysinit.target allows trimming the startup from default systemd 222 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN) Detected virtualization docker. Detected architecture x86-64. Running with unpopulated /etc. Welcome to Fedora 23 (Twenty Three)! Set hostname to <0be9a4736391>. Initializing machine ID from random generator. Populated /etc with preset unit settings. Running in a container, ignoring fstab device entry for /dev/disk/by-uuid/e9c6a283-f8b8-437b-bf5e-9e73b8f44ac8. [ OK ] Reached target Encrypted Volumes. [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. [ OK ] Reached target Paths. [ OK ] Created slice Root Slice. [ OK ] Created slice System Slice. Starting First Boot Wizard... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Journal Service... Starting Rebuild Hardware Database... [ OK ] Listening on /dev/initctl Compatibility Named Pipe. Starting Rebuild Journal Catalog... [ OK ] Started First Boot Wizard. [ OK ] Started Load/Save Random Seed. Starting Create System Users... [ OK ] Started Create System Users. [ OK ] Started Rebuild Journal Catalog. [ OK ] Started Rebuild Dynamic Linker Cache. [ OK ] Started Journal Service. Starting Flush Journal to Persistent Storage... [ OK ] Started Rebuild Hardware Database. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Started Flush Journal to Persistent Storage. Starting Create Volatile Files and Directories... [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting D-Bus System Message Bus... [ OK ] Reached target Minimal target for systemd-based containers. [ OK ] Reached target Timers. to neat systemd 222 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN) Detected virtualization docker. Detected architecture x86-64. Running with unpopulated /etc. Welcome to Fedora 23 (Twenty Three)! Set hostname to <219bdc798753>. Initializing machine ID from random generator. Populated /etc with preset unit settings. Running in a container, ignoring fstab device entry for /dev/disk/by-uuid/e9c6a283-f8b8-437b-bf5e-9e73b8f44ac8. [ OK ] Created slice Root Slice. [ OK ] Created slice System Slice. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Started D-Bus System Message Bus. Starting D-Bus System Message Bus... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Journal Service. [ OK ] Reached target Minimal target for systemd-based containers. However, on rawhide, systemd freezes: systemd 229 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN) Detected virtualization docker. Detected architecture x86-64. Welcome to Fedora 25 (Rawhide)! Set hostname to <8478dc5e84e2>. Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning. Running in a container, ignoring fstab device entry for /dev/disk/by-uuid/2933020c-519a-4dd8-9cbb-98975ad66c95. Failed to isolate default target: Unit sysinit.target is masked. Attempted to remove disk file system, and we can't allow that. [!!!!!!] Failed to isolate default target, freezing. Freezing execution. Version-Release number of selected component (if applicable): both systemd-229-4.fc25.x86_64 and systemd-container-229-13.fc25.x86_64 (the second one if the RUN dnf upgrade line is uncommented. How reproducible: Deterministic. Steps to Reproduce: 1. In current directory, have Dockerfile and container-minimal.target with content as shown above. 2. Run docker build -t systemd:rawhide . 3. Run docker run --rm -ti -v /sys/fs/cgroup:/sys/fs/cgroup:ro systemd:rawhide Actual results: Failed to isolate default target: Unit sysinit.target is masked. Attempted to remove disk file system, and we can't allow that. [!!!!!!] Failed to isolate default target, freezing. Freezing execution. Expected results: [ OK ] Created slice Root Slice. [ OK ] Created slice System Slice. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Started D-Bus System Message Bus. Starting D-Bus System Message Bus... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Journal Service. [ OK ] Reached target Minimal target for systemd-based containers. Additional info:
The fedora:24 image (f9873d530588) has the same problem.
It seems merely masking basic.target causes similar error: Failed to populate /etc with preset unit settings, ignoring: No such file or directory Failed to isolate default target: Unit basic.target is masked. [!!!!!!] Failed to isolate default target, freezing. Freezing execution.
This bug appears to have been reported against 'rawhide' during the Fedora 25 development cycle. Changing version to '25'.
Sorry for not responding more promptly. I think systemd behaviour is correct here: $ systemctl list-dependencies --all container-minimal.target container-minimal.target ● ├─dbus.service ● │ ├─dbus.socket ● │ │ └─sysinit.target All those dependencies are of type Requires, which means that if the dependencies cannot be started, the target unit also fails. The last depenency (socket → sysinit.target) is added automatically to all socket units DefaultDependencies=yes. This behaviour of systemd hasn't changed for a long time (a40eb73224e237 'unit: add DefaultDependencies= setting' Sat Jul 3 19:48:33 2010). So something in the units changed, most likely dbus.service gained a Requires dependency on the socket. But that's certainly allowed, such dependency is reasonable for the service. I cannot reproduce the issue with basic.target: the system boots into container-minimal.target without issue when it is masked. It's possible that something changed in the dependency tree meanwhile. I'll close this as NOTABUG: I think the issue is in the container-minimal.target and it's dependencies. I'm afraid you'll have to mask individual services or ask dbus maintainers to relax their dependencies.
Thank you for the investigating. Even in Fedora 23, the Requires dependency on dbus.socket is there: $ docker exec c-1329124 systemctl list-dependencies --all container-minimal.target | head container-minimal.target ● ├─dbus.service ● │ ├─dbus.socket ● │ │ ├─-.slice ● │ │ └─sysinit.target ● │ │ ├─dev-hugepages.mount ● │ │ ├─dev-mqueue.mount ● │ │ │ └─system.slice ● │ │ │ └─-.slice ● │ │ ├─kmod-static-nodes.service write /dev/stdout: broken pipe $ docker exec c-1329124 systemctl status sysinit.target ● sysinit.target Loaded: masked (/dev/null) Active: inactive (dead) $ docker exec c-1329124 grep Requires /usr/lib/systemd/system/dbus.service Requires=dbus.socket And yet the container starts just fine, with sysinit.target. On fedora:rawhide it fails. What has changed?
You are right about the basic.target -- I am no longer able to see the breakage with basic.target masked.
Oops, it seems I broke it in some refactoring and later on it got fixed. See https://github.com/systemd/systemd/issues/2315 in some history. So current behaviour is as designed. Also, the following commit stopped masking basic.target stopped being a problem: commit cb4c247d48fc195e64dd895a4e9dc8162ae62b23 Author: Lennart Poettering <lennart> Date: Wed Nov 11 16:22:25 2015 +0100 core: change default deps of services to require sysinit.target instead of basic.target
So how can dbus.service or dbus.socket relax the dependency on sysinit.target, in containers? Adding DefaultDependencies=false to either of them does not seem to prevent sysinit.target from being Require'd.
(In reply to Jan Pazdziora from comment #8) > Adding DefaultDependencies=false to either of them does not seem to prevent > sysinit.target from being Require'd. It seems to work here. If it doesn't work for you, please provide a minimal test case and describe exactly what you did and what happened. > So how can dbus.service or dbus.socket relax the dependency on > sysinit.target, in containers? This is a wrong approach. Those dependencies are there for a reason. Instead, you should try to reduce the number of services started as sysinit.target. But there's no magic bullet: each one should be considered and evaluated on its own.
Looking at the original trace from #c0, it seems that systemd-udev is installed in the container: it most likely should not be. Those are the only things on that list which have noticable impact. Also: https://github.com/systemd/systemd/pull/3941.
(In reply to Zbigniew Jędrzejewski-Szmek from comment #9) > (In reply to Jan Pazdziora from comment #8) > > Adding DefaultDependencies=false to either of them does not seem to prevent > > sysinit.target from being Require'd. > It seems to work here. If it doesn't work for you, please provide a minimal > test case and describe exactly what you did and what happened. container-minimal.target file: [Unit] Description=Minimal target for systemd-based containers DefaultDependencies=false AllowIsolate=yes Requires=systemd-tmpfiles-setup.service systemd-journald.service dbus.service After=systemd-tmpfiles-setup.service systemd-journald.service dbus.service Dockerfile: FROM fedora:rawhide ENV container docker VOLUME /tmp /run ENTRYPOINT /usr/sbin/init COPY container-minimal.target /usr/lib/systemd/system/ RUN systemctl set-default container-minimal.target || : # RUN for i in basic.target sysinit.target network.service netconsole.service ; do rm -f /usr/lib/systemd/system/$i && ln -s /dev/null /usr/lib/systemd/system/$i ; done RUN for i in basic.target network.service netconsole.service ; do rm -f /usr/lib/systemd/system/$i && ln -s /dev/null /usr/lib/systemd/system/$i ; done LABEL RUN "docker run -ti -v /sys/fs/cgroup:/sys/fs/cgroup:ro ${NAME}" RUN echo '[Unit]' >> /usr/lib/systemd/user/dbus.service RUN echo DefaultDependencies=false >> /usr/lib/systemd/user/dbus.service RUN echo '[Unit]' >> /usr/lib/systemd/user/dbus.socket RUN echo DefaultDependencies=false >> /usr/lib/systemd/user/dbus.socket The goal is to stop dbus.service and dbus.socket from loading sysinit.target. I build image with docker build -t bz-1329124 . I run container with docker run --name c-1329124 -ti -v /sys/fs/cgroup:/sys/fs/cgroup:ro bz-1329124 In other terminal, I run docker exec c-1329124 systemctl list-dependencies --all container-minimal.target | head and I get container-minimal.target ● ├─dbus.service ● │ ├─dbus.socket ● │ │ ├─-.slice ● │ │ └─sysinit.target ● │ │ ├─dev-hugepages.mount ● │ │ ├─dev-mqueue.mount ● │ │ │ └─system.slice ● │ │ │ └─-.slice ● │ │ ├─ldconfig.service > > So how can dbus.service or dbus.socket relax the dependency on > > sysinit.target, in containers? > This is a wrong approach. Those dependencies are there for a reason. > Instead, you should try to reduce the number of services started as > sysinit.target. But there's no magic bullet: each one should be considered > and evaluated on its own. I'd much prefer to start with clean table and add service as (when) needed, instead of having to continuously watch what new versions of software and distributions add and purge that. I've spent many months trying to get systemd running in containers as minimal environment, with minimal set of services / targets. I believe the container-minimal.target does all I need for my purposes (FreeIPA in container). Now I need way to prevent systemd from automagically adding things back.
(In reply to Zbigniew Jędrzejewski-Szmek from comment #10) > Looking at the original trace from #c0, it seems that systemd-udev is > installed in the container: it most likely should not be. Those are the only > things on that list which have noticable impact. It does not seem to be the case: $ docker exec c-1329124 rpm -q systemd-udev package systemd-udev is not installed for container started from image from Dockerfile in comment 11.
RUN echo '[Unit]' >> /usr/lib/systemd/user/dbus.service RUN echo DefaultDependencies=false >> /usr/lib/systemd/user/dbus.service RUN echo '[Unit]' >> /usr/lib/systemd/user/dbus.socket RUN echo DefaultDependencies=false >> /usr/lib/systemd/user/dbus.socket dbus.{socket,service} are not user units, so the path is wrong. > $ docker exec c-1329124 rpm -q systemd-udev > package systemd-udev is not installed Good. systemd-udev was split out in F24 (IIRC), so you should be OK.
Btw., please don't treat my previous comment as suggesting that I think that adding DefaultDependencies=no to dbus units is a good idea. :]]
(In reply to Zbigniew Jędrzejewski-Szmek from comment #13) > RUN echo '[Unit]' >> /usr/lib/systemd/user/dbus.service > RUN echo DefaultDependencies=false >> /usr/lib/systemd/user/dbus.service > RUN echo '[Unit]' >> /usr/lib/systemd/user/dbus.socket > RUN echo DefaultDependencies=false >> /usr/lib/systemd/user/dbus.socket > > dbus.{socket,service} are not user units, so the path is wrong. Ah, sorry for that. The Dockerfile should be FROM fedora:rawhide ENV container docker VOLUME /tmp /run ENTRYPOINT /usr/sbin/init COPY container-minimal.target /usr/lib/systemd/system/ RUN systemctl set-default container-minimal.target || : # RUN for i in basic.target sysinit.target network.service netconsole.service ; do rm -f /usr/lib/systemd/system/$i && ln -s /dev/null /usr/lib/systemd/system/$i ; done RUN for i in basic.target network.service netconsole.service ; do rm -f /usr/lib/systemd/system/$i && ln -s /dev/null /usr/lib/systemd/system/$i ; done LABEL RUN "docker run -ti -v /sys/fs/cgroup:/sys/fs/cgroup:ro ${NAME}" RUN echo '[Unit]' >> /usr/lib/systemd/system/dbus.service RUN echo DefaultDependencies=false >> /usr/lib/systemd/system/dbus.service RUN echo '[Unit]' >> /usr/lib/systemd/system/dbus.socket RUN echo DefaultDependencies=false >> /usr/lib/systemd/system/dbus.socket With that, I see $ docker exec c-1329124 systemctl list-dependencies --all container-minimal.target container-minimal.target ● ├─dbus.service ● │ ├─dbus.socket ● │ │ └─-.slice ● │ ├─system.slice ● │ │ └─-.slice ● │ └─sysinit.target ● │ ├─dev-hugepages.mount ● │ ├─dev-mqueue.mount ● │ │ └─system.slice ● │ │ └─-.slice ● │ ├─ldconfig.service ● │ │ └─system.slice ● │ │ └─-.slice So the dependency of sysinit.target on dbus.socket is gone but the dependency on dbus.service is still there. How do I remove that?
BTW, the dbus.service file is: $ docker exec c-1329124 cat /usr/lib/systemd/system/dbus.service [Unit] Description=D-Bus System Message Bus Documentation=man:dbus-daemon(1) Requires=dbus.socket [Service] ExecStart=/usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation ExecReload=/usr/bin/dbus-send --print-reply --system --type=method_call --dest=org.freedesktop.DBus / org.freedesktop.DBus.ReloadConfig OOMScoreAdjust=-900 [Unit] DefaultDependencies=false And I see $ docker exec c-1329124 systemctl list-dependencies --all dbus.service | head dbus.service ● ├─dbus.socket ● │ └─-.slice ● ├─system.slice ● │ └─-.slice ● └─sysinit.target ● ├─dev-hugepages.mount ● ├─dev-mqueue.mount ● │ └─system.slice ● │ └─-.slice
(In reply to Zbigniew Jędrzejewski-Szmek from comment #14) > Btw., please don't treat my previous comment as suggesting that I think that > adding DefaultDependencies=no to dbus units is a good idea. :]] I hear you. I appreciate you taking time to investigate and explain the systemd behaviour. In comment 4 you mentioned that DefaultDependencies drive the default dependency of .socket units on sysinit.target and yes, we are able to remove that with DefaultDependencies=no/false. I still want to make sure I'm able to control the dependencies so I'd like to be able to break the dbus.service -- sysinit.target dependency. Could you help me figure out what seems to be creating that?
Hm, that's really strange. DefaultDependencies=no seems to work fine here. I don't have the exact same systemd version and I'm not using docker, but this shouldn't matter. Can you check the logs that for any parsing errors or such?
Ah, now I see the problem -- it's caused by bug 1343136: $ docker run --rm -ti fedora:rawhide ls -la /etc/systemd/system/dbus.service -rw-r--r--. 1 root root 347 May 24 08:31 /etc/systemd/system/dbus.service I was changing file in /usr/lib which was not used at all. Sorry for the noise -- now I have container setup without sysinit.target. Thank you, jan