I'm not sure if this is a bug or intentional behavior, so I'm looking for some advice. $ uname -a Linux worklaptop 5.3.0-1.fc31.x86_64 #1 SMP Mon Sep 16 12:34:42 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux $ rpm -q systemd systemd-243-2.gitfab6f01.fc31.x86_64 $ mount | grep cgroup cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,seclabel,nsdelegate) $ cat /proc/self/cgroup 0::/user.slice/user-1000.slice/... After I run a mock build (via fedpkg mockbuild), the last bit changes $ cat /proc/self/cgroup 1:name=systemd:/ 0::/user.slice/user-1000.slice/... This broke libvirt VM startup: https://bugzilla.redhat.com/show_bug.cgi?id=1751120 podman container startup using crun also seems broken: $ podman run -it alpine sh Error: creating cgroup directory '/sys/fs/cgroup/name=systemd/user.slice/user-1000.slice/user/user.slice/libpod-69952a2d396138de9ffa0d1a5e096ae2d6382c6e17c93a0baf53b762a8f83ff9.scope': Permission denied: OCI runtime error So is this a systemd bug, or do both those apps need to adjust their expectations? Also is their some way to 'undo' this situation without rebooting the host, as a temporary workaround?
mock doesn't matter here, plain systemd-nspawn invocation is enough to cause this (with the right image). systemd-nspawn has code to detect if the image it is about to start has an init that has support for the unified cgroup hierarchy. Specifically, it checks if systemd>230 is installed. If it detects no support, if falls back to mounting legacy hierarchy. This is a bit ugly, but in the beginning almost nothing had support for the unified hierarchy, and we didn't want to break spawning of existing images. Maybe this should be revisited, but I don't think there's any easy solution if we want to keep compatibility with older images. With recent buildroot dependency cleanups, systemd may no longer be installed by mock, so suddenly systemd-nspawn detects no support and falls back to legacy in the booted image. Once that happens, I don't think it is possible to undo without rebooting. It is possible to override the detection by setting UNIFIED_CGROUP_HIERARCHY=yes. You can use that as a temporary work-around. That said, mock actually calls systemd-nspawn with --as-pid2/-a. In this mode, nspawn "knows" that a compatible init will be running (because it is it), so we should skip the detection based on the image. I'll prep a patch.
FEDORA-2019-5cb48e5111 has been submitted as an update to Fedora 31. https://bodhi.fedoraproject.org/updates/FEDORA-2019-5cb48e5111
systemd-243-3.gitef67743.fc31 has been pushed to the Fedora 31 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-5cb48e5111
RHBZ#1756143 not fixed. I did ~40 builds and now it hangs again.
@Artem: This bug is about a very simple thing: whether 'systemd-nspawn --as-pid2' mounts v1 or v2 hierarchy in the container. It is fully deterministic, without any race conditions, so if you're seeing hangs when doing a large number of builds, then it's most likely some different thing.
Oh, sorry. Maybe we should reopen #1754807 then? https://bugzilla.redhat.com/show_bug.cgi?id=1754807 @Vascom wrote that it still happens as well.
#1754807 still persist.
systemd-243-3.gitef67743.fc31 has been pushed to the Fedora 31 stable repository. If problems still persist, please make note of it in this bug report.