Bug 1756143

Summary: after running 'mock', /proc/self/cgroup contains name=systemd. intentional?
Product: [Fedora] Fedora Reporter: Cole Robinson <crobinso>
Component: systemdAssignee: systemd-maint
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 31CC: ego.cordatus, lnykryn, msekleta, praiskup, ssahani, s, systemd-maint, vascom2, yaneti, zbyszek
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Fixed In Version: systemd-243-3.gitef67743.fc31 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-10-26 17:24:42 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Cole Robinson 2019-09-26 20:45:45 UTC
I'm not sure if this is a bug or intentional behavior, so I'm looking for some advice.

$ uname -a
Linux worklaptop 5.3.0-1.fc31.x86_64 #1 SMP Mon Sep 16 12:34:42 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
$ rpm -q systemd
$ mount | grep cgroup
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,seclabel,nsdelegate)
$ cat /proc/self/cgroup

After I run a mock build (via fedpkg mockbuild), the last bit changes
$ cat /proc/self/cgroup

This broke libvirt VM startup: https://bugzilla.redhat.com/show_bug.cgi?id=1751120

podman container startup using crun also seems broken:

$ podman run -it alpine sh
Error: creating cgroup directory '/sys/fs/cgroup/name=systemd/user.slice/user-1000.slice/user@1000.service/user.slice/libpod-69952a2d396138de9ffa0d1a5e096ae2d6382c6e17c93a0baf53b762a8f83ff9.scope': Permission denied: OCI runtime error

So is this a systemd bug, or do both those apps need to adjust their expectations? Also is their some way to 'undo' this situation without rebooting the host, as a temporary workaround?

Comment 1 Zbigniew Jędrzejewski-Szmek 2019-09-27 13:03:48 UTC
mock doesn't matter here, plain systemd-nspawn invocation is enough to cause this
(with the right image).

systemd-nspawn has code to detect if the image it is about to start has an init that
has support for the unified cgroup hierarchy. Specifically, it checks if systemd>230
is installed. If it detects no support, if falls back to mounting legacy hierarchy.
This is a bit ugly, but in the beginning almost nothing had support for the unified
hierarchy, and we didn't want to break spawning of existing images. Maybe this should
be revisited, but I don't think there's any easy solution if we want to keep compatibility
with older images.

With recent buildroot dependency cleanups, systemd may no longer be installed by mock,
so suddenly systemd-nspawn detects no support and falls back to legacy in the booted
image. Once that happens, I don't think it is possible to undo without rebooting.

It is possible to override the detection by setting UNIFIED_CGROUP_HIERARCHY=yes.
You can use that as a temporary work-around.

That said, mock actually calls systemd-nspawn with --as-pid2/-a. In this mode, nspawn
"knows" that a compatible init will be running (because it is it), so we should
skip the detection based on the image. I'll prep a patch.

Comment 2 Fedora Update System 2019-10-10 20:46:35 UTC
FEDORA-2019-5cb48e5111 has been submitted as an update to Fedora 31. https://bodhi.fedoraproject.org/updates/FEDORA-2019-5cb48e5111

Comment 3 Fedora Update System 2019-10-11 16:53:38 UTC
systemd-243-3.gitef67743.fc31 has been pushed to the Fedora 31 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-5cb48e5111

Comment 4 Artem 2019-10-14 07:07:23 UTC
RHBZ#1756143 not fixed. I did ~40 builds and now it hangs again.

Comment 5 Zbigniew Jędrzejewski-Szmek 2019-10-14 07:40:19 UTC
This bug is about a very simple thing: whether 'systemd-nspawn --as-pid2' mounts
v1 or v2 hierarchy in the container. It is fully deterministic, without any race conditions,
so if you're seeing hangs when doing a large number of builds, then it's most likely
some different thing.

Comment 6 Artem 2019-10-14 07:43:54 UTC
Oh, sorry. Maybe we should reopen #1754807 then?


@Vascom wrote that it still happens as well.

Comment 7 Vasiliy Glazov 2019-10-14 07:46:02 UTC
#1754807 still persist.

Comment 8 Fedora Update System 2019-10-26 17:24:42 UTC
systemd-243-3.gitef67743.fc31 has been pushed to the Fedora 31 stable repository. If problems still persist, please make note of it in this bug report.