Bug 1767850

Summary:	moby-engine fails to run systemd in container unless --tmpfs /run and -v /sys/fs/cgroup:... is specified
Product:	[Fedora] Fedora	Reporter:	Jan Pazdziora <jpazdziora>
Component:	moby-engine	Assignee:	Olivier Lemasle <o.lemasle>
Status:	CLOSED EOL	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	31	CC:	jpazdziora, o.lemasle
Target Milestone:	---	Keywords:	Regression
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-11-24 18:14:14 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Jan Pazdziora 2019-11-01 14:19:11 UTC

Description of problem:

With the docker daemon from docker-1.13.1-68.git47e2230.fc30.x86_64 installed on Fedora 31 booted with systemd.unified_cgroup_hierarchy=0 to workaround bug 1746355, it is possible to start a container running systemd with just

# docker run --rm -ti registry.fedoraproject.org/fedora:31 /usr/sbin/init
systemd v243-4.gitef67743.fc31 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=unified)
Detected virtualization container-other.
Detected architecture x86-64.

Welcome to Fedora 31 (Container Image)!

Set hostname to <971e3465fc86>.
[  OK  ] Started Dispatch Password …ts to Console Directory Watch.
[  OK  ] Started Forward Password R…uests to Wall Directory Watch.
[  OK  ] Reached target Local File Systems.
[  OK  ] Reached target Paths.
[  OK  ] Reached target Remote File Systems.
[  OK  ] Reached target Slices.
[  OK  ] Reached target Swap.
[  OK  ] Listening on Process Core Dump Socket.
[  OK  ] Listening on initctl Compatibility Named Pipe.
[  OK  ] Listening on Journal Socket (/dev/log).
[  OK  ] Listening on Journal Socket.
ldconfig.service: unit configures an IP firewall, but the local system does not support BPF/cgroup firewalling.
(This warning is only shown for the first unit using IP firewalling.)
         Starting Rebuild Dynamic Linker Cache...
         Starting Journal Service...
         Starting Create System Users...
[  OK  ] Started Create System Users.
[  OK  ] Started Rebuild Dynamic Linker Cache.
[  OK  ] Started Journal Service.

When the docker package gets replaced with moby-engine, the same command fails.

Version-Release number of selected component (if applicable):

moby-engine-18.09.8-2.ce.git0dd43dd.fc31.x86_64

How reproducible:

Deterministic.

Steps to Reproduce:
1. Have moby-engine-18.09.8-2.ce.git0dd43dd.fc31.x86_64 installed and docker.service provided by it running.
2. Run docker run --rm -ti registry.fedoraproject.org/fedora:31 /usr/sbin/init

Actual results:

Failed to mount tmpfs at /run: Operation not permitted
[!!!!!!] Failed to mount API filesystems.
Exiting PID 1...

Expected results:

systemd v243-4.gitef67743.fc31 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=unified)
Detected virtualization container-other.
Detected architecture x86-64.

Welcome to Fedora 31 (Container Image)!

Set hostname to <971e3465fc86>.
[  OK  ] Started Dispatch Password …ts to Console Directory Watch.
[  OK  ] Started Forward Password R…uests to Wall Directory Watch.
[  OK  ] Reached target Local File Systems.
[  OK  ] Reached target Paths.
[  OK  ] Reached target Remote File Systems.
[  OK  ] Reached target Slices.
[...]

Additional info:

When I add --tmpfs /run option to the docker run, I get

# docker run --rm -ti --tmpfs /run registry.fedoraproject.org/fedora:31 /usr/sbin/init
systemd v243-4.gitef67743.fc31 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=unified)
Detected virtualization container-other.
Detected architecture x86-64.

Welcome to Fedora 31 (Container Image)!

Set hostname to <afe15f905fac>.
Initializing machine ID from random generator.
Failed to create /system.slice/docker-afe15f905fac595614c5bb1dda729d6102d420417c4ed43f6c6f66a756519c4d.scope/init.scope control group: Read-only file system
Failed to allocate manager object: Read-only file system
[!!!!!!] Failed to allocate manager object.
Exiting PID 1...

Only when I add -v /sys/fs/cgroup:/sys/fs/cgroup:ro do I get somehow working systemd in the container:

# docker run --rm -ti --tmpfs /run -v /sys/fs/cgroup:/sys/fs/cgroup:ro registry.fedoraproject.org/fedora:31 /usr/sbin/init
systemd v243-4.gitef67743.fc31 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=unified)
Detected virtualization container-other.
Detected architecture x86-64.

Welcome to Fedora 31 (Container Image)!

Set hostname to <ffbe940928c7>.
Initializing machine ID from random generator.
[  OK  ] Started Dispatch Password Requests to Console Directory Watch.
[  OK  ] Started Forward Password Requests to Wall Directory Watch.
[  OK  ] Reached target Paths.
[  OK  ] Reached target Remote File Systems.
[  OK  ] Reached target Slices.
[  OK  ] Reached target Swap.
[  OK  ] Listening on Process Core Dump Socket.
[  OK  ] Listening on initctl Compatibility Named Pipe.
[  OK  ] Listening on Journal Socket (/dev/log).
[  OK  ] Listening on Journal Socket.
tmp.mount: Directory /tmp to mount over is not empty, mounting anyway.
         Mounting Temporary Directory (/tmp)...
Failed to set devices.allow on /system.slice/docker-ffbe940928c7406f700f36fe8b2e22bb377a811668b769ca0a64c150e9e84add.scope/system.slice/systemd-journald.service: Operation not permitted
         Starting Journal Service...
         Starting Create System Users...
tmp.mount: Mount process exited, code=exited, status=32/n/a
tmp.mount: Failed with result 'exit-code'.
[FAILED] Failed to mount Temporary Directory (/tmp).
See 'systemctl status tmp.mount' for details.
[  OK  ] Reached target Local File Systems.
         Starting Rebuild Dynamic Linker Cache...
[  OK  ] Started Create System Users.
[  OK  ] Started Rebuild Dynamic Linker Cache.
[  OK  ] Started Journal Service.
[...]

Installing oci-systemd-hook-0.2.0-2.git05e6923.fc31.x86_64 does not help.

Comment 1 Ben Cotton 2020-11-03 15:42:37 UTC

This message is a reminder that Fedora 31 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 31 on 2020-11-24.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '31'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 31 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 2 Ben Cotton 2020-11-24 18:14:14 UTC

Fedora 31 changed to end-of-life (EOL) status on 2020-11-24. Fedora 31 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.