Bug 1402264 - Running /usr/sbin/init in container fails with --userns-remap set
Summary: Running /usr/sbin/init in container fails with --userns-remap set
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: docker
Version: 26
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Tom Sweeney
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-12-07 07:18 UTC by Jan Pazdziora
Modified: 2018-05-10 16:40 UTC (History)
18 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-05-10 16:40:36 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Jan Pazdziora 2016-12-07 07:18:44 UTC
Description of problem:

Running

docker run --rm -ti -e container=docker -v /sys/fs/cgroup:/sys/fs/cgroup --tmpfs /run --tmpfs /tmp fedora:24 /usr/sbin/init

starts systemd but it then fails with

Failed to create /system.slice/docker-74aa860461f2033ffc6e9fc0f08ff37893f84adc9591e45fe2c113ff2f30952f.scope/init.scope control group: Permission denied
Failed to allocate manager object: Permission denied
[!!!!!!] Failed to allocate manager object, freezing.
Freezing execution.

Version-Release number of selected component (if applicable):

docker-1.12.3-10.git7b5044b.fc25.x86_64
kernel-4.8.10-300.fc25.x86_64

How reproducible:

Deterministic.

Steps to Reproduce:
1. Have docker installation.
2. Remove /usr/libexec/oci/hooks.d/oci-systemd-hook to avoid hitting bug 1401537.
3. In /etc/sysconfig/docker, set OPTIONS='--icc=false --log-driver=journald --userns-remap=default'
4. systemctl restart docker
5. docker run --rm -ti -e container=docker -v /sys/fs/cgroup:/sys/fs/cgroup --tmpfs /run --tmpfs /tmp fedora:24 /usr/sbin/init

Actual results:

# docker run --rm -ti -e container=docker -v /sys/fs/cgroup:/sys/fs/cgroup --tmpfs /run --tmpfs /tmp fedora:24 /usr/sbin/init
systemd 229 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
Detected virtualization docker.
Detected architecture x86-64.
Running with unpopulated /etc.

Welcome to Fedora 24 (Twenty Four)!

Set hostname to <74aa860461f2>.
Initializing machine ID from random generator.
Failed to read AF_UNIX datagram queue length, ignoring: No such file or directory
Failed to populate /etc with preset unit settings, ignoring: No such file or directory
Failed to create /system.slice/docker-74aa860461f2033ffc6e9fc0f08ff37893f84adc9591e45fe2c113ff2f30952f.scope/init.scope control group: Permission denied
Failed to allocate manager object: Permission denied
[!!!!!!] Failed to allocate manager object, freezing.
Freezing execution.

Expected results:

No error, systemd runs with namespaced uid.

Additional info:

We mount without :ro to avoid bug 1401944.

We disabled --selinux-enabled to avoid bug 1401537.

Comment 1 Jan Pazdziora 2017-05-15 12:08:47 UTC
With

docker-1.12.6-6.gitae7d637.fc25.x86_64
container-selinux-2.10-1.fc25.noarch
oci-systemd-hook-0.1.7-1.git1788cf2.fc25.x86_64

and

OPTIONS='--selinux-enabled --userns-remap=default'

docker run --rm -ti -e container=docker -v /sys/fs/cgroup:/sys/fs/cgroup --tmpfs /run --tmpfs /tmp fedora:24 id

passes but

docker run --rm -ti -e container=docker -v /sys/fs/cgroup:/sys/fs/cgroup --tmpfs /run --tmpfs /tmp fedora:24 /usr/sbin/init

fails with

/usr/bin/docker-current: Error response from daemon: invalid header field value "oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:334: running prestart hook 2 caused \\\"error running hook: exit status 1, stdout: , stderr: \\\"\"\n".

This is with /usr/libexec/oci/hooks.d/oci-systemd-hook in place (as bug 1401537 seems fixed in latest Fedora 25).

With the hook removed, the failure is still

systemd 229 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
Detected virtualization docker.
Detected architecture x86-64.
Running with unpopulated /etc.

Welcome to Fedora 24 (Twenty Four)!

Set hostname to <016ef784dada>.
Initializing machine ID from random generator.
Failed to read AF_UNIX datagram queue length, ignoring: No such file or directory
Failed to populate /etc with preset unit settings, ignoring: No such file or directory
Failed to create /system.slice/docker-016ef784dadaa93cc558c8bcd7269335f6124b1e9407096cb2171c5f99c2a021.scope/init.scope control group: Permission denied
Failed to allocate manager object: Permission denied
[!!!!!!] Failed to allocate manager object, freezing.
Freezing execution.

Comment 2 Fedora End Of Life 2017-11-16 19:24:03 UTC
This message is a reminder that Fedora 25 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 25. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '25'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 25 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

Comment 3 Jan Pazdziora 2017-12-07 11:17:34 UTC
The issue as described in comment 1 is still present in Fedora 26 with

docker-1.13.1-40.git877b6df.fc26.x86_64
selinux-policy-3.13.1-260.14.fc26.noarch
container-selinux-2.28-1.fc26.noarch
oci-systemd-hook-0.1.13-1.gitafe4b4a.fc26.x86_64

# docker run --rm -ti -e container=docker -v /sys/fs/cgroup:/sys/fs/cgroup --tmpfs /run --tmpfs /tmp fedora:26 id
uid=0(root) gid=0(root) groups=0(root)

# docker run --rm -ti -e container=docker -v /sys/fs/cgroup:/sys/fs/cgroup --tmpfs /run --tmpfs /tmp fedora:26 /usr/sbin/init
/usr/bin/docker-current: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "process_linux.go:339: running prestart hook 2 caused \"error running hook: exit status 1, stdout: , stderr: \"".

Therefore, it seems to be an oci-systemd-hook issue. In journal, I can see

oci-systemd-hook[3992]: systemdhook <error>: Failed to mkdir: /var/lib/docker/165536.165536/devicemapper/mnt/7ad6b276cf8f2e22e0cf78503756cd6910507a950e036f25fb47310d3dd1acd2/rootfs/run/lock: Value too large for defined data type

If I run just

# docker run --rm -ti fedora:26 /usr/sbin/init

I get the same error

# docker run --rm -ti fedora:26 /usr/sbin/init
/usr/bin/docker-current: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "process_linux.go:339: running prestart hook 2 caused \"error running hook: exit status 1, stdout: , stderr: \"".

and journal has

oci-systemd-hook[4130]: systemdhook <error>: Failed to remount /var/lib/docker/165536.165536/devicemapper/mnt/a8c07d93143fec71b724f4a3585e166dfae1a25b5af819fe7e01a522d24dc2e6/rootfs//sys/fs/cgroup/systemd readonly: Operation not permitted
oci-systemd-hook[4130]: systemdhook <error>: Failed to bind mount /sys/fs/cgroup/systemd on /var/lib/docker/165536.165536/devicemapper/mnt/a8c07d93143fec71b724f4a3585e166dfae1a25b5af819fe7e01a522d24dc2e6/rootfs//sys/fs/cgroup/systemd: Operation not permitted

This case seems to be tracked in bug 1401944.

So I propose to use this bugzilla to track the case when with

OPTIONS='--selinux-enabled --log-driver=journald --userns-remap=default'

in /etc/sysconfig/docker, and after

setsebool -P container_manage_cgroup 1

running

# docker run --rm -ti --tmpfs /run fedora:26 /usr/sbin/init

fails and produces

oci-systemd-hook[5403]: systemdhook <debug>: rootfs=/var/lib/docker/165536.165536/devicemapper/mnt/e90fc972b648dd4c8ecb9419ee6a5ae6b95962e0c585e0e15d9d7c28716b6c9a/rootfs
oci-systemd-hook[5403]: systemdhook <debug>: GID: 165536
oci-systemd-hook[5403]: systemdhook <debug>: UID: 165536
oci-systemd-hook[5403]: systemdhook <debug>: /run already present as a mount point in container configuration, skipping
systemdhook <error>: Failed to mkdir: /var/lib/docker/165536.165536/devicemapper/mnt/e90fc972b648dd4c8ecb9419ee6a5ae6b95962e0c585e0e15d9d7c28716b6c9a/rootfs/run/lock: Value too large for defined data type

unless we agree that users should not --tmpfs /run, ever.

Comment 4 Fedora End Of Life 2018-05-03 07:59:51 UTC
This message is a reminder that Fedora 26 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 26. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '26'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 26 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

Comment 5 Jan Pazdziora 2018-05-10 08:50:37 UTC
docker run --rm -ti -e container=docker -v /sys/fs/cgroup:/sys/fs/cgroup --tmpfs /run --tmpfs /tmp fedora:26 /usr/sbin/init

still fails with

/usr/bin/docker-current: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "process_linux.go:339: running prestart hook 2 caused \"error running hook: exit status 1, stdout: , stderr: \"".

but plain

docker run --rm fedora:26 /usr/sbin/init

works fine, with

docker-1.13.1-44.git584d391.fc26.x86_64
selinux-policy-3.13.1-260.20.fc26.noarch
container-selinux-2.40-1.fc26.noarch
oci-systemd-hook-0.1.15-1.git2d0b8a3.fc26.x86_64

So I guess this can be closed CURRENTRELEASE.


Note You need to log in before you can comment on or make changes to this bug.