Bug 1401537

Merely running bash fails as well:

# docker run --rm -ti -e container=docker fedora:25 bash
/usr/bin/docker-current: Error response from daemon: invalid header field value "oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:359: container init caused \\\"rootfs_linux.go:54: mounting \\\\\\\"tmpfs\\\\\\\" to rootfs \\\\\\\"/var/lib/docker/886432.886432/devicemapper/mnt/41080130319e94ac09eeafc4e6fe9a405ec8fe1aa769fcadc381f7f12b6bfaaf/rootfs\\\\\\\" at \\\\\\\"/dev\\\\\\\" caused \\\\\\\"permission denied\\\\\\\"\\\"\"\n".

For the record, docker-1.10.3-53.el7.x86_64 on fails as well:

# docker run --rm -ti rhel7 bash
docker: Error response from daemon: Cannot start container 184efbed1030c311ff54877bb5a1c86c2b3eac5f564c974cc15ea708e8f1af3d: [9] System error: fork/exec /proc/self/exe: invalid argument.

Mrunal do you remember anything about user ns in our projectatomic/docker? I remember there was still something not supported

Antonio, It looks like this is related to SELinux. If you take out the selinux build tag then it works fine. 
Also, this is reproducible with docker upstream on Fedora 25. 

Dan, could you take a look?

Is it the /dev/mqueue issue?

Does it work if you run the daemon without --selinux-enabled?

Are there any AVCs?

I know there is an issue with /dev/mqueu not working correctly with user namespace which prevents the SELinux label being set on /dev/mqueue when running with user namespace.

Dan, this isn't related to mqueue from what I can tell. Also, it works without --selinux-enabled and I don't see any AVCs using ausearch -m avc -ts recent.

This is an issue of mounting a tmpfs while in a user namespace using a context="container_file_t" issue.

Basically usernamespace is not allowed to put arbitrary SELinux labels on a mount point.

The question is why does the tmpfs have to be mounted within the UserNamespace?

Opened an issue with runc.

https://github.com/opencontainers/runc/issues/1215

(In reply to Mrunal Patel from comment #7)
> Dan, this isn't related to mqueue from what I can tell. Also, it works
> without --selinux-enabled and I don't see any AVCs using ausearch -m avc -ts
> recent.

Confirming, removing --selinux-enabled helps. And there are no AVC denials logged.

Do we need another bugzilla for tracking the lack of AVC denials in this situation?

Jan this is not an SELinux denial. This is a denial of a user namespace process without SYS_ADMIN attempting to set a label on a tmpfs mount point.  I see this as a runc bug.  Runc can not be mounting the tmpfs's while under a non root user namespace.

(In reply to Daniel Walsh from comment #11)
> Jan this is not an SELinux denial.

OK, thanks.

For systemd in container, we likely need to be able to mount /sys/fs/cgroup to the container. I've disabled oci-systemd-hook (to workaround bug 1401944) but when I try to mount the /sys/fs/cgroup explicitly, it fails:

$ docker run -v /sys/fs/cgroup:/sys/fs/cgroup:ro --rm -ti fedora:25 bash
/usr/bin/docker-current: Error response from daemon: invalid header field value "oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:359: container init caused \\\"rootfs_linux.go:54: mounting \\\\\\\"/sys/fs/cgroup\\\\\\\" to rootfs \\\\\\\"/var/lib/docker/886432.886432/devicemapper/mnt/b79ef8835182d4ef2ccf8dc0f5ef3dfab9d806d1ee83c1213377f7f7c4913761/rootfs\\\\\\\" at \\\\\\\"/var/lib/docker/886432.886432/devicemapper/mnt/b79ef8835182d4ef2ccf8dc0f5ef3dfab9d806d1ee83c1213377f7f7c4913761/rootfs/sys/fs/cgroup\\\\\\\" caused \\\\\\\"operation not permitted\\\\\\\"\\\"\"\n".

Is it the same issue or would you like separate bugzilla for the /sys/fs/cgroup case?

(In reply to Jan Pazdziora from comment #13)
> For systemd in container, we likely need to be able to mount /sys/fs/cgroup
> to the container. I've disabled oci-systemd-hook (to workaround bug 1401944)
> but when I try to mount the /sys/fs/cgroup explicitly, it fails:
> 
> $ docker run -v /sys/fs/cgroup:/sys/fs/cgroup:ro --rm -ti fedora:25 bash
> /usr/bin/docker-current: Error response from daemon: invalid header field
> value "oci runtime error: container_linux.go:247: starting container process
> caused \"process_linux.go:359: container init caused \\\"rootfs_linux.go:54:
> mounting \\\\\\\"/sys/fs/cgroup\\\\\\\" to rootfs
> \\\\\\\"/var/lib/docker/886432.886432/devicemapper/mnt/
> b79ef8835182d4ef2ccf8dc0f5ef3dfab9d806d1ee83c1213377f7f7c4913761/
> rootfs\\\\\\\" at
> \\\\\\\"/var/lib/docker/886432.886432/devicemapper/mnt/
> b79ef8835182d4ef2ccf8dc0f5ef3dfab9d806d1ee83c1213377f7f7c4913761/rootfs/sys/
> fs/cgroup\\\\\\\" caused \\\\\\\"operation not permitted\\\\\\\"\\\"\"\n".
> 
> Is it the same issue or would you like separate bugzilla for the
> /sys/fs/cgroup case?

This issue is tracked upstream at https://github.com/docker/docker/issues/27202 and here in bugzilla as well https://bugzilla.redhat.com/show_bug.cgi?id=1373780

(In reply to Antonio Murdaca from comment #14)
> (In reply to Jan Pazdziora from comment #13)
> > 
> > $ docker run -v /sys/fs/cgroup:/sys/fs/cgroup:ro --rm -ti fedora:25 bash
> > /usr/bin/docker-current: Error response from daemon: invalid header field
> > value "oci runtime error: container_linux.go:247: starting container process
> > caused \"process_linux.go:359: container init caused \\\"rootfs_linux.go:54:
> > mounting \\\\\\\"/sys/fs/cgroup\\\\\\\" to rootfs
> > \\\\\\\"/var/lib/docker/886432.886432/devicemapper/mnt/
> > b79ef8835182d4ef2ccf8dc0f5ef3dfab9d806d1ee83c1213377f7f7c4913761/
> > rootfs\\\\\\\" at
> > \\\\\\\"/var/lib/docker/886432.886432/devicemapper/mnt/
> > b79ef8835182d4ef2ccf8dc0f5ef3dfab9d806d1ee83c1213377f7f7c4913761/rootfs/sys/
> > fs/cgroup\\\\\\\" caused \\\\\\\"operation not permitted\\\\\\\"\\\"\"\n".
> > 
> > Is it the same issue or would you like separate bugzilla for the
> > /sys/fs/cgroup case?
> 
> This issue is tracked upstream at
> https://github.com/docker/docker/issues/27202 and here in bugzilla as well
> https://bugzilla.redhat.com/show_bug.cgi?id=1373780

That issue is about console, not about /sys/fs/cgroup. Note that the command I run here is

docker run -v /sys/fs/cgroup:/sys/fs/cgroup:ro --rm -ti fedora:25 bash

-- I specify bash to be run in the container, there is no systemd involved. Of course, I got to this case by minimizing systemd attempts but I believe this is solely about mounting stuff with --userns-remap.

If you try the upstream issue and the bugzilla I linked you'll get exactly the same issue you're having. See also Mrunal's comment in the upstream issue.

(In reply to Antonio Murdaca from comment #16)
> If you try the upstream issue and the bugzilla I linked you'll get exactly
> the same issue you're having. See also Mrunal's comment in the upstream
> issue.

I that bugzilla https://bugzilla.redhat.com/show_bug.cgi?id=1373780 I point out that the console issue is a regression in fedora:25 and fedora:rawhide against fedora:24 image. That bugzilla setup also does not use --userns-remap.

This issue (from comment 13) I can reproduce on fedora:24:

# docker run -v /sys/fs/cgroup:/sys/fs/cgroup:ro --rm -ti fedora:24 bash
/usr/bin/docker-current: Error response from daemon: invalid header field value "oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:359: container init caused \\\"rootfs_linux.go:54: mounting \\\\\\\"/sys/fs/cgroup\\\\\\\" to rootfs \\\\\\\"/var/lib/docker/886432.886432/devicemapper/mnt/8c7c02a8a7c9d5f36a74540c2001d39be5b4e19c8b930bb5d89a12fe5c285973/rootfs\\\\\\\" at \\\\\\\"/var/lib/docker/886432.886432/devicemapper/mnt/8c7c02a8a7c9d5f36a74540c2001d39be5b4e19c8b930bb5d89a12fe5c285973/rootfs/sys/fs/cgroup\\\\\\\" caused \\\\\\\"operation not permitted\\\\\\\"\\\"\"\n".

And it only happens when I use --userns-remap. So why the problem from this bugzilla's comment 0 or comment 13 should be duplicate of bug 1373780 is really not obvious.

It well may be but to an outsider there don't seem much overlap. Could you elaborate some more?

I agree with Jan, that this is a different issue.  Nothing to do with systemd.

I see at least two different bugs.

oci-systemd-hook does not work with User Namespace, and volume mounting /sys/fs/cgroup into a container does not work with User Namespace.  (They could be the same issue, since both attempt to mount /sys/fs/cgroup into the container.)

I wonder if mounting other file systems into the container could cause the same issue.

If you volume mount in /sys/fs/selinux does that crash the container also?

Is this related at all to Linux kernel commit aad82892af261b9903cc11c55be3ecf5f0b0b4f8 ("selinux: Add support for unprivileged mounts from user namespaces") circa Linux 4.8?  If so, that change does cause SELinux to reject context mounts on user namespace mounts, to prevent unprivileged processes from performing context mounts.

Would seem to make sense.

Why does this need to be a context mount?

Interesting find: while

$ docker run -v /sys/fs/cgroup:/sys/fs/cgroup:ro --rm -ti fedora:24 bash

fails, specifying the mount without :ro passes:

$ docker run -v /sys/fs/cgroup:/sys/fs/cgroup --rm -ti fedora:24 bash
[root@3b65038bb538 /]#

Mounting /sys/fs/selinux passes both for the :ro and no-ro case:

# docker run -v /sys/fs/selinux:/sys/fs/selinux:ro --rm -ti fedora:24 bash
[root@834a43d4e491 /]# exit
# docker run -v /sys/fs/selinux:/sys/fs/selinux --rm -ti fedora:24 bash
[root@76e49ad37f62 /]# exit

Wow that is weird.

Steven this does not need to be a context mount, the only thing we need is the tmpfs to be labeled with the Container MountLabel.  Theoretically we coould just do a setfilecon(/dev, mountlabel)  But I am not sure this will succeed.  If it should based on SELinux labels, then perhaps the kernel change is wrong.  And a non priv process should be able to label the mount object at mount time, if SELinux policy allows him to set the SELinux label.

I think this is wrong.

http://kernel.suse.com/cgit/kernel/commit/?id=aad82892af261b9903cc11c55be3ecf5f0b0b4f8

Why should we be controlling the ability to specify a context mount based on DAC issues,  It should be controlled by MAC.

So this bugzilla is primarily about the oci-systemd-hook issue, while mounting of /sys/fs/cgroup:/sys/fs/cgroup:ro is explicitly tracked in bug 1401944.

Thanks.

Mrunal, just circling back on older bugzillas.  I believe the problem here was runc not matching the options of the mounted file system, when it remounts it read/only.  Do you know if this has been fixed in the latest runc?

This looks to be fixed with the latest oci-systemd-hook.  Asking QA to verify my testing.

[root@localhost ~]# vi /etc/sysconfig/docker
[root@localhost ~]# cat /etc/sysconfig/docker
# /etc/sysconfig/docker

# Modify these options if you want to change the way the docker daemon runs
#OPTIONS='--selinux-enabled --log-driver=journald'
OPTIONS='--icc=false --selinux-enabled --log-driver=journald --userns-remap=default'
# (Further lines edited out for brevity)
[root@localhost ~]# cat test/Dockerfile
FROM fedora:25
RUN dnf install -y /usr/bin/ps

[root@localhost ~]# systemctl restart docker
[root@localhost ~]# vi Dockerfile
[root@localhost ~]# pwd
/root
[root@localhost ~]# mkdir test
[root@localhost ~]# mv Dockerfile test/
[root@localhost ~]# ls test
Dockerfile
[root@localhost ~]# docker build -t test ~/test/
Sending build context to Docker daemon 2.048 kB
Step 1 : FROM fedora:25
Trying to pull repository docker.io/library/fedora ... 
sha256:b44cdaee0feafc85cab454e2023807f66725c727655b6bef260aa6d21dd2b068: Pulling from docker.io/library/fedora
bc5187a39b05: Pull complete 
Digest: sha256:b44cdaee0feafc85cab454e2023807f66725c727655b6bef260aa6d21dd2b068
Status: Downloaded newer image for docker.io/fedora:25
 ---> 4daa661b467f
Step 2 : RUN dnf install -y /usr/bin/ps
 ---> Running in 16afdaafe1b0
Last metadata expiration check: 0:00:21 ago on Sat Apr  1 19:00:44 2017.
Dependencies resolved.
================================================================================
 Package           Arch           Version                  Repository      Size
================================================================================
Installing:
 procps-ng         x86_64         3.3.10-11.fc24           fedora         389 k

Transaction Summary
================================================================================
Install  1 Package

Total download size: 389 k
Installed size: 873 k
Downloading Packages:
--------------------------------------------------------------------------------
Total                                           684 kB/s | 389 kB     00:00     
warning: /var/cache/dnf/fedora-310f9d37d74ceec1/packages/procps-ng-3.3.10-11.fc24.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID fdb19c98: NOKEY
Importing GPG key 0xFDB19C98:
 Userid     : "Fedora 25 Primary (25) <fedora-25-primary>"
 Fingerprint: C437 DCCD 558A 66A3 7D6F 4372 4089 D8F2 FDB1 9C98
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-25-x86_64
Key imported successfully
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Installing  : procps-ng-3.3.10-11.fc24.x86_64                             1/1 
error: unpacking of archive failed on file /usr/bin/free;58dff8f5: cpio: open
Error unpacking rpm package procps-ng-3.3.10-11.fc24.x86_64
Error unpacking rpm package procps-ng-3.3.10-11.fc24.x86_64
procps-ng-3.3.10-11.fc24.x86_64 was supposed to be installed but is not!
  Verifying   : procps-ng-3.3.10-11.fc24.x86_64                             1/1 

Failed:
  procps-ng.x86_64 3.3.10-11.fc24                                               

Complete!
 ---> 3b27323a5dd8
Removing intermediate container 16afdaafe1b0
Successfully built 3b27323a5dd8
[root@localhost ~]# docker build -t test ~/test/
Sending build context to Docker daemon 2.048 kB
Step 1 : FROM fedora:25
 ---> 4daa661b467f
Step 2 : RUN dnf install -y /usr/bin/ps
 ---> Using cache
 ---> 3b27323a5dd8
Successfully built 3b27323a5dd8

[root@localhost ~]# rpm -qa oci-systemd-hook
oci-systemd-hook-0.1.6-1.gitfe22236.fc25.x86_64[root@localhost ~]# uname -a
Linux localhost.localdomain 4.10.5-200.fc25.x86_64 #1 SMP Wed Mar 22 20:37:08 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux[root@localhost ~]# cat /etc/*release
Fedora release 25 (Twenty Five)
NAME=Fedora
VERSION="25 (Server Edition)"
ID=fedora
VERSION_ID=25
PRETTY_NAME="Fedora 25 (Server Edition)"

(In reply to Tom Sweeney from comment #31)

This

> Transaction test succeeded.
> Running transaction
>   Installing  : procps-ng-3.3.10-11.fc24.x86_64                            
> 1/1 
> error: unpacking of archive failed on file /usr/bin/free;58dff8f5: cpio: open
> Error unpacking rpm package procps-ng-3.3.10-11.fc24.x86_64
> Error unpacking rpm package procps-ng-3.3.10-11.fc24.x86_64
> procps-ng-3.3.10-11.fc24.x86_64 was supposed to be installed but is not!

causes for ps not to be actually installed even if the subsequent docker run passes without error:

> [root@localhost ~]# docker build -t test ~/test/
> Sending build context to Docker daemon 2.048 kB
> Step 1 : FROM fedora:25
>  ---> 4daa661b467f
> Step 2 : RUN dnf install -y /usr/bin/ps
>  ---> Using cache
>  ---> 3b27323a5dd8
> Successfully built 3b27323a5dd8

# docker run --rm -ti test ps axuwwf
/usr/bin/docker-current: Error response from daemon: invalid header field value "oci runtime error: container_linux.go:247: starting container process caused \"exec: \\\"ps\\\": executable file not found in $PATH\"\n".

Why can't /usr/bin/free be installed?

OTOH,

   docker run --rm -ti -e container=docker fedora:25 bash

now passes. Do we want separate bugzilla about the installation of /usr/bin/free under --userns-remap?

(In reply to Jan Pazdziora from comment #32)
>
> causes for ps not to be actually installed even if the subsequent docker run
> passes without error:

I wanted to say "even if the subsequente docker *build* passes".

The build seemingly passes but docker run then fails.

Are these errors about oci-systemd-hook or something else?

(In reply to Daniel Walsh from comment #34)
> Are these errors about oci-systemd-hook or something else?

They don't seem to be oci-systemd-hook-related, rather a dac_override AVC denial. Filed bug 1439587 for that.

I see

docker run --rm -ti -e container=docker fedora:25 bash

passing now with released

oci-systemd-hook-0.1.7-1.git1788cf2.fc25.x86_64

and

docker-1.12.6-6.gitae7d637.fc25.x86_64
container-selinux-2.10-1.fc25.noarch
selinux-policy-3.13.1-225.13.fc25.noarch
kernel-4.10.14-200.fc25.x86_64

Could we get Fixed In Version filled and this bugzilla closed, ideally with an errata?

oci-systemd-hook-0.1.9-1.gitaa42622.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-8c49b86e6c

oci-systemd-hook-0.1.9-1.gitaa42622.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-8b041abf0a

oci-systemd-hook-0.1.9-1.gitaa42622.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-8b041abf0a

oci-systemd-hook-0.1.9-1.gitaa42622.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-8c49b86e6c

oci-systemd-hook-0.1.9-1.gitaa42622.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.

oci-systemd-hook-0.1.9-1.gitaa42622.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days