Description of problem: In an attempt to harden my docker installation, I've set OPTIONS='--icc=false --selinux-enabled --log-driver=journald --userns-remap=default' appending the --userns-remap option. After restarting the daemon, I can no longer build images. Version-Release number of selected component (if applicable): docker-1.12.3-10.git7b5044b.fc25.x86_64 How reproducible: Deterministic. Steps to Reproduce: 1. Set /etc/sysconfig/docker option OPTIONS to OPTIONS='--icc=false --selinux-enabled --log-driver=journald --userns-remap=default' 2. systemctl restart docker 3. Have test/Dockerfile FROM fedora:25 RUN dnf install -y /usr/bin/ps 4. docker build -t test ~/test/ Actual results: # docker build -t test ~/test/ Sending build context to Docker daemon 2.048 kB Step 1 : FROM fedora:25 ---> 5bafccc4d5dc Step 2 : RUN dnf install -y /usr/bin/ps ---> Running in e8b291d9d285 invalid header field value "oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:359: container init caused \\\"rootfs_linux.go:54: mounting \\\\\\\"tmpfs\\\\\\\" to rootfs \\\\\\\"/var/lib/docker/886432.886432/devicemapper/mnt/ade9f8c36ec7d1328aa42a34bba0be40c482ca5e3da30c1e17410ca196f46941/rootfs\\\\\\\" at \\\\\\\"/dev\\\\\\\" caused \\\\\\\"permission denied\\\\\\\"\\\"\"\n" Expected results: No error, image built and usable. Additional info:
Merely running bash fails as well: # docker run --rm -ti -e container=docker fedora:25 bash /usr/bin/docker-current: Error response from daemon: invalid header field value "oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:359: container init caused \\\"rootfs_linux.go:54: mounting \\\\\\\"tmpfs\\\\\\\" to rootfs \\\\\\\"/var/lib/docker/886432.886432/devicemapper/mnt/41080130319e94ac09eeafc4e6fe9a405ec8fe1aa769fcadc381f7f12b6bfaaf/rootfs\\\\\\\" at \\\\\\\"/dev\\\\\\\" caused \\\\\\\"permission denied\\\\\\\"\\\"\"\n".
For the record, docker-1.10.3-53.el7.x86_64 on fails as well: # docker run --rm -ti rhel7 bash docker: Error response from daemon: Cannot start container 184efbed1030c311ff54877bb5a1c86c2b3eac5f564c974cc15ea708e8f1af3d: [9] System error: fork/exec /proc/self/exe: invalid argument.
Mrunal do you remember anything about user ns in our projectatomic/docker? I remember there was still something not supported
Antonio, It looks like this is related to SELinux. If you take out the selinux build tag then it works fine. Also, this is reproducible with docker upstream on Fedora 25. Dan, could you take a look?
Is it the /dev/mqueue issue? Does it work if you run the daemon without --selinux-enabled? Are there any AVCs?
I know there is an issue with /dev/mqueu not working correctly with user namespace which prevents the SELinux label being set on /dev/mqueue when running with user namespace.
Dan, this isn't related to mqueue from what I can tell. Also, it works without --selinux-enabled and I don't see any AVCs using ausearch -m avc -ts recent.
This is an issue of mounting a tmpfs while in a user namespace using a context="container_file_t" issue. Basically usernamespace is not allowed to put arbitrary SELinux labels on a mount point. The question is why does the tmpfs have to be mounted within the UserNamespace?
Opened an issue with runc. https://github.com/opencontainers/runc/issues/1215
(In reply to Mrunal Patel from comment #7) > Dan, this isn't related to mqueue from what I can tell. Also, it works > without --selinux-enabled and I don't see any AVCs using ausearch -m avc -ts > recent. Confirming, removing --selinux-enabled helps. And there are no AVC denials logged. Do we need another bugzilla for tracking the lack of AVC denials in this situation?
Jan this is not an SELinux denial. This is a denial of a user namespace process without SYS_ADMIN attempting to set a label on a tmpfs mount point. I see this as a runc bug. Runc can not be mounting the tmpfs's while under a non root user namespace.
(In reply to Daniel Walsh from comment #11) > Jan this is not an SELinux denial. OK, thanks.
For systemd in container, we likely need to be able to mount /sys/fs/cgroup to the container. I've disabled oci-systemd-hook (to workaround bug 1401944) but when I try to mount the /sys/fs/cgroup explicitly, it fails: $ docker run -v /sys/fs/cgroup:/sys/fs/cgroup:ro --rm -ti fedora:25 bash /usr/bin/docker-current: Error response from daemon: invalid header field value "oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:359: container init caused \\\"rootfs_linux.go:54: mounting \\\\\\\"/sys/fs/cgroup\\\\\\\" to rootfs \\\\\\\"/var/lib/docker/886432.886432/devicemapper/mnt/b79ef8835182d4ef2ccf8dc0f5ef3dfab9d806d1ee83c1213377f7f7c4913761/rootfs\\\\\\\" at \\\\\\\"/var/lib/docker/886432.886432/devicemapper/mnt/b79ef8835182d4ef2ccf8dc0f5ef3dfab9d806d1ee83c1213377f7f7c4913761/rootfs/sys/fs/cgroup\\\\\\\" caused \\\\\\\"operation not permitted\\\\\\\"\\\"\"\n". Is it the same issue or would you like separate bugzilla for the /sys/fs/cgroup case?
(In reply to Jan Pazdziora from comment #13) > For systemd in container, we likely need to be able to mount /sys/fs/cgroup > to the container. I've disabled oci-systemd-hook (to workaround bug 1401944) > but when I try to mount the /sys/fs/cgroup explicitly, it fails: > > $ docker run -v /sys/fs/cgroup:/sys/fs/cgroup:ro --rm -ti fedora:25 bash > /usr/bin/docker-current: Error response from daemon: invalid header field > value "oci runtime error: container_linux.go:247: starting container process > caused \"process_linux.go:359: container init caused \\\"rootfs_linux.go:54: > mounting \\\\\\\"/sys/fs/cgroup\\\\\\\" to rootfs > \\\\\\\"/var/lib/docker/886432.886432/devicemapper/mnt/ > b79ef8835182d4ef2ccf8dc0f5ef3dfab9d806d1ee83c1213377f7f7c4913761/ > rootfs\\\\\\\" at > \\\\\\\"/var/lib/docker/886432.886432/devicemapper/mnt/ > b79ef8835182d4ef2ccf8dc0f5ef3dfab9d806d1ee83c1213377f7f7c4913761/rootfs/sys/ > fs/cgroup\\\\\\\" caused \\\\\\\"operation not permitted\\\\\\\"\\\"\"\n". > > Is it the same issue or would you like separate bugzilla for the > /sys/fs/cgroup case? This issue is tracked upstream at https://github.com/docker/docker/issues/27202 and here in bugzilla as well https://bugzilla.redhat.com/show_bug.cgi?id=1373780
(In reply to Antonio Murdaca from comment #14) > (In reply to Jan Pazdziora from comment #13) > > > > $ docker run -v /sys/fs/cgroup:/sys/fs/cgroup:ro --rm -ti fedora:25 bash > > /usr/bin/docker-current: Error response from daemon: invalid header field > > value "oci runtime error: container_linux.go:247: starting container process > > caused \"process_linux.go:359: container init caused \\\"rootfs_linux.go:54: > > mounting \\\\\\\"/sys/fs/cgroup\\\\\\\" to rootfs > > \\\\\\\"/var/lib/docker/886432.886432/devicemapper/mnt/ > > b79ef8835182d4ef2ccf8dc0f5ef3dfab9d806d1ee83c1213377f7f7c4913761/ > > rootfs\\\\\\\" at > > \\\\\\\"/var/lib/docker/886432.886432/devicemapper/mnt/ > > b79ef8835182d4ef2ccf8dc0f5ef3dfab9d806d1ee83c1213377f7f7c4913761/rootfs/sys/ > > fs/cgroup\\\\\\\" caused \\\\\\\"operation not permitted\\\\\\\"\\\"\"\n". > > > > Is it the same issue or would you like separate bugzilla for the > > /sys/fs/cgroup case? > > This issue is tracked upstream at > https://github.com/docker/docker/issues/27202 and here in bugzilla as well > https://bugzilla.redhat.com/show_bug.cgi?id=1373780 That issue is about console, not about /sys/fs/cgroup. Note that the command I run here is docker run -v /sys/fs/cgroup:/sys/fs/cgroup:ro --rm -ti fedora:25 bash -- I specify bash to be run in the container, there is no systemd involved. Of course, I got to this case by minimizing systemd attempts but I believe this is solely about mounting stuff with --userns-remap.
If you try the upstream issue and the bugzilla I linked you'll get exactly the same issue you're having. See also Mrunal's comment in the upstream issue.
(In reply to Antonio Murdaca from comment #16) > If you try the upstream issue and the bugzilla I linked you'll get exactly > the same issue you're having. See also Mrunal's comment in the upstream > issue. I that bugzilla https://bugzilla.redhat.com/show_bug.cgi?id=1373780 I point out that the console issue is a regression in fedora:25 and fedora:rawhide against fedora:24 image. That bugzilla setup also does not use --userns-remap. This issue (from comment 13) I can reproduce on fedora:24: # docker run -v /sys/fs/cgroup:/sys/fs/cgroup:ro --rm -ti fedora:24 bash /usr/bin/docker-current: Error response from daemon: invalid header field value "oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:359: container init caused \\\"rootfs_linux.go:54: mounting \\\\\\\"/sys/fs/cgroup\\\\\\\" to rootfs \\\\\\\"/var/lib/docker/886432.886432/devicemapper/mnt/8c7c02a8a7c9d5f36a74540c2001d39be5b4e19c8b930bb5d89a12fe5c285973/rootfs\\\\\\\" at \\\\\\\"/var/lib/docker/886432.886432/devicemapper/mnt/8c7c02a8a7c9d5f36a74540c2001d39be5b4e19c8b930bb5d89a12fe5c285973/rootfs/sys/fs/cgroup\\\\\\\" caused \\\\\\\"operation not permitted\\\\\\\"\\\"\"\n". And it only happens when I use --userns-remap. So why the problem from this bugzilla's comment 0 or comment 13 should be duplicate of bug 1373780 is really not obvious. It well may be but to an outsider there don't seem much overlap. Could you elaborate some more?
I agree with Jan, that this is a different issue. Nothing to do with systemd.
I see at least two different bugs. oci-systemd-hook does not work with User Namespace, and volume mounting /sys/fs/cgroup into a container does not work with User Namespace. (They could be the same issue, since both attempt to mount /sys/fs/cgroup into the container.) I wonder if mounting other file systems into the container could cause the same issue. If you volume mount in /sys/fs/selinux does that crash the container also?
Is this related at all to Linux kernel commit aad82892af261b9903cc11c55be3ecf5f0b0b4f8 ("selinux: Add support for unprivileged mounts from user namespaces") circa Linux 4.8? If so, that change does cause SELinux to reject context mounts on user namespace mounts, to prevent unprivileged processes from performing context mounts.
Would seem to make sense.
Why does this need to be a context mount?
Interesting find: while $ docker run -v /sys/fs/cgroup:/sys/fs/cgroup:ro --rm -ti fedora:24 bash fails, specifying the mount without :ro passes: $ docker run -v /sys/fs/cgroup:/sys/fs/cgroup --rm -ti fedora:24 bash [root@3b65038bb538 /]#
Mounting /sys/fs/selinux passes both for the :ro and no-ro case: # docker run -v /sys/fs/selinux:/sys/fs/selinux:ro --rm -ti fedora:24 bash [root@834a43d4e491 /]# exit # docker run -v /sys/fs/selinux:/sys/fs/selinux --rm -ti fedora:24 bash [root@76e49ad37f62 /]# exit
Wow that is weird.
Steven this does not need to be a context mount, the only thing we need is the tmpfs to be labeled with the Container MountLabel. Theoretically we coould just do a setfilecon(/dev, mountlabel) But I am not sure this will succeed. If it should based on SELinux labels, then perhaps the kernel change is wrong. And a non priv process should be able to label the mount object at mount time, if SELinux policy allows him to set the SELinux label.
I think this is wrong. http://kernel.suse.com/cgit/kernel/commit/?id=aad82892af261b9903cc11c55be3ecf5f0b0b4f8 Why should we be controlling the ability to specify a context mount based on DAC issues, It should be controlled by MAC.
So this bugzilla is primarily about the oci-systemd-hook issue, while mounting of /sys/fs/cgroup:/sys/fs/cgroup:ro is explicitly tracked in bug 1401944.
Thanks.
Mrunal, just circling back on older bugzillas. I believe the problem here was runc not matching the options of the mounted file system, when it remounts it read/only. Do you know if this has been fixed in the latest runc?
This looks to be fixed with the latest oci-systemd-hook. Asking QA to verify my testing. [root@localhost ~]# vi /etc/sysconfig/docker [root@localhost ~]# cat /etc/sysconfig/docker # /etc/sysconfig/docker # Modify these options if you want to change the way the docker daemon runs #OPTIONS='--selinux-enabled --log-driver=journald' OPTIONS='--icc=false --selinux-enabled --log-driver=journald --userns-remap=default' # (Further lines edited out for brevity) [root@localhost ~]# cat test/Dockerfile FROM fedora:25 RUN dnf install -y /usr/bin/ps [root@localhost ~]# systemctl restart docker [root@localhost ~]# vi Dockerfile [root@localhost ~]# pwd /root [root@localhost ~]# mkdir test [root@localhost ~]# mv Dockerfile test/ [root@localhost ~]# ls test Dockerfile [root@localhost ~]# docker build -t test ~/test/ Sending build context to Docker daemon 2.048 kB Step 1 : FROM fedora:25 Trying to pull repository docker.io/library/fedora ... sha256:b44cdaee0feafc85cab454e2023807f66725c727655b6bef260aa6d21dd2b068: Pulling from docker.io/library/fedora bc5187a39b05: Pull complete Digest: sha256:b44cdaee0feafc85cab454e2023807f66725c727655b6bef260aa6d21dd2b068 Status: Downloaded newer image for docker.io/fedora:25 ---> 4daa661b467f Step 2 : RUN dnf install -y /usr/bin/ps ---> Running in 16afdaafe1b0 Last metadata expiration check: 0:00:21 ago on Sat Apr 1 19:00:44 2017. Dependencies resolved. ================================================================================ Package Arch Version Repository Size ================================================================================ Installing: procps-ng x86_64 3.3.10-11.fc24 fedora 389 k Transaction Summary ================================================================================ Install 1 Package Total download size: 389 k Installed size: 873 k Downloading Packages: -------------------------------------------------------------------------------- Total 684 kB/s | 389 kB 00:00 warning: /var/cache/dnf/fedora-310f9d37d74ceec1/packages/procps-ng-3.3.10-11.fc24.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID fdb19c98: NOKEY Importing GPG key 0xFDB19C98: Userid : "Fedora 25 Primary (25) <fedora-25-primary>" Fingerprint: C437 DCCD 558A 66A3 7D6F 4372 4089 D8F2 FDB1 9C98 From : /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-25-x86_64 Key imported successfully Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Installing : procps-ng-3.3.10-11.fc24.x86_64 1/1 error: unpacking of archive failed on file /usr/bin/free;58dff8f5: cpio: open Error unpacking rpm package procps-ng-3.3.10-11.fc24.x86_64 Error unpacking rpm package procps-ng-3.3.10-11.fc24.x86_64 procps-ng-3.3.10-11.fc24.x86_64 was supposed to be installed but is not! Verifying : procps-ng-3.3.10-11.fc24.x86_64 1/1 Failed: procps-ng.x86_64 3.3.10-11.fc24 Complete! ---> 3b27323a5dd8 Removing intermediate container 16afdaafe1b0 Successfully built 3b27323a5dd8 [root@localhost ~]# docker build -t test ~/test/ Sending build context to Docker daemon 2.048 kB Step 1 : FROM fedora:25 ---> 4daa661b467f Step 2 : RUN dnf install -y /usr/bin/ps ---> Using cache ---> 3b27323a5dd8 Successfully built 3b27323a5dd8 [root@localhost ~]# rpm -qa oci-systemd-hook oci-systemd-hook-0.1.6-1.gitfe22236.fc25.x86_64[root@localhost ~]# uname -a Linux localhost.localdomain 4.10.5-200.fc25.x86_64 #1 SMP Wed Mar 22 20:37:08 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux[root@localhost ~]# cat /etc/*release Fedora release 25 (Twenty Five) NAME=Fedora VERSION="25 (Server Edition)" ID=fedora VERSION_ID=25 PRETTY_NAME="Fedora 25 (Server Edition)"
(In reply to Tom Sweeney from comment #31) This > Transaction test succeeded. > Running transaction > Installing : procps-ng-3.3.10-11.fc24.x86_64 > 1/1 > error: unpacking of archive failed on file /usr/bin/free;58dff8f5: cpio: open > Error unpacking rpm package procps-ng-3.3.10-11.fc24.x86_64 > Error unpacking rpm package procps-ng-3.3.10-11.fc24.x86_64 > procps-ng-3.3.10-11.fc24.x86_64 was supposed to be installed but is not! causes for ps not to be actually installed even if the subsequent docker run passes without error: > [root@localhost ~]# docker build -t test ~/test/ > Sending build context to Docker daemon 2.048 kB > Step 1 : FROM fedora:25 > ---> 4daa661b467f > Step 2 : RUN dnf install -y /usr/bin/ps > ---> Using cache > ---> 3b27323a5dd8 > Successfully built 3b27323a5dd8 # docker run --rm -ti test ps axuwwf /usr/bin/docker-current: Error response from daemon: invalid header field value "oci runtime error: container_linux.go:247: starting container process caused \"exec: \\\"ps\\\": executable file not found in $PATH\"\n". Why can't /usr/bin/free be installed? OTOH, docker run --rm -ti -e container=docker fedora:25 bash now passes. Do we want separate bugzilla about the installation of /usr/bin/free under --userns-remap?
(In reply to Jan Pazdziora from comment #32) > > causes for ps not to be actually installed even if the subsequent docker run > passes without error: I wanted to say "even if the subsequente docker *build* passes". The build seemingly passes but docker run then fails.
Are these errors about oci-systemd-hook or something else?
(In reply to Daniel Walsh from comment #34) > Are these errors about oci-systemd-hook or something else? They don't seem to be oci-systemd-hook-related, rather a dac_override AVC denial. Filed bug 1439587 for that.
I see docker run --rm -ti -e container=docker fedora:25 bash passing now with released oci-systemd-hook-0.1.7-1.git1788cf2.fc25.x86_64 and docker-1.12.6-6.gitae7d637.fc25.x86_64 container-selinux-2.10-1.fc25.noarch selinux-policy-3.13.1-225.13.fc25.noarch kernel-4.10.14-200.fc25.x86_64 Could we get Fixed In Version filled and this bugzilla closed, ideally with an errata?
oci-systemd-hook-0.1.9-1.gitaa42622.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-8c49b86e6c
oci-systemd-hook-0.1.9-1.gitaa42622.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-8b041abf0a
oci-systemd-hook-0.1.9-1.gitaa42622.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-8b041abf0a
oci-systemd-hook-0.1.9-1.gitaa42622.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-8c49b86e6c
oci-systemd-hook-0.1.9-1.gitaa42622.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.
oci-systemd-hook-0.1.9-1.gitaa42622.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days