Description of problem: Docker doesn't run containers Version-Release number of selected component (if applicable): docker-1.12.1-24.git9a3752d.fc26 systemd-232-6.fc26 kernel-4.10.0-0.rc2.git4.1.fc26 How reproducible: Always Steps to Reproduce: 1.docker run -ti fedora bash 2.docker exits with error 3. Actual results: docker: Error response from daemon: oci runtime error: rootfs_linux.go:53: mounting "/sys/fs/cgroup" to rootfs "/var/lib/docker/overlay/b5331adb3bf783718e85bedb706d430d79d52aba138be8e06594826158d29164/merged" caused "no subsystem for mount". Expected results: I get a running container Additional info: Manual mounting of /sys/fs/cgroup works https://github.com/docker/docker/issues/28109
First could you update your system to the latest fedora 26. docker-1.12.5-15.git079fbe3.fc26.x86_64
Updated, rebooted, now I get /usr/bin/docker-current: Error response from daemon: invalid header field value "oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:359: container init caused \\\"rootfs_linux.go:54: mounting \\\\\\\"cgroup\\\\\\\" to rootfs \\\\\\\"/var/lib/docker/overlay/ef15be62037b1eb3bee17bce3006339e16c350759e12ae91bf7710be626a8e26/merged\\\\\\\" at \\\\\\\"/sys/fs/cgroup\\\\\\\" caused \\\\\\\"no subsystem for mount\\\\\\\"\\\"\"\n".
Any chance this is SELinux. setenforce 0 Then try your command? This is working fine for me on Rawhide.
This seems like runc failing to parse cgroups files under /sys/fs/cgroup (or /proc/self/cgroup, I always forget). What systemd version are you using? could you paste your /proc/self/cgroup? are you using cgroup v2 by any chance?
Yes, that could be a problem.
10:cpuset:/ 9:freezer:/ 8:net_cls,net_prio:/ 7:devices:/user.slice 6:hugetlb:/ 5:cpu,cpuacct:/ 4:blkio:/ 3:pids:/user.slice/user-1000.slice/session-1.scope 2:memory:/ 1:perf_event:/ 0::/user.slice/user-1000.slice/session-1.scope
(In reply to Zuzana Svetlikova from comment #6) > 10:cpuset:/ > 9:freezer:/ > 8:net_cls,net_prio:/ > 7:devices:/user.slice > 6:hugetlb:/ > 5:cpu,cpuacct:/ > 4:blkio:/ > 3:pids:/user.slice/user-1000.slice/session-1.scope > 2:memory:/ > 1:perf_event:/ > 0::/user.slice/user-1000.slice/session-1.scope that last line looks suspicious indeed. Not sure that's cgroup v2 though. I have to spin up a rawhide vm to check this out.
Could you please provide systemd and kernel version? according to https://www.kernel.org/doc/Documentation/cgroup-v2.txt that very last line seems to come from cgroup v2 (but I haven't heard of any move to cgroup v2) (or I'm totally mistaken)
Kernel doc says: ``` The entry for cgroup v2 is always in the format "0::$PATH". ```
Upstream runc issue is here though: https://github.com/opencontainers/runc/issues/654
Kernel and systemd versions are systemd-232-6.fc26 and kernel-4.10.0-0.rc2.git4.1.fc26
Here is my rawhide system # uname -r 4.10.0-0.rc2.git2.1.fc26.x86_64 # cat /proc/self/cgroup 11:perf_event:/ 10:memory:/user.slice/user-3267.slice/session-1.scope 9:hugetlb:/ 8:devices:/user.slice 7:cpuset:/ 6:blkio:/ 5:cpu,cpuacct:/user.slice/user-3267.slice/session-1.scope 4:freezer:/ 3:pids:/user.slice/user-3267.slice/session-1.scope 2:net_cls,net_prio:/ 1:name=systemd:/user.slice/user-3267.slice/session-1.scope # rpm -q systemd systemd-232-6.fc26.x86_64 I will reboot into the newer kernel.
With latest kernel, I have the same issue. docker run -ti fedora bash /usr/bin/docker-current: Error response from daemon: invalid header field value "oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:359: container init caused \\\"rootfs_linux.go:54: mounting \\\\\\\"cgroup\\\\\\\" to rootfs \\\\\\\"/var/lib/docker/overlay/e1432a26e33bebbc27619c9802d9218f3da8938b7f1696ca9be0890a2e75ac65/merged\\\\\\\" at \\\\\\\"/sys/fs/cgroup\\\\\\\" caused \\\\\\\"no subsystem for mount\\\\\\\"\\\"\"\n". # cat /proc/self/cgroup 10:hugetlb:/ 9:blkio:/ 8:net_cls,net_prio:/ 7:perf_event:/ 6:memory:/user.slice/user-3267.slice/session-1.scope 5:devices:/user.slice 4:cpu,cpuacct:/user.slice/user-3267.slice/session-1.scope 3:pids:/user.slice/user-3267.slice/session-1.scope 2:cpuset:/ 1:freezer:/ 0::/user.slice/user-3267.slice/session-1.scope uname -r 4.10.0-0.rc2.git4.1.fc26.x86_64
So this is kernel related.
Dan, yes, kernel enabled cgroup v2 but runc doesn't yet support it (https://github.com/opencontainers/runc/issues/654). Maybe Mrunal knows more on this.
Most likely it is failing to parse. I will look into this one.
Could it just be that "systemd" was dropped?
Could be a kernel regression also (just wondering since this is a kernel rc release, not really likely, but still).
I am not seeing this on my updated rawhide [mrunal@dhcp-16-129 ~]$ cat /proc/self/cgroup 11:hugetlb:/ 10:blkio:/ 9:cpuset:/ 8:freezer:/ 7:memory:/ 6:devices:/user.slice 5:pids:/user.slice/user-1000.slice/user 4:perf_event:/ 3:net_cls,net_prio:/ 2:cpu,cpuacct:/ 1:name=systemd:/user.slice/user-1000.slice/user/gnome-terminal-server.service It could be a kernel config or a systemd change to change the cgroup mounts.
I rebooted back into previous kernel and docker is running correctly again. I see 1:name=systemd:/user.slice/user-3267.slice/session-1.scope 4.10.0-0.rc2.git2.1.fc26.x86_64 But when I boot with latest, I get. 0::/user.slice/user-3267.slice/session-1.scope 4.10.0-0.rc2.git4.1.fc26.x86_64
This is fixed in systemd upstream https://github.com/systemd/systemd/pull/4628/files Should we get that backported to systemd 232 for rawhide?
https://github.com/systemd/systemd/pull/4670 is an open PR where legacy name=systemd heirarchy is still maintained.
Either ways, I have created https://github.com/opencontainers/runc/pull/1266 as a fix that we will need sooner or later in runc.
This bug appears to have been reported against 'rawhide' during the Fedora 26 development cycle. Changing version to '26'.
Folks this appears to be an active issue in F26 currently, it looks like the runc change was merged, perhaps an updated runc is needed? 4.11.10-300.fc26.x86_64 runc-1.0.0-9.git6394544.fc26.x86_64 docker-latest-1.13-31.git27e468e.fc26.x86_64 docker-common-1.13.1-19.git27e468e.fc26.x86_64 docker-v1.10-migrator-1.13.1-19.git27e468e.fc26.x86_64 docker-compose-1.14.0-1.fc26.noarch docker-client-4.0.6-5.fc26.noarch docker run -ti fedora bash /usr/bin/docker-latest: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "process_linux.go:359: container init caused \"rootfs_linux.go:54: mounting \\\"cgroup\\\" to rootfs \\\"/var/lib/docker-latest/devicemapper/mnt/b9d9fccaf263a351fcb77796632a0e90d72426567b026cb449644b37c5f88672/rootfs\\\" at \\\"/sys/fs/cgroup\\\" caused \\\"no subsystem for mount\\\"\"".
docker sues an internal docker-runc, not the system runc.
This appears to be fixed, at least for me: docker-latest-1.13-32.git27e468e.fc26.x86_64 But this isn't my bug, so I'll leave it someone else to confirm and close.
Not sure what needs to be done in order to fix this issue but I am hitting this today in Fedora 26. $ uname -r 4.12.8-300.fc26.x86_64 $ cat /proc/self/cgroup 11:hugetlb:/ 10:perf_event:/ 9:net_cls,net_prio:/ 8:memory:/user.slice 7:blkio:/user.slice 6:devices:/user.slice 5:pids:/user.slice/user-1001.slice/user 4:cpuset:/ 3:freezer:/ 2:cpu,cpuacct:/user.slice 1:name=systemd:/user.slice/user-1001.slice/user/gnome-terminal-server.service 0::/user.slice/user-1001.slice/user/gnome-terminal-server.service $ docker version Client: Version: 1.13.1 API version: 1.26 Go version: go1.7.5 Git commit: 092cba3 Built: Wed Feb 8 06:48:37 2017 OS/Arch: linux/amd64 Server: Version: 1.13.1 API version: 1.26 (minimum version 1.12) Go version: go1.7.5 Git commit: 092cba3 Built: Wed Feb 8 06:48:37 2017 OS/Arch: linux/amd64 Experimental: false $ systemctl --version systemd 233 +PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN default-hierarchy=hybrid
Ok, fixed it by upgrading docker to 17.05.0-ce. Not really comfortable with upgrading Docker as it breaks me most of the times I upgrade but now it seems that it helped.