Hide Forgot
Description of problem: When enabling userns-remap=default the kubelet is unable to mount volumes and receives: "oci runtime error: rootfs_linux.go:53: mounting \"/dev/termination-log\" to rootfs \"/var/lib/docker/2197152.2197152/devicemapper/mnt/54a7356f92388a3141e8323d8530e5cab9f6c8bcada674fec62c519232296fde/rootfs\" caused \"stat /var/lib/kubelet/pods/36c58049-7993-11e6-b050-54ee752009cb/containers/hello-nginx-docker-pod/efe8c73e: permission denied\""" Steps to Reproduce: 1. use docker 1.11+ to avoid issues with https://bugzilla.redhat.com/show_bug.cgi?id=1375878 and https://bugzilla.redhat.com/show_bug.cgi?id=1375877 2. start the docker daemon with userns-remap=default 3. start a pod in kubernetes Actual results: "oci runtime error: rootfs_linux.go:53: mounting \"/dev/termination-log\" to rootfs \"/var/lib/docker/2197152.2197152/devicemapper/mnt/54a7356f92388a3141e8323d8530e5cab9f6c8bcada674fec62c519232296fde/rootfs\" caused \"stat /var/lib/kubelet/pods/36c58049-7993-11e6-b050-54ee752009cb/containers/hello-nginx-docker-pod/efe8c73e: permission denied\""" Expected results: mounts work correctly and the container starts
A little more triage. I'm not sure if this issue lies in Kube or in Containers ### # For this error ### rootfs_linux.go:53: mounting "/var/lib/kubelet/pods/f5b1ffe6-7a52-11e6-bde2-54ee752009cb/containers/hello-nginx-docker-pod/68c5eb4c" to rootfs "/var/lib/docker/2197152.2197152/devicemapper/mnt/c704022245b24e cb3bbb91d4d1923ab370aefd861302cd1c575581df7adfa6af/rootfs" at "/dev/termination-log" caused "stat /var/lib/kubelet/pods/f5b1ffe6-7a52-11e6-bde2-54ee752009cb/containers/hello-nginx-docker-pod/68c5eb4c: permission denied" ### # Permissions break down like this. Eventually there is a point # where the "other" access isn't available which is probably what is causing # the stat to fail. ### [pweil@localhost kubernetes]$ sudo ls -l /var/lib/kubelet/pods/f5b1ffe6-7a52-11e6-bde2-54ee752009cb/containers/hello-nginx-docker-pod/68c5eb4c -rw-r--r--. 1 root root 0 Sep 14 04:12 /var/lib/kubelet/pods/f5b1ffe6-7a52-11e6-bde2-54ee752009cb/containers/hello-nginx-docker-pod/68c5eb4c [pweil@localhost kubernetes]$ sudo ls -l /var/lib/kubelet/pods/f5b1ffe6-7a52-11e6-bde2-54ee752009cb/containers/hello-nginx-docker-pod/ total 0 -rw-r--r--. 1 root root 0 Sep 14 04:12 68c5eb4c -rw-r--r--. 1 root root 0 Sep 14 04:12 96a09fd0 [pweil@localhost kubernetes]$ sudo ls -l /var/lib/kubelet/pods/f5b1ffe6-7a52-11e6-bde2-54ee752009cb/containers/ total 4 drwxr-x---. 2 root root 4096 Sep 14 04:12 hello-nginx-docker-pod [pweil@localhost kubernetes]$ sudo ls -l /var/lib/kubelet/pods/f5b1ffe6-7a52-11e6-bde2-54ee752009cb/ total 16 drwxr-x---. 3 root root 4096 Sep 14 04:12 containers -rw-r--r--. 1 2197152 2197152 213 Sep 14 04:12 etc-hosts drwxr-x---. 3 root root 4096 Sep 14 04:12 plugins drwxr-x---. 3 root root 4096 Sep 14 04:12 volumes [pweil@localhost kubernetes]$ sudo ls -l /var/lib/kubelet/pods/ total 28 drwxr-x---. 4 root root 4096 Sep 13 08:48 4a6dceba-79b0-11e6-a66a-54ee752009cb drwxr-x---. 5 root root 4096 Sep 14 03:39 4bc8f4e5-7a4e-11e6-bd9e-54ee752009cb drwxr-x---. 5 root root 4096 Sep 13 10:37 9eb33efb-79bf-11e6-8ecf-54ee752009cb drwxr-x---. 5 root root 4096 Sep 13 08:58 c1d0aa5c-79b1-11e6-9a26-54ee752009cb drwxr-x---. 5 root root 4096 Sep 13 08:51 cec7068e-79b0-11e6-9515-54ee752009cb drwxr-x---. 5 root root 4096 Sep 14 03:50 e0d4337a-7a4f-11e6-b66b-54ee752009cb drwxr-x---. 5 root root 4096 Sep 14 04:12 f5b1ffe6-7a52-11e6-bde2-54ee752009cb [pweil@localhost kubernetes]$ sudo ls -l /var/lib/kubelet/ total 8 drwxr-x---. 2 root root 4096 Jul 19 13:13 plugins drwxr-x---. 9 root root 4096 Sep 14 04:12 pods ### # Workaround ### I added some logging to runc and noticed files were being created by the subuid/subgid values. In the non-remap environment when running with a runAsUser in Kube (which translates to a --user in docker) the files were run with {user}:root. I was able to work around this and get a running pod in Kubernetes by editing the /etc/subgid file so that the dockeremap user used the root group: dockremap:0:65536
Paul could you try this with docker-1.12? Also this might be a need to chown the k8s file to be owned by dockerroot not real root?
I'm seeing what might be the same issue. docker-1.12.6-25.git62520c0.el7 on 7.4 beta, kernel 3.10.0-663.el7 # mkdir -p /tmp/a/b # chmod 700 /tmp/a # docker run --rm -v /tmp/a/b:/tmp:z centos true container_linux.go:247: starting container process caused "process_linux.go:359: container init caused \"rootfs_linux.go:54: mounting \\\"/tmp/a/b\\\" to rootfs \\\"/var/lib/docker/100000.100000/devicemapper/mnt/a3575af331be8252a9499c3e0fc40c001d9b052ea2be7d445d4e3bcaf9719822/rootfs\\\" at \\\"/tmp\\\" caused \\\"stat /tmp/a/b: permission denied\\\"\"" /usr/bin/docker-current: Error response from daemon: invalid header field value "oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:359: container init caused \\\"rootfs_linux.go:54: mounting \\\\\\\"/tmp/a/b\\\\\\\" to rootfs \\\\\\\"/var/lib/docker/100000.100000/devicemapper/mnt/a3575af331be8252a9499c3e0fc40c001d9b052ea2be7d445d4e3bcaf9719822/rootfs\\\\\\\" at \\\\\\\"/tmp\\\\\\\" caused \\\\\\\"stat /tmp/a/b: permission denied\\\\\\\"\\\"\"\n". # cat /etc/sub?id dockremap:100000:65536 dockremap:100000:65536 # cat /proc/sys/user/max_user_namespaces 100