| Summary: | [DOCKER] Permission denied mounting in userns-remap environment | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Paul Weil <pweil> |
| Component: | Containers | Assignee: | Mrunal Patel <mpatel> |
| Status: | CLOSED WONTFIX | QA Contact: | Xiaoli Tian <xtian> |
| Severity: | low | Docs Contact: | |
| Priority: | medium | ||
| Version: | unspecified | CC: | amurdaca, aos-bugs, chuyu, dwalsh, haowang, imcleod, jhonce, jokerman, jpazdziora, mmccomas, nagrawal, santiago |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-04-02 15:18:16 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
|
Description
Paul Weil
2016-09-14 07:21:53 UTC
A little more triage. I'm not sure if this issue lies in Kube or in Containers
###
# For this error
###
rootfs_linux.go:53: mounting "/var/lib/kubelet/pods/f5b1ffe6-7a52-11e6-bde2-54ee752009cb/containers/hello-nginx-docker-pod/68c5eb4c" to rootfs "/var/lib/docker/2197152.2197152/devicemapper/mnt/c704022245b24e cb3bbb91d4d1923ab370aefd861302cd1c575581df7adfa6af/rootfs" at "/dev/termination-log" caused "stat /var/lib/kubelet/pods/f5b1ffe6-7a52-11e6-bde2-54ee752009cb/containers/hello-nginx-docker-pod/68c5eb4c: permission denied"
###
# Permissions break down like this. Eventually there is a point
# where the "other" access isn't available which is probably what is causing
# the stat to fail.
###
[pweil@localhost kubernetes]$ sudo ls -l /var/lib/kubelet/pods/f5b1ffe6-7a52-11e6-bde2-54ee752009cb/containers/hello-nginx-docker-pod/68c5eb4c
-rw-r--r--. 1 root root 0 Sep 14 04:12 /var/lib/kubelet/pods/f5b1ffe6-7a52-11e6-bde2-54ee752009cb/containers/hello-nginx-docker-pod/68c5eb4c
[pweil@localhost kubernetes]$ sudo ls -l /var/lib/kubelet/pods/f5b1ffe6-7a52-11e6-bde2-54ee752009cb/containers/hello-nginx-docker-pod/
total 0
-rw-r--r--. 1 root root 0 Sep 14 04:12 68c5eb4c
-rw-r--r--. 1 root root 0 Sep 14 04:12 96a09fd0
[pweil@localhost kubernetes]$ sudo ls -l /var/lib/kubelet/pods/f5b1ffe6-7a52-11e6-bde2-54ee752009cb/containers/
total 4
drwxr-x---. 2 root root 4096 Sep 14 04:12 hello-nginx-docker-pod
[pweil@localhost kubernetes]$ sudo ls -l /var/lib/kubelet/pods/f5b1ffe6-7a52-11e6-bde2-54ee752009cb/
total 16
drwxr-x---. 3 root root 4096 Sep 14 04:12 containers
-rw-r--r--. 1 2197152 2197152 213 Sep 14 04:12 etc-hosts
drwxr-x---. 3 root root 4096 Sep 14 04:12 plugins
drwxr-x---. 3 root root 4096 Sep 14 04:12 volumes
[pweil@localhost kubernetes]$ sudo ls -l /var/lib/kubelet/pods/
total 28
drwxr-x---. 4 root root 4096 Sep 13 08:48 4a6dceba-79b0-11e6-a66a-54ee752009cb
drwxr-x---. 5 root root 4096 Sep 14 03:39 4bc8f4e5-7a4e-11e6-bd9e-54ee752009cb
drwxr-x---. 5 root root 4096 Sep 13 10:37 9eb33efb-79bf-11e6-8ecf-54ee752009cb
drwxr-x---. 5 root root 4096 Sep 13 08:58 c1d0aa5c-79b1-11e6-9a26-54ee752009cb
drwxr-x---. 5 root root 4096 Sep 13 08:51 cec7068e-79b0-11e6-9515-54ee752009cb
drwxr-x---. 5 root root 4096 Sep 14 03:50 e0d4337a-7a4f-11e6-b66b-54ee752009cb
drwxr-x---. 5 root root 4096 Sep 14 04:12 f5b1ffe6-7a52-11e6-bde2-54ee752009cb
[pweil@localhost kubernetes]$ sudo ls -l /var/lib/kubelet/
total 8
drwxr-x---. 2 root root 4096 Jul 19 13:13 plugins
drwxr-x---. 9 root root 4096 Sep 14 04:12 pods
###
# Workaround
###
I added some logging to runc and noticed files were being created by the subuid/subgid values. In the non-remap environment when running with a runAsUser in Kube (which translates to a --user in docker) the files were run with {user}:root.
I was able to work around this and get a running pod in Kubernetes by editing the /etc/subgid file so that the dockeremap user used the root group:
dockremap:0:65536
Paul could you try this with docker-1.12? Also this might be a need to chown the k8s file to be owned by dockerroot not real root? I'm seeing what might be the same issue. docker-1.12.6-25.git62520c0.el7 on 7.4 beta, kernel 3.10.0-663.el7
# mkdir -p /tmp/a/b
# chmod 700 /tmp/a
# docker run --rm -v /tmp/a/b:/tmp:z centos true
container_linux.go:247: starting container process caused "process_linux.go:359: container init caused \"rootfs_linux.go:54: mounting \\\"/tmp/a/b\\\" to rootfs \\\"/var/lib/docker/100000.100000/devicemapper/mnt/a3575af331be8252a9499c3e0fc40c001d9b052ea2be7d445d4e3bcaf9719822/rootfs\\\" at \\\"/tmp\\\" caused \\\"stat /tmp/a/b: permission denied\\\"\""
/usr/bin/docker-current: Error response from daemon: invalid header field value "oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:359: container init caused \\\"rootfs_linux.go:54: mounting \\\\\\\"/tmp/a/b\\\\\\\" to rootfs \\\\\\\"/var/lib/docker/100000.100000/devicemapper/mnt/a3575af331be8252a9499c3e0fc40c001d9b052ea2be7d445d4e3bcaf9719822/rootfs\\\\\\\" at \\\\\\\"/tmp\\\\\\\" caused \\\\\\\"stat /tmp/a/b: permission denied\\\\\\\"\\\"\"\n".
# cat /etc/sub?id
dockremap:100000:65536
dockremap:100000:65536
# cat /proc/sys/user/max_user_namespaces
100
|