Hide Forgot
Description of problem: Pods fail to start if they specify a file as a volume subPath to mount Version-Release number of selected component (if applicable): How reproducible: Always Steps to Reproduce: 1. Create a pod that mounts a volume's file using subPath Actual results: Pod reports "mkdir" error and fails to start Expected results: Pod enters Running state
https://github.com/openshift/origin/pull/14193
Tested on openshift v3.5.5.23 Pod failed to start given the subPath is not a directory, it failed with mount error: ``` Failed to start container with docker id a906aa02f938 with error: Error response from daemon: {"message":"invalid header field value \"oci runtime error: container_linux.go:247: starting container process caused \\\"process_linux.go:359: container init caused \\\\\\\"rootfs_linux.go:54: mounting \\\\\\\\\\\\\\\"/var/lib/origin/openshift.local.volumes/pods/3a5cc9d1-49bb-11e7-a437-fa163eed6cea/volumes/kubernetes.io~nfs/pvol/file\\\\\\\\\\\\\\\" to rootfs \\\\\\\\\\\\\\\"/var/lib/docker/devicemapper/mnt/c39df92ec2572d6c793b2e8402064b45fce1b4e5ae1c0cd5648cfa53219ff874/rootfs\\\\\\\\\\\\\\\" at \\\\\\\\\\\\\\\"/var/lib/docker/devicemapper/mnt/c39df92ec2572d6c793b2e8402064b45fce1b4e5ae1c0cd5648cfa53219ff874/rootfs/opt\\\\\\\\\\\\\\\" caused \\\\\\\\\\\\\\\"not a directory\\\\\\\\\\\\\\\"\\\\\\\"\\\"\\n\""} ``` Changing to assigned since we expect the it to be in Running state. 1. Create Pod { "apiVersion": "v1", "kind": "Pod", "metadata": { "name": "mypod1", "labels": { "name": "frontendhttp" } }, "spec": { "containers": [{ "name": "myfrontend", "image": "jhou/hello-openshift", "imagePullPolicy": "IfNotPresent", "ports": [{ "containerPort": 80, "name": "http-server" }], "volumeMounts": [{ "mountPath": "/opt", "subPath": "file", "name": "pvol" }] }], "volumes": [{ "name": "pvol", "nfs": { "server": "xxx", "path": "/nfs", "readOnly": false } }] } } 2. oc get pods NAME READY STATUS RESTARTS AGE mypod1 0/1 CrashLoopBackOff 6 9m 3. oc describe pod mypod1 11m 11m 1 {default-scheduler } Normal Scheduled Successfully assigned mypod1 to host-8-175-116.host.centralci.eng.rdu2.redhat.com 11m 11m 1 {kubelet host-8-175-116.host.centralci.eng.rdu2.redhat.com} spec.containers{myfrontend} Normal Created Created container with docker id a2133de5a048; Security:[seccomp=unconfined] 11m 11m 1 {kubelet host-8-175-116.host.centralci.eng.rdu2.redhat.com} spec.containers{myfrontend} Warning Failed Failed to start container with docker id a2133de5a048 with error: Error response from daemon: {"message":"invalid header field value \"oci runtime error: container_linux.go:247: starting container process caused \\\"process_linux.go:359: container init caused \\\\\\\"rootfs_linux.go:54: mounting \\\\\\\\\\\\\\\"/var/lib/origin/openshift.local.volumes/pods/3a5cc9d1-49bb-11e7-a437-fa163eed6cea/volumes/kubernetes.io~nfs/pvol/file\\\\\\\\\\\\\\\" to rootfs \\\\\\\\\\\\\\\"/var/lib/docker/devicemapper/mnt/9c59a37dc240dbbe6012ad02018a54ea376167b455126384b99106776d2cca1b/rootfs\\\\\\\\\\\\\\\" at \\\\\\\\\\\\\\\"/var/lib/docker/devicemapper/mnt/9c59a37dc240dbbe6012ad02018a54ea376167b455126384b99106776d2cca1b/rootfs/opt\\\\\\\\\\\\\\\" caused \\\\\\\\\\\\\\\"not a directory\\\\\\\\\\\\\\\"\\\\\\\"\\\"\\n\""} 11m 11m 1 {kubelet host-8-175-116.host.centralci.eng.rdu2.redhat.com} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "myfrontend" with RunContainerError: "runContainer: Error response from daemon: {\"message\":\"invalid header field value \\\"oci runtime error: container_linux.go:247: starting container process caused \\\\\\\"process_linux.go:359: container init caused \\\\\\\\\\\\\\\"rootfs_linux.go:54: mounting \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"/var/lib/origin/openshift.local.volumes/pods/3a5cc9d1-49bb-11e7-a437-fa163eed6cea/volumes/kubernetes.io~nfs/pvol/file\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\" to rootfs \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"/var/lib/docker/devicemapper/mnt/9c59a37dc240dbbe6012ad02018a54ea376167b455126384b99106776d2cca1b/rootfs\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\" at \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"/var/lib/docker/devicemapper/mnt/9c59a37dc240dbbe6012ad02018a54ea376167b455126384b99106776d2cca1b/rootfs/opt\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\" caused \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"not a directory\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"\\\\\\\\\\\\\\\"\\\\\\\"\\\\n\\\"\"}"
Jianwei, please try editing the pod spec from "mountPath": "/opt" to "mountPath": "/opt/file"? That error occurs because "/opt" already exists in the container, and "mountPath": "/opt", "subPath": "file", means "mount /nfs/file /opt", not "mount /nfs/file /opt/file" as one might expect.
(In reply to Matthew Wong from comment #5) > Jianwei, please try editing the pod spec from "mountPath": "/opt" to > "mountPath": "/opt/file"? That error occurs because "/opt" already exists in > the container, and > > "mountPath": "/opt", > "subPath": "file", > > means "mount /nfs/file /opt", not "mount /nfs/file /opt/file" as one might > expect. Sorry for the incorrect mountPath. I've verified it following your instruction and now it works. Thank you.
This bug is verified according to comment 6
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:1425