Bug 1451007 - Pods fail to start if they specify a file as a volume subPath to mount
Summary: Pods fail to start if they specify a file as a volume subPath to mount
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 3.5.1
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 3.5.z
Assignee: Matthew Wong
QA Contact: Jianwei Hou
URL:
Whiteboard:
Depends On:
Blocks: 1451008
TreeView+ depends on / blocked
 
Reported: 2017-05-15 14:43 UTC by Matthew Wong
Modified: 2020-09-10 10:34 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
undefined
Clone Of:
: 1451008 (view as bug list)
Environment:
Last Closed: 2017-06-15 18:39:31 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:1425 0 normal SHIPPED_LIVE OpenShift Container Platform 3.5, 3.4, 3.3, and 3.2 bug fix update 2017-06-15 22:35:53 UTC

Description Matthew Wong 2017-05-15 14:43:27 UTC
Description of problem:

Pods fail to start if they specify a file as a volume subPath to mount

Version-Release number of selected component (if applicable):

How reproducible:

Always

Steps to Reproduce:
1. Create a pod that mounts a volume's file using subPath

Actual results:

Pod reports "mkdir" error and fails to start

Expected results:

Pod enters Running state

Comment 1 Matthew Wong 2017-05-15 14:44:19 UTC
https://github.com/openshift/origin/pull/14193

Comment 4 Jianwei Hou 2017-06-05 07:05:04 UTC
Tested on openshift v3.5.5.23

Pod failed to start given the subPath is not a directory, it failed with mount error:
```
Failed to start container with docker id a906aa02f938 with error: Error response from daemon: {"message":"invalid header field value \"oci runtime error: container_linux.go:247: starting container process caused \\\"process_linux.go:359: container init caused \\\\\\\"rootfs_linux.go:54: mounting \\\\\\\\\\\\\\\"/var/lib/origin/openshift.local.volumes/pods/3a5cc9d1-49bb-11e7-a437-fa163eed6cea/volumes/kubernetes.io~nfs/pvol/file\\\\\\\\\\\\\\\" to rootfs \\\\\\\\\\\\\\\"/var/lib/docker/devicemapper/mnt/c39df92ec2572d6c793b2e8402064b45fce1b4e5ae1c0cd5648cfa53219ff874/rootfs\\\\\\\\\\\\\\\" at \\\\\\\\\\\\\\\"/var/lib/docker/devicemapper/mnt/c39df92ec2572d6c793b2e8402064b45fce1b4e5ae1c0cd5648cfa53219ff874/rootfs/opt\\\\\\\\\\\\\\\" caused \\\\\\\\\\\\\\\"not a directory\\\\\\\\\\\\\\\"\\\\\\\"\\\"\\n\""}
```

Changing to assigned since we expect the it to be in Running state.

1. Create Pod
{
    "apiVersion": "v1",
    "kind": "Pod",
    "metadata": {
        "name": "mypod1",
        "labels": {
            "name": "frontendhttp"
        }
    },
    "spec": {
        "containers": [{
            "name": "myfrontend",
            "image": "jhou/hello-openshift",
            "imagePullPolicy": "IfNotPresent",
            "ports": [{
                "containerPort": 80,
                "name": "http-server"
            }],
            "volumeMounts": [{
                "mountPath": "/opt",
                "subPath": "file",
                "name": "pvol"
            }]
        }],
        "volumes": [{
            "name": "pvol",
            "nfs": {
                "server": "xxx",
                "path": "/nfs",
                "readOnly": false
            }
        }]
    }
}
2. oc get pods
NAME      READY     STATUS             RESTARTS   AGE
mypod1    0/1       CrashLoopBackOff   6          9m

3. oc describe pod mypod1
  11m		11m		1	{default-scheduler }									Normal		Scheduled	Successfully assigned mypod1 to host-8-175-116.host.centralci.eng.rdu2.redhat.com
  11m		11m		1	{kubelet host-8-175-116.host.centralci.eng.rdu2.redhat.com}	spec.containers{myfrontend}	Normal		Created		Created container with docker id a2133de5a048; Security:[seccomp=unconfined]
  11m		11m		1	{kubelet host-8-175-116.host.centralci.eng.rdu2.redhat.com}	spec.containers{myfrontend}	Warning		Failed		Failed to start container with docker id a2133de5a048 with error: Error response from daemon: {"message":"invalid header field value \"oci runtime error: container_linux.go:247: starting container process caused \\\"process_linux.go:359: container init caused \\\\\\\"rootfs_linux.go:54: mounting \\\\\\\\\\\\\\\"/var/lib/origin/openshift.local.volumes/pods/3a5cc9d1-49bb-11e7-a437-fa163eed6cea/volumes/kubernetes.io~nfs/pvol/file\\\\\\\\\\\\\\\" to rootfs \\\\\\\\\\\\\\\"/var/lib/docker/devicemapper/mnt/9c59a37dc240dbbe6012ad02018a54ea376167b455126384b99106776d2cca1b/rootfs\\\\\\\\\\\\\\\" at \\\\\\\\\\\\\\\"/var/lib/docker/devicemapper/mnt/9c59a37dc240dbbe6012ad02018a54ea376167b455126384b99106776d2cca1b/rootfs/opt\\\\\\\\\\\\\\\" caused \\\\\\\\\\\\\\\"not a directory\\\\\\\\\\\\\\\"\\\\\\\"\\\"\\n\""}
  11m		11m		1	{kubelet host-8-175-116.host.centralci.eng.rdu2.redhat.com}				Warning		FailedSync	Error syncing pod, skipping: failed to "StartContainer" for "myfrontend" with RunContainerError: "runContainer: Error response from daemon: {\"message\":\"invalid header field value \\\"oci runtime error: container_linux.go:247: starting container process caused \\\\\\\"process_linux.go:359: container init caused \\\\\\\\\\\\\\\"rootfs_linux.go:54: mounting \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"/var/lib/origin/openshift.local.volumes/pods/3a5cc9d1-49bb-11e7-a437-fa163eed6cea/volumes/kubernetes.io~nfs/pvol/file\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\" to rootfs \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"/var/lib/docker/devicemapper/mnt/9c59a37dc240dbbe6012ad02018a54ea376167b455126384b99106776d2cca1b/rootfs\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\" at \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"/var/lib/docker/devicemapper/mnt/9c59a37dc240dbbe6012ad02018a54ea376167b455126384b99106776d2cca1b/rootfs/opt\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\" caused \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"not a directory\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"\\\\\\\\\\\\\\\"\\\\\\\"\\\\n\\\"\"}"

Comment 5 Matthew Wong 2017-06-05 16:26:50 UTC
Jianwei, please try editing the pod spec from "mountPath": "/opt" to "mountPath": "/opt/file"? That error occurs because "/opt" already exists in the container, and 

                "mountPath": "/opt",
                "subPath": "file",

means "mount /nfs/file /opt", not "mount /nfs/file /opt/file" as one might expect.

Comment 6 Jianwei Hou 2017-06-06 03:33:56 UTC
(In reply to Matthew Wong from comment #5)
> Jianwei, please try editing the pod spec from "mountPath": "/opt" to
> "mountPath": "/opt/file"? That error occurs because "/opt" already exists in
> the container, and 
> 
>                 "mountPath": "/opt",
>                 "subPath": "file",
> 
> means "mount /nfs/file /opt", not "mount /nfs/file /opt/file" as one might
> expect.

Sorry for the incorrect mountPath. I've verified it following your instruction and now it works. Thank you.

Comment 7 Jianwei Hou 2017-06-07 02:21:29 UTC
This bug is verified according to comment 6

Comment 9 errata-xmlrpc 2017-06-15 18:39:31 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1425


Note You need to log in before you can comment on or make changes to this bug.