Bug 1209625

Summary: docker-compose fails to build container when attempting to bind mount a file ("can't create volume there")
Product: Red Hat Enterprise Linux 7 Reporter: B. Warsing <dayglojesus>
Component: dockerAssignee: Daniel Walsh <dwalsh>
Status: CLOSED ERRATA QA Contact: Luwen Su <lsu>
Severity: low Docs Contact:
Priority: unspecified    
Version: 7.1CC: lsm5, mjenner, sghosh
Target Milestone: rcKeywords: Extras
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: Default labeling handling was broken Consequence: Unable to mount certain directories into containers. Fix: Code was fixed. Result: You can now mount any directory into container.
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-06-23 09:29:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description B. Warsing 2015-04-07 20:02:22 UTC
Description of problem:

When attempting to bind mount a file (not directory) using `docker-compose`, the container build fails with:

"docker create_container <- (name=u'webcluster_test_1', image=u'nginx:latest', environment={}, volumes={u'/etc/nginx/conf.d/default.conf': {}}, detach=True, ports=[u'80'])
file exists at %!s(MISSING), can't create volume there"

Version-Release number of selected component (if applicable):

docker-1.5.0-28.el7.centos

How reproducible:

highly

Steps to Reproduce:

1. Install Docker: `yum install docker-1.5.0-28.el7.centos`

2. Install docker-compose: https://docs.docker.com/compose/install/

3. Create a simple docker-compose service:

# docker-compose.yml
---
test:
  image: nginx
  ports:
    - "80:80"
  volumes:
    - sites/test.conf:/etc/nginx/conf.d/default.conf

3. Run: `docker-compose --verbose up -d`

Actual results:

The build process will consistently produce the error:

"file exists at %!s(MISSING), can't create volume there" because it cannot bind mouth a file.

Expected results:

Docker should build the container successfully at it did in previous yum packages (docker-1.5.0-1.el7).

Additional info:

See Discussion:  https://github.com/docker/compose/issues/424#issuecomment-90689698

1. docker-1.5.0-1.el7 is NOT affected by this

2. This is a bug that does not exist in the 1.5.0 release of Docker, but is present in the 1.5.0-28 CentOS yum repo because of some additional patches that have been applied.

Comment 1 Lokesh Mandvekar 2015-04-07 20:30:24 UTC
(In reply to B. Warsing from comment #0)
> See Discussion: 
> https://github.com/docker/compose/issues/424#issuecomment-90689698
> 
> 1. docker-1.5.0-1.el7 is NOT affected by this

Just so it's clear to everyone, docker-1.5.0-1.el7 mentioned here is the one that I maintain as part of CentOS virt SIG, and only includes the upstream docker release 1.5.0

Comment 2 Daniel Walsh 2015-04-08 13:25:52 UTC
I take it that this is equivalent of doing


docker run -v sites/test.conf:/etc/nginx/conf.d/default.conf ...

Comment 3 Daniel Walsh 2015-04-08 13:28:17 UTC
Lokesh can you build the latest docker-1.6 packages and see if B. Warsing can reproduce there.


I don't use docker-compose.

Comment 4 B. Warsing 2015-04-08 13:30:35 UTC
(In reply to Daniel Walsh from comment #2)
> I take it that this is equivalent of doing
> 
> 
> docker run -v sites/test.conf:/etc/nginx/conf.d/default.conf ...

Yes, but oddly enough, the equivalent `docker run` command will succeed where `docker-compose` fails.

Comment 5 Daniel Walsh 2015-04-08 21:23:21 UTC
Could you patch the exact error message you are seeing?

THis looks like something you typed.


"file exists at %!s(MISSING), can't create volume there" because it cannot bind mouth a file.

Since I doubt we printed "bind mouth"

Comment 6 B. Warsing 2015-04-08 21:37:33 UTC
(In reply to Daniel Walsh from comment #5)
> Could you patch the exact error message you are seeing?
> 
> THis looks like something you typed.
> 
> 
> "file exists at %!s(MISSING), can't create volume there" because it cannot
> bind mouth a file.
> 
> Since I doubt we printed "bind mouth"

# docker-compose --verbose up -d
Compose version 1.1.0
Docker base_url: http+unix://var/run/docker.sock
Docker version: KernelVersion=3.10.0-123.el7.x86_64, Arch=amd64, ApiVersion=1.18, Version=1.5.0-dev, GitCommit=fc0329b/1.5.0, Os=linux, GoVersion=go1.3.3
docker containers <- (all=True)
docker containers -> (list with 0 items)
Creating munkicluster_proxy_1...
docker containers <- (all=True)
docker containers -> (list with 0 items)
docker create_container <- (name=u'webcluster_proxy_1', image=u'jwilder/nginx-proxy:latest', environment={}, volumes={u'/app/nginx.tmpl': {}}, detach=True, ports=[u'443'])
file exists at %!s(MISSING), can't create volume there

Comment 7 Daniel Walsh 2015-04-08 21:48:39 UTC
func (container *Container) parseVolumeMountConfig() (map[string]*Mount, error) {
	var mounts = make(map[string]*Mount)
	// Get all the bind mounts
	for _, spec := range container.hostConfig.Binds {
		path, mountToPath, writable, err := parseBindMountSpec(spec)
		if err != nil {
			return nil, err
		}
		// Check if a bind mount has already been specified for the same container path
		if m, exists := mounts[mountToPath]; exists {
			return nil, fmt.Errorf("Duplicate volume %q: %q already in use, mounted from %q", path, mountToPath, m.volume.Path)
		}
		// Check if a volume already exists for this and use it
		vol, err := container.daemon.volumes.FindOrCreateVolume(path, writable)
		if err != nil {
			return nil, err
		}
		mounts[mountToPath] = &Mount{
			container:   container,
			volume:      vol,
			MountToPath: mountToPath,
			Writable:    writable,
			isBind:      true, // in case the volume itself is a normal volume, but is being mounted in as a bindmount here
		}
	}

	// Get the rest of the volumes
	for path := range container.Config.Volumes {
		// Check if this is already added as a bind-mount
		path = filepath.Clean(path)
		if _, exists := mounts[path]; exists {
			continue
		}

		// Check if this has already been created
		if _, exists := container.Volumes[path]; exists {
			continue
		}
		realPath, err := container.getResourcePath(path)
		if err != nil {
			return nil, fmt.Errorf("failed to evaluate the absolute path of symlink")
		}
		if stat, err := os.Stat(realPath); err == nil {
			if !stat.IsDir() {
				return nil, fmt.Errorf("file exists at %s, can't create volume there", realPath)
			}
		}

This is where the code is blocking. It looks like it is checking if an inode exists and then is blocking the mount on it unless it is a directory.

Comment 8 Daniel Walsh 2015-04-14 14:32:13 UTC
Should open this as an issue with upstream.  I have no idea what the intended behaviour is.

Comment 9 B. Warsing 2015-04-16 18:14:33 UTC
I am not sure what you mean. Are you asking me to do re-open this ticket somewhere else?

Comment 10 Daniel Walsh 2015-04-17 12:07:18 UTC
Yes open it on github with docker-compose or docker.  The problem is I am not sure of the intended behaviour, also since docker-compose is not supported on RHEL, this is not something we even look at.

Comment 12 Daniel Walsh 2015-06-02 18:23:03 UTC
Could you check this with docker-1.6.2

Comment 14 Luwen Su 2015-06-15 06:17:15 UTC
In docker-1.6.2-10.el7.x86_64, the error doesn't exist

Compose version 1.2.0
Docker base_url: http+unix://var/run/docker.sock
Docker version: KernelVersion=3.10.0-243.el7.x86_64, Arch=amd64, ApiVersion=1.18, Version=1.6.2, GitCommit=b79465d/1.6.2, Os=linux, GoVersion=go1.4.2
docker containers <- (all=True)
docker containers -> (list with 0 items)
Creating test_test_1...
docker containers <- (all=True)
docker containers -> (list with 0 items)
docker create_container <- (name=u'test_test_1', image=u'nginx:latest', host_config={'Binds': ['/root/test/sites/test.conf:/etc/nginx/conf.d/default.conf:rw'], 'PortBindings': {'80/tcp': [{'HostPort': '80', 'HostIp': ''}]}, 'NetworkMode': u'bridge', 'Links': [], 'VolumesFrom': []}, volumes={u'/etc/nginx/conf.d/default.conf': {}}, detach=True, ports=[u'80'])
docker create_container -> {u'Id': u'34f457928267bea591b83b9617e50eb9b2e51fcefddedf80ca3596aa9adb2cb9',
 u'Warnings': None}
docker inspect_container <- (u'34f457928267bea591b83b9617e50eb9b2e51fcefddedf80ca3596aa9adb2cb9')
docker inspect_container -> {u'AppArmorProfile': u'',
 u'Args': [u'-g', u'daemon off;'],
 u'Config': {u'AttachStderr': False,
             u'AttachStdin': False,
             u'AttachStdout': False,
             u'Cmd': [u'nginx', u'-g', u'daemon off;'],
             u'CpuShares': 0,
             u'Cpuset': u'',
             u'Domainname': u'',
             u'Entrypoint': None,
...
docker start <- (u'34f457928267bea591b83b9617e50eb9b2e51fcefddedf80ca3596aa9adb2cb9')
docker start -> None
docker containers <- (all=False)
docker containers -> (list with 1 items)

# curl -v 10.66.4.160:80
* About to connect() to 10.66.4.160 port 80 (#0)
*   Trying 10.66.4.160...
* Connected to 10.66.4.160 (10.66.4.160) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 10.66.4.160
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: nginx/1.9.1
< Date: Mon, 15 Jun 2015 06:15:22 GMT
< Content-Type: text/html
< Content-Length: 612
< Last-Modified: Tue, 26 May 2015 15:02:09 GMT
< Connection: keep-alive
< ETag: "55648af1-264"
< Accept-Ranges: bytes
< 
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
* Connection #0 to host 10.66.4.160 left intact


**but since there is no support from rhel docker(no :z/:Z), it gets permission denied if not set the right label in the host.See https://github.com/rhatdan/docker/commit/b79465d2e9a11c61dcba47965fad2dad83f6c31e, the label won't be changed if no z:Z given....
# ls -aZ sites/test.conf 
-rw-r--r--. root root system_u:object_r:svirt_sandbox_file_t:s0 sites/test.conf

Comment 16 errata-xmlrpc 2015-06-23 09:29:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-1167.html