Bug 1326532

Summary:	Running `docker cp` clobbers important mount points on the host
Product:	[Fedora] Fedora	Reporter:	Stephen Gallagher <sgallagh>
Component:	docker	Assignee:	Daniel Walsh <dwalsh>
Status:	CLOSED INSUFFICIENT_DATA	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	24	CC:	adimania, admiller, agoldste, amurdaca, ccoleman, dwalsh, ichavero, jcajka, jchaloup, lsm5, marianne, miminar, nalin, pmorie, vbatts
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-09-22 18:14:09 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Stephen Gallagher 2016-04-13 00:55:06 UTC

Description of problem:
Running `docker cp imagename:/path/to/file /path/to/local/file` results in /sys, /proc, /dev and several other critical mount points being lost from the host operating system. Once this happens, only a hard-reboot of the system is possible (since access to systemd for a clean reboot is unavailable).

The mount points are restored at boot, but until the machine is rebooted, it is essentially entirely unable to function.

Version-Release number of selected component (if applicable):
docker-1.10.3-4.gitf8a9a2a.fc24.x86_64

How reproducible:
Every time

Steps to Reproduce:
1. Install docker
2. `systemctl start docker.service`
3. From a terminal:
sudo docker run --name c7 -it --privileged --net=host --pid=host \
-v /:/rootfs:ro -v /var/run:/var/run:rw -v /sys:/sys:ro -v \
/var/lib/docker:/var/lib/docker:rw centos:centos7 bash
4. From a separate terminal:
sudo docker cp c7:/etc/centos-release /tmp/centos-release
5. Once `docker cp` returns: `ls /proc`

Actual results:
/proc, /sys and /dev are all empty and most actions on the host are now impossible.

Expected results:
The host operating system should not be affected.

Additional info:

Upstream bug: https://github.com/docker/docker/issues/20670

The upstream bug suggests the workaround of adding `MountFlags=slave` to the [Service] section of the docker.service file. This has the effect of preventing the docker containers from propagating mount/unmount activity up to the container host. I don't have sufficient knowledge to comment on what performance and/or functionality effects this might have.

Comment 1 Daniel Walsh 2016-04-13 11:59:53 UTC

Vivek, any ideas what is going on here?

Comment 2 Andy Goldstein 2016-04-13 12:23:30 UTC

There are lots of calls to container.UnmountVolumes in https://github.com/docker/docker/blob/master/daemon/archive.go... I'd start looking there perhaps?

Comment 3 Andy Goldstein 2016-04-13 13:29:46 UTC

Note, this is blocking hack/test-end-to-end-docker.sh in OpenShift.

Comment 4 Daniel Walsh 2016-04-13 13:34:42 UTC

Well you can add the slave mount to docker.service, but this will block being able to do 

docker run -v /source/target:shared fedora /bin/sh

Comment 5 Andy Goldstein 2016-04-13 13:42:45 UTC

Right, but that presumably won't work for us. I'm guessing a better option is to stop docker from unmounting bind mounts?

Comment 6 Vivek Goyal 2016-04-13 13:47:04 UTC

Question is how unmounting in container rootfs leads to unmounting of /proc/ /sys  on host. I don't understand that yet.

Comment 7 Andy Goldstein 2016-04-13 13:50:20 UTC

Is it because e.g /sys is bind mounted from the host into the container? container.UnmountVolumes unmounts all volumes in the container's list of mounts, which includes bind mounts.

Comment 8 Vivek Goyal 2016-04-13 14:16:26 UTC

Sure, but that should just remove the bind mount and the not source of the bind mount.

Comment 9 Vivek Goyal 2016-04-13 14:23:57 UTC

For example, bind mount /sys on a dir and unmount then it only unmounts that particular mount point and not source of bind mount.

# mkdir /root/sys-dest
# mount --bind /sys /root/sys-dest
# umount /root/sys-dest

Now /sys is still around and has not been unmounted.

So something is going on here. Trying to debug it.

Comment 10 Vivek Goyal 2016-04-13 15:57:19 UTC

I see UnmountVolume() does lazy unmount. And that probably means that all the sub mounts will be unmounted. If those submounts are shared with host then these
will be unmounted on host too.

So looks like when "docker cp" mounts the volume, we need to always mount them as "private" or "rprivate" so that after copying the files out of container when these mount points are unmounted, they don't destroy host's mount points.

I will track down where actual mounting is taking place.

Comment 11 Vivek Goyal 2016-04-13 18:33:39 UTC

Proposed a fix.

https://github.com/docker/docker/pull/22009

Comment 12 Vivek Goyal 2016-04-27 17:48:06 UTC

Fixes were merged upstream and backported in fedora. This should be fixed now.

Comment 13 Daniel Walsh 2016-04-27 20:58:59 UTC

Lokesh we need a new version of docker with these fixes.

Comment 14 Antonio Murdaca 2016-04-28 07:26:23 UTC

The docker(s) I built yesterday contains the fix for this.

Comment 15 Daniel Walsh 2016-04-28 13:24:03 UTC

What version? Mark this bug as modified by that verison.

Comment 16 Antonio Murdaca 2016-04-28 13:27:13 UTC

Fixed into docker-1.10.3-6.git964eda6.fc24
could you please test it out? Thx

Comment 17 Fedora Admin XMLRPC Client 2016-06-08 14:11:05 UTC

This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.