# docker run --rm -v /:/rootfs -v /var/lib/docker:/var/lib/docker \ docker.io/fedora:latest findmnt -R -n --list / |grep /overlay2 /rootfs/sysroot/ostree/deploy/fedora-atomic/var/lib/docker/overlay2 /dev/mapper/atomicos-docker--root--lv[/overlay2] xfs rw,relatime,seclabel,attr2,inode64,noquota # docker run --rm -v /:/rootfs -v /var/lib/docker:/var/lib/docker \ ls -l /rootfs/sysroot/ostree/deploy/fedora-atomic/var/lib/docker/overlay2 [lots of hashes] Environment: docker-1.13.1-22.gitb5e3294.fc26.x86_64 on Fedora Atomic Host 26
Do you have oci-umount installed?
oci-umount-2.0.0-2.gitf90b64c.fc26.x86_64 This is on Fedora Atomic Host with `atomic host upgrade` as of 2017-10-03 PM. # atomic host status State: idle Deployments: ? fedora-atomic:fedora/26/x86_64/atomic-host Version: 26.131 (2017-09-19 22:29:04) Commit: 98088cb6ed2a4b3f7e4e7bf6d34f9e137c296bc43640b4c1967631f22fe1802f GPGSignature: Valid signature by E641850B77DF435378D1D7E2812A6B4B64DAB85D fedora-atomic:fedora/26/x86_64/atomic-host Version: 26.83 (2017-07-05 13:04:12) Commit: 2e8d8dab1c794ff61ba79e60652a687cd1b3c43ad97f0aac38fd271553e58ee2 GPGSignature: Valid signature by E641850B77DF435378D1D7E2812A6B4B64DAB85D I greatly regret that I am going offline now and will not be able to reply again until, at the hopeful earliest, Friday.
cat /etc/oci-umount.conf
So there are two paths inside container. /var/lib/docker/overlay2 got unmounted but /rootfs/....../overlay2 did not. I think oci-umount logic expected /rootfs/var/lib/docker/overlay2 to be present because atomic does so much of magic with mounts, its not there. Instead, "/rootfs/sysroot/ostree/deploy/fedora-atomic/var/lib/docker/overlay2", is there. But this sounds like an atomic problem. If I do "-v /:/rootfs", shouldn't I see a mount point on /rootfs/var/lib/docker/overlay2 inside container?
Ed, please also paste journalctl logs which are prefixed with umounthook.
I think the /rootfs/sysroot/ostree/deploy/fedora-atomic/var/lib/docker/overlay2 is unrelated and is not a leaked mount point into this container. It is a mount point on the host, but unrelated to the container.
/etc/oci-umount.conf: # This contains a list of paths on host which will be unmounted inside # container. (If they are mounted inside container). # If there is a "/*" at the end, that means only mounts underneath that # mounts (submounts) will be unmounted but top level mount will remain # in place. /var/lib/docker/overlay2 /var/lib/docker/overlay /var/lib/docker/devicemapper /var/lib/docker/containers/* /var/lib/docker-latest/overlay2 /var/lib/docker-latest/overlay /var/lib/docker-latest/devicemapper /var/lib/docker-latest/containers/* /var/lib/containers/storage/lvm /var/lib/containers/storage/devicemapper /var/lib/containers/storage/overlay /var/run/containers/storage
Created attachment 1335366 [details] journalctl -f | grep umounthook Attached: journalctl -f | grep umounthook command run in other window: docker run --rm -v /:/rootfs -v /var/lib/docker:/var/lib/docker docker.io/fedora:latest findmnt -R -n --list / |grep /overlay2
Ed is this still the case with the latest oci-umount?
Dan: yes, it looks like it: # docker run --rm -v /:/rootfs -v /var/lib/docker:/var/lib/docker docker.io/fedora:latest findmnt -R -n --list / |grep /overlay2 / overlay overlay rw,relatime,context="system_u:object_r:container_file_t:s0:c9,c93",lowerdir=/var/lib/docker/overlay2/l/KNWJRV376GNMPNXU2SMUUJGH2F:/var/lib/docker/overlay2/l/THD7KDPGXPUUNNDWWYLPZXM2TJ,upperdir=/var/lib/docker/overlay2/11f60c6ee54e427f4cfd20e30c6a24fb38cdc0e1e745fcf5bb2196344b73fb2d/diff,workdir=/var/lib/docker/overlay2/11f60c6ee54e427f4cfd20e30c6a24fb38cdc0e1e745fcf5bb2196344b73fb2d/work /rootfs/sysroot/ostree/deploy/fedora-atomic/var/lib/docker/overlay2 /dev/mapper/atomicos-root[/ostree/deploy/fedora-atomic/var/lib/docker/overlay2] xfs rw,relatime,seclabel,attr2,inode64,noquota # docker run --rm -v /:/rootfs -v /var/lib/docker:/var/lib/docker docker.io/fedora:latest ls -l /rootfs/sysroot/ostree/deploy/fedora-atomic/var/lib/docker/overlay2 ls: cannot access '/rootfs/sysroot/ostree/deploy/fedora-atomic/var/lib/docker/overlay2/backingFsBlockDev': Permission denied total 0 drwx------. 5 root root 69 Jan 22 21:37 5ddb64baba16bf80ed436f40b46c3e7e0afaf195fb41c397290249698f72e311 drwx------. 5 root root 69 Jan 22 21:37 5ddb64baba16bf80ed436f40b46c3e7e0afaf195fb41c397290249698f72e311-init b?????????? ? ? ? ? ? backingFsBlockDev drwx------. 3 root root 30 Jan 22 21:35 d254fd08ae8efabbfcd5d5c5f4e98457b15dd921466758f2a13cd0eb4502c8b3 drwx------. 2 root root 108 Jan 22 21:37 l host details: # atomic host status State: idle Deployments: ? fedora-atomic:fedora/27/x86_64/atomic-host Version: 27.61 (2018-01-17 15:52:47) Commit: 772ab185b0752b0d6bc8b2096d08955660d80ed95579e13e136e6a54e3559ca9 GPGSignature: Valid signature by 860E19B0AFA800A1751881A6F55E7430F5282EE4 rpms: docker-1.13.1-44.git584d391.fc27.x86_64 oci-umount-2.3.2-1.git3025b19.fc27.x86_64 oci-umount.conf : # This contains a list of paths on host which will be unmounted inside # container. (If they are mounted inside container). # If there is a "/*" at the end, that means only mounts underneath that # mounts (submounts) will be unmounted but top level mount will remain # in place. /var/lib/docker/overlay2 /var/lib/docker/overlay /var/lib/docker/devicemapper /var/lib/docker/containers/* /var/lib/docker-latest/overlay2 /var/lib/docker-latest/overlay /var/lib/docker-latest/devicemapper /var/lib/docker-latest/containers/*
Vivek thoughts?
I think it is a problem where source mount is visible on host in multiple paths and we don't have any mechanism to deal with it. For example, on a regular fedora host, do following. # mkdir /root/foo/foo2 # mount --rbind /var/lib/ /root/foo/foo2 # docker run -ti -v /:/rootfs fedora bash Now inside the container you will still see the overlay2 mount through /rootfs/foo/foo2/docker/ovlerlay2 oci-umount has already taken out two mounts. One at /var/lib/docker/overlay2 and another one at /rootfs/var/lib/docker/overlay2. But it does not know anything about source path on host "/root/foo/foo2/docker/overlay2" and when that path leaks inside container due to "-v /:/rootfs", oci-umount can't do anything about it. To get rid of it, oci-umount will have to be told about it explicitly. So put something like this in /etc/oci-umount.conf /root/foo/foo2/docker/overlay2 Similarly for atomic case, if we put something like this in oci-umount.conf, that should take care of it. /sysroot/ostree/deploy/fedora-atomic/var/lib/docker/overlay2 Ed, can you try modifying /etc/oci-umount.conf and put above and see if it helps.
Yes, it helps: # docker run --rm -v /:/rootfs -v /var/lib/docker:/var/lib/docker docker.io/fedora:latest findmnt -R -n --list / |grep /overlay2 / overlay overlay rw,relatime,context="system_u:object_r:container_file_t:s0:c297,c565",lowerdir=/var/lib/docker/overlay2/l/MU22NS3P366FKOLXVPQSWEDPEH:/var/lib/docker/overlay2/l/EQG7KJX7LIONH4KNW4W6QO4CYC,upperdir=/var/lib/docker/overlay2/1f209110ec4de4a7a504f278b2c6c5fb47650909b36ad9be87f488d89ec00197/diff,workdir=/var/lib/docker/overlay2/1f209110ec4de4a7a504f278b2c6c5fb47650909b36ad9be87f488d89ec00197/work But as you imply, this is just a workaround. The real issue, as I see it, is that oci-umount works on a blacklist basis ("everything is OK except these few") -- there will always be ways to bypass that, almost always unintentionally.
I think that's the right fix. (its not a workaround). No, oci-umount does not work on blacklist basis. It just works on the list of paths (source paths on host) in /etc/oci-umount.conf and if that path is mounted inside container, it unmounts that. It does not care about other mounts other than listed in /etc/oci-umount.conf. /sysroot/ostree/deploy/fedora-atomic/var/lib/docker/overlay2 is not same as /var/lib/docker/overlay2. So we will have to explicitly specify it in /etc/oci-umount.conf. I think atomic kickstart might have to be modified for this. Colin, WDYT.
The /sysroot thing seems like a plain ostree bug - offhand, I think we should be making /sysroot slave and not shared. Let me do some testing.
Upstream PR: https://github.com/ostreedev/ostree/pull/1438
Since this is fixed upstream, we don't need to fix this in oci-umount? Is that correct? Do we have this issue in RHEL 7?