docker-latest-1.12.0-4.el7.x86_64 -- pretty much nothing works: # docker run fedora echo hi /usr/bin/docker-latest: Error response from daemon: oci runtime error: rootfs ("/var/lib/docker-latest/devicemapper/mnt/35b92e43e1d11408a6add14576512f506cb9806b8b498b26304762d79c8ae5e5/rootfs") does not exist. I've tried restarting docker-containerd; tried rm -rf /var/lib/docker* and docker-latest-storage-setup --reset; setenforce 0.
Hmm, weird, seems to work with the "docker" 1.12.0 at https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=507118 ..Let me check for packaging bugs, or name mismatches RE: appending "-latest"
removing MountFlags=slave from the unitfile seems to make this work.
> removing MountFlags=slave from the unitfile seems to make this work. Confirmed.
This is not the correct solution. It will cause other issues.
I think this is caused by oci-register-machine. Basically if the oci-register-machine sends a message to systemd, it will tell it the container is running with a specific mount point. If this directory is not mounted in systemd mount namespace, oci-register-machine will fail. I think it is probably better that we disable oci-register-machine on rhel for now.
Ok lets remove oci-register machine from RHEL7 at this time. Or at least disable it from docker-latest requires. oci-register-machine tells systemd that a container PID is about to run on some path. /var/lib/docker-latest/devicemapper/mnt/67d94e161c93131596eb64238705733dbf31e2a24c82ee5dfd56b06c1cc9eeb5/rootfs systemd checks the path and if it does not exist throws an error back at oci-register-machine. which kills the container. Since docker is in a different namespace then systemd, the path only exists in dockers mount namespace. Removing MountFlags fixed this issue, but opens us up to other issues. I think we are better off holding off on oci-register-machine for RHEL until we can run docker in the host namespace
Yes please, let's back it out. See also https://lists.projectatomic.io/projectatomic-archives/atomic-devel/2016-May/msg00050.html
Well this is actually far worse then just oci-register-machine. We have a problem with the mount namespace implementation the kernel in RHEL7. There is a bug which is being actively worked on in the kernel but will not be fixed by rhel7.2.7 Maybe not by rhel7.3. Basically this bug causes docker to fail to properly start and cleanup containers when it runs in the host mnt namespace. We have solved this problem in docker-1.10 by running the docker daemon in its own mount namespace. In docker-latest-1.12 we have tried the same procedure, but it does not work. The problem is that the docker daemon is no longer the parent of the new container, docker-containerd is. If we configure docker daemon to run in its own namespace, it will setup the containers rootfs and then send a message to docker-containerd to run the container, but docker-containerd is in a different namespace so it fails to see the rootfs and the container fails. The only way so far we have figured out to fix this is to run both daemons on the host namespace (As we do in Fedora), but this triggers the bug in the RHEL kernel. Since this is the way we run these daemons in Fedora, we have not seen this issue. We have tried playing around with systemd directives JoinNamespaceOf, but so far have not been able to get this to work. We have also noticed that docker daemon will start containerd as a child if the containerd is not running. This means that the containerd ends up being in the namespace of the docker daemon, so that stuff works. But we would need to remove the docker-containerd unit file. We have not experimented fully with this yet.
THis is really a docker issue in that docker can not be setting the mount points up in a different namespace then systemd is in, since it will fail to see the mountpoints.
oci-register-machine is now disabled in rhel7 release. If you move docker daemon back into the host namespace, it should work properly.
docker-latest-1.12.1-2.el7.x86_64 does not demonstrate this problem.