Bug 1366268

Hmm, weird, seems to work with the "docker" 1.12.0 at https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=507118 ..Let me check for packaging bugs, or name mismatches RE: appending "-latest"

removing MountFlags=slave from the unitfile seems to make this work.

> removing MountFlags=slave from the unitfile seems to make this work.

Confirmed.

This is not the correct solution.  It will cause other issues.

I think this is caused by oci-register-machine.  Basically if the oci-register-machine sends a message to systemd, it will tell it the container is running with a specific mount point.  If this directory is not mounted in systemd mount namespace, oci-register-machine will fail.

I think it is probably better that we disable oci-register-machine on rhel for now.

Ok lets remove oci-register machine from RHEL7 at this time.  Or at least disable it from  docker-latest requires.

oci-register-machine tells systemd that a container PID is about to run on some path.  /var/lib/docker-latest/devicemapper/mnt/67d94e161c93131596eb64238705733dbf31e2a24c82ee5dfd56b06c1cc9eeb5/rootfs

systemd checks the path and if it does not exist throws an error back at oci-register-machine. which kills the container.

Since docker is in a different namespace then systemd, the path only exists in dockers mount namespace.

Removing MountFlags fixed this issue, but opens us up to other issues.

I think we are better off holding off on oci-register-machine for RHEL until we can run docker in the host namespace

Yes please, let's back it out.  See also https://lists.projectatomic.io/projectatomic-archives/atomic-devel/2016-May/msg00050.html

Well this is actually far worse then just oci-register-machine.

We have a problem with the mount namespace implementation the kernel in
RHEL7.  There is a bug which is being actively worked on in the kernel
but will not be fixed by rhel7.2.7 Maybe not by rhel7.3.  Basically this
bug causes docker to fail to properly start and cleanup containers when
it runs in the host mnt namespace.  We have solved this problem in
docker-1.10 by running the docker daemon in its own mount namespace. 

In docker-latest-1.12 we have tried the same procedure, but it does not
work.  The problem is that the docker daemon is no longer the parent of
the new container, docker-containerd is.
If we configure docker daemon to run in its own namespace, it will setup
the containers rootfs and then send a message to docker-containerd to
run the container, but docker-containerd is in a different namespace so
it fails to see the rootfs and the container fails.  The only way so far
we have figured out to fix this is to run both daemons on the host
namespace (As we do in Fedora), but this triggers the bug in the RHEL
kernel.  Since this is the way we run these daemons in Fedora, we have
not seen this issue.  We have tried playing around with systemd
directives JoinNamespaceOf, but so far have not been able to get this to
work.  We have also noticed that docker daemon will start containerd as
a child if the containerd is not running.  This means that the
containerd ends up being in the namespace of the docker daemon, so that
stuff works.  But we would need to remove the docker-containerd unit
file. We have not experimented fully with this yet.

THis is really a docker issue in that docker can not be setting the mount points up in a different namespace then systemd is in, since it will fail to see the mountpoints.

oci-register-machine is now disabled in rhel7 release.  If you move docker daemon back into the host namespace, it should work properly.

docker-latest-1.12.1-2.el7.x86_64 does not demonstrate this problem.