Description of problem:

Creating multiple (>= 2) containers with the "--rm" option in parallel, frequently leads to some of them to fail to be properly clean up, giving a response to the user like this:

Error response from daemon: Unable to remove filesystem for 815a98e314769f5bedf57c56ffcc9b034e4b12f30c1dc06ab0c0939992cdca36: remove /var/lib/docker/containers/815a98e314769f5bedf57c56ffcc9b034e4b12f30c1dc06ab0c0939992cdca36/shm: device or resource busy


Version-Release number of selected component (if applicable):

Tested with docker-1.12.6-28.git1398f24.el7.x86_64


How reproducible:

~1/3 of the time


Steps to Reproduce:

Manually:
1. Use an application which allows you to control multiple terminals simultaneously (like Terminator).
2. Run "docker run --rm busybox" from 2 or more terminals in parallel.

Scripted:
#! /bin/bash -e

for i in $(seq 5); do
  (
    while true; do
      docker run --rm busybox
      sleep 1
    done
  )&
done

sleep 10
pkill -f test.sh


Actual results:

Some containers will fail to clean up with an error like this:

Error response from daemon: Unable to remove filesystem for 815a98e314769f5bedf57c56ffcc9b034e4b12f30c1dc06ab0c0939992cdca36: remove /var/lib/docker/containers/815a98e314769f5bedf57c56ffcc9b034e4b12f30c1dc06ab0c0939992cdca36/shm: device or resource busy

Additionally, if using the devmapper graphdriver, each time clean up fails, an internal DM device will be leaked (see also BZ #1460728).


Expected results:

Clean up shouldn't fail.


Additional info:

These symptoms are caused by multiple issues:

1. dockerd unmounts container's "/dev/shm" lazily (using syscall.MNT_DETACH), which causes os.RemoveAll call from daemon/delete.go:cleanupContainer to fail with EBUSY.

2. When using the devmapper graphdriver, dockerd also unmounts lazily the DM devices, causing the os.RemoveAll call from daemon/graphdriver/devmapper/driver.go:Remove to fail with EBUSY.

3. When using the devmapper graphdriver with dm.use_deferred_removal=true and dm.use_deferred_deletion=fail (the default options in RHEL7.3), the removal of the DM device is deferred, so it may be still present when reaching the devicemapper.DeleteDevice call in daemon/graphdriver/devmapper/deviceset.go:deleteTransaction, causing it to fail with EBUSY.

Issues (1) and (2) are fixed by upstream's "Do not remove containers from memory on error" (54dcbab25ea4771da303fa95e0c26f2d39487b49). 

Issue (3) can be fixed or worked around with one of these options:

1. Disabling dm.use_deferred_removal. The non-deferred removal path has a retry loop to deal with potential EBUSY.

2. Enabling dm.use_deferred_deletion. This way the delete operation is also deferred, making the whole operation coherent with dm.use_deferred_removal=true.

3. Implement a retry loop for devicemapper.DeleteDevice in daemon/graphdriver/devmapper/deviceset.go:deleteTransaction.