Bug 1037634

Summary: The devicemapper volumes are not removed after container is removed
Product: [Fedora] Fedora Reporter: Michal Fojtik <mfojtik>
Component: docker-ioAssignee: Lokesh Mandvekar <lsm5>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 20CC: agk, alexl, dwalsh, golang-updates, jkeck, lsm5, mattdm, mgoldman, raina, ricardo.arguello, vbatts
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-05-28 18:19:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Michal Fojtik 2013-12-03 14:10:39 UTC 
Description of problem:

After removing all containers in Docker, some of the devicemapper volumes are not removed and they are showed in 'df -h'

Version-Release number of selected component (if applicable):

[root@localhost postgresql]# docker -v
Docker version 0.7.0, build 0ff9bc1/0.7.0


How reproducible:

Create some containers and then use 'docker rm CONTAINER_ID' and remove them all. When the list of all containers (docker ps -a), is empty,
run 'df -h'. There are some devicemapper volumes from removed container still mounted in system.
After restarting docker service they seems to be properly removed.

Steps to Reproduce:
1. docker run ... (create some containers)
2. docker rm ALL_CONTAINERS
3. df -h (-> will show several devicemapper volumes mounted)
4. systemctl restart docker
5. df -h (-> all devicemapper volumes are erased)

Actual results:

[root@localhost postgresql]# docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
[root@localhost postgresql]#

[root@localhost postgresql]# df -h | grep mapper
/dev/mapper/fedora-root                                                                               9.2G  5.0G  3.7G  58% /
/dev/dm-3                                                                                             9.8G  390M  8.9G   5% /var/lib/docker/devicemapper/mnt/64e0619c473e974ac710568673fa0907fe9bc78f6998888ff41913cf6a69af0b
/dev/dm-4                                                                                             9.8G  388M  8.9G   5% /var/lib/docker/devicemapper/mnt/1bdd590de956281c7a5b1394cc0801ee879bc220be52e399eb61a9c3f2810c0d
/dev/dm-5                                                                                             9.8G  235M  9.0G   3% /var/lib/docker/devicemapper/mnt/97fc5bf7f8d42606fa896e1d391a0b882f78322ce0ff77c03fbd7f8e3b7a73ed
/dev/dm-6                                                                                             9.8G  388M  8.9G   5% /var/lib/docker/devicemapper/mnt/a567e6f3d26a5736d00a5e9be9a609b44ab13f0e43fc466f45beaa7ea9054766
/dev/mapper/docker-253:1-15108-7ee96a4f5167932745b68de0d026d750d12afa911a0e2591edd48efebdc8eb2f-init  9.8G  390M  8.9G   5% /var/lib/docker/devicemapper/mnt/7ee96a4f5167932745b68de0d026d750d12afa911a0e2591edd48efebdc8eb2f-init
/dev/mapper/docker-253:1-15108-c8c302f735e607c55ac4ae542fda9081959f128abbe3b126fa9afa541c84d52d-init  9.8G  518M  8.7G   6% /var/lib/docker/devicemapper/mnt/c8c302f735e607c55ac4ae542fda9081959f128abbe3b126fa9afa541c84d52d-init
/dev/mapper/docker-253:1-15108-5d0694e3a73324e395adfd03d7524c76f126c7449a4f5b73bac64b7bf26f51ef-init  9.8G  556M  8.7G   6% /var/lib/docker/devicemapper/mnt/5d0694e3a73324e395adfd03d7524c76f126c7449a4f5b73bac64b7bf26f51ef-init
/dev/mapper/docker-253:1-15108-f8a7e54667e09c1e9f3edf104124279d2e34fa1bade1ccbd8772a85369132cb6-init  9.8G  591M  8.7G   7% /var/lib/docker/devicemapper/mnt/f8a7e54667e09c1e9f3edf104124279d2e34fa1bade1ccbd8772a85369132cb6-init
/dev/mapper/docker-253:1-15108-fcf2d918ff47b9f24b1a9dcd6ea2829d4b20a5736f06d0a6d987c5cce74d4a55-init  9.8G  591M  8.7G   7% /var/lib/docker/devicemapper/mnt/fcf2d918ff47b9f24b1a9dcd6ea2829d4b20a5736f06d0a6d987c5cce74d4a55-init
/dev/mapper/docker-253:1-15108-004e09e25c19d765db7c75a767fffead530cac52aefd82d6acec5a9e765d4149-init  9.8G  591M  8.7G   7% /var/lib/docker/devicemapper/mnt/004e09e25c19d765db7c75a767fffead530cac52aefd82d6acec5a9e765d4149-init
/dev/mapper/docker-253:1-15108-5f32ced8736080d768cf72f210cda8ae06ef23d7018e5f07819c4c59607ae488-init  9.8G  591M  8.7G   7% /var/lib/docker/devicemapper/mnt/5f32ced8736080d768cf72f210cda8ae06ef23d7018e5f07819c4c59607ae488-init
Comment 1 Michal Fojtik 2013-12-03 14:11:41 UTC 
[root@localhost postgresql]# docker images -a
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
mattdm/fedora       f19                 a567e6f3d26a        5 days ago          348.4 MB (virtual 348.4 MB)
mattdm/fedora       latest              64e0619c473e        6 days ago          349.8 MB (virtual 349.8 MB)
mattdm/fedora       f20                 1bdd590de956        5 weeks ago         348 MB (virtual 348 MB)
mattdm/fedora       f20rc3.small        97fc5bf7f8d4        10 weeks ago        198.4 MB (virtual 198.4 MB)
Comment 2 Alexander Larsson 2013-12-03 14:37:13 UTC 
There are two issues here. 
First, there is a bug where we never remove the -init layers. 
And secondly, there are mounts for the images, which will go away if you remove them (rmi), but these mounts are kinda unnecessary as they are not currenly in use, and we should probably unmount them to keep the system "cleaner".
Comment 3 Michal Fojtik 2013-12-03 16:47:55 UTC 
Also there might be additional bug, related to this. I just discovered that my /var/lib/docker folder use 11G of space, with no containers running and no extra images... Seems like they will need to be collected as well :-)

in /var/lib/docker:

11G     ./devicemapper
....
3.2G    ./devicemapper/mnt
11G     ./devicemapper
Comment 4 Alexander Larsson 2013-12-04 08:07:17 UTC 
Michal: Not sure what you mean, you seem to have some mounts with 3.2 gigs of data in /var/lib/devicemapper/mnt, which implies that you have either a container or an image. Plus, the loopback mounts in the devicemapper folder are not zero size even if there are no containers/images in it.
Comment 5 Alexander Larsson 2013-12-06 11:42:39 UTC 
The -init layer deletion is fixed in master and will be in 0.7.1

This will make us unmount not used devices: https://github.com/dotcloud/docker/pull/3073