Bug 1321640 - docker recovery after destruction of /var/lib/docker
Summary: docker recovery after destruction of /var/lib/docker
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: doc-RHEL-Atomic
Version: 7.4
Hardware: Unspecified
OS: Unspecified
Target Milestone: rc
: ---
Assignee: Yoana Ruseva
QA Contact: Vikram Goyal
Vikram Goyal
Depends On:
TreeView+ depends on / blocked
Reported: 2016-03-28 17:50 UTC by Eric Paris
Modified: 2019-03-06 01:20 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Last Closed: 2016-05-16 10:49:16 UTC
Target Upstream Version:

Attachments (Terms of Use)

Description Eric Paris 2016-03-28 17:50:44 UTC
I stopped docker and ran rm -rf /var/lib/docker
I then tried to start docker and it failed with:

level=fatal msg="Error starting daemon: error initializing graphdriver: Unable to take ownership of thin-pool (dockerVG-docker--pool) that already has used data blocks"

Eventually Vgoyal pointed me to:


Which describes deleting the lv and restarting docker. This solved my problem.

However this document was completely undiscoverable. The error message does not show up in the doc. Nor does the doc describe that deleting /var/lib/docker requires deleting the lv.

I would like for searched on the error message from docker to point me to a solution.

Comment 4 Chris Negus 2016-04-27 03:46:20 UTC
I wrote a KCS solution (https://access.redhat.com/solutions/2281281) that includes the issue (deleting the /var/lib/docker directory), the error message that occurs when starting docker, and a pointer to the docs that addresses the problem. Yoana, could you check with Eric Paris and make sure this addresses the issue? Thanks!

Comment 5 Eric Paris 2016-04-27 12:20:10 UTC
I feel like the kcs kinda misses telling users the actual problem. Nor does it really make it clear the solution.

IF you are using device mapper (instead of loopback) /var/lib/docker contains metadata informing docker about the contents of the device mapper storage area. If you delete /var/lib/docker that metadata is lost. Docker is then able to detect that the thin pool has data but docker is unable to make use of that information. The only solution is to delete the thin pool and recreate it so that both the thin pool and the metadata in /var/lib/docker will be empty.

Comment 6 Chris Negus 2016-04-27 20:02:23 UTC
I took another shot at correcting this solution: https://access.redhat.com/solutions/2281281. Eric, let me know if that covers the issue or requires further changes. Feel free to make edits yourself. Once you okay it, I'll publish it as verified.

Comment 7 Yoana Ruseva 2016-05-02 15:09:22 UTC
Hi Eric, does the KCS solution cover what you had in mind for this BZ? I think in this case the guide won't need updating since the article is more to the point.

Comment 8 Eric Paris 2016-05-02 15:22:15 UTC
It is good. I do not believe one line is quite right
# rm /etc/sysconfig/docker-storage-setup

The directions should likely tell you to
- stop docker
- lvremove
- configure /etc/sysconfig/docker-storage-setup
- start docker

Comment 9 Yoana Ruseva 2016-05-12 10:45:15 UTC
Eric, what do you mean with "configure /etc/docker-storage-setup"? Edit the file directly?

Comment 10 Eric Paris 2016-05-12 13:32:52 UTC
That line might be better written as:

- stop docker
- lvremove
- configure docker-storage-setup according to https://access.redhat.com/documentation/en/red-hat-enterprise-linux-atomic-host/version-7/getting-started-with-containers/#managing_storage_with_docker_formatted_containers
- start docker

Comment 11 Yoana Ruseva 2016-05-12 14:02:08 UTC
Ok, edited. Thanks, Eric.


Comment 12 Eric Paris 2016-05-12 14:05:26 UTC
looks great!

Note You need to log in before you can comment on or make changes to this bug.