Bug 1321640 - docker recovery after destruction of /var/lib/docker
docker recovery after destruction of /var/lib/docker
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: doc-RHEL-Atomic (Show other bugs)
7.4
Unspecified Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: Yoana Ruseva
Vikram Goyal
Vikram Goyal
: Documentation
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-03-28 13:50 EDT by Eric Paris
Modified: 2016-05-16 06:49 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-05-16 06:49:16 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Eric Paris 2016-03-28 13:50:44 EDT
I stopped docker and ran rm -rf /var/lib/docker
I then tried to start docker and it failed with:

level=fatal msg="Error starting daemon: error initializing graphdriver: Unable to take ownership of thin-pool (dockerVG-docker--pool) that already has used data blocks"

Eventually Vgoyal pointed me to:

https://access.redhat.com/documentation/en/red-hat-enterprise-linux-atomic-host/version-7/getting-started-with-containers/#changing_docker_storage_configuration

Which describes deleting the lv and restarting docker. This solved my problem.

However this document was completely undiscoverable. The error message does not show up in the doc. Nor does the doc describe that deleting /var/lib/docker requires deleting the lv.

I would like for searched on the error message from docker to point me to a solution.
Comment 4 Chris Negus 2016-04-26 23:46:20 EDT
I wrote a KCS solution (https://access.redhat.com/solutions/2281281) that includes the issue (deleting the /var/lib/docker directory), the error message that occurs when starting docker, and a pointer to the docs that addresses the problem. Yoana, could you check with Eric Paris and make sure this addresses the issue? Thanks!
Comment 5 Eric Paris 2016-04-27 08:20:10 EDT
I feel like the kcs kinda misses telling users the actual problem. Nor does it really make it clear the solution.

IF you are using device mapper (instead of loopback) /var/lib/docker contains metadata informing docker about the contents of the device mapper storage area. If you delete /var/lib/docker that metadata is lost. Docker is then able to detect that the thin pool has data but docker is unable to make use of that information. The only solution is to delete the thin pool and recreate it so that both the thin pool and the metadata in /var/lib/docker will be empty.
Comment 6 Chris Negus 2016-04-27 16:02:23 EDT
I took another shot at correcting this solution: https://access.redhat.com/solutions/2281281. Eric, let me know if that covers the issue or requires further changes. Feel free to make edits yourself. Once you okay it, I'll publish it as verified.
Comment 7 Yoana Ruseva 2016-05-02 11:09:22 EDT
Hi Eric, does the KCS solution cover what you had in mind for this BZ? I think in this case the guide won't need updating since the article is more to the point.
Comment 8 Eric Paris 2016-05-02 11:22:15 EDT
It is good. I do not believe one line is quite right
```
# rm /etc/sysconfig/docker-storage-setup
```

The directions should likely tell you to
- stop docker
- lvremove
- configure /etc/sysconfig/docker-storage-setup
- start docker
Comment 9 Yoana Ruseva 2016-05-12 06:45:15 EDT
Eric, what do you mean with "configure /etc/docker-storage-setup"? Edit the file directly?
Comment 10 Eric Paris 2016-05-12 09:32:52 EDT
That line might be better written as:


- stop docker
- lvremove
- configure docker-storage-setup according to https://access.redhat.com/documentation/en/red-hat-enterprise-linux-atomic-host/version-7/getting-started-with-containers/#managing_storage_with_docker_formatted_containers
- start docker
Comment 11 Yoana Ruseva 2016-05-12 10:02:08 EDT
Ok, edited. Thanks, Eric.

https://access.redhat.com/solutions/2281281
Comment 12 Eric Paris 2016-05-12 10:05:26 EDT
looks great!

Note You need to log in before you can comment on or make changes to this bug.