Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1321640

Summary: docker recovery after destruction of /var/lib/docker
Product: Red Hat Enterprise Linux 7 Reporter: Eric Paris <eparis>
Component: doc-RHEL-AtomicAssignee: Yoana Ruseva <yruseva>
Status: CLOSED CURRENTRELEASE QA Contact: Vikram Goyal <vigoyal>
Severity: unspecified Docs Contact: Vikram Goyal <vigoyal>
Priority: unspecified    
Version: 7.4CC: cnegus, eparis, rhel-docs, yruseva, zdover
Target Milestone: rcKeywords: Documentation
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-05-16 10:49:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Eric Paris 2016-03-28 17:50:44 UTC
I stopped docker and ran rm -rf /var/lib/docker
I then tried to start docker and it failed with:

level=fatal msg="Error starting daemon: error initializing graphdriver: Unable to take ownership of thin-pool (dockerVG-docker--pool) that already has used data blocks"

Eventually Vgoyal pointed me to:

https://access.redhat.com/documentation/en/red-hat-enterprise-linux-atomic-host/version-7/getting-started-with-containers/#changing_docker_storage_configuration

Which describes deleting the lv and restarting docker. This solved my problem.

However this document was completely undiscoverable. The error message does not show up in the doc. Nor does the doc describe that deleting /var/lib/docker requires deleting the lv.

I would like for searched on the error message from docker to point me to a solution.

Comment 4 Chris Negus 2016-04-27 03:46:20 UTC
I wrote a KCS solution (https://access.redhat.com/solutions/2281281) that includes the issue (deleting the /var/lib/docker directory), the error message that occurs when starting docker, and a pointer to the docs that addresses the problem. Yoana, could you check with Eric Paris and make sure this addresses the issue? Thanks!

Comment 5 Eric Paris 2016-04-27 12:20:10 UTC
I feel like the kcs kinda misses telling users the actual problem. Nor does it really make it clear the solution.

IF you are using device mapper (instead of loopback) /var/lib/docker contains metadata informing docker about the contents of the device mapper storage area. If you delete /var/lib/docker that metadata is lost. Docker is then able to detect that the thin pool has data but docker is unable to make use of that information. The only solution is to delete the thin pool and recreate it so that both the thin pool and the metadata in /var/lib/docker will be empty.

Comment 6 Chris Negus 2016-04-27 20:02:23 UTC
I took another shot at correcting this solution: https://access.redhat.com/solutions/2281281. Eric, let me know if that covers the issue or requires further changes. Feel free to make edits yourself. Once you okay it, I'll publish it as verified.

Comment 7 Yoana Ruseva 2016-05-02 15:09:22 UTC
Hi Eric, does the KCS solution cover what you had in mind for this BZ? I think in this case the guide won't need updating since the article is more to the point.

Comment 8 Eric Paris 2016-05-02 15:22:15 UTC
It is good. I do not believe one line is quite right
```
# rm /etc/sysconfig/docker-storage-setup
```

The directions should likely tell you to
- stop docker
- lvremove
- configure /etc/sysconfig/docker-storage-setup
- start docker

Comment 9 Yoana Ruseva 2016-05-12 10:45:15 UTC
Eric, what do you mean with "configure /etc/docker-storage-setup"? Edit the file directly?

Comment 10 Eric Paris 2016-05-12 13:32:52 UTC
That line might be better written as:


- stop docker
- lvremove
- configure docker-storage-setup according to https://access.redhat.com/documentation/en/red-hat-enterprise-linux-atomic-host/version-7/getting-started-with-containers/#managing_storage_with_docker_formatted_containers
- start docker

Comment 11 Yoana Ruseva 2016-05-12 14:02:08 UTC
Ok, edited. Thanks, Eric.

https://access.redhat.com/solutions/2281281

Comment 12 Eric Paris 2016-05-12 14:05:26 UTC
looks great!