Bug 1441560

Summary: "atomic storage modify --add-device" silently erases all images
Product: [Fedora] Fedora Reporter: Marius Vollmer <mvollmer>
Component: atomicAssignee: smahajan <smahajan>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 26CC: adimania, amurdaca, dwalsh, fkluknav, lsm5, walters
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: atomic-1.18.1-3.fc26 atomic-1.18.1-2.fc25 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-06-15 03:01:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Marius Vollmer 2017-04-12 08:44:01 UTC
Description of problem:

Running "atomic storage modify --add-device /dev/sda" will succeed and the given device can be used for Docker images, but it also plants a time bomb: All images disappear after the next Docker restart.

This happens because "atomic storage modify" will call container-storage-setup, and container-storage-setup will write out a storage configuration for Docker that uses devmapper.  On the next start, Docker will use that configuration instead of the previous overlay2 one.

I think this issue arises because container-storage-setup assumes that Docker uses devmapper (with loopback) by default, but now Docker seems to use overlay2 by default.  (Or I might be confused, I haven't investigated closely.)


Version-Release number of selected component (if applicable):
container-storage-setup-0.2.0-2.fc26.noarch
atomic-1.16.5-1.fc26.x86_64
docker-1.13.1-6.git5be1549.fc26.x86_64

How reproducible:
Always

Steps to Reproduce:
# docker images
REPOSITORY                             TAG                 IMAGE ID            CREATED             SIZE
cockpit/base                           latest              b9b9d906729a        2 days ago          287 MB
docker.io/fedora                       25                  4daa661b467f        12 days ago         231 MB

# atomic storage modify --add-device /dev/sda

# docker pull busybox
Using default tag: latest
Trying to pull repository docker.io/library/busybox ... 
sha256:32f093055929dbc23dec4d03e09dfe971f5973a9ca5cf059cbfb644c206aa83f: Pulling from docker.io/library/busybox
Digest: sha256:32f093055929dbc23dec4d03e09dfe971f5973a9ca5cf059cbfb644c206aa83f
Status: Image is up to date for docker.io/busybox:latest

# docker images
REPOSITORY                             TAG                 IMAGE ID            CREATED             SIZE
cockpit/base                           latest              b9b9d906729a        2 days ago          287 MB
docker.io/fedora                       25                  4daa661b467f        12 days ago         231 MB
docker.io/busybox                      latest              00f017a8c2a6        4 weeks ago         1.11 MB

# systemctl restart docker

# docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
<empty>

Actual results:

"atomic storage modify --add-device" succeeds and images disappear after restarting Docker.

Expected results:

Either "atomic storage modify --add-device" fails (and tells the user that a "storage reset" is needed), or the images don't disappear.

Comment 1 Marius Vollmer 2017-04-12 08:49:36 UTC
Once again, from a clean slate:

# echo STORAGE_DRIVER=overlay2 >>/etc/sysconfig/docker-storage-setup

# docker images
REPOSITORY                             TAG                 IMAGE ID            CREATED             SIZE
cockpit/base                           latest              b9b9d906729a        2 days ago          287 MB
docker.io/fedora                       25                  4daa661b467f        12 days ago         231 MB

# atomic storage modify --add-device /dev/sda

# systemctl restart docker

# docker images
REPOSITORY                             TAG                 IMAGE ID            CREATED             SIZE
cockpit/base                           latest              b9b9d906729a        2 days ago          287 MB
docker.io/fedora                       25                  4daa661b467f        12 days ago         231 MB

Thus, putting "STORAGE_DRIVER=overlay2" into /etc/sysconfig/docker-storage-setup causes container-storage-setup to do the right thing.

Comment 2 smahajan@redhat.com 2017-04-12 17:54:35 UTC
That is correct, docker has moved it's default graph driver to overlay2.
Taking a look into this one.

Marius, thanks for pointing it out !

Shishir

Comment 3 smahajan@redhat.com 2017-04-12 20:41:46 UTC
We have made STORAGE_DRIVER=overlay2 as default in container-storage-setup on some distributions. So I am wondering which Linux distribution are you on ? where devicemapper is still the default.

Shishir

Comment 4 Marius Vollmer 2017-04-13 07:32:19 UTC
(In reply to smahajan from comment #3)
> We have made STORAGE_DRIVER=overlay2 as default in container-storage-setup
> on some distributions.

Ahh, that is great to hear.

>  So I am wondering which Linux distribution are you on
> ? where devicemapper is still the default.

This is on the not-yet-released Fedora 26.  Package versions are in comment 0 above.

Comment 5 Daniel Walsh 2017-04-13 12:12:42 UTC
Fedora 26 is supposed to default to overlay2

Comment 6 smahajan@redhat.com 2017-04-13 16:18:33 UTC
I just tried this on a fedora 27 (Server Edition) and overlay2 is the default.

STEPS I did:

1) dnf remove docker.
2) rm -rf /var/lib/docker
3) rm /etc/sysconfig/docker-storage-setup and rm /etc/sysconfig/docker-storage

I did steps (2) and (3) just to make sure I install on a clean slate.

4) dnf install docker.
5) systemctl start docker
6) cat /etc/sysconfig/docker-storage-setup
   >> STORAGE_DRIVER=overlay2
7) cat /etc/sysconfig/docker-storage
   >> DOCKER_STORAGE_OPTIONS="--storage-driver overlay2 "

docker info is showing that docker is running on overlay2.

Will try on Fedora 26 now.

Shishir

Comment 7 Marius Vollmer 2017-04-18 07:03:13 UTC
Please follow the steps in comment 0 when trying to reproduce this.  Running "atomic storage modify --add-device /dev/somedev" is essential.

Comment 8 Marius Vollmer 2017-04-18 07:04:17 UTC
(In reply to Daniel Walsh from comment #5)
> Fedora 26 is supposed to default to overlay2

The docker daemon defaults to overlay2, but container-storage-setup defaults to devmapper.  I guess the bug is that these two defaults are not in synch.

Comment 9 Marius Vollmer 2017-04-18 07:08:38 UTC
(In reply to Marius Vollmer from comment #7)
> Please follow the steps in comment 0 when trying to reproduce this.  Running
> "atomic storage modify --add-device /dev/somedev" is essential.

Hmm, maybe it isn't.  I didn't read comment 6 carefully enough...

Comment 10 Fedora Update System 2017-06-02 10:21:52 UTC
atomic-1.18.1-2.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-d95fc1ce23

Comment 11 Fedora Update System 2017-06-02 10:22:21 UTC
atomic-1.18.1-2.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-fd61a2effa

Comment 12 Fedora Update System 2017-06-03 04:37:20 UTC
atomic-1.18.1-2.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-d95fc1ce23

Comment 13 Fedora Update System 2017-06-04 19:40:15 UTC
atomic-1.18.1-2.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-fd61a2effa

Comment 14 Fedora Update System 2017-06-05 23:15:07 UTC
atomic-1.18.1-3.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-f0db18f2e1

Comment 15 Fedora Update System 2017-06-06 19:05:30 UTC
atomic-1.18.1-3.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-f0db18f2e1

Comment 16 Fedora Update System 2017-06-15 03:01:12 UTC
atomic-1.18.1-3.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.

Comment 17 Fedora Update System 2017-06-15 09:22:41 UTC
atomic-1.18.1-2.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.