Bug 1459700 - 'overlay2' is not supported over btrfs
Summary: 'overlay2' is not supported over btrfs
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: docker
Version: 26
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Antonio Murdaca
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-06-07 22:10 UTC by Evan Nemerson
Modified: 2017-07-19 16:21 UTC (History)
13 users (show)

Fixed In Version: docker-1.13.1-19.git27e468e.fc26
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-06-24 03:05:45 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Evan Nemerson 2017-06-07 22:10:50 UTC
Docker fails to start on Fedora 26 when using btrfs.  `systemctl start docker` reports a failure, `journalctl -xe` says:

- Unit docker.service has begun starting up.
Jun 07 15:09:14 peltast dockerd-current[3678]: time="2017-06-07T15:09:14-07:00" level=info msg="SUSE:secrets :: enabled"
Jun 07 15:09:15 peltast dockerd-current[3678]: time="2017-06-07T15:09:15.032034670-07:00" level=error msg="'overlay2' is not supported over btrfs"
Jun 07 15:09:15 peltast dockerd-current[3678]: Error starting daemon: error initializing graphdriver: backing file system is unsupported for this graph driver
Jun 07 15:09:15 peltast systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Jun 07 15:09:15 peltast systemd[1]: Failed to start Docker Application Container Engine.
-- Subject: Unit docker.service has failed
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit docker.service has failed.

Comment 1 Daniel Walsh 2017-06-08 13:51:21 UTC
You have to change the driver to something other then overlay*

Maybe try btrfs.

Comment 2 Evan Nemerson 2017-06-08 14:19:33 UTC
This worked out of the box on F25, it seems like a regression that I can no longer simply `dnf install docker` and have a working installation.

It's also highly non-obvious what it should be, or where the it should be set.  I didn't even know "btrfs" was an option, and changing the value in /etc/sysconfig/docker-storage-setup has no effect.

Comment 3 Daniel Walsh 2017-06-08 14:20:55 UTC
In F25 we shipped the default as devicemapper, in F26 we changed the default to overlay2 which matches the upstream default.

Comment 4 Vivek Goyal 2017-06-08 14:29:14 UTC
Yep, you need to go back to using devicemapper graph driver for docker. You can try following.

- systemctl stop docker
- atomic storage reset
- Edit /etc/sysconfig/docker-storage-setup and add following
  STORAGE_DRIVER=devicemapper
- systemctl start docker

This should work as long as you have free space in your root volume group. Otherwise you will need to add additional disk to your VM/system and specify it 
in /etc/sysconfig/docker-storage-setup

DEVS="/dev/vdb"
STORAGE_DRIVER=devicemapper

Comment 5 Evan Nemerson 2017-06-08 18:33:07 UTC
(In reply to Daniel Walsh from comment #3)
> In F25 we shipped the default as devicemapper, in F26 we changed the default
> to overlay2 which matches the upstream default.

I understand, but as a user what I see is that, with a btrfs system, docker used to work and now it doesn't, which seems like an undesirable regression to me.  Even upgrading (with dnf system-upgrade) will break a working installation.  

Maybe an appropriate storage backend should be detected automatically instead of being set to overlay2?  If not, a better error message would be quite helpful.

(In reply to Vivek Goyal from comment #4)
> Yep, you need to go back to using devicemapper graph driver for docker. You
> can try following.
> 
> - systemctl stop docker
> - atomic storage reset
> - Edit /etc/sysconfig/docker-storage-setup and add following
>   STORAGE_DRIVER=devicemapper
> - systemctl start docker

This doesn't work; /etc/sysconfi/docker-storage doesn't use $STORAGE_DRIVER, it hard-codes overlay2.  The entire contents of that file are:

    DOCKER_STORAGE_OPTIONS="--storage-driver overlay2 "

This is where you have to change the storage driver, and s/overlay2/devicemapper/ there does work as expected.

It's rather difficult to discover this, or even /etc/sysconfig/docker-storage-setup.  Docker's documentation seems to expect you to set it in /etc/docker/daemon.js, but doing so results in failure (an error about the value conflicting with a command line argument, IIRC).

Comment 6 Vivek Goyal 2017-06-08 19:07:45 UTC
I want you to edit /etc/sysconfig/docker-storage-setup and not /etc/sysconfig/docker-storage.

There is subtle difference. Notice extra *setup*.

Comment 7 Vivek Goyal 2017-06-08 19:08:50 UTC
I noticed that upstream docker recently removed the restriction of overlay2 on top of btrfs. Saying it should work for kernel 4.7 onwards. May be we should backport that patch.

https://github.com/moby/moby/commit/f64a4ad008e68996afcec3ab34a869887716f944


Support overlay2 on btrfs
OverlayFS is supported on top of btrfs as of Linux Kernel 4.7.
Skip the hard enforcement when on kernel 4.7 or newer and
respect the kernel check override flag on older kernels.
https://btrfs.wiki.kernel.org/index.php/Changelog#By_feature

Signed-off-by: Derek McGowan <derek>

Comment 8 Evan Nemerson 2017-06-08 19:47:21 UTC
(In reply to Vivek Goyal from comment #6)
> I want you to edit /etc/sysconfig/docker-storage-setup and not
> /etc/sysconfig/docker-storage.
> 
> There is subtle difference. Notice extra *setup*.

Yes, that's what I was trying to point out.  I tried changing the value in /etc/sysconfig/docker-storage-setup and it didn't do anything.  The file exists, and there is a STORAGE_DRIVER variable, but setting it has no effect.

However, /etc/sysconfig/docker-storage also exists, and uses the value "overlay2" (instead of referencing the STORAGE_DRIVER variable from /etc/sysconfig/docker-storage-setup.  Changing the value in /etc/sysconfig/docker-storage-setup from overlay2 to devicemapper *does* work.

This seems like a bug in the configuration scripts for the docker package.  Should I file a separate issue for that?

> I noticed that upstream docker recently removed the restriction of overlay2
> on top of btrfs. Saying it should work for kernel 4.7 onwards. May be we
> should backport that patch.

I'd be fine with that.

I know btrfs isn't the default filesystem, and I understand that there are complications with docker storage drivers and filesystems, but that doesn't justify failing like this.  `dnf install docker && systemctl start docker` should just work, even on btrfs, but it doesn't.  Which storage driver to use is an implementation detail I shouldn't need to know, and certainly shouldn't need to tell the OS.

Comment 9 Vivek Goyal 2017-06-08 20:16:55 UTC
(In reply to Evan Nemerson from comment #8)
> (In reply to Vivek Goyal from comment #6)
> > I want you to edit /etc/sysconfig/docker-storage-setup and not
> > /etc/sysconfig/docker-storage.
> > 
> > There is subtle difference. Notice extra *setup*.
> 
> Yes, that's what I was trying to point out.  I tried changing the value in
> /etc/sysconfig/docker-storage-setup and it didn't do anything.  The file
> exists, and there is a STORAGE_DRIVER variable, but setting it has no effect.
> 
> However, /etc/sysconfig/docker-storage also exists, and uses the value
> "overlay2" (instead of referencing the STORAGE_DRIVER variable from
> /etc/sysconfig/docker-storage-setup.  Changing the value in
> /etc/sysconfig/docker-storage-setup from overlay2 to devicemapper *does*
> work.

Can you check your journal logs and look for messages from container-storage-setup.

Did you do "atomic storage reset" before setting STORAGE_DRIVER=. atomic storage reset should have removed /etc/sysconfig/docker-storage file. If it is still there, that means you tried to change STORAGE_DRIVER, without first resetting existing storage. That means container storage setup should exit with an error and that might be in journal logs.

We probably should make docker fail by default if container-storage-setup failed. (If we are not already doing it).

Ideally following should happen.

- atomic storage reset will cleanup existing storage. And will remove /etc/sysconfig/docker-storage file.

- Then you will put STORAGE_DRIVER=devicemapper in /etc/sysconfig/docker-storage-setup

- And restarting docker will start docker-storage-setup.service and that will setup devicemapper storage (as long as there is free space in root volume group). And this will also generate a new /etc/sysconfig/docker-storage file
with storage driver devicemapper.

Not sure why it is not working for you that way. 

> 
> This seems like a bug in the configuration scripts for the docker package. 
> Should I file a separate issue for that?

Anything is fine. Either we can try to first narrow it down in this bz and then track in a separate bz or you can create a new one now.

Comment 10 Daniel Walsh 2017-06-09 11:33:00 UTC
Even, I agree that this is not ideal, but I want to point out something you got wrong.  If you had upgraded from a previous version of docker to this one, the previous storage would have stayed.  So if in f25 you were running with docker, you would have been on devicemapper and when you upgraded to F26 it would have stayed on devicemapper.

I think changing from overlay to devicemapper is not working because you have some kind of existing storage that is causing docker to use overlay.  If you remove the storage it should switch to devicemapper.

systemctl stop docker
atomic storage modify --driver devicemapper
atomic storage reset 
systemctl start docker

Should switch your storage.

I think removing the check about btrfs and overlay is the best solution.

Antonio can you make this change?

Comment 11 Antonio Murdaca 2017-06-09 12:06:33 UTC
Yes, I'm working to backport this asap.

Comment 12 Antonio Murdaca 2017-06-09 18:48:51 UTC
Backported to 1.12.x and 1.13.x branch. Needs a rebuild in fedora now.

Frantisek can you rebuild docker in Fedora 26 from the latest commit here: https://github.com/projectatomic/docker/commits/docker-1.12.6

Comment 13 Frantisek Kluknavsky 2017-06-15 13:39:36 UTC
Maybe... Rebase is in git. Koji does not build for s390 at the moment. Wish for something quickly, a mainframe is falling.

Comment 14 Fedora Update System 2017-06-15 14:20:26 UTC
docker-1.13.1-18.git27e468e.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-740382b040

Comment 15 Fedora Update System 2017-06-15 15:33:31 UTC
docker-latest-1.13-31.git27e468e.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-dd12db5cac

Comment 16 Fedora Update System 2017-06-16 22:24:11 UTC
docker-1.13.1-18.git27e468e.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-740382b040

Comment 17 Fedora Update System 2017-06-16 22:24:37 UTC
docker-latest-1.13-31.git27e468e.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-dd12db5cac

Comment 18 Fedora Update System 2017-06-19 19:53:23 UTC
docker-1.13.1-19.git27e468e.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-36d05500ee

Comment 19 Fedora Update System 2017-06-21 06:21:43 UTC
docker-1.13.1-19.git27e468e.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-36d05500ee

Comment 20 Fedora Update System 2017-06-24 03:05:45 UTC
docker-1.13.1-19.git27e468e.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.

Comment 21 Fedora Update System 2017-07-19 16:21:08 UTC
docker-latest-1.13-31.git27e468e.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.