Docker fails to start on Fedora 26 when using btrfs. `systemctl start docker` reports a failure, `journalctl -xe` says: - Unit docker.service has begun starting up. Jun 07 15:09:14 peltast dockerd-current[3678]: time="2017-06-07T15:09:14-07:00" level=info msg="SUSE:secrets :: enabled" Jun 07 15:09:15 peltast dockerd-current[3678]: time="2017-06-07T15:09:15.032034670-07:00" level=error msg="'overlay2' is not supported over btrfs" Jun 07 15:09:15 peltast dockerd-current[3678]: Error starting daemon: error initializing graphdriver: backing file system is unsupported for this graph driver Jun 07 15:09:15 peltast systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE Jun 07 15:09:15 peltast systemd[1]: Failed to start Docker Application Container Engine. -- Subject: Unit docker.service has failed -- Defined-By: systemd -- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit docker.service has failed.
You have to change the driver to something other then overlay* Maybe try btrfs.
This worked out of the box on F25, it seems like a regression that I can no longer simply `dnf install docker` and have a working installation. It's also highly non-obvious what it should be, or where the it should be set. I didn't even know "btrfs" was an option, and changing the value in /etc/sysconfig/docker-storage-setup has no effect.
In F25 we shipped the default as devicemapper, in F26 we changed the default to overlay2 which matches the upstream default.
Yep, you need to go back to using devicemapper graph driver for docker. You can try following. - systemctl stop docker - atomic storage reset - Edit /etc/sysconfig/docker-storage-setup and add following STORAGE_DRIVER=devicemapper - systemctl start docker This should work as long as you have free space in your root volume group. Otherwise you will need to add additional disk to your VM/system and specify it in /etc/sysconfig/docker-storage-setup DEVS="/dev/vdb" STORAGE_DRIVER=devicemapper
(In reply to Daniel Walsh from comment #3) > In F25 we shipped the default as devicemapper, in F26 we changed the default > to overlay2 which matches the upstream default. I understand, but as a user what I see is that, with a btrfs system, docker used to work and now it doesn't, which seems like an undesirable regression to me. Even upgrading (with dnf system-upgrade) will break a working installation. Maybe an appropriate storage backend should be detected automatically instead of being set to overlay2? If not, a better error message would be quite helpful. (In reply to Vivek Goyal from comment #4) > Yep, you need to go back to using devicemapper graph driver for docker. You > can try following. > > - systemctl stop docker > - atomic storage reset > - Edit /etc/sysconfig/docker-storage-setup and add following > STORAGE_DRIVER=devicemapper > - systemctl start docker This doesn't work; /etc/sysconfi/docker-storage doesn't use $STORAGE_DRIVER, it hard-codes overlay2. The entire contents of that file are: DOCKER_STORAGE_OPTIONS="--storage-driver overlay2 " This is where you have to change the storage driver, and s/overlay2/devicemapper/ there does work as expected. It's rather difficult to discover this, or even /etc/sysconfig/docker-storage-setup. Docker's documentation seems to expect you to set it in /etc/docker/daemon.js, but doing so results in failure (an error about the value conflicting with a command line argument, IIRC).
I want you to edit /etc/sysconfig/docker-storage-setup and not /etc/sysconfig/docker-storage. There is subtle difference. Notice extra *setup*.
I noticed that upstream docker recently removed the restriction of overlay2 on top of btrfs. Saying it should work for kernel 4.7 onwards. May be we should backport that patch. https://github.com/moby/moby/commit/f64a4ad008e68996afcec3ab34a869887716f944 Support overlay2 on btrfs OverlayFS is supported on top of btrfs as of Linux Kernel 4.7. Skip the hard enforcement when on kernel 4.7 or newer and respect the kernel check override flag on older kernels. https://btrfs.wiki.kernel.org/index.php/Changelog#By_feature Signed-off-by: Derek McGowan <derek>
(In reply to Vivek Goyal from comment #6) > I want you to edit /etc/sysconfig/docker-storage-setup and not > /etc/sysconfig/docker-storage. > > There is subtle difference. Notice extra *setup*. Yes, that's what I was trying to point out. I tried changing the value in /etc/sysconfig/docker-storage-setup and it didn't do anything. The file exists, and there is a STORAGE_DRIVER variable, but setting it has no effect. However, /etc/sysconfig/docker-storage also exists, and uses the value "overlay2" (instead of referencing the STORAGE_DRIVER variable from /etc/sysconfig/docker-storage-setup. Changing the value in /etc/sysconfig/docker-storage-setup from overlay2 to devicemapper *does* work. This seems like a bug in the configuration scripts for the docker package. Should I file a separate issue for that? > I noticed that upstream docker recently removed the restriction of overlay2 > on top of btrfs. Saying it should work for kernel 4.7 onwards. May be we > should backport that patch. I'd be fine with that. I know btrfs isn't the default filesystem, and I understand that there are complications with docker storage drivers and filesystems, but that doesn't justify failing like this. `dnf install docker && systemctl start docker` should just work, even on btrfs, but it doesn't. Which storage driver to use is an implementation detail I shouldn't need to know, and certainly shouldn't need to tell the OS.
(In reply to Evan Nemerson from comment #8) > (In reply to Vivek Goyal from comment #6) > > I want you to edit /etc/sysconfig/docker-storage-setup and not > > /etc/sysconfig/docker-storage. > > > > There is subtle difference. Notice extra *setup*. > > Yes, that's what I was trying to point out. I tried changing the value in > /etc/sysconfig/docker-storage-setup and it didn't do anything. The file > exists, and there is a STORAGE_DRIVER variable, but setting it has no effect. > > However, /etc/sysconfig/docker-storage also exists, and uses the value > "overlay2" (instead of referencing the STORAGE_DRIVER variable from > /etc/sysconfig/docker-storage-setup. Changing the value in > /etc/sysconfig/docker-storage-setup from overlay2 to devicemapper *does* > work. Can you check your journal logs and look for messages from container-storage-setup. Did you do "atomic storage reset" before setting STORAGE_DRIVER=. atomic storage reset should have removed /etc/sysconfig/docker-storage file. If it is still there, that means you tried to change STORAGE_DRIVER, without first resetting existing storage. That means container storage setup should exit with an error and that might be in journal logs. We probably should make docker fail by default if container-storage-setup failed. (If we are not already doing it). Ideally following should happen. - atomic storage reset will cleanup existing storage. And will remove /etc/sysconfig/docker-storage file. - Then you will put STORAGE_DRIVER=devicemapper in /etc/sysconfig/docker-storage-setup - And restarting docker will start docker-storage-setup.service and that will setup devicemapper storage (as long as there is free space in root volume group). And this will also generate a new /etc/sysconfig/docker-storage file with storage driver devicemapper. Not sure why it is not working for you that way. > > This seems like a bug in the configuration scripts for the docker package. > Should I file a separate issue for that? Anything is fine. Either we can try to first narrow it down in this bz and then track in a separate bz or you can create a new one now.
Even, I agree that this is not ideal, but I want to point out something you got wrong. If you had upgraded from a previous version of docker to this one, the previous storage would have stayed. So if in f25 you were running with docker, you would have been on devicemapper and when you upgraded to F26 it would have stayed on devicemapper. I think changing from overlay to devicemapper is not working because you have some kind of existing storage that is causing docker to use overlay. If you remove the storage it should switch to devicemapper. systemctl stop docker atomic storage modify --driver devicemapper atomic storage reset systemctl start docker Should switch your storage. I think removing the check about btrfs and overlay is the best solution. Antonio can you make this change?
Yes, I'm working to backport this asap.
Backported to 1.12.x and 1.13.x branch. Needs a rebuild in fedora now. Frantisek can you rebuild docker in Fedora 26 from the latest commit here: https://github.com/projectatomic/docker/commits/docker-1.12.6
Maybe... Rebase is in git. Koji does not build for s390 at the moment. Wish for something quickly, a mainframe is falling.
docker-1.13.1-18.git27e468e.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-740382b040
docker-latest-1.13-31.git27e468e.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-dd12db5cac
docker-1.13.1-18.git27e468e.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-740382b040
docker-latest-1.13-31.git27e468e.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-dd12db5cac
docker-1.13.1-19.git27e468e.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-36d05500ee
docker-1.13.1-19.git27e468e.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-36d05500ee
docker-1.13.1-19.git27e468e.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.
docker-latest-1.13-31.git27e468e.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.