Bug 1508376 - docker-storage-setup fails to start
Summary: docker-storage-setup fails to start
Alias: None
Product: Fedora
Classification: Fedora
Component: docker
Version: 26
Hardware: x86_64
OS: Linux
Target Milestone: ---
Assignee: Frantisek Kluknavsky
QA Contact: Fedora Extras Quality Assurance
Depends On:
TreeView+ depends on / blocked
Reported: 2017-11-01 10:31 UTC by Panos Kavalagios
Modified: 2018-01-17 16:18 UTC (History)
13 users (show)

Clone Of:
Last Closed: 2017-12-09 22:25:39 UTC

Attachments (Terms of Use)

Description Panos Kavalagios 2017-11-01 10:31:29 UTC
Description of problem:
The service docker-storage-setup.service is failing.

Version-Release number of selected component (if applicable):

How reproducible:
Start docker.

Steps to Reproduce:
1. systemctl start docker

Actual results:
● docker-storage-setup.service loaded failed failed Docker Storage Setup

Expected results:
The service should have started succesfully.

Additional info:
● docker-storage-setup.service - Docker Storage Setup
   Loaded: loaded (/usr/lib/systemd/system/docker-storage-setup.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Wed 2017-11-01 12:26:29 EET; 4min 29s ago
  Process: 7074 ExecStart=/usr/bin/container-storage-setup (code=exited, status=1/FAILURE)
 Main PID: 7074 (code=exited, status=1/FAILURE)

Νοέ 01 12:26:29 bb229 systemd[1]: Starting Docker Storage Setup...
Νοέ 01 12:26:29 bb229 container-storage-setup[7074]: INFO: Volume group backing root filesystem could not be determined
Νοέ 01 12:26:29 bb229 container-storage-setup[7074]: ERROR: No valid volume group found. Exiting.
Νοέ 01 12:26:29 bb229 systemd[1]: docker-storage-setup.service: Main process exited, code=exited, status=1/FAILURE
Νοέ 01 12:26:29 bb229 systemd[1]: Failed to start Docker Storage Setup.
Νοέ 01 12:26:29 bb229 systemd[1]: docker-storage-setup.service: Unit entered failed state.
Νοέ 01 12:26:29 bb229 systemd[1]: docker-storage-setup.service: Failed with result 'exit-code'.

Comment 1 Vivek Goyal 2017-11-01 14:11:13 UTC
Can you paste output of lsblk.

docker-storage-setup will need a volume group (with free space) from where it can carve out a logical volume. There is no volume group. So you need to configure it right.

You need to have an extra disk and pass that to docker-storage-setup.

# cat /etc/sysconfig/docker-storage-setup


Comment 2 Panos Kavalagios 2017-11-01 14:36:19 UTC
The command output follows:

$ lsblk
loop0                      7:0    0   100G  0 loop 
└─docker-8:1-677656-pool 253:0    0   100G  0 dm   
loop1                      7:1    0     2G  0 loop 
└─docker-8:1-677656-pool 253:0    0   100G  0 dm   
sda                        8:0    0 447.1G  0 disk 
├─sda1                     8:1    0    50G  0 part /
├─sda2                     8:2    0     4G  0 part [SWAP]
└─sda3                     8:3    0 393.1G  0 part /home
sdb                        8:16   0 931.5G  0 disk 
└─sdb1                     8:17   0 931.5G  0 part /opt
sr0                       11:0    1  1024M  0 rom  

If there is no volume group defined, it should have used the defaults for the root device, right? My /etc/sysconfig/docker-storage-setup is empty.

Comment 3 Vivek Goyal 2017-11-01 14:56:09 UTC
(In reply to Panos Kavalagios from comment #2)
> If there is no volume group defined, it should have used the defaults for
> the root device, right? My /etc/sysconfig/docker-storage-setup is empty.

Right but in your case there is no volume group backing rootfs. Root disk seems to be just partitioned and /dev/sda1 is root. 

That's why docker-storage-setup needs a disk and volume group to work with.

Comment 4 Panos Kavalagios 2017-11-01 15:14:46 UTC
Adding a specific partition is not supported. Is there any other solution for systems without LVM?

Comment 5 Vivek Goyal 2017-11-01 15:48:08 UTC
Create a VG yourself and pass that VG to docker-storage-setup.

# vgcreate docker-vg /dev/foo-part

# cat /etc/docker-storage-setup

Comment 6 Ed Santiago 2017-11-01 17:00:44 UTC
I think there's something else going on here. docker on Ff26 defaults to overlay2. The only way I can reproduce the original problem is by manually changing /etc/sysconfig/docker-storage-setup to add STORAGE_DRIVER=devicemapper

WAG: is it possible that this system was originally installed as f25, and docker was installed but never used, and the system was upgraded to f26 without docker ever having been run?

Comment 7 Panos Kavalagios 2017-11-01 17:12:57 UTC
The VG creation requires empty disk partitions of "Linux LVM" type to be added to the group, correct? So, in a system without LVM and already defined partitions is not possible (ok only with gparted probably).

I confirm that docker was installed on F25 and without any custom configuration, but a docker image of MySQL was successfully tested.

Comment 8 Ed Santiago 2017-11-01 20:42:32 UTC
Quick clarification: you do not have "a system without LVM". docker requires lvm2, so you have the lvm tools installed even if you have no lvm-managed partitions. But this is just a nit.

I've been unable to reproduce this: starting with f25, installing and starting docker, 'docker info' shows thinpool as expected. Upgrading to f26, same thing (because /var/lib/docker was untouched). Stop docker, rm -rf /var/lib/docker, start docker, and now it's running overlay2. @Panos Kavalagios, can you think of any other factors that may be involved here?

Comment 9 Panos Kavalagios 2017-11-02 08:45:53 UTC
LVM is also required by overlay2 storage driver? I don't have spare partition to create any LVM configuration and I would prefer overlayFS:

# docker info
Containers: 2
 Running: 0
 Paused: 0
 Stopped: 2
Images: 1
Server Version: 1.13.1
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: journald
Cgroup Driver: systemd
 Volume: local
 Network: bridge host macvlan null overlay
 Authorization: rhel-push-plugin
Swarm: inactive
Runtimes: oci runc
Default Runtime: oci
Init Binary: /usr/libexec/docker/docker-init-current
containerd version:  (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: N/A (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: N/A (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
  WARNING: You're not using the default seccomp profile
  Profile: /etc/docker/seccomp.json
Kernel Version: 4.13.9-200.fc26.x86_64
Operating System: Fedora 26 (Twenty Six)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 3
CPUs: 4
Total Memory: 15.57 GiB
Name: bb229
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Http Proxy: http://squid.eurodyn.com:8080/
No Proxy: localhost,,.eurodyn.com
Registry: https://registry.fedoraproject.org/v1/
Experimental: false
Insecure Registries:
Live Restore Enabled: false
Registries: registry.fedoraproject.org (secure), registry.access.redhat.com (secure), docker.io (secure)

How I can get rid of the docker-storage-setup.service startup failure with the above configuration without affecting docker normal operation?

Comment 10 Vivek Goyal 2017-11-02 11:47:43 UTC
You can specify to make use of overlay2 driver and then it should succeed.

#cat /etc/sysconfig/docker-storage-setup

Comment 11 Panos Kavalagios 2017-11-02 11:59:42 UTC
OK, thanks that worked. 

Now, if the default has changed to overlay2 in Fedora 26 from devicemapper, shouldn't STORAGE_DRIVER have overlay2 as the default option as well? Or at least docker of F26 to include /etc/sysconfig/docker-storage-setup file with that already set to overlay2.

Comment 12 Vivek Goyal 2017-11-02 12:04:27 UTC
That will happen if you install a fresh F26. But not over upgrade. Over upgrade user config files are supposed to remain same and new files are not supposed to replace existing files/settings.

Comment 13 Panos Kavalagios 2017-11-02 12:40:05 UTC
I would propose something like the following in the RPM spec and %files section:

%config(noreplace) /etc/sysconfig/docker-storage-setup

so that the F26 could be notified for the new option in the .rpmnew version of the file. That would help to avoid configuration bugs like mine.

I have performed no configuration in the docker sysconfig files in F25. I have only removed the /var/lib/docker/ contents to re-initialise the environment. I have also issued "dnf reinstall docker" in order to resolve the reported problem to no avail. Probably the "dnf remove docker" and then "dnf install docker" could have helped instead, but if there was a docker-storage-setup.rpmnew out there I would be fine. Just a simple change that would help the less power docker users after the migration to F26 to see that a specific systemd startup is failing.

Comment 14 Daniel Walsh 2017-11-02 16:00:30 UTC
yes this config should be set that way.

%config(noreplace) /etc/sysconfig/docker-storage-setup

Comment 15 Daniel Walsh 2017-11-02 16:01:11 UTC
Franticek could you make sure that this is the setting?

Comment 16 Vivek Goyal 2017-11-02 16:35:55 UTC

Should "dnf remove docker" have removed /etc/sysconfig/docker-storage-setup, so that fresh installation of docker brings in new file?

What about /etc/sysconfig/docker-storage?

Comment 17 Daniel Walsh 2017-11-02 18:46:20 UTC
Only if they are listed in payload and not modified. (If they are config(noreplace) I believe.

Comment 18 Frantisek Kluknavsky 2017-11-13 16:10:15 UTC
It is "%ghost %config(noreplace) %{_sysconfdir}/sysconfig/%{repo}-storage-setup" in f26 and later since february 2017. F25 lacks %ghost but is still %config(noreplace).

docker-storage is also "%config(noreplace) %{_sysconfdir}/sysconfig/%{repo}-storage" in f26 and later.

Comment 19 Vivek Goyal 2017-11-14 18:39:55 UTC
Ok, I tested on F26, and removal of docker package does not remove /etc/sysconfig/docker-storage-setup. And that's a bug.

%ghost means that file is not part of the package. If that's the case then specifying config(noreplace) after that does not make any sense. I suspect that could be part of the problem. If we just say.

%ghost %{_sysconfdir}/sysconfig/%{repo}-storage-setup

it might work. 

Remember, it needs to be done only if custom_storage needs to be set.

Comment 20 Vivek Goyal 2017-11-14 18:56:02 UTC
I removed config(noreplace) and tested and that fixed the problem. And now file is removed when docker package is removed.

So we need following.

# d-s-s specific
%if %{custom_storage}
%ghost %{_sysconfdir}/sysconfig/%{name}-storage-setup

Comment 21 Daniel Walsh 2017-11-14 19:24:40 UTC
Franticek or Lokesh can you make the change.

Comment 22 Fedora Update System 2017-11-23 16:02:52 UTC
docker-1.13.1-43.git4402c09.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-c0b1fae074

Comment 23 Frantisek Kluknavsky 2017-11-23 16:09:30 UTC
It feels wrong to have a config file not marked as %config, but it seems to do what we want.

Maybe we learn more at https://bugzilla.redhat.com/show_bug.cgi?id=1516938

Comment 24 Fedora Update System 2017-11-25 00:36:43 UTC
docker-1.13.1-43.git4402c09.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-abfe613b7c

Comment 25 Fedora Update System 2017-11-25 01:35:28 UTC
docker-1.13.1-43.git4402c09.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-c0b1fae074

Comment 26 Fedora Update System 2017-12-09 22:25:39 UTC
docker-1.13.1-44.git584d391.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.

Comment 27 Fedora Update System 2018-01-17 16:18:54 UTC
docker-1.13.1-44.git584d391.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.