Bug 1771078 - Failed to start docker: Devices cgroup isn't mounted
Summary: Failed to start docker: Devices cgroup isn't mounted
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Fedora
Classification: Fedora
Component: docker
Version: 31
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Daniel Walsh
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-11-11 20:43 UTC by Juan Orti
Modified: 2020-08-31 12:57 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-08-31 12:57:27 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github docker cli issues 2104 0 None open dockerd: failed to start daemon: Devices cgroup isn't mounted 2020-10-08 01:30:59 UTC

Description Juan Orti 2019-11-11 20:43:36 UTC
Description of problem:
After upgrading to F31, the docker daemon doesn't start.

dockerd-current[843]: Error starting daemon: Devices cgroup isn't mounted

Version-Release number of selected component (if applicable):
docker-1.13.1-68.git47e2230.fc30.x86_64
systemd-243-4.gitef67743.fc31.x86_64
kernel-5.3.8-300.fc31.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Have docker running in F30
2. dnf system-upgrade to F31

Actual results:
Docker doesn't start anymore

Expected results:
A working docker

Additional info:

systemctl status -l docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Mon 2019-11-11 21:18:13 CET; 23min ago
     Docs: http://docs.docker.com
  Process: 796 ExecStart=/usr/bin/dockerd-current --add-runtime oci=/usr/libexec/docker/docker-runc-current --default-runtime=oci --authorization-plugin=rhel-push-plugin --containerd /run/containerd.sock --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --init-path=/usr/libexec/docker/docker-init-current --seccomp-profile=/etc/docker/seccomp.json $OPTIONS $DOCKER_STORAGE_OPTIONS $DOCKER_NETWORK_OPTIONS $ADD_REGISTRY $BLOCK>
 Main PID: 796 (code=exited, status=1/FAILURE)
      CPU: 467ms

nov 11 21:18:07 server.example.com systemd[1]: Starting Docker Application Container Engine...
nov 11 21:18:08 server.example.com dockerd-current[796]: time="2019-11-11T21:18:08.564732494+01:00" level=warning msg="could not change group /var/run/docker.sock to docker: group docker not found"
nov 11 21:18:13 server.example.com dockerd-current[796]: time="2019-11-11T21:18:13.867403996+01:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
nov 11 21:18:13 server.example.com dockerd-current[796]: time="2019-11-11T21:18:13.868712430+01:00" level=warning msg="Your kernel does not support cgroup memory limit"
nov 11 21:18:13 server.example.com dockerd-current[796]: time="2019-11-11T21:18:13.868734694+01:00" level=warning msg="Unable to find cpu cgroup in mounts"
nov 11 21:18:13 server.example.com dockerd-current[796]: time="2019-11-11T21:18:13.868769408+01:00" level=warning msg="Unable to find blkio cgroup in mounts"
nov 11 21:18:13 server.example.com dockerd-current[796]: time="2019-11-11T21:18:13.868784490+01:00" level=warning msg="Unable to find cpuset cgroup in mounts"
nov 11 21:18:13 server.example.com dockerd-current[796]: time="2019-11-11T21:18:13.868840496+01:00" level=warning msg="mountpoint for pids not found"
nov 11 21:18:13 server.example.com dockerd-current[796]: Error starting daemon: Devices cgroup isn't mounted
nov 11 21:18:13 server.example.com systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
nov 11 21:18:13 server.example.com systemd[1]: docker.service: Failed with result 'exit-code'.
nov 11 21:18:13 server.example.com systemd[1]: Failed to start Docker Application Container Engine.


# mount | grep cgroup
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,seclabel,nsdelegate)

Comment 1 greg.hellings 2019-12-18 14:41:15 UTC
Currently docker doesn't support cgroupsv2, which became the default in Fedora 31. Until it does, you can disable cgroupsv2 using the following:


Add GRUB_CMDLINE_LINUX="systemd.unified_cgroup_hierarchy=0" to /etc/default/grub
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
sudo reboot

This has fixed it for me, at least for the time being.

Comment 2 Hin-Tak Leung 2020-03-20 22:42:38 UTC
Argh, me too. Have docker running for a few years, but only use it every couple of months, and just discovered that it is broken, and the message is identical (except the first line - I don't have the "group docker not found" line).

Is there any plan for an forthcoming docker update to suppot cgroups v2 ?

Comment 3 greg.hellings 2020-03-21 03:19:26 UTC
Upstream is aware of it. Upstream is not yet moving to fix it.

You can follow along at the Github link I added to this bug.

Comment 4 Daniel Walsh 2020-03-21 11:40:12 UTC
Any reason not to try Podman, which works pretty much the same as docker and works on cgroup V2.

Comment 5 Lennart Jern 2020-03-21 15:20:01 UTC
(In reply to Daniel Walsh from comment #4)
> Any reason not to try Podman, which works pretty much the same as docker and
> works on cgroup V2.

Podman works fine as a replacement when I just want to run a container, but say I want to play with Kubernetes, then things become tricky. For example Kubernetes in Docker (KIND) only works with docker as far as I know. K3s is another example where Podman doesn't help.

Comment 6 Daniel Walsh 2020-03-22 01:54:45 UTC
CRI-O is for the use case of kubernetes.  Also Podman now has grown in APIV2 the ability to listen on /run/docker.socket and start containers via the API.

Comment 7 Timur Kristóf 2020-04-20 14:18:16 UTC
A feasible workaround to this problem is to use the following kernel parameter: systemd.unified_cgroup_hierarchy=0


Note You need to log in before you can comment on or make changes to this bug.