Bug 1757078

Summary: Docker cannot start in Fedora 31
Product: [Fedora] Fedora Reporter: Ooyama Yosiyuki <sujiniku>
Component: moby-engineAssignee: Olivier Lemasle <o.lemasle>
Status: CLOSED DUPLICATE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 31CC: adimania, admiller, amurdaca, awilliam, dustymabe, dvukovic, dwalsh, fedoraproject, flast, frantisek.kluknavsky, gwync, ichavero, jcajka, jpopelka, lslebodn, nalin, o.lemasle, ondrej.kolin, pasik, pbokoc, philbates35, rabin, santiago, thomas, vbatts, znmeb
Target Milestone: ---Keywords: CommonBugs
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: https://fedoraproject.org/wiki/Common_F31_bugs#docker-moby-engine
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-10-22 17:29:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Ooyama Yosiyuki 2019-09-30 14:11:25 UTC
Description of problem:
In Fedora 31, If I do the following command in terminal, Docker doesn't start and failed with error.

In my memory, in CentOS 7, Docker started with the same command.
Therefore I think guess that it's perhaps a bug in Fedora 31.

Version-Release number of selected component (if applicable):


How reproducible:
everytimes

Steps to Reproduce:
1. sudo systemctl start docker 
2.
3.

Actual results:
[sujiniku@localhost ~]$ sudo systemctl start docker 
[sudo] password for sujiniku: 
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details.


Expected results:
Docker starts after entering the command "sudo systemctl start docker " .

Additional info:

Comment 1 Daniel Walsh 2019-10-01 11:43:24 UTC
Docker is not a fedora 31 package.  
We have replaced most of the functionality with Podman.

moby-engine is the packaged version of Docker upstream for Fedora, but it does not support the cgroups V2 change in Fedora.

If you boot the machine with the systemd.unified_cgroup_hierarchy=0 it should boot with the older cgroups and docker should work.

Comment 2 M. Edward (Ed) Borasky 2019-10-21 19:55:24 UTC
(In reply to Daniel Walsh from comment #1)
> Docker is not a fedora 31 package.  
> We have replaced most of the functionality with Podman.
> 
> moby-engine is the packaged version of Docker upstream for Fedora, but it
> does not support the cgroups V2 change in Fedora.
> 
> If you boot the machine with the systemd.unified_cgroup_hierarchy=0 it
> should boot with the older cgroups and docker should work.

This needs to be in the release notes. My workflow depends on Docker CE - it has to work everywhere Docker CE does (Windows 10, MacOS and supported Linux distros).

Comment 3 Dusty Mabe 2019-10-22 17:29:25 UTC
(In reply to Daniel Walsh from comment #1)
> Docker is not a fedora 31 package.  
> We have replaced most of the functionality with Podman.
> 
> moby-engine is the packaged version of Docker upstream for Fedora, but it
> does not support the cgroups V2 change in Fedora.
> 
> If you boot the machine with the systemd.unified_cgroup_hierarchy=0 it
> should boot with the older cgroups and docker should work.

I can confirm that setting `systemd.unified_cgroup_hierarchy=0` on the kernel command line enables docker.service (provided by the moby-engine package) to start successfully in Fedora 31.

I agree we do need this highlighted in the release notes, but I'll close this bug out since it's not a bug.

Comment 4 Dusty Mabe 2019-10-22 23:13:55 UTC
Note that I persistently edited the kernel args by opening  /boot/grub2/grubenv and adding `systemd.unified_cgroup_hierarchy=0` to the `kernelopts` line.

Comment 5 Petr Bokoc 2019-10-24 23:11:37 UTC
Hi, regarding Release Notes: This bug has been included in the Common Bugs page on the Wiki, here: https://fedoraproject.org/wiki/Common_F31_bugs#Docker_package_no_longer_available_and_will_not_run_by_default_.28due_to_switch_to_cgroups_v2.29

The Release Notes are a separate document (for historical reasons) that covers high level new features and important changes, however, the common bugs page is always linked in the table of contents in the sidebar, which I believe provides enough visibility.

Comment 6 Adam Williamson 2019-10-29 00:43:56 UTC
I would have preferred this be mentioned in the release notes or release announcement, though, as it is not really a *bug*. It is an intentional change. The Common Bugs page, as the name suggests, is intended to cover bugs - unintentional deficiencies - not things we did on purpose and with full knowledge of the consequences.

Comment 7 Adam Williamson 2019-10-29 02:00:51 UTC
*** Bug 1746355 has been marked as a duplicate of this bug. ***

Comment 8 Daniel Walsh 2019-10-29 12:56:42 UTC
I would say the BUG is that moby-engine/runc do not currently support the settings. once they are updated, and I believe it will happen fairly quickly this will no longer be an issue.

Comment 9 Petr Bokoc 2019-10-29 13:10:50 UTC
(In reply to Adam Williamson from comment #6)
> I would have preferred this be mentioned in the release notes or release
> announcement, though, as it is not really a *bug*. It is an intentional
> change. The Common Bugs page, as the name suggests, is intended to cover
> bugs - unintentional deficiencies - not things we did on purpose and with
> full knowledge of the consequences.

Well, my rationale is that people who run into this might consider it a bug. I'd call it a type of known issue, with the issue being "people are confused by this" - it's just that the common bugs page, while it's used to document basically known issues, isn't named that way and instead the name suggests it's specifically about bugs.

Eh, whatever, I can put it into our relnotes, we do talk about package/feature removals in other parts already so it won't be out of place.

Comment 10 Dusty Mabe 2019-10-29 13:34:23 UTC
(In reply to Adam Williamson from comment #6)
> I would have preferred this be mentioned in the release notes or release
> announcement, though, as it is not really a *bug*. It is an intentional
> change. The Common Bugs page, as the name suggests, is intended to cover
> bugs - unintentional deficiencies - not things we did on purpose and with
> full knowledge of the consequences.

I agree. In the very least mentioning the switch to c groups V2 and that some
container tools might not work (with link to the common bugs entry) would be
useful for peoples discoverability.

Comment 11 Adam Williamson 2019-10-29 15:29:32 UTC
yeah, that - I meant the release notes should treat the switch to CGroups v2 as a significant change to be mentioned, and perhaps mention the impact on Docker in a bit of detail as it's a widely used thing. We could keep the Common Bugs entry also from the "docker not supporting CGroups v2 is a bug" perspective, as Dan suggested.

Comment 12 Gwyn Ciesla 2019-10-29 19:36:13 UTC
I just upgraded a machine to f31, and applied the systemd.unified_cgroup_hierarchy=0 option, and switched to moby-engine, and I can't start docker:

Oct 29 14:33:14 emandlo dockerd[1507]: Error starting daemon: Devices cgroup isn't mounted

Did I miss something?

Comment 13 Daniel Walsh 2019-10-29 21:31:45 UTC
Are you sure the option worked?

cat /proc/self/cgroup 
0::/user.slice/user-3267.slice/user/gnome-terminal-server.service

On a cgroup V1 system this will have many lines.

Comment 14 Gwyn Ciesla 2019-10-29 21:39:01 UTC
Looks like no. 0::/user.slice/user-1002.slice/session-4.scope

Comment 15 Adam Williamson 2019-10-29 23:25:27 UTC
How exactly did you try and set it, Gwyn? Remember in recent Fedoras we have all the BLS shenanigans going on; you need to set kernel args in the grub environment using grub2-editenv , I think.

Comment 16 Ondřej Kolín 2019-10-30 07:44:53 UTC
This affects LXC containers as well.

Comment 17 Daniel Walsh 2019-10-30 13:01:48 UTC
I would be surprised if LXC containers could not handle cgoup V2 yet?

Gwyn you could always take this opportunity to try to use Podman and friends as a replacement for Docker.

BTW there is a lot of work and changes going into runc/libcontainer to allow it to work with cgroup V2, which is great.

Comment 18 Gwyn Ciesla 2019-10-30 13:26:16 UTC
I used the grub2-editenv command in the release notes. And I am looking at podman. :)

Comment 19 Ondřej Kolín 2019-10-30 13:37:58 UTC
(In reply to Daniel Walsh from comment #17)
> I would be surprised if LXC containers could not handle cgoup V2 yet?
> 
> Gwyn you could always take this opportunity to try to use Podman and friends
> as a replacement for Docker.
> 
> BTW there is a lot of work and changes going into runc/libcontainer to allow
> it to work with cgroup V2, which is great.
I tested it in my Fedora 31, upgraded from 30, forum topic is here:
https://ask.fedoraproject.org/t/lxc-problems-after-upgrade-to-31/3874

Comment 20 Adam Williamson 2019-10-30 15:08:42 UTC
Gwyn: so, if you mean the one in Common Bugs, this one:

sudo grub2-editenv - set "$(grub2-editenv - list | grep kernelopts) systemd.unified_cgroup_hierarchy=0"

I should admit that I wrote that but I didn't actually test it, so I might have messed it up :|  Let me try it out now.

...

well, I just did, and it seemed to work. At least, `grub2-editenv - list` afterwards showed the change correctly. But you could try this version instead, I guess:

grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=0"

let me know if that works better, and I'll update the common bugs page...

Comment 21 Gwyn Ciesla 2019-10-30 15:18:20 UTC
Yes, that's the one. The new one works.

Comment 22 Adam Williamson 2019-10-30 15:21:50 UTC
Thanks! I've updated the common bugs page. Sorry for the error.

Comment 24 davu 2019-11-23 07:55:04 UTC
> This affects LXC containers as well.

I can partly confirm this. I installed the lxd snap package. Containers do start and one is able to attach  to them, but those containers don't get an IPv4 IP and stopping those containers only works with the `--force` option. After changing the kernel command line option to use "systemd.unified_cgroup_hierarchy=0" everything worked as expected. 

I know that this is a "me too" without logs. If anything else is needed, please let me know what you might need.