Bug 1757078

Docker is not a fedora 31 package.  
We have replaced most of the functionality with Podman.

moby-engine is the packaged version of Docker upstream for Fedora, but it does not support the cgroups V2 change in Fedora.

If you boot the machine with the systemd.unified_cgroup_hierarchy=0 it should boot with the older cgroups and docker should work.

(In reply to Daniel Walsh from comment #1)
> Docker is not a fedora 31 package.  
> We have replaced most of the functionality with Podman.
> 
> moby-engine is the packaged version of Docker upstream for Fedora, but it
> does not support the cgroups V2 change in Fedora.
> 
> If you boot the machine with the systemd.unified_cgroup_hierarchy=0 it
> should boot with the older cgroups and docker should work.

This needs to be in the release notes. My workflow depends on Docker CE - it has to work everywhere Docker CE does (Windows 10, MacOS and supported Linux distros).

(In reply to Daniel Walsh from comment #1)
> Docker is not a fedora 31 package.  
> We have replaced most of the functionality with Podman.
> 
> moby-engine is the packaged version of Docker upstream for Fedora, but it
> does not support the cgroups V2 change in Fedora.
> 
> If you boot the machine with the systemd.unified_cgroup_hierarchy=0 it
> should boot with the older cgroups and docker should work.

I can confirm that setting `systemd.unified_cgroup_hierarchy=0` on the kernel command line enables docker.service (provided by the moby-engine package) to start successfully in Fedora 31.

I agree we do need this highlighted in the release notes, but I'll close this bug out since it's not a bug.

Note that I persistently edited the kernel args by opening  /boot/grub2/grubenv and adding `systemd.unified_cgroup_hierarchy=0` to the `kernelopts` line.

Hi, regarding Release Notes: This bug has been included in the Common Bugs page on the Wiki, here: https://fedoraproject.org/wiki/Common_F31_bugs#Docker_package_no_longer_available_and_will_not_run_by_default_.28due_to_switch_to_cgroups_v2.29

The Release Notes are a separate document (for historical reasons) that covers high level new features and important changes, however, the common bugs page is always linked in the table of contents in the sidebar, which I believe provides enough visibility.

I would have preferred this be mentioned in the release notes or release announcement, though, as it is not really a *bug*. It is an intentional change. The Common Bugs page, as the name suggests, is intended to cover bugs - unintentional deficiencies - not things we did on purpose and with full knowledge of the consequences.

*** Bug 1746355 has been marked as a duplicate of this bug. ***

I would say the BUG is that moby-engine/runc do not currently support the settings. once they are updated, and I believe it will happen fairly quickly this will no longer be an issue.

(In reply to Adam Williamson from comment #6)
> I would have preferred this be mentioned in the release notes or release
> announcement, though, as it is not really a *bug*. It is an intentional
> change. The Common Bugs page, as the name suggests, is intended to cover
> bugs - unintentional deficiencies - not things we did on purpose and with
> full knowledge of the consequences.

Well, my rationale is that people who run into this might consider it a bug. I'd call it a type of known issue, with the issue being "people are confused by this" - it's just that the common bugs page, while it's used to document basically known issues, isn't named that way and instead the name suggests it's specifically about bugs.

Eh, whatever, I can put it into our relnotes, we do talk about package/feature removals in other parts already so it won't be out of place.

(In reply to Adam Williamson from comment #6)
> I would have preferred this be mentioned in the release notes or release
> announcement, though, as it is not really a *bug*. It is an intentional
> change. The Common Bugs page, as the name suggests, is intended to cover
> bugs - unintentional deficiencies - not things we did on purpose and with
> full knowledge of the consequences.

I agree. In the very least mentioning the switch to c groups V2 and that some
container tools might not work (with link to the common bugs entry) would be
useful for peoples discoverability.

yeah, that - I meant the release notes should treat the switch to CGroups v2 as a significant change to be mentioned, and perhaps mention the impact on Docker in a bit of detail as it's a widely used thing. We could keep the Common Bugs entry also from the "docker not supporting CGroups v2 is a bug" perspective, as Dan suggested.

I just upgraded a machine to f31, and applied the systemd.unified_cgroup_hierarchy=0 option, and switched to moby-engine, and I can't start docker:

Oct 29 14:33:14 emandlo dockerd[1507]: Error starting daemon: Devices cgroup isn't mounted

Did I miss something?

Are you sure the option worked?

cat /proc/self/cgroup 
0::/user.slice/user-3267.slice/user/gnome-terminal-server.service

On a cgroup V1 system this will have many lines.

Looks like no. 0::/user.slice/user-1002.slice/session-4.scope

How exactly did you try and set it, Gwyn? Remember in recent Fedoras we have all the BLS shenanigans going on; you need to set kernel args in the grub environment using grub2-editenv , I think.

This affects LXC containers as well.

I would be surprised if LXC containers could not handle cgoup V2 yet?

Gwyn you could always take this opportunity to try to use Podman and friends as a replacement for Docker.

BTW there is a lot of work and changes going into runc/libcontainer to allow it to work with cgroup V2, which is great.

I used the grub2-editenv command in the release notes. And I am looking at podman. :)

(In reply to Daniel Walsh from comment #17)
> I would be surprised if LXC containers could not handle cgoup V2 yet?
> 
> Gwyn you could always take this opportunity to try to use Podman and friends
> as a replacement for Docker.
> 
> BTW there is a lot of work and changes going into runc/libcontainer to allow
> it to work with cgroup V2, which is great.
I tested it in my Fedora 31, upgraded from 30, forum topic is here:
https://ask.fedoraproject.org/t/lxc-problems-after-upgrade-to-31/3874

Gwyn: so, if you mean the one in Common Bugs, this one:

sudo grub2-editenv - set "$(grub2-editenv - list | grep kernelopts) systemd.unified_cgroup_hierarchy=0"

I should admit that I wrote that but I didn't actually test it, so I might have messed it up :|  Let me try it out now.

...

well, I just did, and it seemed to work. At least, `grub2-editenv - list` afterwards showed the change correctly. But you could try this version instead, I guess:

grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=0"

let me know if that works better, and I'll update the common bugs page...

Yes, that's the one. The new one works.

Thanks! I've updated the common bugs page. Sorry for the error.

> This affects LXC containers as well.

I can partly confirm this. I installed the lxd snap package. Containers do start and one is able to attach  to them, but those containers don't get an IPv4 IP and stopping those containers only works with the `--force` option. After changing the kernel command line option to use "systemd.unified_cgroup_hierarchy=0" everything worked as expected. 

I know that this is a "me too" without logs. If anything else is needed, please let me know what you might need.