Bug 1765821 - LXC container do not start on Fedora 31
Summary: LXC container do not start on Fedora 31
Keywords:
Status: CLOSED EOL
Alias: None
Product: Fedora
Classification: Fedora
Component: lxc
Version: 31
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Thomas Moschny
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-10-26 11:31 UTC by bengt
Modified: 2020-11-24 20:13 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-11-24 20:13:33 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github lxc lxc issues 2991 0 'None' closed lxc-start failed on cgroup_no_v1=all 2020-12-09 14:59:50 UTC

Internal Links: 1787209

Description bengt 2019-10-26 11:31:40 UTC
Description of problem:
LXC container do not start on Fedora 31

Version-Release number of selected component (if applicable):
lxc-3.0.4-2.fc31.x86_64
lxc-start: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=34a5f559890743039487239ae52e65d885b509c8, for GNU/Linux 3.2.0, stripped

How reproducible:
Very easy

Steps to Reproduce:
1. Fresh install or upgraded Fedora 31
2. lxc-create -n fedora -t download -- -d fedora -r 31 -a amd64
3. lxc-start -n fedora -F

Actual results:
Fail to start container:
lxc-start: fedora: cgroups/cgfsng.c: cg_legacy_set_data: 2299 Failed to setup limits for the "devices" controller. The controller seems to be unused by "cgfsng" cgroup driver or not enabled on the cgroup hierarchy
lxc-start: fedora: start.c: lxc_spawn: 1865 Failed to setup legacy device cgroup controller limits
lxc-start: fedora: start.c: lxc_abort: 1103 No such file or directory - Failed to send SIGKILL to 1040
lxc-start: fedora: start.c: __lxc_start: 2019 Failed to spawn container "fedora"
lxc-start: fedora: tools/lxc_start.c: main: 329 The container failed to start
lxc-start: fedora: tools/lxc_start.c: main: 334 Additional information can be obtained by setting the --logfile and --logpriority options


Expected results:
Container starting.

Additional info:
CGroupsV2 is the new default.
Set kernel commandline option: systemd.unified_cgroup_hierarchy=0 to retain the old default and lxc-start start container.

Comment 1 RafaelRS 2019-11-05 10:09:22 UTC
Mine doesn't work too.

Just FYI here is lxc-checkconfig:
# lxc-checkconfig 
Kernel configuration not found at /proc/config.gz; searching...
Kernel configuration found at /boot/config-5.3.7-301.fc31.x86_64
--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
Warning: newuidmap is not setuid-root
Warning: newgidmap is not setuid-root
Network namespace: enabled

--- Control groups ---
Cgroups: enabled

Cgroup v1 mount points: 


Cgroup v2 mount points: 
/sys/fs/cgroup

Cgroup v1 systemd controller: missing
Cgroup v1 freezer controller: missing
Cgroup namespace: required
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled

--- Misc ---
Veth pair device: enabled, loaded
Macvlan: enabled, not loaded
Vlan: enabled, not loaded
Bridges: enabled, loaded
Advanced netfilter: enabled, not loaded
CONFIG_NF_NAT_IPV4: missing
CONFIG_NF_NAT_IPV6: missing
CONFIG_IP_NF_TARGET_MASQUERADE: enabled, not loaded
CONFIG_IP6_NF_TARGET_MASQUERADE: enabled, not loaded
CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled, loaded
CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled, not loaded
FUSE (for use with lxcfs): enabled, loaded

--- Checkpoint/Restore ---
checkpoint restore: enabled
CONFIG_FHANDLE: enabled
CONFIG_EVENTFD: enabled
CONFIG_EPOLL: enabled
CONFIG_UNIX_DIAG: enabled
CONFIG_INET_DIAG: enabled
CONFIG_PACKET_DIAG: enabled
CONFIG_NETLINK_DIAG: enabled
File capabilities: 

Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig

Comment 2 Oleg Girko 2019-11-06 16:18:39 UTC
Absolutely the same symptoms with kernel 5.3.8-300.fc31.x86_64. Error message is the same (only container name is different), lxc-checkconfig output is the same (only kernel version is different).

Comment 3 Thomas Moschny 2019-11-09 10:39:46 UTC
Discussed here: https://github.com/lxc/lxc/issues/2991

Comment 4 Ryutaroh Matsumoto 2019-12-02 01:48:21 UTC
According to another github thread
https://github.com/lxc/lxc/issues/3183

LXC *3.21* seems to work on CGroup V2 with suitable additions to config as


lxc.cgroup.devices.allow =
lxc.cgroup.devices.deny =
lxc.mount.auto = proc:mixed sys:mixed cgroup:rw:force

Comment 5 Ryutaroh Matsumoto 2019-12-09 21:09:10 UTC
Non-root users cannot start LXC containers as reported by
a Fedora 31 user at https://github.com/lxc/lxc/issues/3221
which is also observed on Ubuntu Eoan.
It is caused by another bug because
another workaround can suppress the symptom as discussed
at https://github.com/lxc/lxc/issues/3221#issuecomment-563413881

Comment 6 Ryutaroh Matsumoto 2020-01-01 00:31:40 UTC
When both host and guest Linux distros are Fedora 31,
lxc-start works fine after doing steps in
https://bugzilla.redhat.com/show_bug.cgi?id=1787093
for the following RPM package versions:
# dnf list --installed | grep lxc
lxc.x86_64                                   3.0.4-2.fc31                           @fedora            
lxc-libs.x86_64                              3.0.4-2.fc31                           @fedora            
lxc-templates.x86_64                         3.0.4-2.fc31                           @fedora

Maybe it is enough to provide some documentation telling that
lxc.cgroup.devices config items must be turned off on CGroup V2 host Linux.

Comment 7 Ryutaroh Matsumoto 2020-01-03 01:49:20 UTC
I have filed bug report against the lack of documentation at
https://bugzilla.redhat.com/show_bug.cgi?id=1787209

Comment 8 naaa 2020-01-29 22:14:59 UTC
I was able to get Ubuntu Bionic/Xenial i386 started and attached using Fedora 31 64 bit non-root (unprivileged).

It seems to me that it may not exactly be a Fedora/Red Hat problem. It took some work to get past various problems which from what I can see are a result of lxc 3.0 changes that were made.

Although, I gave up working with lxc 3.0 containers following that because Ubuntu apt-get has problems retrieving packages so I can't use it but it seems to connect to the internet as it finds and asks to install a package that I want - it just won't get it following that though. The problem there looks to be far too much of a time sink for me.

I'm not going to list the steps as my notes are too convoluted for that and need to be narrowed down. The starting point that matters is here: https://linuxcontainers.org/lxc/getting-started/ and also need the following as well because lxc-net is missing but is definitely needed https://wiki.debian.org/LXC/SimpleBridge. Those two should get people farther along to lxc 3.0 containers here.

Again, I don't think this is a Fedora problem. It seems the lxc devs did some real breaking with 3.0 and people have less understanding.

Also this is normal and doesn't stop you from getting lxc container running:
Cgroup v1 systemd controller: missing
Cgroup v1 freezer controller: missing
Cgroup namespace: required

The Cgroup namespace was removed in 3.0 according to lxc dev https://bugs.archlinux.org/task/30369

So a lot of crazy unknowns with lxc 3.0 lol. I'm looking for a way to install lxc 2.0 now as lxc 3.0 is garbage imo (not to offend lxc devs). Course, I'm assuming 2.0 works fine still. If not, then that will probably be the end of lxc for me.

Comment 9 Ryutaroh Matsumoto 2020-01-30 00:09:46 UTC
(In reply to naaa from comment #8)
> I was able to get Ubuntu Bionic/Xenial i386 started and attached using
> Fedora 31 64 bit non-root (unprivileged).

I wrote a procedure to use lxc on Fedora 31 and newer at
https://bugzilla.redhat.com/show_bug.cgi?id=1787209

If you find something missing, could you add a comment there (or here)?

> So a lot of crazy unknowns with lxc 3.0 lol. I'm looking for a way to
> install lxc 2.0 now as lxc 3.0 is garbage imo (not to offend lxc devs).
> Course, I'm assuming 2.0 works fine still. If not, then that will probably
> be the end of lxc for me.

It seems to me that LXC developer(s) claimed support of pure CGroup V2 hierarchy
without running lxc on a Linux host with pure CGroup V2 hierarchy,
as there are many problems in LXC 3.0.4 and 3.2.1 with pure CGroup V2 hierarchy.
I made tons of complains and issue reports to the LXC github, and most of them
were fixed in the latest github LXC source...

Comment 10 Ben Cotton 2020-11-03 16:54:06 UTC
This message is a reminder that Fedora 31 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 31 on 2020-11-24.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '31'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 31 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 11 Ben Cotton 2020-11-24 20:13:33 UTC
Fedora 31 changed to end-of-life (EOL) status on 2020-11-24. Fedora 31 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.


Note You need to log in before you can comment on or make changes to this bug.