Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Description of problem:
It has been observed Podman fails to create containers on RHEL 8 if CGroups V2 is enabled. If CGroup V2 is enabled and if a container is created and run on RHEL 8, it fails with a message :
~~~
container create failed: container_linux.go:336: starting container process caused "process_linux.go:293: applying cgroup configuration for process caused \"mountpoint for devices not found\""
: internal libpod error
~~~
Version-Release number of selected component (if applicable):
OS Version : Red Hat Enterprise Linux release 8.0 Beta (Ootpa)
Kernel Version : 4.18.0-64.el8.x86_64
Podman Version : podman-1.0.0-1.git82e8011.module+el8+2696+e59f0461.x86_64
How reproducible:
Always
Steps to Reproduce:
1. Install podman on RHEL 8
2. Enable CGroup V2
3. Create/run a container on it
Following are the testing results :
#> cat /proc/cmdline
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-64.el8.x86_64 root=UUID=3c9c2c15-73da-488a-8f19-f229bed2590c ro crashkernel=auto resume=UUID=4fa587d1-bc73-434a-b7fc-f63571ab9b45 systemd.unified_cgroup_hierarchy=1
#> findmnt -R /sys/fs/cgroup
TARGET SOURCE FSTYPE OPTIONS
/sys/fs/cgroup cgroup2 cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate
#> podman images
REPOSITORY TAG IMAGE ID CREATED SIZE
registry.redhat.io/rhel8-beta/rhel latest a80dad1c1953 3 months ago 210 MB
#> podman run --name rhel8test registry.redhat.io/rhel8-beta/rhel ls
container create failed: container_linux.go:336: starting container process caused "process_linux.go:293: applying cgroup configuration for process caused \"mountpoint for devices not found\""
: internal libpod error
Actual results:
Container creation fails with an error :
~~~
container create failed: container_linux.go:336: starting container process caused "process_linux.go:293: applying cgroup configuration for process caused \"mountpoint for devices not found\""
: internal libpod error
~~~
Expected results:
A container should be created successfully.
Additional info:
If CGroup V1 is enabled, the container will be created correctly.
#> cat /proc/cmdline
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-64.el8.x86_64 root=UUID=3c9c2c15-73da-488a-8f19-f229bed2590c ro crashkernel=auto resume=UUID=4fa587d1-bc73-434a-b7fc-f63571ab9b45
#> findmnt -R /sys/fs/cgroup
TARGET SOURCE FSTYPE OPTIONS
/sys/fs/cgroup tmpfs tmpfs ro,nosuid,nodev,noexec,mode=755
├─/sys/fs/cgroup/systemd cgroup cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-age
├─/sys/fs/cgroup/cpu,cpuacct cgroup cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct
├─/sys/fs/cgroup/pids cgroup cgroup rw,nosuid,nodev,noexec,relatime,pids
├─/sys/fs/cgroup/rdma cgroup cgroup rw,nosuid,nodev,noexec,relatime,rdma
├─/sys/fs/cgroup/net_cls,net_prio cgroup cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio
├─/sys/fs/cgroup/cpuset cgroup cgroup rw,nosuid,nodev,noexec,relatime,cpuset
├─/sys/fs/cgroup/devices cgroup cgroup rw,nosuid,nodev,noexec,relatime,devices
├─/sys/fs/cgroup/hugetlb cgroup cgroup rw,nosuid,nodev,noexec,relatime,hugetlb
├─/sys/fs/cgroup/freezer cgroup cgroup rw,nosuid,nodev,noexec,relatime,freezer
├─/sys/fs/cgroup/perf_event cgroup cgroup rw,nosuid,nodev,noexec,relatime,perf_event
├─/sys/fs/cgroup/memory cgroup cgroup rw,nosuid,nodev,noexec,relatime,memory
└─/sys/fs/cgroup/blkio cgroup cgroup rw,nosuid,nodev,noexec,relatime,blkio
#> podman run --name rhel8test -it a80dad1c1953
bash-4.4# cat /etc/redhat-release
Red Hat Enterprise Linux release 8.0 Beta (Ootpa)
This is a known issue. Can we make this public? No reason that I see to hide it.
Containers tools/specifically runc do not support CGroupsV2 yet. There is a large effort to add support going on right now, but there will need to be backported changes to the kernel, runc, podman and conmon to make it work.
This will not happen that quickly. I would guess RHEL8.2 we might get support.
We have opened a Change Request for Fedora 31 to move to CGroupsV2 by default to try to spur on the effort.
https://fedoraproject.org/wiki/Changes/CGroupsV2
(In reply to Daniel Walsh from comment #1)
> This is a known issue. Can we make this public? No reason that I see to
> hide it.
Sure, I have made the BZ public.
Dan and Giuseppe I think crun is set for RHEL 8.3, is there anything further than Jindrich needs to do? Or am I off base and we need to push this to 8.4?
Derrick, I'll let Matt or Jindrick correct me if I'm off base, but this is a crun only update at this point. The changes necessary for Podman to run with crun were put into place in earlier versions of Podman.
Also worked on the following components w/ CgroupV2 enabled.
[root@ibm-x3650m4-01-vm-15 ~]# rpm -q runc crun podman kernel
runc-1.0.0-68.rc92.module+el8.3.1+8686+2a59bca3.x86_64
crun-0.15.1-1.module+el8.3.1+8686+2a59bca3.x86_64
podman-2.1.1-3.module+el8.3.1+8686+2a59bca3.x86_64
kernel-4.18.0-240.3.1.el8_3.x86_64
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (Moderate: container-tools:rhel8 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHSA-2021:0531