RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1678546 - Podman fails to create containers on RHEL 8 if CGroups V2 is enabled.
Summary: Podman fails to create containers on RHEL 8 if CGroups V2 is enabled.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: podman
Version: 8.0
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: rc
: 8.3
Assignee: Jindrich Novy
QA Contact: Alex Jia
URL:
Whiteboard:
Depends On: 1844322
Blocks: 1186913 1823908
TreeView+ depends on / blocked
 
Reported: 2019-02-19 04:12 UTC by Ishan Kulkarni
Modified: 2023-09-07 19:45 UTC (History)
19 users (show)

Fixed In Version: podman-2.0.5 or newer
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-02-16 14:21:45 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Ishan Kulkarni 2019-02-19 04:12:57 UTC 
Description of problem:

It has been observed Podman fails to create containers on RHEL 8 if CGroups V2 is enabled. If CGroup V2 is enabled and if a container is created and run on RHEL 8, it fails with a message :

~~~
container create failed: container_linux.go:336: starting container process caused "process_linux.go:293: applying cgroup configuration for process caused \"mountpoint for devices not found\""
: internal libpod error
~~~


Version-Release number of selected component (if applicable):

OS Version     : Red Hat Enterprise Linux release 8.0 Beta (Ootpa)
Kernel Version : 4.18.0-64.el8.x86_64
Podman Version : podman-1.0.0-1.git82e8011.module+el8+2696+e59f0461.x86_64


How reproducible:

Always

Steps to Reproduce:
1. Install podman on RHEL 8
2. Enable CGroup V2
3. Create/run a container on it

Following are the testing results :

#> cat /proc/cmdline
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-64.el8.x86_64 root=UUID=3c9c2c15-73da-488a-8f19-f229bed2590c ro crashkernel=auto resume=UUID=4fa587d1-bc73-434a-b7fc-f63571ab9b45 systemd.unified_cgroup_hierarchy=1
 
#> findmnt -R /sys/fs/cgroup
TARGET         SOURCE  FSTYPE  OPTIONS
/sys/fs/cgroup cgroup2 cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate
 
#> podman images
REPOSITORY                           TAG      IMAGE ID       CREATED        SIZE
registry.redhat.io/rhel8-beta/rhel   latest   a80dad1c1953   3 months ago   210 MB
 
#> podman run --name rhel8test registry.redhat.io/rhel8-beta/rhel ls
container create failed: container_linux.go:336: starting container process caused "process_linux.go:293: applying cgroup configuration for process caused \"mountpoint for devices not found\""
: internal libpod error



Actual results:

Container creation fails with an error :

~~~
container create failed: container_linux.go:336: starting container process caused "process_linux.go:293: applying cgroup configuration for process caused \"mountpoint for devices not found\""
: internal libpod error
~~~


Expected results:

A container should be created successfully.


Additional info:

If CGroup V1 is enabled, the container will be created correctly.

#> cat /proc/cmdline
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-64.el8.x86_64 root=UUID=3c9c2c15-73da-488a-8f19-f229bed2590c ro crashkernel=auto resume=UUID=4fa587d1-bc73-434a-b7fc-f63571ab9b45
 
 
#> findmnt -R /sys/fs/cgroup
TARGET                            SOURCE FSTYPE OPTIONS
/sys/fs/cgroup                    tmpfs  tmpfs  ro,nosuid,nodev,noexec,mode=755
├─/sys/fs/cgroup/systemd          cgroup cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-age
├─/sys/fs/cgroup/cpu,cpuacct      cgroup cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct
├─/sys/fs/cgroup/pids             cgroup cgroup rw,nosuid,nodev,noexec,relatime,pids
├─/sys/fs/cgroup/rdma             cgroup cgroup rw,nosuid,nodev,noexec,relatime,rdma
├─/sys/fs/cgroup/net_cls,net_prio cgroup cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio
├─/sys/fs/cgroup/cpuset           cgroup cgroup rw,nosuid,nodev,noexec,relatime,cpuset
├─/sys/fs/cgroup/devices          cgroup cgroup rw,nosuid,nodev,noexec,relatime,devices
├─/sys/fs/cgroup/hugetlb          cgroup cgroup rw,nosuid,nodev,noexec,relatime,hugetlb
├─/sys/fs/cgroup/freezer          cgroup cgroup rw,nosuid,nodev,noexec,relatime,freezer
├─/sys/fs/cgroup/perf_event       cgroup cgroup rw,nosuid,nodev,noexec,relatime,perf_event
├─/sys/fs/cgroup/memory           cgroup cgroup rw,nosuid,nodev,noexec,relatime,memory
└─/sys/fs/cgroup/blkio            cgroup cgroup rw,nosuid,nodev,noexec,relatime,blkio
 
 
#> podman run --name rhel8test -it a80dad1c1953
bash-4.4# cat /etc/redhat-release
Red Hat Enterprise Linux release 8.0 Beta (Ootpa)
Comment 1 Daniel Walsh 2019-02-19 22:17:58 UTC 
This is a known issue.  Can we make this public?  No reason that I see to hide it.

Containers tools/specifically runc do not support CGroupsV2 yet.  There is a large effort to add support going on right now, but there will need to be backported changes to the kernel, runc, podman and conmon to make it work.
This will not happen that quickly.  I would guess RHEL8.2 we might get support.

We have opened a Change Request for Fedora 31 to move to CGroupsV2 by default to try to spur on the effort.

https://fedoraproject.org/wiki/Changes/CGroupsV2
Comment 2 Ishan Kulkarni 2019-02-20 01:18:34 UTC 
(In reply to Daniel Walsh from comment #1)
> This is a known issue.  Can we make this public?  No reason that I see to
> hide it.

Sure, I have made the BZ public.
Comment 3 Daniel Walsh 2019-02-20 16:37:01 UTC 
Thanks now, people looking for this bugzilla will be able to find it.
Comment 4 Daniel Walsh 2019-03-01 11:14:45 UTC 
Had meeting on this, this week and pretty good progress is being made.
Comment 5 Daniel Walsh 2019-08-14 10:49:45 UTC 
Giueseppe, this should be fixed in podman-1.5, correct?  As long as crun is part of RHEL8.1?  Do you know if it has been added as a package?
Comment 6 Giuseppe Scrivano 2019-08-14 14:01:52 UTC 
Not fully, there are still changes going on to support cgroups v2.

I am not sure about the status of the crun package.  Lokesh, do we have it now?
Comment 16 Giuseppe Scrivano 2019-11-22 11:48:46 UTC 
moving to 8.3
Comment 17 Tom Sweeney 2020-06-08 21:11:09 UTC 
Dan and Giuseppe I think crun is set for RHEL 8.3, is there anything further than Jindrich needs to do?   Or am I off base and we need to push this to 8.4?
Comment 18 Daniel Walsh 2020-06-08 21:29:25 UTC 
If Jindrich can confirm that crun is in, then we are all set.
Comment 19 Jindrich Novy 2020-06-09 07:23:01 UTC 
I don't see crun imported yet. Giuseppe is working on it. Let me know if you need any help with this one.
Comment 21 Alex Jia 2020-09-22 10:13:05 UTC 

[root@kvm-08-guest29 ~]# rpm -q podman kernel
podman-2.0.5-4.module+el8.3.0+8152+c5c3262e.x86_64
kernel-4.18.0-239.el8.x86_64

[root@kvm-08-guest29 ~]# cat /proc/cmdline 
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-239.el8.x86_64 root=/dev/mapper/rhel_kvm--08--guest29-root ro crashkernel=auto resume=/dev/mapper/rhel_kvm--08--guest29-swap rd.lvm.lv=rhel_kvm-08-guest29/root rd.lvm.lv=rhel_kvm-08-guest29/swap console=ttyS0,115200 systemd.unified_cgroup_hierarchy=1

[root@kvm-08-guest29 ~]# findmnt -R /sys/fs/cgroup
TARGET         SOURCE  FSTYPE  OPTIONS
/sys/fs/cgroup cgroup2 cgroup2 rw,nosuid,nodev,noexec,relatime,seclabel,nsdelegate

[root@kvm-08-guest29 ~]# podman info | grep -i -A2 runtime 
  ociRuntime:
    name: crun
    package: crun-0.14.1-2.module+el8.3.0+8152+c5c3262e.x86_64

[root@kvm-08-guest29 ~]# podman run --runtime=`which crun` registry.redhat.io/rhel8-beta/rhel ls
Trying to pull registry.redhat.io/rhel8-beta/rhel...
Getting image source signatures
Copying blob 386105ae8b62 done  
Copying blob 619051b1fc41 done  
Writing manifest to image destination
Storing signatures
bin
boot
dev
etc
home
lib
lib64
lost+found
media
mnt
opt
proc
root
run
sbin
srv
sys
tmp
usr
var
Comment 29 Derrick Ornelas 2020-09-24 20:10:54 UTC 
Was there any change to the podman code needed here, or is this really just about getting crun in?
Comment 30 Tom Sweeney 2020-09-25 13:45:43 UTC 
Derrick, I'll let Matt or Jindrick correct me if I'm off base, but this is a crun only update at this point.  The changes necessary for Podman to run with crun were put into place in earlier versions of Podman.
Comment 36 Alex Jia 2020-11-10 14:09:46 UTC 
Also worked on the following components w/ CgroupV2 enabled. 

[root@ibm-x3650m4-01-vm-15 ~]# rpm -q runc crun podman kernel
runc-1.0.0-68.rc92.module+el8.3.1+8686+2a59bca3.x86_64
crun-0.15.1-1.module+el8.3.1+8686+2a59bca3.x86_64
podman-2.1.1-3.module+el8.3.1+8686+2a59bca3.x86_64
kernel-4.18.0-240.3.1.el8_3.x86_64
Comment 38 errata-xmlrpc 2021-02-16 14:21:45 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: container-tools:rhel8 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:0531
Note You need to log in before you can comment on or make changes to this bug.