Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1947999

Summary: rootless podman --cgroup-manager=cgroupfs run command causes OCI permission error when CGroups V2 is enabled
Product: Red Hat Enterprise Linux 8 Reporter: Alex Jia <ajia>
Component: runcAssignee: Jindrich Novy <jnovy>
Status: CLOSED ERRATA QA Contact: Alex Jia <ajia>
Severity: high Docs Contact:
Priority: high    
Version: 8.4CC: dwalsh, gscrivan, jnovy, leiwang, mheon, tsweeney
Target Milestone: betaFlags: pm-rhel: mirror+
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: podman-3.3.0-0.13.el8 or newer Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-11-09 17:37:47 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Alex Jia 2021-04-09 17:29:41 UTC
Description of problem:
podman --cgroup-manager=cgroupfs run command causes OCI permission denied in rootless mode w/ cgroupv2 enabled.

Version-Release number of selected component (if applicable):
[test@kvm-02-guest02 ~]$ cat /etc/redhat-release
Red Hat Enterprise Linux release 8.4 (Ootpa)

[test@kvm-02-guest02 ~]$  rpm -q podman containers-common runc crun kernel
podman-3.0.1-6.module+el8.4.0+10614+dd38312c.x86_64
containers-common-1.2.2-7.module+el8.4.0+10614+dd38312c.x86_64
runc-1.0.0-70.rc92.module+el8.4.0+10614+dd38312c.x86_64
crun-0.18-2.module+el8.4.0+10614+dd38312c.x86_64
kernel-4.18.0-302.el8.x86_64
kernel-4.18.0-304.el8.x86_64

How reproducible:
always

Steps to Reproduce:
1. configure rootless
2. configure cgroupv2 and reboot
3.  podman --cgroup-manager=cgroupfs --runtime=runc run --name myc quay.io/libpod/testimage:20200929 true

Actual results:
$ mount|grep cgroup
cgroup2 on /sys/fs/cgroup type cgroup2
(rw,nosuid,nodev,noexec,relatime,seclabel)

$ podman unshare cat /proc/self/uid_map
         0       1000          1
         1     100000      65536

$ podman --cgroup-manager=cgroupfs run --name myc quay.io/libpod/testimage:20200929 true
Error: container_linux.go:370: starting container process caused: process_linux.go:326: applying cgroup configuration for process caused: mkdir /sys/fs/cgroup/libpod_parent/libpod-8fea41ce357783321a158534ef44ca8ab5ba76b1f6807a31e7b1f12e049a7805: permission denied: OCI permission denied

Expected results:
fix permission denied error

Additional info:

1. it's okay w/o --cgroup-manager=cgroupfs option for runc runtime
[test@kvm-04-guest15 ~]$ podman --runtime=runc run --name myc quay.io/libpod/testimage:20200929 true
[test@kvm-04-guest15 ~]$ echo $?
0
[test@kvm-04-guest15 ~]$ podman ps -a
CONTAINER ID  IMAGE                              COMMAND  CREATED        STATUS                    PORTS   NAMES
8b0fc0cae0ad  quay.io/libpod/testimage:20200929  true     8 seconds ago  Exited (0) 9 seconds ago          myc

2. it's okay for rootfull mode w/ runc runtime
[test@kvm-04-guest15 ~]$ sudo podman --cgroup-manager=cgroupfs --runtime=runc run --name myc quay.io/libpod/testimage:20200929 true
[test@kvm-04-guest15 ~]$ echo $?
0
[test@kvm-04-guest15 ~]$ sudo podman ps -a
CONTAINER ID  IMAGE                              COMMAND  CREATED        STATUS                    PORTS   NAMES
b7cb9a7ae2b1  quay.io/libpod/testimage:20200929  true     8 seconds ago  Exited (0) 9 seconds ago          myc

3. it's okay for crun runtime
[test@kvm-04-guest15 ~]$ podman --cgroup-manager=cgroupfs --runtime=crun run --name myc quay.io/libpod/testimage:20200929 true
[test@kvm-04-guest15 ~]$ echo $?
0
[test@kvm-04-guest15 ~]$ podman ps -a
CONTAINER ID  IMAGE                              COMMAND  CREATED        STATUS                    PORTS   NAMES
7e3ca80467f5  quay.io/libpod/testimage:20200929  true     5 seconds ago  Exited (0) 5 seconds ago          myc

Comment 1 Tom Sweeney 2021-04-09 21:40:50 UTC
Matt, can you take a look at this early next week and see if it's at all related to https://bugzilla.redhat.com/show_bug.cgi?id=1947432?

Comment 3 Matthew Heon 2021-04-12 14:26:36 UTC
I see no reason to believe these two BZs are related. This is a CGroups issue, the previous one was a readonly-paths issue. This looks like, specifically, a permission denied on making a directory in cgroupfs while part of a `podman unshare` shell?

I'm reassigning to Giuseppe given cgroups are more his area of expertise.

Comment 4 Giuseppe Scrivano 2021-04-12 14:56:51 UTC
opened a PR: https://github.com/containers/podman/pull/9996

Comment 5 Alex Jia 2021-05-11 13:22:03 UTC
This bug has been fixed on podman-3.2.0-0.11.

[test@kvm-07-guest01 ~]$ cat /etc/redhat-release
Red Hat Enterprise Linux release 8.5 Beta (Ootpa)
[test@kvm-07-guest01 ~]$ rpm -q podman crun runc kernel
podman-3.2.0-0.11.module+el8.5.0+10952+e23bdbc3.x86_64
crun-0.19.1-1.module+el8.5.0+10952+e23bdbc3.x86_64
runc-1.0.0-72.rc93.module+el8.5.0+10952+e23bdbc3.x86_64
kernel-4.18.0-305.1.el8.x86_64
[test@kvm-07-guest01 ~]$ podman unshare cat /proc/self/uid_map
         0       1000          1
         1     100000      65536
[test@kvm-07-guest01 ~]$ grep cgroup /proc/mounts
cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate 0 0

[test@kvm-07-guest01 ~]$ podman info --format json | jq .host.ociRuntime
{
  "name": "runc",
  "package": "runc-1.0.0-72.rc93.module+el8.5.0+10952+e23bdbc3.x86_64",
  "path": "/usr/bin/runc",
  "version": "runc version spec: 1.0.2-dev\ngo: go1.16.1\nlibseccomp: 2.5.1"
}

[test@kvm-07-guest01 ~]$ podman --cgroup-manager=cgroupfs --runtime=runc run --name myc quay.io/libpod/testimage:20200929 true
Trying to pull quay.io/libpod/testimage:20200929...
Getting image source signatures
Copying blob 5c10a2f1fe01 done
Copying config 766ff5a3a7 done
Writing manifest to image destination
Storing signatures

[test@kvm-07-guest01 ~]$ podman --cgroup-manager=cgroupfs --runtime=crun run --name mycnt quay.io/libpod/testimage:20200929 true
[test@kvm-07-guest01 ~]$ echo $?
0
[test@kvm-07-guest01 ~]$ podman ps
CONTAINER ID  IMAGE   COMMAND  CREATED  STATUS  PORTS   NAMES
[test@kvm-07-guest01 ~]$ podman ps -a
CONTAINER ID  IMAGE                              COMMAND  CREATED        STATUS                    PORTS   NAMES
93f6ff60824f  quay.io/libpod/testimage:20200929  true     2 minutes ago  Exited (0) 2 minutes ago          myc
699d3c3a69a1  quay.io/libpod/testimage:20200929  true     7 seconds ago  Exited (0) 7 seconds ago          mycnt

Comment 8 Alex Jia 2021-08-03 09:55:10 UTC
This bug has been verified on runc-1.0.1-3.module+el8.5.0+12014+438a5746.


[test@kvm-02-guest15 ~]$ cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.5 Beta (Ootpa)

[test@kvm-02-guest15 ~]$ rpm -q runc podman kernel
runc-1.0.1-3.module+el8.5.0+12014+438a5746.x86_64
podman-3.3.0-0.17.module+el8.5.0+12014+438a5746.x86_64
kernel-4.18.0-325.el8.x86_64

[test@kvm-02-guest15 ~]$ grep cgroup /proc/mounts
cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate 0 0

[test@kvm-02-guest15 ~]$ podman unshare cat /proc/self/uid_map
         0       1000          1
         1     100000      65536

[test@kvm-02-guest15 ~]$ podman info --format json|jq .host.ociRuntime
{
  "name": "runc",
  "package": "runc-1.0.1-3.module+el8.5.0+12014+438a5746.x86_64",
  "path": "/usr/bin/runc",
  "version": "runc version unknown\nspec: 1.0.2-dev\ngo: go1.16.6\nlibseccomp: 2.5.1"
}

[test@kvm-02-guest15 ~]$ podman --cgroup-manager=cgroupfs run --name myc quay.io/libpod/testimage:20200929 true
Trying to pull quay.io/libpod/testimage:20200929...
Getting image source signatures
Copying blob 5c10a2f1fe01 done  
Copying config 766ff5a3a7 done  
Writing manifest to image destination
Storing signatures
[test@kvm-02-guest15 ~]$ echo $?
0
[test@kvm-02-guest15 ~]$ podman ps -a
CONTAINER ID  IMAGE                              COMMAND     CREATED         STATUS                     PORTS       NAMES
494d7cfbd768  quay.io/libpod/testimage:20200929  true        21 seconds ago  Exited (0) 22 seconds ago              myc

[test@kvm-02-guest15 ~]$ podman --cgroup-manager=cgroupfs --runtime=crun run --name mycnt quay.io/libpod/testimage:20200929 true
[test@kvm-02-guest15 ~]$ echo $?
0

Comment 10 errata-xmlrpc 2021-11-09 17:37:47 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: container-tools:rhel8 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:4154