RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1947999 - rootless podman --cgroup-manager=cgroupfs run command causes OCI permission error when CGroups V2 is enabled
Summary: rootless podman --cgroup-manager=cgroupfs run command causes OCI permission e...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: runc
Version: 8.4
Hardware: x86_64
OS: Linux
high
high
Target Milestone: beta
: ---
Assignee: Jindrich Novy
QA Contact: Alex Jia
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-04-09 17:29 UTC by Alex Jia
Modified: 2021-11-09 19:37 UTC (History)
6 users (show)

Fixed In Version: podman-3.3.0-0.13.el8 or newer
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-11-09 17:37:47 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2021:4154 0 None None None 2021-11-09 17:38:28 UTC

Description Alex Jia 2021-04-09 17:29:41 UTC 
Description of problem:
podman --cgroup-manager=cgroupfs run command causes OCI permission denied in rootless mode w/ cgroupv2 enabled.

Version-Release number of selected component (if applicable):
[test@kvm-02-guest02 ~]$ cat /etc/redhat-release
Red Hat Enterprise Linux release 8.4 (Ootpa)

[test@kvm-02-guest02 ~]$  rpm -q podman containers-common runc crun kernel
podman-3.0.1-6.module+el8.4.0+10614+dd38312c.x86_64
containers-common-1.2.2-7.module+el8.4.0+10614+dd38312c.x86_64
runc-1.0.0-70.rc92.module+el8.4.0+10614+dd38312c.x86_64
crun-0.18-2.module+el8.4.0+10614+dd38312c.x86_64
kernel-4.18.0-302.el8.x86_64
kernel-4.18.0-304.el8.x86_64

How reproducible:
always

Steps to Reproduce:
1. configure rootless
2. configure cgroupv2 and reboot
3.  podman --cgroup-manager=cgroupfs --runtime=runc run --name myc quay.io/libpod/testimage:20200929 true

Actual results:
$ mount|grep cgroup
cgroup2 on /sys/fs/cgroup type cgroup2
(rw,nosuid,nodev,noexec,relatime,seclabel)

$ podman unshare cat /proc/self/uid_map
         0       1000          1
         1     100000      65536

$ podman --cgroup-manager=cgroupfs run --name myc quay.io/libpod/testimage:20200929 true
Error: container_linux.go:370: starting container process caused: process_linux.go:326: applying cgroup configuration for process caused: mkdir /sys/fs/cgroup/libpod_parent/libpod-8fea41ce357783321a158534ef44ca8ab5ba76b1f6807a31e7b1f12e049a7805: permission denied: OCI permission denied

Expected results:
fix permission denied error

Additional info:

1. it's okay w/o --cgroup-manager=cgroupfs option for runc runtime
[test@kvm-04-guest15 ~]$ podman --runtime=runc run --name myc quay.io/libpod/testimage:20200929 true
[test@kvm-04-guest15 ~]$ echo $?
0
[test@kvm-04-guest15 ~]$ podman ps -a
CONTAINER ID  IMAGE                              COMMAND  CREATED        STATUS                    PORTS   NAMES
8b0fc0cae0ad  quay.io/libpod/testimage:20200929  true     8 seconds ago  Exited (0) 9 seconds ago          myc

2. it's okay for rootfull mode w/ runc runtime
[test@kvm-04-guest15 ~]$ sudo podman --cgroup-manager=cgroupfs --runtime=runc run --name myc quay.io/libpod/testimage:20200929 true
[test@kvm-04-guest15 ~]$ echo $?
0
[test@kvm-04-guest15 ~]$ sudo podman ps -a
CONTAINER ID  IMAGE                              COMMAND  CREATED        STATUS                    PORTS   NAMES
b7cb9a7ae2b1  quay.io/libpod/testimage:20200929  true     8 seconds ago  Exited (0) 9 seconds ago          myc

3. it's okay for crun runtime
[test@kvm-04-guest15 ~]$ podman --cgroup-manager=cgroupfs --runtime=crun run --name myc quay.io/libpod/testimage:20200929 true
[test@kvm-04-guest15 ~]$ echo $?
0
[test@kvm-04-guest15 ~]$ podman ps -a
CONTAINER ID  IMAGE                              COMMAND  CREATED        STATUS                    PORTS   NAMES
7e3ca80467f5  quay.io/libpod/testimage:20200929  true     5 seconds ago  Exited (0) 5 seconds ago          myc
Comment 1 Tom Sweeney 2021-04-09 21:40:50 UTC 
Matt, can you take a look at this early next week and see if it's at all related to https://bugzilla.redhat.com/show_bug.cgi?id=1947432?
Comment 3 Matthew Heon 2021-04-12 14:26:36 UTC 
I see no reason to believe these two BZs are related. This is a CGroups issue, the previous one was a readonly-paths issue. This looks like, specifically, a permission denied on making a directory in cgroupfs while part of a `podman unshare` shell?

I'm reassigning to Giuseppe given cgroups are more his area of expertise.
Comment 4 Giuseppe Scrivano 2021-04-12 14:56:51 UTC 
opened a PR: https://github.com/containers/podman/pull/9996
Comment 5 Alex Jia 2021-05-11 13:22:03 UTC 
This bug has been fixed on podman-3.2.0-0.11.

[test@kvm-07-guest01 ~]$ cat /etc/redhat-release
Red Hat Enterprise Linux release 8.5 Beta (Ootpa)
[test@kvm-07-guest01 ~]$ rpm -q podman crun runc kernel
podman-3.2.0-0.11.module+el8.5.0+10952+e23bdbc3.x86_64
crun-0.19.1-1.module+el8.5.0+10952+e23bdbc3.x86_64
runc-1.0.0-72.rc93.module+el8.5.0+10952+e23bdbc3.x86_64
kernel-4.18.0-305.1.el8.x86_64
[test@kvm-07-guest01 ~]$ podman unshare cat /proc/self/uid_map
         0       1000          1
         1     100000      65536
[test@kvm-07-guest01 ~]$ grep cgroup /proc/mounts
cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate 0 0

[test@kvm-07-guest01 ~]$ podman info --format json | jq .host.ociRuntime
{
  "name": "runc",
  "package": "runc-1.0.0-72.rc93.module+el8.5.0+10952+e23bdbc3.x86_64",
  "path": "/usr/bin/runc",
  "version": "runc version spec: 1.0.2-dev\ngo: go1.16.1\nlibseccomp: 2.5.1"
}

[test@kvm-07-guest01 ~]$ podman --cgroup-manager=cgroupfs --runtime=runc run --name myc quay.io/libpod/testimage:20200929 true
Trying to pull quay.io/libpod/testimage:20200929...
Getting image source signatures
Copying blob 5c10a2f1fe01 done
Copying config 766ff5a3a7 done
Writing manifest to image destination
Storing signatures

[test@kvm-07-guest01 ~]$ podman --cgroup-manager=cgroupfs --runtime=crun run --name mycnt quay.io/libpod/testimage:20200929 true
[test@kvm-07-guest01 ~]$ echo $?
0
[test@kvm-07-guest01 ~]$ podman ps
CONTAINER ID  IMAGE   COMMAND  CREATED  STATUS  PORTS   NAMES
[test@kvm-07-guest01 ~]$ podman ps -a
CONTAINER ID  IMAGE                              COMMAND  CREATED        STATUS                    PORTS   NAMES
93f6ff60824f  quay.io/libpod/testimage:20200929  true     2 minutes ago  Exited (0) 2 minutes ago          myc
699d3c3a69a1  quay.io/libpod/testimage:20200929  true     7 seconds ago  Exited (0) 7 seconds ago          mycnt
Comment 8 Alex Jia 2021-08-03 09:55:10 UTC 
This bug has been verified on runc-1.0.1-3.module+el8.5.0+12014+438a5746.


[test@kvm-02-guest15 ~]$ cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.5 Beta (Ootpa)

[test@kvm-02-guest15 ~]$ rpm -q runc podman kernel
runc-1.0.1-3.module+el8.5.0+12014+438a5746.x86_64
podman-3.3.0-0.17.module+el8.5.0+12014+438a5746.x86_64
kernel-4.18.0-325.el8.x86_64

[test@kvm-02-guest15 ~]$ grep cgroup /proc/mounts
cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate 0 0

[test@kvm-02-guest15 ~]$ podman unshare cat /proc/self/uid_map
         0       1000          1
         1     100000      65536

[test@kvm-02-guest15 ~]$ podman info --format json|jq .host.ociRuntime
{
  "name": "runc",
  "package": "runc-1.0.1-3.module+el8.5.0+12014+438a5746.x86_64",
  "path": "/usr/bin/runc",
  "version": "runc version unknown\nspec: 1.0.2-dev\ngo: go1.16.6\nlibseccomp: 2.5.1"
}

[test@kvm-02-guest15 ~]$ podman --cgroup-manager=cgroupfs run --name myc quay.io/libpod/testimage:20200929 true
Trying to pull quay.io/libpod/testimage:20200929...
Getting image source signatures
Copying blob 5c10a2f1fe01 done  
Copying config 766ff5a3a7 done  
Writing manifest to image destination
Storing signatures
[test@kvm-02-guest15 ~]$ echo $?
0
[test@kvm-02-guest15 ~]$ podman ps -a
CONTAINER ID  IMAGE                              COMMAND     CREATED         STATUS                     PORTS       NAMES
494d7cfbd768  quay.io/libpod/testimage:20200929  true        21 seconds ago  Exited (0) 22 seconds ago              myc

[test@kvm-02-guest15 ~]$ podman --cgroup-manager=cgroupfs --runtime=crun run --name mycnt quay.io/libpod/testimage:20200929 true
[test@kvm-02-guest15 ~]$ echo $?
0
Comment 10 errata-xmlrpc 2021-11-09 17:37:47 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: container-tools:rhel8 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:4154
Note You need to log in before you can comment on or make changes to this bug.