(In reply to Johnny Westerlund from comment #0)

> Steps to Reproduce:
> 1. enable cgroup v2 - https://access.redhat.com/solutions/3777261
> 2. as non-root run a container with a cpu limit - podman run -ti
> --memory=500M --cpus=1 quay.io/jwesterl/getfattr bash
> 
> 
> Actual results:
> Error: runc: container_linux.go:380: starting container process caused:
> process_linux.go:545: container init caused: process_linux.go:508: setting
> cgroup config for procHooks process caused: open
> /sys/fs/cgroup/user.slice/user-1000.slice/user/user.slice/
> libpod-38ba1a1b2125de815ece97adfb2b8038c7e3dbba690f951f4b3e938c55368525.
> scope/cpu.max: no such file or directory: OCI runtime attempted to invoke a
> command that was not found

I got a different error, it should be related to systemd and kernel version I think,
a similar bug 1897579 is still a NEW state.

1. runc runtime
[test@kvm-07-guest13 ~]$ podman run -ti --memory=500M --cpus=1 quay.io/jwesterl/getfattr bash
Error: runc: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: process_linux.go:508: setting cgroup config for procHooks process caused: open /sys/fs/cgroup/user.slice/user-1000.slice/user/user.slice/libpod-dd8de2e8415f15ef1c5d0779950250c57a168b1fe27121336f47d272473d1c17.scope/pids.max: no such file or directory: OCI runtime attempted to invoke a command that was not found

2. crun runtime
[test@kvm-07-guest13 ~]$ podman run --runtime crun -ti --memory=500M --cpus=1 quay.io/jwesterl/getfattr bash
Error: crun: opening file `memory.max` for writing: No such file or directory: OCI runtime attempted to invoke a command that was not found

[test@kvm-07-guest13 ~]$ cat /sys/fs/cgroup/cgroup.controllers
cpuset cpu io memory hugetlb pids rdma

[test@kvm-07-guest13 ~]$ cat /etc/systemd/system/user@.service.d/delegate.conf
cat: /etc/systemd/system/user@.service.d/delegate.conf: No such file or directory

[test@kvm-07-guest13 ~]$ cat /proc/self/cgroup
0::/user.slice/user-1000.slice/session-6.scope

[test@kvm-07-guest13 ~]$ cat /proc/cgroups
#subsys_name	hierarchy	num_cgroups	enabled
cpuset	0	149	1
cpu	0	149	1
cpuacct	0	149	1
blkio	0	149	1
memory	0	149	1
devices	0	149	1
freezer	0	149	1
net_cls	0	149	1
perf_event	0	149	1
net_prio	0	149	1
hugetlb	0	149	1
pids	0	149	1
rdma	0	149	1

[test@kvm-07-guest13 ~]$ findmnt -R /sys/fs/cgroup/
TARGET         SOURCE  FSTYPE  OPTIONS
/sys/fs/cgroup cgroup2 cgroup2 rw,nosuid,nodev,noexec,relatime,seclabel,nsdelegate

[test@kvm-07-guest13 ~]$ rpm -q podman runc systemd kernel
podman-4.0.2-6.module+el8.6.0+14877+f643d2d6.x86_64
runc-1.0.3-2.module+el8.6.0+14877+f643d2d6.x86_64
systemd-239-58.el8_6.1.x86_64
kernel-4.18.0-372.11.1.el8_6.x86_64

Can you provide a full `podman info` from the affected system?

[johnny@cgroup2 ~]$ podman info
host:
  arch: amd64
  buildahVersion: 1.24.1
  cgroupControllers: []
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.0-1.module+el8.6.0+14877+f643d2d6.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: 87b7a9037cbd1d81288bdf2d6705dfda889f7cf9'
  cpus: 2
  distribution:
    distribution: '"rhel"'
    version: "8.6"
  eventLogger: file
  hostname: cgroup2.home.lab
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 4.18.0-372.9.1.el8.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 1354113024
  memTotal: 1900539904
  networkBackend: cni
  ociRuntime:
    name: runc
    package: runc-1.0.3-2.module+el8.6.0+14877+f643d2d6.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.3
      spec: 1.0.2-dev
      go: go1.17.7
      libseccomp: 2.5.2
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.8-2.module+el8.6.0+14877+f643d2d6.x86_64
    version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 0
  swapTotal: 0
  uptime: 1m 27.47s
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/johnny/.config/containers/storage.conf
  containerStore:
    number: 6
    paused: 0
    running: 0
    stopped: 6
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/johnny/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /tmp/podman-run-1000/containers
  volumePath: /home/johnny/.local/share/containers/storage/volumes
version:
  APIVersion: 4.0.2
  Built: 1650363392
  BuiltTime: Tue Apr 19 12:16:32 2022
  GitCommit: ""
  GoVersion: go1.17.7
  OsArch: linux/amd64
  Version: 4.0.2

I checked Alex Jia's comment and I also tried to run a container without defining a memory/cpu limit,

podman run -ti quay.io/jwesterl/simple-webpage:1.0 bash
Error: runc: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: process_linux.go:508: setting cgroup config for procHooks process caused: open /sys/fs/cgroup/user.slice/user-1000.slice/user/user.slice/libpod-94d298d87a7518c387b516225896aaa759bfc2918960595eea23aef0cacac220.scope/pids.max: no such file or directory: OCI runtime attempted to invoke a command that was not found

So it might be unrelated to defining limits.

Can you try a container using the `crun` OCI runtime? (Install the `crun` package and use `--runtime=crun` when creating the container)

[johnny@cgroup2 ~]$ podman --runtime=crun run -p 8080:8080 quay.io/jwesterl/simple-webpage:1.0
Error: OCI runtime error: crun: the requested cgroup controller `pids` is not available

[johnny@cgroup2 ~]$ podman run --runtime=crun -ti quay.io/jwesterl/simple-webpage:1.0 bash
Error: OCI runtime error: crun: the requested cgroup controller `pids` is not available
[johnny@cgroup2 ~]$ 

Error message is different but doesn't seem to work

I think this is a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1897579

@jwesterl does the documented workaround here: https://github.com/containers/podman/blob/main/troubleshooting.md#26-running-containers-with-cpu-limits-fails-with-a-permissions-error fix the issue for you?  If so, we'll dupe this to the other BZ that Giuseppe pointed at.

I'm unable to follow those instructions (not sure what I'm doing wrong)..

I've created the systemd directory / file as per instructions, the directory and file did not exist at all on my system. (Which might explain why it's not working for me)

[johnny@cgroup2 user]$ cat /etc/systemd/system/user@.service.d/delegate.conf
[Service]
Delegate=memory pids cpu io

I've logged out an logged in, I did systemctl daemon-reload and rebooted the machine.

[johnny@cgroup2 ~]$ cat "/sys/fs/cgroup/user.slice/user-$(id -u).slice/user@$(id -u).service/cgroup.controllers"
[johnny@cgroup2 ~]$ 

[johnny@cgroup2 ~]$ podman run -ti quay.io/jwesterl/simple-webpage:1.0 bash
Error: runc: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: process_linux.go:508: setting cgroup config for procHooks process caused: open /sys/fs/cgroup/user.slice/user-1000.slice/user/user.slice/libpod-8a87e8804e40086733050c43b77736c326d8db5d9dd04009dedd0647035e6c59.scope/pids.max: no such file or directory: OCI runtime attempted to invoke a command that was not found

(Johnnys customer with the original problem here)

The work-around described did not work for us.

A bit of additional details:
* minimal command that reproduces issue: "podman run hello-world"
* Fedora 36 tested for reference, does not have the issue
* Stepped Fedora back to 31 and it does not have the issue (podman 2.x and systemd-243)
* I was unable to get a setup on Fedora 30 to work enough to try to reproduce the problem.

By chance we discovered that doing (as root) systemctl stop, start, enable on gpm.service "often" results in a system (RHEL8) that works as expected (the missing cgroup directories are correctly created.)

It seems clear to me that the systemd in RHEL8(.5,.6) does not properly handle it's cgroup-v2 duties / is buggy.

Testing installing / enabling gpm as per Peter's comment #11
It seems doing so, correctly configures the system to work with cgroupv2 and podman.


[johnny@cgroup2 ~]$ cat "/sys/fs/cgroup/user.slice/user-$(id -u).slice/user@$(id -u).service/cgroup.controllers"
[johnny@cgroup2 ~]$ podman run --cpus=1 -ti quay.io/jwesterl/simple-webpage:1.0 bash
Error: runc: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: process_linux.go:508: setting cgroup config for procHooks process caused: open /sys/fs/cgroup/user.slice/user-1000.slice/user/user.slice/libpod-9848a1a4888417dc98e73786742811fbc81fb8f8382824a445e2b383ad16c621.scope/pids.max: no such file or directory: OCI runtime attempted to invoke a command that was not found
[johnny@cgroup2 ~]$ cat "/sys/fs/cgroup/user.slice/user-$(id -u).slice/user@$(id -u).service/cgroup.controllers"
[johnny@cgroup2 ~]$ su -
Password: 
Last login: tor jun  2 12:35:33 CEST 2022 on pts/0
[root@cgroup2 ~]# dnf install gpm
Updating Subscription Management repositories.
Last metadata expiration check: 1:57:32 ago on Thu 02 Jun 2022 01:28:58 PM CEST.
Dependencies resolved.
==============================================================================================================================================================================================================================================
 Package                                                  Architecture                                  Version                                                 Repository                                                               Size
==============================================================================================================================================================================================================================================
Installing:
 gpm                                                      x86_64                                        1.20.7-17.el8                                           rhel-8-for-x86_64-appstream-rpms                                        200 k
Installing dependencies:
 SDL                                                      x86_64                                        1.2.15-39.el8                                           rhel-8-for-x86_64-appstream-rpms                                        218 k
 gpm-libs                                                 x86_64                                        1.20.7-17.el8                                           rhel-8-for-x86_64-appstream-rpms                                         39 k
 linuxconsoletools                                        x86_64                                        1.6.0-4.el8                                             rhel-8-for-x86_64-appstream-rpms                                         82 k

Transaction Summary
==============================================================================================================================================================================================================================================
Install  4 Packages

Total download size: 539 k
Installed size: 1.1 M
Is this ok [y/N]: y
Downloading Packages:
(1/4): gpm-libs-1.20.7-17.el8.x86_64.rpm                                                                                                                                                                       73 kB/s |  39 kB     00:00    
(2/4): linuxconsoletools-1.6.0-4.el8.x86_64.rpm                                                                                                                                                               144 kB/s |  82 kB     00:00    
(3/4): gpm-1.20.7-17.el8.x86_64.rpm                                                                                                                                                                           319 kB/s | 200 kB     00:00    
(4/4): SDL-1.2.15-39.el8.x86_64.rpm                                                                                                                                                                           282 kB/s | 218 kB     00:00    
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                         409 kB/s | 539 kB     00:01     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                                                      1/1 
  Installing       : SDL-1.2.15-39.el8.x86_64                                                                                                                                                                                             1/4 
  Running scriptlet: SDL-1.2.15-39.el8.x86_64                                                                                                                                                                                             1/4 
  Installing       : linuxconsoletools-1.6.0-4.el8.x86_64                                                                                                                                                                                 2/4 
  Installing       : gpm-libs-1.20.7-17.el8.x86_64                                                                                                                                                                                        3/4 
  Running scriptlet: gpm-libs-1.20.7-17.el8.x86_64                                                                                                                                                                                        3/4 
  Installing       : gpm-1.20.7-17.el8.x86_64                                                                                                                                                                                             4/4 
  Running scriptlet: gpm-1.20.7-17.el8.x86_64                                                                                                                                                                                             4/4 
  Verifying        : linuxconsoletools-1.6.0-4.el8.x86_64                                                                                                                                                                                 1/4 
  Verifying        : gpm-1.20.7-17.el8.x86_64                                                                                                                                                                                             2/4 
  Verifying        : gpm-libs-1.20.7-17.el8.x86_64                                                                                                                                                                                        3/4 
  Verifying        : SDL-1.2.15-39.el8.x86_64                                                                                                                                                                                             4/4 
Installed products updated.

Installed:
  SDL-1.2.15-39.el8.x86_64                               gpm-1.20.7-17.el8.x86_64                               gpm-libs-1.20.7-17.el8.x86_64                               linuxconsoletools-1.6.0-4.el8.x86_64                              

Complete!
[root@cgroup2 ~]# systemctl restart gpm.service
[root@cgroup2 ~]# logout
[johnny@cgroup2 ~]$ cat "/sys/fs/cgroup/user.slice/user-$(id -u).slice/user@$(id -u).service/cgroup.controllers"
cpu io memory pids
[johnny@cgroup2 ~]$ podman run --cpus=1 -ti quay.io/jwesterl/simple-webpage:1.0 bash
bash-5.1$ 
exit
[johnny@cgroup2 ~]$

THanks for the info all.  I've documented Peter K's discovery in the other Bugzilla, https://bugzilla.redhat.com/show_bug.cgi?id=1897579, and I'm closing this one as a duplicate of that.

*** This bug has been marked as a duplicate of bug 1897579 ***