Bug 2091505
| Summary: | [cgroup_v2] rootless containers cannot define CPU limits | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | Johnny Westerlund <jwesterl> |
| Component: | container-tools-4.0-module | Assignee: | Matthew Heon <mheon> |
| Status: | CLOSED DUPLICATE | QA Contact: | atomic-bugs <atomic-bugs> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 8.6 | CC: | ajia, gscrivan, jligon, jnovy, lsm5, mheon, peter.kjellstrom, tsweeney |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-06-02 18:52:43 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Johnny Westerlund
2022-05-30 06:58:57 UTC
(In reply to Johnny Westerlund from comment #0) > Steps to Reproduce: > 1. enable cgroup v2 - https://access.redhat.com/solutions/3777261 > 2. as non-root run a container with a cpu limit - podman run -ti > --memory=500M --cpus=1 quay.io/jwesterl/getfattr bash > > > Actual results: > Error: runc: container_linux.go:380: starting container process caused: > process_linux.go:545: container init caused: process_linux.go:508: setting > cgroup config for procHooks process caused: open > /sys/fs/cgroup/user.slice/user-1000.slice/user/user.slice/ > libpod-38ba1a1b2125de815ece97adfb2b8038c7e3dbba690f951f4b3e938c55368525. > scope/cpu.max: no such file or directory: OCI runtime attempted to invoke a > command that was not found I got a different error, it should be related to systemd and kernel version I think, a similar bug 1897579 is still a NEW state. 1. runc runtime [test@kvm-07-guest13 ~]$ podman run -ti --memory=500M --cpus=1 quay.io/jwesterl/getfattr bash Error: runc: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: process_linux.go:508: setting cgroup config for procHooks process caused: open /sys/fs/cgroup/user.slice/user-1000.slice/user/user.slice/libpod-dd8de2e8415f15ef1c5d0779950250c57a168b1fe27121336f47d272473d1c17.scope/pids.max: no such file or directory: OCI runtime attempted to invoke a command that was not found 2. crun runtime [test@kvm-07-guest13 ~]$ podman run --runtime crun -ti --memory=500M --cpus=1 quay.io/jwesterl/getfattr bash Error: crun: opening file `memory.max` for writing: No such file or directory: OCI runtime attempted to invoke a command that was not found [test@kvm-07-guest13 ~]$ cat /sys/fs/cgroup/cgroup.controllers cpuset cpu io memory hugetlb pids rdma [test@kvm-07-guest13 ~]$ cat /etc/systemd/system/user@.service.d/delegate.conf cat: /etc/systemd/system/user@.service.d/delegate.conf: No such file or directory [test@kvm-07-guest13 ~]$ cat /proc/self/cgroup 0::/user.slice/user-1000.slice/session-6.scope [test@kvm-07-guest13 ~]$ cat /proc/cgroups #subsys_name hierarchy num_cgroups enabled cpuset 0 149 1 cpu 0 149 1 cpuacct 0 149 1 blkio 0 149 1 memory 0 149 1 devices 0 149 1 freezer 0 149 1 net_cls 0 149 1 perf_event 0 149 1 net_prio 0 149 1 hugetlb 0 149 1 pids 0 149 1 rdma 0 149 1 [test@kvm-07-guest13 ~]$ findmnt -R /sys/fs/cgroup/ TARGET SOURCE FSTYPE OPTIONS /sys/fs/cgroup cgroup2 cgroup2 rw,nosuid,nodev,noexec,relatime,seclabel,nsdelegate [test@kvm-07-guest13 ~]$ rpm -q podman runc systemd kernel podman-4.0.2-6.module+el8.6.0+14877+f643d2d6.x86_64 runc-1.0.3-2.module+el8.6.0+14877+f643d2d6.x86_64 systemd-239-58.el8_6.1.x86_64 kernel-4.18.0-372.11.1.el8_6.x86_64 Can you provide a full `podman info` from the affected system? [johnny@cgroup2 ~]$ podman info
host:
arch: amd64
buildahVersion: 1.24.1
cgroupControllers: []
cgroupManager: systemd
cgroupVersion: v2
conmon:
package: conmon-2.1.0-1.module+el8.6.0+14877+f643d2d6.x86_64
path: /usr/bin/conmon
version: 'conmon version 2.1.0, commit: 87b7a9037cbd1d81288bdf2d6705dfda889f7cf9'
cpus: 2
distribution:
distribution: '"rhel"'
version: "8.6"
eventLogger: file
hostname: cgroup2.home.lab
idMappings:
gidmap:
- container_id: 0
host_id: 1000
size: 1
- container_id: 1
host_id: 100000
size: 65536
uidmap:
- container_id: 0
host_id: 1000
size: 1
- container_id: 1
host_id: 100000
size: 65536
kernel: 4.18.0-372.9.1.el8.x86_64
linkmode: dynamic
logDriver: k8s-file
memFree: 1354113024
memTotal: 1900539904
networkBackend: cni
ociRuntime:
name: runc
package: runc-1.0.3-2.module+el8.6.0+14877+f643d2d6.x86_64
path: /usr/bin/runc
version: |-
runc version 1.0.3
spec: 1.0.2-dev
go: go1.17.7
libseccomp: 2.5.2
os: linux
remoteSocket:
path: /run/user/1000/podman/podman.sock
security:
apparmorEnabled: false
capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: true
seccompEnabled: true
seccompProfilePath: /usr/share/containers/seccomp.json
selinuxEnabled: true
serviceIsRemote: false
slirp4netns:
executable: /usr/bin/slirp4netns
package: slirp4netns-1.1.8-2.module+el8.6.0+14877+f643d2d6.x86_64
version: |-
slirp4netns version 1.1.8
commit: d361001f495417b880f20329121e3aa431a8f90f
libslirp: 4.4.0
SLIRP_CONFIG_VERSION_MAX: 3
libseccomp: 2.5.2
swapFree: 0
swapTotal: 0
uptime: 1m 27.47s
plugins:
log:
- k8s-file
- none
- passthrough
- journald
network:
- bridge
- macvlan
- ipvlan
volume:
- local
registries:
search:
- registry.fedoraproject.org
- registry.access.redhat.com
- registry.centos.org
- docker.io
store:
configFile: /home/johnny/.config/containers/storage.conf
containerStore:
number: 6
paused: 0
running: 0
stopped: 6
graphDriverName: overlay
graphOptions: {}
graphRoot: /home/johnny/.local/share/containers/storage
graphStatus:
Backing Filesystem: xfs
Native Overlay Diff: "true"
Supports d_type: "true"
Using metacopy: "false"
imageCopyTmpDir: /var/tmp
imageStore:
number: 1
runRoot: /tmp/podman-run-1000/containers
volumePath: /home/johnny/.local/share/containers/storage/volumes
version:
APIVersion: 4.0.2
Built: 1650363392
BuiltTime: Tue Apr 19 12:16:32 2022
GitCommit: ""
GoVersion: go1.17.7
OsArch: linux/amd64
Version: 4.0.2
I checked Alex Jia's comment and I also tried to run a container without defining a memory/cpu limit, podman run -ti quay.io/jwesterl/simple-webpage:1.0 bash Error: runc: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: process_linux.go:508: setting cgroup config for procHooks process caused: open /sys/fs/cgroup/user.slice/user-1000.slice/user/user.slice/libpod-94d298d87a7518c387b516225896aaa759bfc2918960595eea23aef0cacac220.scope/pids.max: no such file or directory: OCI runtime attempted to invoke a command that was not found So it might be unrelated to defining limits. Can you try a container using the `crun` OCI runtime? (Install the `crun` package and use `--runtime=crun` when creating the container) [johnny@cgroup2 ~]$ podman --runtime=crun run -p 8080:8080 quay.io/jwesterl/simple-webpage:1.0 Error: OCI runtime error: crun: the requested cgroup controller `pids` is not available [johnny@cgroup2 ~]$ podman run --runtime=crun -ti quay.io/jwesterl/simple-webpage:1.0 bash Error: OCI runtime error: crun: the requested cgroup controller `pids` is not available [johnny@cgroup2 ~]$ Error message is different but doesn't seem to work I think this is a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1897579 @jwesterl does the documented workaround here: https://github.com/containers/podman/blob/main/troubleshooting.md#26-running-containers-with-cpu-limits-fails-with-a-permissions-error fix the issue for you? If so, we'll dupe this to the other BZ that Giuseppe pointed at. I'm unable to follow those instructions (not sure what I'm doing wrong).. I've created the systemd directory / file as per instructions, the directory and file did not exist at all on my system. (Which might explain why it's not working for me) [johnny@cgroup2 user]$ cat /etc/systemd/system/user@.service.d/delegate.conf [Service] Delegate=memory pids cpu io I've logged out an logged in, I did systemctl daemon-reload and rebooted the machine. [johnny@cgroup2 ~]$ cat "/sys/fs/cgroup/user.slice/user-$(id -u).slice/user@$(id -u).service/cgroup.controllers" [johnny@cgroup2 ~]$ [johnny@cgroup2 ~]$ podman run -ti quay.io/jwesterl/simple-webpage:1.0 bash Error: runc: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: process_linux.go:508: setting cgroup config for procHooks process caused: open /sys/fs/cgroup/user.slice/user-1000.slice/user/user.slice/libpod-8a87e8804e40086733050c43b77736c326d8db5d9dd04009dedd0647035e6c59.scope/pids.max: no such file or directory: OCI runtime attempted to invoke a command that was not found (Johnnys customer with the original problem here) The work-around described did not work for us. A bit of additional details: * minimal command that reproduces issue: "podman run hello-world" * Fedora 36 tested for reference, does not have the issue * Stepped Fedora back to 31 and it does not have the issue (podman 2.x and systemd-243) * I was unable to get a setup on Fedora 30 to work enough to try to reproduce the problem. By chance we discovered that doing (as root) systemctl stop, start, enable on gpm.service "often" results in a system (RHEL8) that works as expected (the missing cgroup directories are correctly created.) It seems clear to me that the systemd in RHEL8(.5,.6) does not properly handle it's cgroup-v2 duties / is buggy. Testing installing / enabling gpm as per Peter's comment #11 It seems doing so, correctly configures the system to work with cgroupv2 and podman. [johnny@cgroup2 ~]$ cat "/sys/fs/cgroup/user.slice/user-$(id -u).slice/user@$(id -u).service/cgroup.controllers" [johnny@cgroup2 ~]$ podman run --cpus=1 -ti quay.io/jwesterl/simple-webpage:1.0 bash Error: runc: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: process_linux.go:508: setting cgroup config for procHooks process caused: open /sys/fs/cgroup/user.slice/user-1000.slice/user/user.slice/libpod-9848a1a4888417dc98e73786742811fbc81fb8f8382824a445e2b383ad16c621.scope/pids.max: no such file or directory: OCI runtime attempted to invoke a command that was not found [johnny@cgroup2 ~]$ cat "/sys/fs/cgroup/user.slice/user-$(id -u).slice/user@$(id -u).service/cgroup.controllers" [johnny@cgroup2 ~]$ su - Password: Last login: tor jun 2 12:35:33 CEST 2022 on pts/0 [root@cgroup2 ~]# dnf install gpm Updating Subscription Management repositories. Last metadata expiration check: 1:57:32 ago on Thu 02 Jun 2022 01:28:58 PM CEST. Dependencies resolved. ============================================================================================================================================================================================================================================== Package Architecture Version Repository Size ============================================================================================================================================================================================================================================== Installing: gpm x86_64 1.20.7-17.el8 rhel-8-for-x86_64-appstream-rpms 200 k Installing dependencies: SDL x86_64 1.2.15-39.el8 rhel-8-for-x86_64-appstream-rpms 218 k gpm-libs x86_64 1.20.7-17.el8 rhel-8-for-x86_64-appstream-rpms 39 k linuxconsoletools x86_64 1.6.0-4.el8 rhel-8-for-x86_64-appstream-rpms 82 k Transaction Summary ============================================================================================================================================================================================================================================== Install 4 Packages Total download size: 539 k Installed size: 1.1 M Is this ok [y/N]: y Downloading Packages: (1/4): gpm-libs-1.20.7-17.el8.x86_64.rpm 73 kB/s | 39 kB 00:00 (2/4): linuxconsoletools-1.6.0-4.el8.x86_64.rpm 144 kB/s | 82 kB 00:00 (3/4): gpm-1.20.7-17.el8.x86_64.rpm 319 kB/s | 200 kB 00:00 (4/4): SDL-1.2.15-39.el8.x86_64.rpm 282 kB/s | 218 kB 00:00 ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Total 409 kB/s | 539 kB 00:01 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Installing : SDL-1.2.15-39.el8.x86_64 1/4 Running scriptlet: SDL-1.2.15-39.el8.x86_64 1/4 Installing : linuxconsoletools-1.6.0-4.el8.x86_64 2/4 Installing : gpm-libs-1.20.7-17.el8.x86_64 3/4 Running scriptlet: gpm-libs-1.20.7-17.el8.x86_64 3/4 Installing : gpm-1.20.7-17.el8.x86_64 4/4 Running scriptlet: gpm-1.20.7-17.el8.x86_64 4/4 Verifying : linuxconsoletools-1.6.0-4.el8.x86_64 1/4 Verifying : gpm-1.20.7-17.el8.x86_64 2/4 Verifying : gpm-libs-1.20.7-17.el8.x86_64 3/4 Verifying : SDL-1.2.15-39.el8.x86_64 4/4 Installed products updated. Installed: SDL-1.2.15-39.el8.x86_64 gpm-1.20.7-17.el8.x86_64 gpm-libs-1.20.7-17.el8.x86_64 linuxconsoletools-1.6.0-4.el8.x86_64 Complete! [root@cgroup2 ~]# systemctl restart gpm.service [root@cgroup2 ~]# logout [johnny@cgroup2 ~]$ cat "/sys/fs/cgroup/user.slice/user-$(id -u).slice/user@$(id -u).service/cgroup.controllers" cpu io memory pids [johnny@cgroup2 ~]$ podman run --cpus=1 -ti quay.io/jwesterl/simple-webpage:1.0 bash bash-5.1$ exit [johnny@cgroup2 ~]$ THanks for the info all. I've documented Peter K's discovery in the other Bugzilla, https://bugzilla.redhat.com/show_bug.cgi?id=1897579, and I'm closing this one as a duplicate of that. *** This bug has been marked as a duplicate of bug 1897579 *** |