Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Description of problem:
Rootless podman can not run systemd container due to "Failed to create root cgroup hierarchy: Permission denied".
Version-Release number of selected component (if applicable):
kernel-3.10.0-1062.12.1.el7.x86_64
podman-1.4.4-4.el7.centos.x86_64
runc-1.0.0-65.rc8.el7.centos.x86_64
shadow-utils-4.6-5.el7.x86_64
oci-systemd-hook-0.2.0-1.git05e6923.el7_6.x86_64
slirp4netns-0.3.0-1.el7.x86_64
[ user.max_user_namespaces=28633 ]
How reproducible:
100%
Steps to Reproduce:
following: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_atomic_host/7/html/managing_containers/index
1. login as non-root user.
2. podman pull centos:7.7.1908
3. podman run -it --rm --name centos7 centos:7.7.1908 /sbin/init
Actual results:
[tester@centos7 ~]$ podman run -it --rm --name centos7 --log-level=debug centos:7.7.1908 /sbin/init
INFO[0000] running as rootless
DEBU[0000] Initializing boltdb state at /home/tester/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver vfs
DEBU[0000] Using graph root /home/tester/.local/share/containers/storage
DEBU[0000] Using run root /tmp/run-1004
DEBU[0000] Using static dir /home/tester/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /tmp/run-1004/libpod/tmp
DEBU[0000] Using volume path /home/tester/.local/share/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "vfs"
DEBU[0000] Initializing event backend journald
DEBU[0000] parsed reference into "[vfs@/home/tester/.local/share/containers/storage+/tmp/run-1004]docker.io/library/centos:7.7.1908"
DEBU[0000] reference "[vfs@/home/tester/.local/share/containers/storage+/tmp/run-1004]docker.io/library/centos:7.7.1908" does not resolve to an image ID
DEBU[0000] parsed reference into "[vfs@/home/tester/.local/share/containers/storage+/tmp/run-1004]localhost/centos:7.7.1908"
DEBU[0000] reference "[vfs@/home/tester/.local/share/containers/storage+/tmp/run-1004]localhost/centos:7.7.1908" does not resolve to an image ID
DEBU[0000] parsed reference into "[vfs@/home/tester/.local/share/containers/storage+/tmp/run-1004]@08d05d1d5859ebcfb3312d246e2082e46cb307f0e896c9ac097185f0b0b19e56"
DEBU[0000] exporting opaque data as blob "sha256:08d05d1d5859ebcfb3312d246e2082e46cb307f0e896c9ac097185f0b0b19e56"
DEBU[0000] parsed reference into "[vfs@/home/tester/.local/share/containers/storage+/tmp/run-1004]@08d05d1d5859ebcfb3312d246e2082e46cb307f0e896c9ac097185f0b0b19e56"
DEBU[0000] exporting opaque data as blob "sha256:08d05d1d5859ebcfb3312d246e2082e46cb307f0e896c9ac097185f0b0b19e56"
DEBU[0000] parsed reference into "[vfs@/home/tester/.local/share/containers/storage+/tmp/run-1004]@08d05d1d5859ebcfb3312d246e2082e46cb307f0e896c9ac097185f0b0b19e56"
DEBU[0000] Got mounts: []
DEBU[0000] Got volumes: []
DEBU[0000] Using slirp4netns netmode
DEBU[0000] setting container name centos7
DEBU[0000] created OCI spec and options for new container
DEBU[0000] Allocated lock 0 for container d3db1a4cf7db3413ed94b1fc7027f6ea68c702e7167afb46891c733045dd6765
DEBU[0000] parsed reference into "[vfs@/home/tester/.local/share/containers/storage+/tmp/run-1004]@08d05d1d5859ebcfb3312d246e2082e46cb307f0e896c9ac097185f0b0b19e56"
DEBU[0000] exporting opaque data as blob "sha256:08d05d1d5859ebcfb3312d246e2082e46cb307f0e896c9ac097185f0b0b19e56"
DEBU[0001] created container "d3db1a4cf7db3413ed94b1fc7027f6ea68c702e7167afb46891c733045dd6765"
DEBU[0001] container "d3db1a4cf7db3413ed94b1fc7027f6ea68c702e7167afb46891c733045dd6765" has work directory "/home/tester/.local/share/containers/storage/vfs-containers/d3db1a4cf7db3413ed94b1fc7027f6ea68c702e7167afb46891c733045dd6765/userdata"
DEBU[0001] container "d3db1a4cf7db3413ed94b1fc7027f6ea68c702e7167afb46891c733045dd6765" has run directory "/tmp/run-1004/vfs-containers/d3db1a4cf7db3413ed94b1fc7027f6ea68c702e7167afb46891c733045dd6765/userdata"
DEBU[0001] New container created "d3db1a4cf7db3413ed94b1fc7027f6ea68c702e7167afb46891c733045dd6765"
DEBU[0001] container "d3db1a4cf7db3413ed94b1fc7027f6ea68c702e7167afb46891c733045dd6765" has CgroupParent "/libpod_parent/libpod-d3db1a4cf7db3413ed94b1fc7027f6ea68c702e7167afb46891c733045dd6765"
DEBU[0001] Handling terminal attach
DEBU[0001] mounted container "d3db1a4cf7db3413ed94b1fc7027f6ea68c702e7167afb46891c733045dd6765" at "/home/tester/.local/share/containers/storage/vfs/dir/f73c48e6ff00ed414814af449d373a16334b0387d4761612cc5931453bed1c97"
DEBU[0001] Created root filesystem for container d3db1a4cf7db3413ed94b1fc7027f6ea68c702e7167afb46891c733045dd6765 at /home/tester/.local/share/containers/storage/vfs/dir/f73c48e6ff00ed414814af449d373a16334b0387d4761612cc5931453bed1c97
DEBU[0001] /etc/system-fips does not exist on host, not mounting FIPS mode secret
DEBU[0001] Created OCI spec for container d3db1a4cf7db3413ed94b1fc7027f6ea68c702e7167afb46891c733045dd6765 at /home/tester/.local/share/containers/storage/vfs-containers/d3db1a4cf7db3413ed94b1fc7027f6ea68c702e7167afb46891c733045dd6765/userdata/config.json
DEBU[0001] /usr/libexec/podman/conmon messages will be logged to syslog
DEBU[0001] running conmon: /usr/libexec/podman/conmon args="[-c d3db1a4cf7db3413ed94b1fc7027f6ea68c702e7167afb46891c733045dd6765 -u d3db1a4cf7db3413ed94b1fc7027f6ea68c702e7167afb46891c733045dd6765 -n centos7 -r /usr/bin/runc -b /home/tester/.local/share/containers/storage/vfs-containers/d3db1a4cf7db3413ed94b1fc7027f6ea68c702e7167afb46891c733045dd6765/userdata -p /tmp/run-1004/vfs-containers/d3db1a4cf7db3413ed94b1fc7027f6ea68c702e7167afb46891c733045dd6765/userdata/pidfile --exit-dir /tmp/run-1004/libpod/tmp/exits --conmon-pidfile /tmp/run-1004/vfs-containers/d3db1a4cf7db3413ed94b1fc7027f6ea68c702e7167afb46891c733045dd6765/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /home/tester/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /tmp/run-1004 --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg cgroupfs --exit-command-arg --tmpdir --exit-command-arg /tmp/run-1004/libpod/tmp --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg vfs --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg d3db1a4cf7db3413ed94b1fc7027f6ea68c702e7167afb46891c733045dd6765 --socket-dir-path /tmp/run-1004/libpod/tmp/socket -t -l k8s-file:/home/tester/.local/share/containers/storage/vfs-containers/d3db1a4cf7db3413ed94b1fc7027f6ea68c702e7167afb46891c733045dd6765/userdata/ctr.log --log-level debug --syslog]"
WARN[0001] Failed to add conmon to cgroupfs sandbox cgroup: error creating cgroup for cpu: mkdir /sys/fs/cgroup/cpu/libpod_parent: permission denied
DEBU[0001] Received container pid: 20053
DEBU[0001] Created container d3db1a4cf7db3413ed94b1fc7027f6ea68c702e7167afb46891c733045dd6765 in OCI runtime
DEBU[0001] Attaching to container d3db1a4cf7db3413ed94b1fc7027f6ea68c702e7167afb46891c733045dd6765
DEBU[0001] connecting to socket /tmp/run-1004/libpod/tmp/socket/d3db1a4cf7db3413ed94b1fc7027f6ea68c702e7167afb46891c733045dd6765/attach
DEBU[0001] Starting container d3db1a4cf7db3413ed94b1fc7027f6ea68c702e7167afb46891c733045dd6765 with command [/sbin/init]
DEBU[0001] Received a resize event: {Width:132 Height:38}
DEBU[0001] Started container d3db1a4cf7db3413ed94b1fc7027f6ea68c702e7167afb46891c733045dd6765
DEBU[0001] Enabling signal proxying
systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
Detected virtualization other.
Detected architecture x86-64.
Welcome to CentOS Linux 7 (Core)!
Set hostname to <d3db1a4cf7db>.
Initializing machine ID from random generator.
Failed to read AF_UNIX datagram queue length, ignoring: No such file or directory
Failed to create root cgroup hierarchy: Permission denied
Failed to allocate manager object: Permission denied
[!!!!!!] Failed to allocate manager object, freezing.
Expected results:
~]$ podman run -it --rm --name centos7 centos:7.7.1908 /sbin/init
systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
Detected virtualization other.
Detected architecture x86-64.
Welcome to CentOS Linux 7 (Core)!
Set hostname to <2426740762f5>.
Initializing machine ID from random generator.
Failed to read AF_UNIX datagram queue length, ignoring: No such file or directory
[ OK ] Created slice Root Slice.
[ OK ] Listening on Journal Socket.
[ OK ] Started Dispatch Password Requests to Console Directory Watch.
[ OK ] Created slice User and Session Slice.
[ OK ] Created slice System Slice.
Starting Journal Service...
Starting Load/Save Random Seed...
Starting Rebuild Hardware Database...
[ OK ] Listening on Delayed Shutdown Socket.
[ OK ] Reached target Slices.
[ OK ] Started Forward Password Requests to Wall Directory Watch.
[ OK ] Created slice system-getty.slice.
[ OK ] Reached target Remote File Systems.
[ OK ] Reached target Swap.
[ OK ] Reached target Local File Systems (Pre).
[ OK ] Reached target Local File Systems.
[ OK ] Listening on /dev/initctl Compatibility Named Pipe.
[ OK ] Reached target Paths.
Starting Rebuild Journal Catalog...
[ OK ] Reached target Local Encrypted Volumes.
[ OK ] Started Journal Service.
[ OK ] Started Load/Save Random Seed.
[ OK ] Started Rebuild Hardware Database.
[ OK ] Started Rebuild Journal Catalog.
Starting Update is Completed...
Starting Flush Journal to Persistent Storage...
[ OK ] Started Update is Completed.
[ OK ] Started Flush Journal to Persistent Storage.
Starting Create Volatile Files and Directories...
[ OK ] Started Create Volatile Files and Directories.
Starting Update UTMP about System Boot/Shutdown...
[ OK ] Started Update UTMP about System Boot/Shutdown.
[ OK ] Reached target System Initialization.
[ OK ] Started Daily Cleanup of Temporary Directories.
[ OK ] Reached target Timers.
[ OK ] Listening on D-Bus System Message Bus Socket.
[ OK ] Reached target Sockets.
[ OK ] Reached target Basic System.
Starting Permit User Sessions...
[ OK ] Started D-Bus System Message Bus.
Starting Login Service...
[ OK ] Started Permit User Sessions.
Starting Cleanup of Temporary Directories...
[ OK ] Started Console Getty.
[ OK ] Reached target Login Prompts.
[ OK ] Started Login Service.
[ OK ] Reached target Multi-User System.
Starting Update UTMP about System Runlevel Changes...
[ OK ] Started Cleanup of Temporary Directories.
[ OK ] Started Update UTMP about System Runlevel Changes.
CentOS Linux 7 (Core)
Kernel 4.18.0-147.el8.x86_64 on an x86_64
2426740762f5 login:
Additional info:
~]$ fgrep tester /etc/* 2> /dev/null
/etc/group:tester:x:1004:
/etc/passwd:tester:x:1004:1004::/home/tester:/bin/bash
/etc/subgid:tester:296608:65536
/etc/subuid:tester:296608:65536
~]$ podman unshare cat /proc/self/uid_map
0 1004 1
1 296608 65536
On RHEL7, this is not supported. Systemd needs to be able to write to cgroups file system, which is not allowed for non root users. It really requires cgroupsV2 for full support.
So on RHEL7 you need to run systemd containers as root. This should be doable on RHEL8.