BTW I cannot see any issues with cgroups V1

can you share the output for?

podman run --name test -ti --rm registry.access.redhat.com/ubi8-init:latest

(In reply to Giuseppe Scrivano from comment #3)
> can you share the output for?
> 
> podman run --name test -ti --rm registry.access.redhat.com/ubi8-init:latest

there is not any difference with `-i`

sh# podman run --name test2 -ti --rm registry.access.redhat.com/ubi8-init:latest
Failed to mount tmpfs at /run: Operation not permitted
[!!!!!!] Failed to mount API filesystems, freezing.
Freezing execution.

sh-5.0# podman --log-level=debug run --name test99 -ti --rm registry.access.redhat.com/ubi8-init:latest
DEBU[0000] using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /var/lib/containers/storage
DEBU[0000] Using run root /var/run/containers/storage
DEBU[0000] Using static dir /var/lib/containers/storage/libpod
DEBU[0000] Using tmp dir /var/run/libpod
DEBU[0000] Using volume path /var/lib/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] cached value indicated that overlay is supported
DEBU[0000] cached value indicated that metacopy is being used
DEBU[0000] cached value indicated that native-diff is not being used
WARN[0000] Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled
DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=true
DEBU[0000] Initializing event backend journald
DEBU[0000] using runtime "/usr/bin/runc"
DEBU[0000] using runtime "/usr/bin/crun"
INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]registry.access.redhat.com/ubi8-init:latest"
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@1de7d7b3f53173fc798c858a750a243bc7f40f4022f4b1e1534f58a11e14bba9"
DEBU[0000] exporting opaque data as blob "sha256:1de7d7b3f53173fc798c858a750a243bc7f40f4022f4b1e1534f58a11e14bba9"
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@1de7d7b3f53173fc798c858a750a243bc7f40f4022f4b1e1534f58a11e14bba9"
DEBU[0000] exporting opaque data as blob "sha256:1de7d7b3f53173fc798c858a750a243bc7f40f4022f4b1e1534f58a11e14bba9"
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@1de7d7b3f53173fc798c858a750a243bc7f40f4022f4b1e1534f58a11e14bba9"
DEBU[0000] No hostname set; container's hostname will default to runtime default
DEBU[0000] Using bridge netmode
DEBU[0000] setting container name test99
DEBU[0000] created OCI spec and options for new container
DEBU[0000] Allocated lock 6 for container 97eca1cf7e97f1b0ef1b41bb27f9140816c14ea77e7f84af6685f5c140632a70
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@1de7d7b3f53173fc798c858a750a243bc7f40f4022f4b1e1534f58a11e14bba9"
DEBU[0000] exporting opaque data as blob "sha256:1de7d7b3f53173fc798c858a750a243bc7f40f4022f4b1e1534f58a11e14bba9"
DEBU[0000] created container "97eca1cf7e97f1b0ef1b41bb27f9140816c14ea77e7f84af6685f5c140632a70"
DEBU[0000] container "97eca1cf7e97f1b0ef1b41bb27f9140816c14ea77e7f84af6685f5c140632a70" has work directory "/var/lib/containers/storage/overlay-containers/97eca1cf7e97f1b0ef1b41bb27f9140816c14ea77e7f84af6685f5c140632a70/userdata"
DEBU[0000] container "97eca1cf7e97f1b0ef1b41bb27f9140816c14ea77e7f84af6685f5c140632a70" has run directory "/var/run/containers/storage/overlay-containers/97eca1cf7e97f1b0ef1b41bb27f9140816c14ea77e7f84af6685f5c140632a70/userdata"
DEBU[0000] New container created "97eca1cf7e97f1b0ef1b41bb27f9140816c14ea77e7f84af6685f5c140632a70"
DEBU[0000] container "97eca1cf7e97f1b0ef1b41bb27f9140816c14ea77e7f84af6685f5c140632a70" has CgroupParent "machine.slice/libpod-97eca1cf7e97f1b0ef1b41bb27f9140816c14ea77e7f84af6685f5c140632a70.scope"
DEBU[0000] Handling terminal attach
DEBU[0000] Made network namespace at /var/run/netns/cni-31d86b2e-df60-ec69-0db5-b6ce081e4367 for container 97eca1cf7e97f1b0ef1b41bb27f9140816c14ea77e7f84af6685f5c140632a70
INFO[0000] Got pod network &{Name:test99 Namespace:test99 ID:97eca1cf7e97f1b0ef1b41bb27f9140816c14ea77e7f84af6685f5c140632a70 NetNS:/var/run/netns/cni-31d86b2e-df60-ec69-0db5-b6ce081e4367 Networks:[] RuntimeConfig:map[podman:{IP: PortMappings:[] Bandwidth:<nil> IpRanges:[]}]}
INFO[0000] About to add CNI network cni-loopback (type=loopback)
DEBU[0000] overlay: mount_data=nodev,metacopy=on,lowerdir=/var/lib/containers/storage/overlay/l/HTMDFB72XGQMODCEULLCVV7VEA:/var/lib/containers/storage/overlay/l/ZUKAHCZWLYEW6FRXQJINTPDZGS:/var/lib/containers/storage/overlay/l/EQLO5MJDL6NKWCTBRPUDD5E3FB,upperdir=/var/lib/containers/storage/overlay/5bb4e70a9054ac5fead6f3a6dcecb16d663e0046f2176aa8c4fb8edc3588d319/diff,workdir=/var/lib/containers/storage/overlay/5bb4e70a9054ac5fead6f3a6dcecb16d663e0046f2176aa8c4fb8edc3588d319/work,context="system_u:object_r:container_file_t:s0:c241,c454"
DEBU[0000] mounted container "97eca1cf7e97f1b0ef1b41bb27f9140816c14ea77e7f84af6685f5c140632a70" at "/var/lib/containers/storage/overlay/5bb4e70a9054ac5fead6f3a6dcecb16d663e0046f2176aa8c4fb8edc3588d319/merged"
DEBU[0000] Created root filesystem for container 97eca1cf7e97f1b0ef1b41bb27f9140816c14ea77e7f84af6685f5c140632a70 at /var/lib/containers/storage/overlay/5bb4e70a9054ac5fead6f3a6dcecb16d663e0046f2176aa8c4fb8edc3588d319/merged
INFO[0000] Got pod network &{Name:test99 Namespace:test99 ID:97eca1cf7e97f1b0ef1b41bb27f9140816c14ea77e7f84af6685f5c140632a70 NetNS:/var/run/netns/cni-31d86b2e-df60-ec69-0db5-b6ce081e4367 Networks:[] RuntimeConfig:map[podman:{IP: PortMappings:[] Bandwidth:<nil> IpRanges:[]}]}
INFO[0000] About to add CNI network podman (type=bridge)
DEBU[0000] [0] CNI result: Interfaces:[{Name:cni-podman0 Mac:86:2d:ad:45:3c:bd Sandbox:} {Name:vethbaa14575 Mac:aa:7e:9c:54:e3:33 Sandbox:} {Name:eth0 Mac:fa:e7:48:1a:ec:42 Sandbox:/var/run/netns/cni-31d86b2e-df60-ec69-0db5-b6ce081e4367}], IP:[{Version:4 Interface:0xc000485b08 Address:{IP:10.88.0.7 Mask:ffff0000} Gateway:10.88.0.1}], Routes:[{Dst:{IP:0.0.0.0 Mask:00000000} GW:<nil>}], DNS:{Nameservers:[] Domain: Search:[] Options:[]}
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode secret
DEBU[0000] Setting CGroups for container 97eca1cf7e97f1b0ef1b41bb27f9140816c14ea77e7f84af6685f5c140632a70 to machine.slice:libpod:97eca1cf7e97f1b0ef1b41bb27f9140816c14ea77e7f84af6685f5c140632a70
DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d
DEBU[0000] reading hooks from /etc/containers/oci/hooks.d
DEBU[0000] Created OCI spec for container 97eca1cf7e97f1b0ef1b41bb27f9140816c14ea77e7f84af6685f5c140632a70 at /var/lib/containers/storage/overlay-containers/97eca1cf7e97f1b0ef1b41bb27f9140816c14ea77e7f84af6685f5c140632a70/userdata/config.json
DEBU[0000] /usr/bin/conmon messages will be logged to syslog
DEBU[0000] running conmon: /usr/bin/conmon               args="[--api-version 1 -s -c 97eca1cf7e97f1b0ef1b41bb27f9140816c14ea77e7f84af6685f5c140632a70 -u 97eca1cf7e97f1b0ef1b41bb27f9140816c14ea77e7f84af6685f5c140632a70 -r /usr/bin/crun -b /var/lib/containers/storage/overlay-containers/97eca1cf7e97f1b0ef1b41bb27f9140816c14ea77e7f84af6685f5c140632a70/userdata -p /var/run/containers/storage/overlay-containers/97eca1cf7e97f1b0ef1b41bb27f9140816c14ea77e7f84af6685f5c140632a70/userdata/pidfile -l k8s-file:/var/lib/containers/storage/overlay-containers/97eca1cf7e97f1b0ef1b41bb27f9140816c14ea77e7f84af6685f5c140632a70/userdata/ctr.log --exit-dir /var/run/libpod/exits --socket-dir-path /var/run/libpod/socket --log-level debug --syslog -t --conmon-pidfile /var/run/containers/storage/overlay-containers/97eca1cf7e97f1b0ef1b41bb27f9140816c14ea77e7f84af6685f5c140632a70/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /var/run/containers/storage --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /var/run/libpod --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev,metacopy=on --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg 97eca1cf7e97f1b0ef1b41bb27f9140816c14ea77e7f84af6685f5c140632a70]"
INFO[0000] Running conmon under slice machine.slice and unitName libpod-conmon-97eca1cf7e97f1b0ef1b41bb27f9140816c14ea77e7f84af6685f5c140632a70.scope
DEBU[0000] Received: 30139
INFO[0000] Got Conmon PID as 30133
DEBU[0000] Created container 97eca1cf7e97f1b0ef1b41bb27f9140816c14ea77e7f84af6685f5c140632a70 in OCI runtime
DEBU[0000] Attaching to container 97eca1cf7e97f1b0ef1b41bb27f9140816c14ea77e7f84af6685f5c140632a70
DEBU[0000] connecting to socket /var/run/libpod/socket/97eca1cf7e97f1b0ef1b41bb27f9140816c14ea77e7f84af6685f5c140632a70/attach
DEBU[0000] Starting container 97eca1cf7e97f1b0ef1b41bb27f9140816c14ea77e7f84af6685f5c140632a70 with command [/sbin/init]
DEBU[0000] Received a resize event: {Width:187 Height:52}
DEBU[0000] Started container 97eca1cf7e97f1b0ef1b41bb27f9140816c14ea77e7f84af6685f5c140632a70
DEBU[0000] Enabling signal proxying
Failed to mount tmpfs at /run: Operation not permitted
[!!!!!!] Failed to mount API filesystems, freezing.
Freezing execution.


sh-5.0# podman logs test99
Failed to mount tmpfs at /run: Operation not permitted
[!!!!!!] Failed to mount API filesystems, freezing.
Freezing execution.
sh-5.0# podman exec test99 systemctl status
System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to connect to bus: Host is down
Error: non zero exit code: 1: OCI runtime error

I bisected podman and the issue is caused by a recent change (3ba3e1c7510d1780b6527a4aa52e40ac2c5b576a) that look up the full path for enabling systemd mode.  We need to modify the registry.access.redhat.com/ubi8-init image to use "/usr/sbin/init" as command, not "init".

To confirm it, could you try if: "# podman run --name test99 -ti --rm registry.access.redhat.com/ubi8-init:latest /usr/sbin/init" works?

(In reply to Giuseppe Scrivano from comment #6)
> I bisected podman and the issue is caused by a recent change
> (3ba3e1c7510d1780b6527a4aa52e40ac2c5b576a) that look up the full path for
> enabling systemd mode.  We need to modify the
> registry.access.redhat.com/ubi8-init image to use "/usr/sbin/init" as
> command, not "init".
> 
> To confirm it, could you try if: "# podman run --name test99 -ti --rm
> registry.access.redhat.com/ubi8-init:latest /usr/sbin/init" works?

I can confirm it works with /usr/sbin/init

However, full path is used by default. Just different one due to /usr/sbin and /sbin merge
https://fedoraproject.org/wiki/Features/UsrMove

sh# podman inspect registry.access.redhat.com/ubi8-init:latest | jq .[0].Config.Cmd
[
  "/sbin/init"
]

I've opened a PR here: https://github.com/containers/libpod/pull/4267