Hide Forgot
Description of problem: Since last week's Fedora 33 updates, I cannot create/run new toolboxes any more. The ones created with an earlier release still work. Version-Release number of selected component (if applicable): toolbox-0.0.95-1.fc33.x86_64 podman-2.1.0-0.179.dev.git43f2771.fc33.x86_64 conmon-2.0.21-3.fc33.x86_64 kernel-5.8.10-300.fc33.x86_64 How reproducible: Always Steps to Reproduce: 1. toolbox --verbose create -c devel (full output at [1]) 2. toolbox --verbose enter -c devel (full output at [2]) Actual results: 2. fails with "Error: failed to start container devel". Trying to start with podman directly: $ podman --log-level=debug start devel [...] DEBU[0000] running conmon: /usr/bin/conmon args="[--api-version 1 -c 19b1ecdff41b5aa68b8f4b4d5c6e35ae1843a012dfd325a1f980ac55168b581d -u 19b1ecdff41b5aa68b8f4b4d5c6e35ae1843a012dfd325a1f980ac55168b581d -r /usr/bin/crun -b /var/home/martin/.local/share/containers/storage/overlay-containers/19b1ecdff41b5aa68b8f4b4d5c6e35ae1843a012dfd325a1f980ac55168b581d/userdata -p /run/user/1000/containers/overlay-containers/19b1ecdff41b5aa68b8f4b4d5c6e35ae1843a012dfd325a1f980ac55168b581d/userdata/pidfile -n devel --exit-dir /run/user/1000/libpod/tmp/exits --socket-dir-path /run/user/1000/libpod/tmp/socket -s -l k8s-file:/var/home/martin/.local/share/containers/storage/overlay-containers/19b1ecdff41b5aa68b8f4b4d5c6e35ae1843a012dfd325a1f980ac55168b581d/userdata/ctr.log --log-level debug --syslog --conmon-pidfile /run/user/1000/containers/overlay-containers/19b1ecdff41b5aa68b8f4b4d5c6e35ae1843a012dfd325a1f980ac55168b581d/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/home/martin/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1000/containers --exit-command-arg --log-level --exit-command-arg error --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --runtime --exit-command-arg /usr/bin/crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg --events-backend --exit-command-arg file --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 19b1ecdff41b5aa68b8f4b4d5c6e35ae1843a012dfd325a1f980ac55168b581d]" [conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied DEBU[0000] Received: -1 DEBU[0000] Cleaning up container 19b1ecdff41b5aa68b8f4b4d5c6e35ae1843a012dfd325a1f980ac55168b581d DEBU[0000] Network is already cleaned up, skipping... DEBU[0000] unmounted container "19b1ecdff41b5aa68b8f4b4d5c6e35ae1843a012dfd325a1f980ac55168b581d" Error: unable to start container "19b1ecdff41b5aa68b8f4b4d5c6e35ae1843a012dfd325a1f980ac55168b581d": open /proc/sys/net/ipv4/ping_group_range: Permission denied: OCI runtime permission denied error This error does not happen when I directly try "podman run -it --rm registry.fedoraproject.org/f33/fedora-toolbox:33", so toolbox configures the container to do something special. But I dont' see that "ping_group_range" thing in "podman inspect devel". Expected results: container starts properly. Additional info: [1] DEBU Running as real user ID 1000 DEBU Resolved absolute path to the executable as /usr/bin/toolbox DEBU Running on a cgroups v2 host DEBU Checking if /etc/subgid and /etc/subuid have entries for user martin DEBU TOOLBOX_PATH is /usr/bin/toolbox DEBU Toolbox config directory is /var/home/martin/.config/toolbox DEBU Current Podman version is 2.1.0-dev DEBU Old Podman version is 2.1.0-dev DEBU Migration not needed: Podman version 2.1.0-dev is unchanged DEBU Resolving container and image names DEBU Container: 'devel' DEBU Image: '' DEBU Release: '' DEBU Resolved container and image names DEBU Container: 'devel' DEBU Image: 'fedora-toolbox:33' DEBU Release: '33' DEBU Checking if container devel already exists DEBU Looking for image fedora-toolbox:33 DEBU Resolving fully qualified name for image fedora-toolbox:33 DEBU Resolved image fedora-toolbox:33 to registry.fedoraproject.org/f33/fedora-toolbox:33 DEBU Checking if 'podman create' supports '--ulimit host' DEBU 'podman create' supports '--ulimit host' DEBU Resolving path to the D-Bus system socket DEBU Calling org.freedesktop.Flatpak.SessionHelper.RequestSession DEBU /var/home/martin canonicalized to /var/home/martin DEBU Checking if /usr is mounted read-only or read-write DEBU Mount-point of /usr is /usr DEBU Mount flags of /usr on the host are ro,relatime,seclabel DEBU Resolving path to the KCM socket DEBU failed to find a SOCK_STREAM socket for sssd-kcm.socket DEBU Checking if /media is a symbolic link to /run/media DEBU /media is a symbolic link to /run/media DEBU Checking if /mnt is a symbolic link to /var/mnt DEBU /mnt is a symbolic link to /var/mnt DEBU Looking for toolbox.sh DEBU Found /etc/profile.d/toolbox.sh DEBU Checking if /home is a symbolic link to /var/home DEBU /home is a symbolic link to /var/home DEBU Creating container devel: DEBU podman DEBU --log-level DEBU error DEBU create DEBU --dns DEBU none DEBU --env DEBU TOOLBOX_PATH=/usr/bin/toolbox DEBU --hostname DEBU toolbox DEBU --ipc DEBU host DEBU --label DEBU com.github.containers.toolbox=true DEBU --label DEBU com.github.debarshiray.toolbox=true DEBU --name DEBU devel DEBU --network DEBU host DEBU --no-hosts DEBU --pid DEBU host DEBU --privileged DEBU --security-opt DEBU label=disable DEBU --ulimit DEBU host DEBU --userns=keep-id DEBU --user DEBU root:root DEBU --volume DEBU /etc:/run/host/etc DEBU --volume DEBU /dev:/dev:rslave DEBU --volume DEBU /run:/run/host/run:rslave DEBU --volume DEBU /tmp:/run/host/tmp:rslave DEBU --volume DEBU /var:/run/host/var:rslave DEBU --volume DEBU /run/dbus/system_bus_socket:/run/dbus/system_bus_socket DEBU --volume DEBU /run/user/1000/.flatpak-helper/monitor:/run/host/monitor DEBU --volume DEBU /var/home/martin:/var/home/martin:rslave DEBU --volume DEBU /usr/bin/toolbox:/usr/bin/toolbox:ro DEBU --volume DEBU /usr:/run/host/usr:ro,rslave DEBU --volume DEBU /run/user/1000:/run/user/1000 DEBU --volume DEBU /run/media:/run/media:rslave DEBU --volume DEBU /etc/profile.d/toolbox.sh:/etc/profile.d/toolbox.sh:ro DEBU registry.fedoraproject.org/f33/fedora-toolbox:33 DEBU toolbox DEBU --verbose DEBU init-container DEBU --home DEBU /var/home/martin DEBU --home-link DEBU --media-link DEBU --mnt-link DEBU --monitor-host DEBU --shell DEBU /bin/bash DEBU --uid DEBU 1000 DEBU --user DEBU martin Created container: devel Enter with: toolbox enter devel [2] DEBU Running as real user ID 1000 DEBU Resolved absolute path to the executable as /usr/bin/toolbox DEBU Running on a cgroups v2 host DEBU Checking if /etc/subgid and /etc/subuid have entries for user martin DEBU TOOLBOX_PATH is /usr/bin/toolbox DEBU Toolbox config directory is /var/home/martin/.config/toolbox DEBU Current Podman version is 2.1.0-dev DEBU Old Podman version is 2.1.0-dev DEBU Migration not needed: Podman version 2.1.0-dev is unchanged DEBU Resolving container and image names DEBU Container: 'devel' DEBU Image: '' DEBU Release: '' DEBU Resolved container and image names DEBU Container: 'devel' DEBU Image: 'fedora-toolbox:33' DEBU Release: '33' DEBU Checking if container devel exists DEBU Calling org.freedesktop.Flatpak.SessionHelper.RequestSession DEBU Starting container devel Error: failed to start container devel
I forgot: Since last week's update, podman and toolbox remained the same. The meaningful package updates in "sudo rpm-ostree db diff" are: conmon 2:2.0.21-0.3.dev.git5a6b2ac.fc33 -> 2:2.0.21-3.fc33 kernel 5.8.8-300.fc33 -> 5.8.10-300.fc33
CC'ing Debarshi as that affects toolbox.
Hmm, I tried to downgrade to the previous conmon: curl https://kojipkgs.fedoraproject.org//packages/conmon/2.0.21/0.3.dev.git5a6b2ac.fc33/x86_64/conmon-2.0.21-0.3.dev.git5a6b2ac.fc33.x86_64.rpm | rpm2cpio | cpio -ivdD /tmp/ ./usr/bin/conmon sudo mount -o bind /tmp/usr/bin/conmon /usr/bin/conmon sudo setenforce 0 But that doesn't seem to help, still the same error. Conversely, I booted back into the previous deployment and ran the conmon binary from 2:2.0.21-3.fc33, and that works fine. So it's not conmon after all. Looks like toolbox needs to adjust its configuration to some recent kernel changes somehow?
It's this Podman issue: https://github.com/containers/podman/issues/7766
Thanks Debarshi! Confirming the bug with recent updates: podman-2.1.1-2.fc33.x86_64 runc-1.0.0-279.dev.gitdedadbf.fc33.x86_64 kernel-5.8.12-300.fc33.x86_64 containers-common-1.1.1-10.fc33.x86_64 crun-0.15-3.fc33.x86_64 I don't have a containers.conf anywhere (that's not created by default apparently), and notice that https://github.com/containers/podman/issues/7766 is slightly different: That has Error: write to /proc/sys/net/ipv4/ping_group_range: Invalid argument: OCI runtime error and running "podman run --uidmap 0:10000:10000 quay.io/libpod/testimage:20200902 true" *does* work. The bug here is open /proc/sys/net/ipv4/ping_group_range: Permission denied: OCI runtime permission denied error Nevertheless, it's certainly related, as creating a containers.conf and disabling the sysctl works: mkdir ~/.config/containers; printf '[containers]\ndefault_sysctls = []\n' > ~/.config/containers/containers.conf This is a good enough workaround for now \o/ which unblocks the workflow, and doesn't need me to go back to an old ostree to recreate toolboxes \o/ This somehow seems to be part of a container definition. If I do rm ~/.config/containers/containers.conf toolbox create -c noconf printf '[containers]\ndefault_sysctls = []\n' > ~/.config/containers/containers.conf toolbox create -c emptysysctl Then noconf fails, and emptysysctl succeeds a "toolbox enter". I looked at diff -u <(podman inspect x) <(podman inspect y) which unfortunately has a lot of noise due to the unsorted Mounts and Binds maps, but after sorting these a little there really is no significant difference between. .local/share/containers/storage/overlay-containers/<uuid>/ is identical for both, they just have an empty userdata/artifacts/ subdir (other containers have a config.json and ctr.log). The config in .local/share/containers/storage/overlay-containers/containers.json also looks very similar, the only difference that isn't trivial (like UUID or timestamps) is "flags": { - "MountLabel": "system_u:object_r:container_file_t:s0:c816,c938", + "MountLabel": "system_u:object_r:container_file_t:s0:c304,c863", "ProcessLabel": "" } (Not sure if anything meaningful is encoded there)
We believe this is fixed in podman-2.1.1-10.fc33 and newer. Can you verify?
Confirmed. I updated to latest Fedora, including podman-2.1.1-11.fc33.x86_64, and it once again works with the default configuration. Thank you!