Fedora Account System
Red Hat Associate
Red Hat Customer
Created attachment 1609085 [details] podman_info.log Description of problem: Rootless podman won't start containers Version-Release number of selected component (if applicable): podman-1.5.1-2.16.dev.gitce64c14.fc31.x86_64 How reproducible: Always Steps to Reproduce: 1. Run `sudo podman run --rm -ti fedora:30` - container starts 2. Run `podman -- run --rm -ti fedora:30` Actual results: ... WARN[0000] Failed to add conmon to systemd sandbox cgroup: read unix @->/run/systemd/private: read: connection reset by peer ... Journalctl: Aug 28 19:35:58 finn podman[18049]: 2019-08-28 19:35:58.381249541 +0200 CEST m=+0.107888353 container create 924058e58955f2895d698bcd2f49d0dfc00a10272339b06c69200219e5db50ca (image=docker.io/library/fedora:30, name=trusting_lovelace) Aug 28 19:35:58 finn systemd[1]: Incoming private connection from unprivileged client, refusing: Operation not permitted Aug 28 19:35:58 finn conmon[18068]: conmon 924058e58955f2895d69 <ndebug>: failed to write to /proc/self/oom_score_adj: Permission denied Aug 28 19:35:58 finn conmon[18069]: conmon 924058e58955f2895d69 <ninfo>: addr{sun_family=AF_UNIX, sun_path=/tmp/conmon-term.MOY36Z} Aug 28 19:35:58 finn conmon[18069]: conmon 924058e58955f2895d69 <ninfo>: attach sock path: /run/user/1000/libpod/tmp/socket/924058e58955f2895d698bcd2f49d0dfc00a10272339b06c69200219e5db50ca/attach Aug 28 19:35:58 finn conmon[18069]: conmon 924058e58955f2895d69 <ninfo>: addr{sun_family=AF_UNIX, sun_path=/run/user/1000/libpod/tmp/socket/924058e58955f2895d698bcd2f49d0dfc00a10272339b06c69200219e5db50ca/attach} Aug 28 19:35:58 finn conmon[18069]: conmon 924058e58955f2895d69 <ninfo>: ctl fifo path: /var/home/vrutkovs/.local/share/containers/storage/overlay-containers/924058e58955f2895d698bcd2f49d0dfc00a10272339b06c69200219e5db50ca/userdata/ctl Aug 28 19:35:58 finn conmon[18069]: conmon 924058e58955f2895d69 <ninfo>: terminal_ctrl_fd: 14 Aug 28 19:35:58 finn systemd[1]: Incoming private connection from unprivileged client, refusing: Operation not permitted Aug 28 19:35:58 finn conmon[18069]: conmon 924058e58955f2895d69 <error>: Failed to create container: exit status 1 Aug 28 19:35:58 finn podman[18049]: 2019-08-28 19:35:58.439851561 +0200 CEST m=+0.166490404 container remove 924058e58955f2895d698bcd2f49d0dfc00a10272339b06c69200219e5db50ca (image=docker.io/library/fedora:30, name=trusting_lovelace) Expected results: Container starts Additional info:
Progress with `runtime=crun`, new containers can be created via `podman run --rm -ti fedora:30`, but toolbox still doesn't work: $ toolbox create -i registry.fedoraproject.org/f30/fedora-toolbox:30 -c fedora-toolbox-31 $ podman --log-level debug start fedora-toolbox-31 INFO[0000] running as rootless DEBU[0000] using conmon: "/usr/libexec/podman/conmon" DEBU[0000] Initializing boltdb state at /var/home/vrutkovs/.local/share/containers/storage/libpod/bolt_state.db DEBU[0000] Using graph driver overlay DEBU[0000] Using graph root /var/home/vrutkovs/.local/share/containers/storage DEBU[0000] Using run root /run/user/1000 DEBU[0000] Using static dir /var/home/vrutkovs/.local/share/containers/storage/libpod DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp DEBU[0000] Using volume path /var/home/vrutkovs/.local/share/containers/storage/volumes DEBU[0000] Set libpod namespace to "" DEBU[0000] [graphdriver] trying provided driver "overlay" DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false DEBU[0000] Initializing event backend journald DEBU[0000] using runtime "/usr/bin/runc" DEBU[0000] using runtime "/usr/bin/crun" DEBU[0000] overlay: mount_data=lowerdir=/var/home/vrutkovs/.local/share/containers/storage/overlay/l/HUMX3LMSCUBQE2GPB3SIN7IASM:/var/home/vrutkovs/.local/share/containers/storage/overlay/l/2GKKO4HNSEUHPU5J33CD5SMVUD,upperdir=/var/home/vrutkovs/.local/share/containers/storage/overlay/b909fecf5fd450e56884d7aab6d76e2797f75c54227927113705e379d8cf4a03/diff,workdir=/var/home/vrutkovs/.local/share/containers/storage/overlay/b909fecf5fd450e56884d7aab6d76e2797f75c54227927113705e379d8cf4a03/work,context="system_u:object_r:container_file_t:s0:c556,c1016" DEBU[0000] mounted container "1e5f54f25b5713c173b80339ae745a325e6568132cec149bf809a5073acb0f53" at "/var/home/vrutkovs/.local/share/containers/storage/overlay/b909fecf5fd450e56884d7aab6d76e2797f75c54227927113705e379d8cf4a03/merged" DEBU[0000] Created root filesystem for container 1e5f54f25b5713c173b80339ae745a325e6568132cec149bf809a5073acb0f53 at /var/home/vrutkovs/.local/share/containers/storage/overlay/b909fecf5fd450e56884d7aab6d76e2797f75c54227927113705e379d8cf4a03/merged DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode secret DEBU[0000] Setting CGroups for container 1e5f54f25b5713c173b80339ae745a325e6568132cec149bf809a5073acb0f53 to user.slice:libpod:1e5f54f25b5713c173b80339ae745a325e6568132cec149bf809a5073acb0f53 DEBU[0000] set root propagation to "rslave" DEBU[0000] Created OCI spec for container 1e5f54f25b5713c173b80339ae745a325e6568132cec149bf809a5073acb0f53 at /var/home/vrutkovs/.local/share/containers/storage/overlay-containers/1e5f54f25b5713c173b80339ae745a325e6568132cec149bf809a5073acb0f53/userdata/config.json DEBU[0000] /usr/libexec/podman/conmon messages will be logged to syslog DEBU[0000] running conmon: /usr/libexec/podman/conmon args="[--api-version 1 -s -c 1e5f54f25b5713c173b80339ae745a325e6568132cec149bf809a5073acb0f53 -u 1e5f54f25b5713c173b80339ae745a325e6568132cec149bf809a5073acb0f53 -r /usr/bin/crun -b /var/home/vrutkovs/.local/share/containers/storage/overlay-containers/1e5f54f25b5713c173b80339ae745a325e6568132cec149bf809a5073acb0f53/userdata -p /run/user/1000/overlay-containers/1e5f54f25b5713c173b80339ae745a325e6568132cec149bf809a5073acb0f53/userdata/pidfile -l k8s-file:/var/home/vrutkovs/.local/share/containers/storage/overlay-containers/1e5f54f25b5713c173b80339ae745a325e6568132cec149bf809a5073acb0f53/userdata/ctr.log --exit-dir /run/user/1000/libpod/tmp/exits --socket-dir-path /run/user/1000/libpod/tmp/socket --log-level debug --syslog --conmon-pidfile /run/user/1000/overlay-containers/1e5f54f25b5713c173b80339ae745a325e6568132cec149bf809a5073acb0f53/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/home/vrutkovs/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1000 --exit-command-arg --log-level --exit-command-arg error --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 1e5f54f25b5713c173b80339ae745a325e6568132cec149bf809a5073acb0f53]" INFO[0000] Running conmon under slice user.slice and unitName libpod-conmon-1e5f54f25b5713c173b80339ae745a325e6568132cec149bf809a5073acb0f53.scope WARN[0000] Failed to add conmon to systemd sandbox cgroup: read unix @->/run/systemd/private: read: connection reset by peer [conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied DEBU[0000] Received: -1 DEBU[0000] Cleaning up container 1e5f54f25b5713c173b80339ae745a325e6568132cec149bf809a5073acb0f53 DEBU[0000] Network is already cleaned up, skipping... DEBU[0000] unmounted container "1e5f54f25b5713c173b80339ae745a325e6568132cec149bf809a5073acb0f53" ERRO[0000] unable to start container "fedora-toolbox-31": setrlimit (RLIM_MEMLOCK): Operation not permitted: OCI runtime error Journal: Aug 28 19:50:11 finn podman[25873]: 2019-08-28 19:50:11.762559633 +0200 CEST m=+0.084201611 container create 1e5f54f25b5713c173b80339ae745a325e6568132cec149bf809a5073acb0f53 (image=registry.fedoraproject.org/f30/fedora-toolbox:30, name=fedora-toolbox-31) Aug 28 19:50:24 finn systemd[1]: Incoming private connection from unprivileged client, refusing: Operation not permitted Aug 28 19:50:24 finn conmon[25999]: conmon 1e5f54f25b5713c173b8 <ndebug>: failed to write to /proc/self/oom_score_adj: Permission denied Aug 28 19:50:24 finn conmon[26000]: conmon 1e5f54f25b5713c173b8 <ninfo>: attach sock path: /run/user/1000/libpod/tmp/socket/1e5f54f25b5713c173b80339ae745a325e6568132cec149bf809a5073acb0f53/attach Aug 28 19:50:24 finn conmon[26000]: conmon 1e5f54f25b5713c173b8 <ninfo>: addr{sun_family=AF_UNIX, sun_path=/run/user/1000/libpod/tmp/socket/1e5f54f25b5713c173b80339ae745a325e6568132cec149bf809a5073acb0f53/attach} Aug 28 19:50:24 finn conmon[26000]: conmon 1e5f54f25b5713c173b8 <ninfo>: ctl fifo path: /var/home/vrutkovs/.local/share/containers/storage/overlay-containers/1e5f54f25b5713c173b80339ae745a325e6568132cec149bf809a5073acb0f53/userdata/ctl Aug 28 19:50:24 finn conmon[26000]: conmon 1e5f54f25b5713c173b8 <ninfo>: terminal_ctrl_fd: 12 Aug 28 19:50:24 finn systemd[1038]: Started libcrun container. Aug 28 19:50:24 finn conmon[26000]: conmon 1e5f54f25b5713c173b8 <error>: Failed to create container: exit status 1 Aug 28 19:50:24 finn systemd[1038]: libpod-1e5f54f25b5713c173b80339ae745a325e6568132cec149bf809a5073acb0f53.scope: Succeeded. Aug 28 19:50:25 finn podman[26000]: 2019-08-28 19:50:25.016370196 +0200 CEST m=+0.052472609 container cleanup 1e5f54f25b5713c173b80339ae745a325e6568132cec149bf809a5073acb0f53 (image=registry.fedoraproject.org/f30/fedora-toolbox:30, name=fedora-toolbox-31)
Dan - I think we need to get a patch into F31 RPMs to set the default runtime in libpod.conf to crun.
Can you try podman-1.5.1-2.17 [1] ? [1] https://bodhi.fedoraproject.org/updates/FEDORA-2019-0a601cf11c
Matt, already have. Ed points to it above. Fixed in podman-1.5.1-2.17 Update karma if this works for you. We need to move forward on Buildah support as well.
Rootless toolbox container still fails to start: $ podman --log-level debug run --rm -ti fedora:30 ... <works> ... $ toolbox create -i registry.fedoraproject.org/f30/fedora-toolbox:30 -c fedora-toolbox-31 $ podman --log-level debug start fedora-toolbox-31 ... DEBU[0000] running conmon: /usr/libexec/podman/conmon args="[--api-version 1 -s -c ca9dc0430321d737701e8debcdb0a0e826408d7ae00a07cecf46be991d34cdb7 -u ca9dc0430321d737701e8debcdb0a0e826408d7ae00a07cecf46be991d34cdb7 -r /usr/bin/crun -b /var/home/vrutkovs/.local/share/containers/storage/overlay-containers/ca9dc0430321d737701e8debcdb0a0e826408d7ae00a07cecf46be991d34cdb7/userdata -p /run/user/1000/overlay-containers/ca9dc0430321d737701e8debcdb0a0e826408d7ae00a07cecf46be991d34cdb7/userdata/pidfile -l k8s-file:/var/home/vrutkovs/.local/share/containers/storage/overlay-containers/ca9dc0430321d737701e8debcdb0a0e826408d7ae00a07cecf46be991d34cdb7/userdata/ctr.log --exit-dir /run/user/1000/libpod/tmp/exits --socket-dir-path /run/user/1000/libpod/tmp/socket --log-level debug --syslog --conmon-pidfile /run/user/1000/overlay-containers/ca9dc0430321d737701e8debcdb0a0e826408d7ae00a07cecf46be991d34cdb7/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/home/vrutkovs/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1000 --exit-command-arg --log-level --exit-command-arg error --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg ca9dc0430321d737701e8debcdb0a0e826408d7ae00a07cecf46be991d34cdb7]" INFO[0000] Running conmon under slice user.slice and unitName libpod-conmon-ca9dc0430321d737701e8debcdb0a0e826408d7ae00a07cecf46be991d34cdb7.scope WARN[0000] Failed to add conmon to systemd sandbox cgroup: read unix @->/run/systemd/private: read: connection reset by peer [conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied DEBU[0000] Received: -1 DEBU[0000] Cleaning up container ca9dc0430321d737701e8debcdb0a0e826408d7ae00a07cecf46be991d34cdb7 DEBU[0000] Network is already cleaned up, skipping... DEBU[0000] unmounted container "ca9dc0430321d737701e8debcdb0a0e826408d7ae00a07cecf46be991d34cdb7" ERRO[0000] unable to start container "fedora-toolbox-31": setrlimit (RLIM_MEMLOCK): Operation not permitted: OCI runtime error podman-1.5.1-2.17.dev.gitce64c14.fc31.x86_64, runtime set to `crun`, `rpm-ostree kargs` has `systemd.unified_cgroup_hierarchy=1`
Here too: $ rpm-ostree status | grep 31 ● ostree://fedora:fedora/31/x86_64/silverblue Version: 31.20190907.n.0 (2019-09-07T08:28:31Z) $ rpm -q podman podman-1.5.1-2.17.dev.gitce64c14.fc31 $ podman run -it fedora:latest Error: time="2019-09-08T12:13:34+02:00" level=warning msg="signal: killed" time="2019-09-08T12:13:34+02:00" level=error msg="container_linux.go:346: starting container process caused \"process_linux.go:297: applying cgroup configuration for process caused \\\"mountpoint for cgroup not found\\\"\"\n" container_linux.go:346: starting container process caused "process_linux.go:297: applying cgroup configuration for process caused \"mountpoint for cgroup not found\"": OCI runtime error $ podman --runtime /usr/bin/crun run -it fedora:latest Error: create '/sys/fs/cgroup/libpod_parent': Permission denied: OCI runtime error $ sudo podman run -it fedora:latest [root@55a2b73fa224 /]# With sudo (root), `ps ax` output suggests that crun is being used, yes. (Indeed `sudo podman --runtime /usr/bin/runc run -it fedora:latest` fails.) However for user (rootless) the errors make this unclear.
that is fixed upstream with https://github.com/containers/libpod/commit/b101a8d3664f054157a9e3f08a6bf8db0144041c a rootless user cannot use eBPF, so we need to make sure we don't set that in th OCI conf file.
Thanks, can't reproduce anymore in podman-1.5.2-0.60.dev.gitf500feb.fc32.x86_64.rpm
Yep I also tried latest rawhide podman-1.5.2-dev today and looks fixed there. (`podman run -it hello-world` works in Fedora Rawhide WS Live)
Discussed during the 2019-09-09 blocker review meeting: [0] The decision to classify this bug as a "RejectedFreezeException" was made as this is obviously a significant bug worth fixing, however we can't see any justification for why it needs to be a Freeze Exception rather than a normal update. [0] https://meetbot.fedoraproject.org/fedora-blocker-review/2019-09-09/f31-blocker-review.2019-09-09-16.00.txt
Ahoy, maintainer - can you please do a build for F31 and submit it as an update? This is still broken in F31, I believe.
Dan, don't we have a new podman build for F31 now?
I see a build in Koji: https://koji.fedoraproject.org/koji/buildinfo?buildID=1377243 but it does not appear to have been submitted to Bodhi as an update. It will not actually reach any F31 repo until that happens.
I just put it in updates.
Comment #13 points to podman-1.5.1-2.18.dev.gitce64c14.fc31 and comment #14 says it's been put in updates. I have podman-1.5.1-3.git0005792.fc31.x86_64 which is presumably newer, and still get the same error message when running a rootless container: Error: container_linux.go:345: starting container process caused "process_linux.go:281: applying cgroup configuration for process caused \"mountpoint for cgroup not found\"": OCI runtime error Running with sudo does work.
looks like you are using runc as the runtime, you'll need to switch to crun. You can change it in ~/.config/containers/libpod.conf and select crun instead of runc, or you could rm -rf ~/.config/containers, next time Podman will create the configuration then it should pick crun by default
> looks like you are using runc as the runtime, you'll need to switch to crun. You can change it in ~/.config/containers/libpod.conf and select crun instead of runc, or you could rm -rf ~/.config/containers, next time Podman will create the configuration then it should pick crun by default Indeed I had `runtime = "runc"` in that file. (side note: it's pretty terrible of podman to create a config file like that silently, and then not change the defaults automatically if the user had never touched that file) I removed ~/.config/containers as you suggested, and now… podman fails in a new way: Error: creating cgroup directory '/sys/fs/cgroup/name=systemd/user.slice/user-1000.slice/user/user.slice/libpod-31c3365fddaa2612eda0069e27a92eee21c923b49288e6ce1330841a9870d210.scope': Permission denied: OCI runtime error Should I open a new issue?
Trying podman-1.6.0-2.gitca5ff03 from Koji (which wasn't sent to Bodhi?) I get: Error: creating cgroup directory '/sys/fs/cgroup/name=systemd/user.slice/user-1000.slice/user/user.slice/libpod-24c3111f6b6ed6d64dfabe54522d7017a1a77e0cdfd4b63c756a6cecc5963b8c.scope': Permission denied: OCI runtime permission denied error Which is essentially the same error (just the word "permission" added to the string)
that should be already fixed upstream by afd0818326aa37f03a3bc74f0269a06a403db16d as a workaround, you need to ensure the unprivileged user owns the current cgroup: "systemd-run --scope --user podman .." or more easily let it wrap your entire session with "systemd-run --scope --user $SHELL"
Fedora 31 has not been released yet. We are working on fixing all of the issues that are caused by switching the defalut cgroups to V2. Thanks for your patience and helping us with the transition.
I am not sure if I should report new bug but let me put my observations while working on getting toolbox working on Fedora Silverblue 31. Below are packages on my system, some are layered. podman-1.6.0-2.gitca5ff03.fc31.x86_64 crun-0.9.1-1.fc31.x86_64 toolbox-0.0.14-1.fc31.noarch I have also done now rm -rf ~/.config/containers/ unset NOTIFY_SOCKET now lets attempt to start toolbox for F30 [parag@localhost ~]$ toolbox -v enter -r 30 toolbox: resolved absolute path for /usr/bin/toolbox to /usr/bin/toolbox toolbox: checking if /etc/subgid and /etc/subuid have entries for user parag toolbox: TOOLBOX_PATH is /usr/bin/toolbox toolbox: checking if 'podman system migrate' exists toolbox: migration not needed: 1.6.0-rc1 is unchanged toolbox: Fedora generational core is f30 toolbox: base image is fedora-toolbox:30 toolbox: container is fedora-toolbox-30 toolbox: checking if container fedora-toolbox-30 exists toolbox: container fedora-toolbox-30 not found toolbox: found 0 containers No toolbox containers found. Create now? [y/N] y toolbox: Fedora generational core is f30 toolbox: base image is fedora-toolbox:30 toolbox: container is fedora-toolbox-30 toolbox: checking value /run/.heim_org.h5l.kcm-socket (Stream) of property Listen in sssd-kcm.socket toolbox: parsing value /run/.heim_org.h5l.kcm-socket (Stream) of property Listen in sssd-kcm.socket toolbox: checking if 'podman create' supports --dns=none and --no-hosts toolbox: 'podman create' supports --dns=none and --no-hosts toolbox: checking if 'podman create' supports --ulimit host toolbox: 'podman create' supports --ulimit host toolbox: looking for image localhost/fedora-toolbox:30 toolbox: looking for image registry.fedoraproject.org/f30/fedora-toolbox:30 toolbox: base image fedora-toolbox:30 resolved to registry.fedoraproject.org/f30/fedora-toolbox:30 toolbox: checking if container fedora-toolbox-30 already exists toolbox: /var/home/parag canonicalized to /var/home/parag toolbox: checking if /home is a symbolic link to /var/home toolbox: /home is a symbolic link to /var/home toolbox: calling org.freedesktop.Flatpak.SessionHelper.RequestSession toolbox: creating container fedora-toolbox-30 toolbox: calling org.freedesktop.Flatpak.SessionHelper.RequestSession toolbox: starting container fedora-toolbox-30 toolbox: /etc/profile.d/toolbox.sh already mounted in container fedora-toolbox-30 Error: unable to start container "fedora-toolbox-30": remount '/var/home/parag/.local/share/containers/storage/overlay/97487dd314498096217b546a5c3e8e0f85c1dbf4c8296bb0109203d813de16b2/merged/run/host/usr': Operation not permitted: OCI runtime permission denied error toolbox: failed to start container fedora-toolbox-30 Can anyone give some hint how fix this issue to get toolbox working? I believe the podman rpm I am using already have upstream commit afd0818326aa37f03a3bc74f0269a06a403db16d
Note another tester tried podman on F31 and got no joy, though the errors don't look the same: https://bugzilla.redhat.com/show_bug.cgi?id=1756059 just another reference point that it seems pretty broken right now :/
(In reply to Giuseppe Scrivano from comment #19) > that should be already fixed upstream by > afd0818326aa37f03a3bc74f0269a06a403db16d > > as a workaround, you need to ensure the unprivileged user owns the current > cgroup: "systemd-run --scope --user podman .." or more easily let it wrap > your entire session with "systemd-run --scope --user $SHELL" Thanks! That error is gone, although the toolbox still won't start ``` $ systemd-run --scope --user toolbox -v enter -c fedora-toolbox-31 Running scope as unit: run-rc9ac55d551f747e29256150b8e032b3e.scope toolbox: resolved absolute path for /usr/bin/toolbox to /usr/bin/toolbox toolbox: checking if /etc/subgid and /etc/subuid have entries for user vrutkovs toolbox: TOOLBOX_PATH is /usr/bin/toolbox toolbox: checking if 'podman system migrate' exists toolbox: migration not needed: 1.5.1 is old toolbox: Fedora generational core is f31 toolbox: base image is fedora-toolbox:31 toolbox: container is fedora-toolbox-31 toolbox: checking if container fedora-toolbox-31 exists toolbox: calling org.freedesktop.Flatpak.SessionHelper.RequestSession toolbox: starting container fedora-toolbox-31 toolbox: /etc/profile.d/toolbox.sh already mounted in container fedora-toolbox-31 Error: unable to start container "fedora-toolbox-31": error reading container (probably exited) json message: EOF toolbox: failed to start container fedora-toolbox-31 ``` podman-1.5.1-3.git0005792.fc31.x86_64 conmon-2.0.0-2.fc31.x86_64 crun-0.9.1-1.fc31.x86_64
Is 1.6 going to ship in F31 GA? If so it would be better to push the RC to testing IMO.
Waiting on a few more fixes - `podman build` is still broken on CGroups v2, for example.
https://bodhi.fedoraproject.org/updates/FEDORA-2019-b34fd3c00a
With crun-0.10-1.fc31.x86_64 podman-1.6.1-2.fc31.x86_64 now hitting the same error as comment #21 IIUC https://github.com/containers/crun/pull/120 would fix that on Silverblue
https://bodhi.fedoraproject.org/updates/FEDORA-2019-845efd1d30 (crun-0.10.1)
I updated all latest packages available on koji for F31 Silverblue system. But then I still get [parag@f31sb ~]$ toolbox -v enter toolbox: running as real user ID 1000 toolbox: resolved absolute path for /usr/bin/toolbox to /usr/bin/toolbox toolbox: checking if /etc/subgid and /etc/subuid have entries for user parag toolbox: TOOLBOX_PATH is /usr/bin/toolbox toolbox: migration not needed: 1.6.1 is unchanged toolbox: Fedora generational core is f31 toolbox: base image is fedora-toolbox:31 toolbox: container is fedora-toolbox-31 toolbox: checking if container fedora-toolbox-31 exists toolbox: calling org.freedesktop.Flatpak.SessionHelper.RequestSession toolbox: starting container fedora-toolbox-31 toolbox: /etc/profile.d/toolbox.sh already mounted in container fedora-toolbox-31 Error: unable to start container "fedora-toolbox-31": setrlimit (RLIM_MEMLOCK): Operation not permitted: OCI runtime permission denied error toolbox: failed to start container fedora-toolbox-31 [parag@f31sb ~]$ Fresh toolbox is working fine but I need old f31 toolbox which was working fine on Fedora 30 system. Also, for f30 toolbox, I get this Error: container 201ae7031d6910ae92281a6004ce076a971b8b017ed4d86e227b74be703dd484 was created with OCI runtime runc, but that runtime is not available in the current configuration: internal libpod error
Error: unable to start container "fedora-toolbox-31": setrlimit (RLIM_MEMLOCK): Operation not permitted: OCI runtime permission denied error It looks like we are trying to use the devices cgroup, that is still a privileged operation on cgroups v2. I think you need a newer Podman that knows to not use the devices cgroup for rootless on cgroups v2.
On IRC, I got hint from mheon to try changing runc line with crun binary paths in libpod.conf. For F31 toolbox the error remained same (RLIM_MEMLOCK) but for F30 toolbox the error changed to Error: unable to start container "fedora-toolbox-30": cannot chown run directory /run/user/1000/overlay-containers/201ae7031d6910ae92281a6004ce076a971b8b017ed4d86e227b74be703dd484/userdata: chown /run/user/1000/overlay-containers/201ae7031d6910ae92281a6004ce076a971b8b017ed4d86e227b74be703dd484/userdata: no such file or directory Anyways, I can create my initially created toolboxes on Fedora 30 Silverblue. I have already using rebased system to Fedora 31 Silverblue. [parag@f31sb ~]$ rpm -qa podman crun runc toolbox crun-0.10.1-1.fc31.x86_64 toolbox-0.0.15-1.fc31.noarch podman-1.6.1-2.fc31.x86_64 I see runc got removed from F31 SB yesterday.
So, is there still a problem here with all the updates that got pushed stable last week or so, or can we close this?
I've happily used toolbox (and thus user podman containers) on current F31 for about a week without problems now, thanks! I didn't test a proper migration though, I just wiped my ~/.config/containers/ and ~/.local/share/containers/ after the upgrade. That's probably not something that most users ought to be required to do, but that would be a separate bz anyway.
Rootless fedora / toolbox containers worked great here after recent updates. Closing this, lets open more specific reports in the future