Description of problem: When I try to run the current fedora-toolbox:34 or fedora-toolbox:33 image in Fedora 35 or C9S, after doing `toolbox create` I get an error for `toolbox enter`. Version-Release number of selected component (if applicable): toolbox-0.0.99.2^3.git075b9a8d2779-1.fc35 toolbox-0.0.99.3-2.module_el9+96+b062886b How reproducible: 100% Steps to Reproduce: 1. In Fedora 35 or Centos 9 Stream 2. toolbox create --release 34 3. toolbox enter Actual results: Error: invalid entry point PID of container fedora-toolbox-34 Expected results: No error Additional info: fedora-toolbox-35 runs okay in F35 and C9S toolbox. Is there any way to get toolbox to give a more informative error message? Even with `toolbox -vv enter` I could not see any clear failure cause in the log.
Hi Jens. This is probably caused by the same problem as here: https://github.com/containers/toolbox/issues/821 I worked in a patch for that here: https://github.com/containers/toolbox/pull/829 It is still pending review. Anyway I will check the exact error you're reporting here to be sure this is the real reason.
This is the output i got: [user@fedora ~]$ toolbox create --release 34 Image required to create toolbox container. Download registry.fedoraproject.org/fedora-toolbox:34 (500MB)? [y/N]: y Creating container fedora-toolbox-34: | Created container: fedora-toolbox-34 Enter with: toolbox enter fedora-toolbox-34 [user@fedora ~]$ toolbox enter fedora-toolbox-34 Error: invalid entry point PID of container fedora-toolbox-34 [user@fedora ~]$ toolbox enter fedora-toolbox-34 -v DEBU Running as real user ID 1000 DEBU Resolved absolute path to the executable as /usr/bin/toolbox DEBU Running on a cgroups v2 host DEBU Checking if /etc/subgid and /etc/subuid have entries for user user DEBU Validating sub-ID file /etc/subuid DEBU Validating sub-ID file /etc/subgid DEBU TOOLBOX_PATH is /usr/bin/toolbox DEBU Migrating to newer Podman DEBU Toolbox config directory is /var/home/user/.config/toolbox DEBU Current Podman version is 3.3.0-rc1 DEBU Creating runtime directory /run/user/1000/toolbox DEBU Old Podman version is 3.3.0-rc1 DEBU Migration not needed: Podman version 3.3.0-rc1 is unchanged DEBU Setting up configuration DEBU Setting up configuration: file /var/home/user/.config/containers/toolbox.conf not found DEBU Resolving image name DEBU Distribution (CLI): '' DEBU Image (CLI): '' DEBU Release (CLI): '' DEBU Resolved image name DEBU Image: 'fedora-toolbox:36' DEBU Release: '36' DEBU Resolving container name DEBU Container: '' DEBU Image: 'fedora-toolbox:36' DEBU Release: '36' DEBU Resolved container name DEBU Container: 'fedora-toolbox-36' DEBU Resolving image name DEBU Distribution (CLI): '' DEBU Image (CLI): '' DEBU Release (CLI): '' DEBU Resolved image name DEBU Image: 'fedora-toolbox:36' DEBU Release: '36' DEBU Resolving container name DEBU Container: 'fedora-toolbox-34' DEBU Image: 'fedora-toolbox:36' DEBU Release: '36' DEBU Resolved container name DEBU Container: 'fedora-toolbox-34' DEBU Checking if container fedora-toolbox-34 exists DEBU Inspecting mounts of container fedora-toolbox-34 DEBU Starting container fedora-toolbox-34 DEBU Inspecting entry point of container fedora-toolbox-34 DEBU Entry point PID is a float64 DEBU Entry point of container fedora-toolbox-34 is toolbox (PID=0) Error: invalid entry point PID of container fedora-toolbox-34 The way to see the actual error is: [user@fedora ~]$ podman start --attach fedora-toolbox-34 toolbox: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by toolbox) And yes, seems the problem is because of the issue #821
Proposed as a Blocker for 35-final by Fedora user cheeselee using the blocker tracking app because: Old Toolbox containers are not runnable after updated to f35. That breaks lives of users of Atomic Host who put their real work in Toolbox containers.
Another option to fully fix this one is to do https://github.com/containers/toolbox/issues/832
Is this bug only related to `toolbox` or does it somehow affect general `podman` usage as well?
It is only related to Toolbox
#agreed 1995439 - The decision to classify this bug as an "AcceptedFreezeException (Beta)" was made as it is a noticeable issue that could impact user experience. We will punt on the other classifications for now.
Discussed during the 2021-09-20 blocker review meeting: [0] The decision to classify this bug as a "RejectedBlocker (Final)" was made as this does not violate any criteria. It's significant primarily for user-facing ostree builds, and none of those are prelease-blocking yet. [0] https://meetbot.fedoraproject.org/fedora-blocker-review/2021-09-20/f35-blocker-review.2021-09-20-16.00.txt
I've been told this is the cause of my issue: https://github.com/fedora-silverblue/issue-tracker/issues/213 But from other comments it seem this should not affect podman, so I'm not sure
Yes. It is a duplicate. The error you're getting in podman is unrelated. You have some keys that are not being read correctly by podman, but the error of not being able to enter the toolbox is only toolbox related.
any temporary workaround for this issue please?
The simplest workaround I know currently is to install f34 toolbox in f35.
(In reply to Jens Petersen from comment #12) > The simplest workaround I know currently is to install f34 toolbox in f35. thanks it's working again by installing: dnf install toolbox-0.0.99.1-1.fc34.x86_64.rpm
Created attachment 1835307 [details] workaround I did a scratch build here https://koji.fedoraproject.org/koji/taskinfo?taskID=77577111 that moves the toolbox command to libexecdir and replaces it with a shell script wrapper that forces it to use the libc and loader from /run/host if /run/host is mounted on the system. Not sure if that's the route rishi wants to go or not, but it at least gets things going for me.
*** Bug 1942576 has been marked as a duplicate of this bug. ***
FEDORA-2021-ec38b7db52 has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2021-ec38b7db52
Proposing this for a Final FE, then. Definitely seems worth fixing. Thanks for the fix.
FEDORA-2021-ec38b7db52 has been pushed to the Fedora 35 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-ec38b7db52` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-ec38b7db52 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
(In reply to Fedora Update System from comment #19) > FEDORA-2021-ec38b7db52 has been pushed to the Fedora 35 testing repository. > Soon you'll be able to install the update with the following command: > `sudo dnf upgrade --enablerepo=updates-testing > --advisory=FEDORA-2021-ec38b7db52` > You can provide feedback for this update here: > https://bodhi.fedoraproject.org/updates/FEDORA-2021-ec38b7db52 > > See also https://fedoraproject.org/wiki/QA:Updates_Testing for more > information on how to test updates. This change breaks all the old Toolbox containers, even F-35 containers. I don't think it is a good solution. I still prefer toolbox-0.0.99.2-7.fc34.x86_64.
Thanks for the feedback, Robin! Does this make things better for you: https://github.com/containers/toolbox/pull/904 ?
(In reply to Debarshi Ray from comment #21) > Thanks for the feedback, Robin! > > Does this make things better for you: > https://github.com/containers/toolbox/pull/904 ? Looks more elaborate. I am willing to test a new build.
(In reply to Robin Lee from comment #22) > (In reply to Debarshi Ray from comment #21) > > Thanks for the feedback, Robin! > > > > Does this make things better for you: > > https://github.com/containers/toolbox/pull/904 ? > > Looks more elaborate. I am willing to test a new build. Here's a scratch build: https://koji.fedoraproject.org/koji/taskinfo?taskID=77792541 Given how the Fedora process work, I think we should create a new build for this specific backwards compatibility issue. That way we can use the new bug to get this into Fedora 35 as a Final FE if needed.
(In reply to Debarshi Ray from comment #23) > (In reply to Robin Lee from comment #22) > > (In reply to Debarshi Ray from comment #21) > > > Thanks for the feedback, Robin! > > > > > > Does this make things better for you: > > > https://github.com/containers/toolbox/pull/904 ? > > > > Looks more elaborate. I am willing to test a new build. > > Here's a scratch build: > https://koji.fedoraproject.org/koji/taskinfo?taskID=77792541 > > Given how the Fedora process work, I think we should create a new build for > this specific backwards compatibility issue. That way we can use the new bug > to get this into Fedora 35 as a Final FE if needed. I meant, a "new bug", not a "new build". Sorry.
We don't necessarily need a new bug. I was not going to pull in the existing update since the issue Robin reported seemed bad; I would pull in an update that both fixes this bug and doesn't trigger a new problem for Robin (or any other new problem).
(In reply to Adam Williamson from comment #25) > We don't necessarily need a new bug. I was not going to pull in the existing > update since the issue Robin reported seemed bad; I would pull in an update > that both fixes this bug and doesn't trigger a new problem for Robin (or any > other new problem). Ok, great. :) I misread "This update has been submitted for stable by bodhi." as "This update has been pushed to stable". Sorry for the confusion.
+4 for FE in https://pagure.io/fedora-qa/blocker-review/issue/447 , marking accepted FE.
Note I'm planning to do an RC later today (or at latest tomorrow morning, but ideally later today), so if we want this pulled into it, please edit the official update to include the fix for Robin's issue...thanks!
Given the really tight timeline on this, I am going to go ahead and do an official build now. Folks can give feedback on the Bodhi ticket. Thanks for doing all the legwork, Adam!
FEDORA-2021-c86060570a has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2021-c86060570a
Downloaded koji build and tested. Works as expected
+1 This LGTM too and now allows older toolbox containers to be entered.
FEDORA-2021-c86060570a has been pushed to the Fedora 35 stable repository. If problem still persists, please make note of it in this bug report.