Bug 1995439 - /usr/bin/toolbox linked against glibc-2.34 doesn't run on older glibc
Summary: /usr/bin/toolbox linked against glibc-2.34 doesn't run on older glibc
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: toolbox
Version: 35
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Debarshi Ray
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: https://ask.fedoraproject.org/t/we-ar...
: 1942576 (view as bug list)
Depends On:
Blocks: F35FinalFreezeException
TreeView+ depends on / blocked
 
Reported: 2021-08-19 06:45 UTC by Jens Petersen
Modified: 2021-12-03 01:02 UTC (History)
20 users (show)

Fixed In Version: toolbox-0.0.99.2^3.git075b9a8d2779-4.fc35
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-10-28 18:28:14 UTC
Type: Bug


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github containers toolbox issues 821 0 None closed /usr/bin/toolbox linked against glibc-2.34 doesn't run on older glibc 2021-10-27 12:10:26 UTC

Description Jens Petersen 2021-08-19 06:45:23 UTC
Description of problem:
When I try to run the current fedora-toolbox:34 or fedora-toolbox:33
image in Fedora 35 or C9S, after doing `toolbox create` I get an error
for `toolbox enter`.

Version-Release number of selected component (if applicable):
toolbox-0.0.99.2^3.git075b9a8d2779-1.fc35
toolbox-0.0.99.3-2.module_el9+96+b062886b

How reproducible:
100%

Steps to Reproduce:
1. In Fedora 35 or Centos 9 Stream
2. toolbox create --release 34
3. toolbox enter

Actual results:
Error: invalid entry point PID of container fedora-toolbox-34

Expected results:
No error

Additional info:
fedora-toolbox-35 runs okay in F35 and C9S toolbox.

Is there any way to get toolbox to give a more informative error message?
Even with `toolbox -vv enter` I could not see any clear failure cause in the log.

Comment 1 Oliver Gutiérrez 2021-08-19 09:16:23 UTC
Hi Jens.

This is probably caused by the same problem as here: https://github.com/containers/toolbox/issues/821

I worked in a patch for that here: https://github.com/containers/toolbox/pull/829

It is still pending review.

Anyway I will check the exact error you're reporting here to be sure this is the real reason.

Comment 2 Oliver Gutiérrez 2021-08-19 09:28:56 UTC
This is the output i got:

[user@fedora ~]$ toolbox create --release 34
Image required to create toolbox container.
Download registry.fedoraproject.org/fedora-toolbox:34 (500MB)? [y/N]: y
Creating container fedora-toolbox-34: | Created container: fedora-toolbox-34
Enter with: toolbox enter fedora-toolbox-34
[user@fedora ~]$  toolbox enter fedora-toolbox-34
Error: invalid entry point PID of container fedora-toolbox-34
[user@fedora ~]$  toolbox enter fedora-toolbox-34 -v
DEBU Running as real user ID 1000                 
DEBU Resolved absolute path to the executable as /usr/bin/toolbox 
DEBU Running on a cgroups v2 host                 
DEBU Checking if /etc/subgid and /etc/subuid have entries for user user 
DEBU Validating sub-ID file /etc/subuid           
DEBU Validating sub-ID file /etc/subgid           
DEBU TOOLBOX_PATH is /usr/bin/toolbox             
DEBU Migrating to newer Podman                    
DEBU Toolbox config directory is /var/home/user/.config/toolbox 
DEBU Current Podman version is 3.3.0-rc1          
DEBU Creating runtime directory /run/user/1000/toolbox 
DEBU Old Podman version is 3.3.0-rc1              
DEBU Migration not needed: Podman version 3.3.0-rc1 is unchanged 
DEBU Setting up configuration                     
DEBU Setting up configuration: file /var/home/user/.config/containers/toolbox.conf not found 
DEBU Resolving image name                         
DEBU Distribution (CLI): ''                       
DEBU Image (CLI): ''                              
DEBU Release (CLI): ''                            
DEBU Resolved image name                          
DEBU Image: 'fedora-toolbox:36'                   
DEBU Release: '36'                                
DEBU Resolving container name                     
DEBU Container: ''                                
DEBU Image: 'fedora-toolbox:36'                   
DEBU Release: '36'                                
DEBU Resolved container name                      
DEBU Container: 'fedora-toolbox-36'               
DEBU Resolving image name                         
DEBU Distribution (CLI): ''                       
DEBU Image (CLI): ''                              
DEBU Release (CLI): ''                            
DEBU Resolved image name                          
DEBU Image: 'fedora-toolbox:36'                   
DEBU Release: '36'                                
DEBU Resolving container name                     
DEBU Container: 'fedora-toolbox-34'               
DEBU Image: 'fedora-toolbox:36'                   
DEBU Release: '36'                                
DEBU Resolved container name                      
DEBU Container: 'fedora-toolbox-34'               
DEBU Checking if container fedora-toolbox-34 exists 
DEBU Inspecting mounts of container fedora-toolbox-34 
DEBU Starting container fedora-toolbox-34         
DEBU Inspecting entry point of container fedora-toolbox-34 
DEBU Entry point PID is a float64                 
DEBU Entry point of container fedora-toolbox-34 is toolbox (PID=0) 
Error: invalid entry point PID of container fedora-toolbox-34

The way to see the actual error is:

[user@fedora ~]$ podman start --attach fedora-toolbox-34
toolbox: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by toolbox)


And yes, seems the problem is because of the issue #821

Comment 3 Fedora Blocker Bugs Application 2021-09-11 11:23:53 UTC
Proposed as a Blocker for 35-final by Fedora user cheeselee using the blocker tracking app because:

 Old Toolbox containers are not runnable after updated to f35. That breaks lives of users of Atomic Host who put their real work in Toolbox containers.

Comment 4 Timothée Ravier 2021-09-13 07:45:38 UTC
Another option to fully fix this one is to do https://github.com/containers/toolbox/issues/832

Comment 5 Kamil Páral 2021-09-13 10:36:30 UTC
Is this bug only related to `toolbox` or does it somehow affect general `podman` usage as well?

Comment 6 Oliver Gutiérrez 2021-09-13 10:44:19 UTC
It is only related to Toolbox

Comment 7 Lukas Ruzicka 2021-09-13 17:55:27 UTC
#agreed 1995439 -  The decision to classify this bug as an "AcceptedFreezeException (Beta)" was made as it is a noticeable issue that could impact user experience. We will punt on the other classifications for now.

Comment 8 Geoffrey Marr 2021-09-20 17:46:51 UTC
Discussed during the 2021-09-20 blocker review meeting: [0]

The decision to classify this bug as a "RejectedBlocker (Final)" was made as this does not violate any criteria. It's significant primarily for user-facing ostree builds, and none of those are prelease-blocking yet.

[0] https://meetbot.fedoraproject.org/fedora-blocker-review/2021-09-20/f35-blocker-review.2021-09-20-16.00.txt

Comment 9 kxra 2021-10-01 14:15:57 UTC
I've been told this is the cause of my issue: https://github.com/fedora-silverblue/issue-tracker/issues/213

But from other comments it seem this should not affect podman, so I'm not sure

Comment 10 Oliver Gutiérrez 2021-10-01 15:34:41 UTC
Yes. It is a duplicate. The error you're getting in podman is unrelated. You have some keys that are not being read correctly by podman, but the error of not being able to enter the toolbox is only toolbox related.

Comment 11 Biji 2021-10-11 06:51:55 UTC
any temporary workaround for this issue please?

Comment 12 Jens Petersen 2021-10-11 08:52:19 UTC
The simplest workaround I know currently is to install f34 toolbox in f35.

Comment 13 Biji 2021-10-11 12:03:53 UTC
(In reply to Jens Petersen from comment #12)
> The simplest workaround I know currently is to install f34 toolbox in f35.

thanks it's working again by installing:

dnf install toolbox-0.0.99.1-1.fc34.x86_64.rpm

Comment 14 Ray Strode [halfline] 2021-10-20 18:56:11 UTC
Created attachment 1835307 [details]
workaround

I did a scratch build here https://koji.fedoraproject.org/koji/taskinfo?taskID=77577111 that moves the toolbox command to libexecdir and replaces it with a shell script wrapper that forces it to use the libc and loader from /run/host if /run/host is mounted on the system.

Not sure if that's the route rishi wants to go or not, but it at least gets things going for me.

Comment 15 Ray Strode [halfline] 2021-10-20 19:11:36 UTC
*** Bug 1942576 has been marked as a duplicate of this bug. ***

Comment 17 Fedora Update System 2021-10-22 01:00:21 UTC
FEDORA-2021-ec38b7db52 has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2021-ec38b7db52

Comment 18 Adam Williamson 2021-10-22 16:53:43 UTC
Proposing this for a Final FE, then. Definitely seems worth fixing. Thanks for the fix.

Comment 19 Fedora Update System 2021-10-23 02:23:32 UTC
FEDORA-2021-ec38b7db52 has been pushed to the Fedora 35 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-ec38b7db52`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-ec38b7db52

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 20 Robin Lee 2021-10-23 20:23:20 UTC
(In reply to Fedora Update System from comment #19)
> FEDORA-2021-ec38b7db52 has been pushed to the Fedora 35 testing repository.
> Soon you'll be able to install the update with the following command:
> `sudo dnf upgrade --enablerepo=updates-testing
> --advisory=FEDORA-2021-ec38b7db52`
> You can provide feedback for this update here:
> https://bodhi.fedoraproject.org/updates/FEDORA-2021-ec38b7db52
> 
> See also https://fedoraproject.org/wiki/QA:Updates_Testing for more
> information on how to test updates.

This change breaks all the old Toolbox containers, even F-35 containers. I don't think it is a good solution. I still prefer toolbox-0.0.99.2-7.fc34.x86_64.

Comment 21 Debarshi Ray 2021-10-25 11:04:21 UTC
Thanks for the feedback, Robin!

Does this make things better for you:
https://github.com/containers/toolbox/pull/904 ?

Comment 22 Robin Lee 2021-10-25 11:24:49 UTC
(In reply to Debarshi Ray from comment #21)
> Thanks for the feedback, Robin!
> 
> Does this make things better for you:
> https://github.com/containers/toolbox/pull/904 ?

Looks more elaborate. I am willing to test a new build.

Comment 23 Debarshi Ray 2021-10-25 11:42:32 UTC
(In reply to Robin Lee from comment #22)
> (In reply to Debarshi Ray from comment #21)
> > Thanks for the feedback, Robin!
> > 
> > Does this make things better for you:
> > https://github.com/containers/toolbox/pull/904 ?
> 
> Looks more elaborate. I am willing to test a new build.

Here's a scratch build:
https://koji.fedoraproject.org/koji/taskinfo?taskID=77792541

Given how the Fedora process work, I think we should create a new build for this specific backwards compatibility issue. That way we can use the new bug to get this into Fedora 35 as a Final FE if needed.

Comment 24 Debarshi Ray 2021-10-25 11:44:05 UTC
(In reply to Debarshi Ray from comment #23)
> (In reply to Robin Lee from comment #22)
> > (In reply to Debarshi Ray from comment #21)
> > > Thanks for the feedback, Robin!
> > > 
> > > Does this make things better for you:
> > > https://github.com/containers/toolbox/pull/904 ?
> > 
> > Looks more elaborate. I am willing to test a new build.
> 
> Here's a scratch build:
> https://koji.fedoraproject.org/koji/taskinfo?taskID=77792541
> 
> Given how the Fedora process work, I think we should create a new build for
> this specific backwards compatibility issue. That way we can use the new bug
> to get this into Fedora 35 as a Final FE if needed.

I meant, a "new bug", not a "new build". Sorry.

Comment 25 Adam Williamson 2021-10-25 16:09:37 UTC
We don't necessarily need a new bug. I was not going to pull in the existing update since the issue Robin reported seemed bad; I would pull in an update that both fixes this bug and doesn't trigger a new problem for Robin (or any other new problem).

Comment 26 Debarshi Ray 2021-10-25 16:18:48 UTC
(In reply to Adam Williamson from comment #25)
> We don't necessarily need a new bug. I was not going to pull in the existing
> update since the issue Robin reported seemed bad; I would pull in an update
> that both fixes this bug and doesn't trigger a new problem for Robin (or any
> other new problem).

Ok, great. :)

I misread "This update has been submitted for stable by bodhi." as "This update has been pushed to stable". Sorry for the confusion.

Comment 27 Adam Williamson 2021-10-25 19:41:47 UTC
+4 for FE in https://pagure.io/fedora-qa/blocker-review/issue/447 , marking accepted FE.

Comment 28 Adam Williamson 2021-10-25 20:16:22 UTC
Note I'm planning to do an RC later today (or at latest tomorrow morning, but ideally later today), so if we want this pulled into it, please edit the official update to include the fix for Robin's issue...thanks!

Comment 29 Debarshi Ray 2021-10-25 22:01:33 UTC
Given the really tight timeline on this, I am going to go ahead and do an official build now. Folks can give feedback on the Bodhi ticket.

Thanks for doing all the legwork, Adam!

Comment 30 Fedora Update System 2021-10-25 23:25:40 UTC
FEDORA-2021-c86060570a has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2021-c86060570a

Comment 31 sumantro 2021-10-26 09:10:39 UTC
Downloaded koji build and tested. Works as expected

Comment 32 Jens Petersen 2021-10-26 10:57:47 UTC
+1 This LGTM too and now allows older toolbox containers to be entered.

Comment 33 Fedora Update System 2021-10-28 18:28:14 UTC
FEDORA-2021-c86060570a has been pushed to the Fedora 35 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.