Bug 2000807

Summary: /usr/bin/toolbox linked against glibc-2.34 doesn't run on older glibc
Product: Red Hat Enterprise Linux 9 Reporter: Alex Jia <ajia>
Component: toolboxAssignee: Debarshi Ray <debarshir>
Status: CLOSED ERRATA QA Contact: Martin Krajnak <mkrajnak>
Severity: high Docs Contact:
Priority: high    
Version: 9.0CC: ddarrah, petersen, tpopela
Target Milestone: rcKeywords: Triaged
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: toolbox-0.0.99.3-2.el9_0 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-05-17 12:31:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2016901, 2031652    

Description Alex Jia 2021-09-03 03:31:42 UTC
Description of problem:
Can't run toolbox command successfully and got error like this "Error: invalid entry point PID of container support-tools-latest".

Version-Release number of selected component (if applicable):

[root@kvm-08-guest02 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux release 9.0 Beta (Plow)

[root@kvm-08-guest02 ~]# rpm -q toolbox podman crun systemd kernel
toolbox-0.0.99.3-3.module+el9.0.0+12467+378c8264.x86_64
podman-3.3.1-6.module+el9.0.0+12467+378c8264.x86_64
crun-1.0-1.module+el9.0.0+12467+378c8264.x86_64
systemd-249-4.el9.x86_64
kernel-5.14.0-0.rc7.54.el9.x86_64

How reproducible:
always

Steps to Reproduce:
1. podman login registry.redhat.io
2. toolbox


Actual results:

[root@kvm-08-guest02 ~]# toolbox
No toolbox containers found. Create now? [y/N] y
Image required to create toolbox container.
Download registry.redhat.io/rhel8/support-tools:latest (500MB)? [y/N]: y
Error: invalid entry point PID of container support-tools-latest

[root@kvm-08-guest02 ~]# podman ps -a
CONTAINER ID  IMAGE                                          COMMAND               CREATED             STATUS                         PORTS       NAMES
d67764882c91  registry.redhat.io/rhel8/support-tools:latest  toolbox --log-lev...  About a minute ago  Exited (1) About a minute ago              support-tools-latest


Expected results:
fix it.

Additional info:

[root@kvm-08-guest02 ~]# toolbox
Error: invalid entry point PID of container support-tools-latest
[root@kvm-08-guest02 ~]# toolbox -v
DEBU Running as real user ID 0
DEBU Resolved absolute path to the executable as /usr/bin/toolbox
DEBU Running on a cgroups v2 host
DEBU TOOLBOX_PATH is /usr/bin/toolbox
DEBU Migrating to newer Podman
DEBU Toolbox config directory is /root/.config/toolbox
DEBU Current Podman version is 3.3.1
DEBU Creating runtime directory /run/toolbox
DEBU Old Podman version is 3.3.1
DEBU Migration not needed: Podman version 3.3.1 is unchanged
DEBU Setting up configuration
DEBU Setting up configuration: file /root/.config/containers/toolbox.conf not found
DEBU Resolving image name
DEBU Distribution (CLI): ''
DEBU Image (CLI): ''
DEBU Release (CLI): ''
DEBU Resolved image name
DEBU Image: 'registry.redhat.io/rhel8/support-tools:latest'
DEBU Release: 'latest'
DEBU Resolving container name
DEBU Container: ''
DEBU Image: 'registry.redhat.io/rhel8/support-tools:latest'
DEBU Release: 'latest'
DEBU Resolved container name
DEBU Container: 'support-tools-latest'
DEBU Resolving image name
DEBU Distribution (CLI): ''
DEBU Image (CLI): ''
DEBU Release (CLI): ''
DEBU Resolved image name
DEBU Image: 'registry.redhat.io/rhel8/support-tools:latest'
DEBU Release: 'latest'
DEBU Resolving container name
DEBU Container: ''
DEBU Image: 'registry.redhat.io/rhel8/support-tools:latest'
DEBU Release: 'latest'
DEBU Resolved container name
DEBU Container: 'support-tools-latest'
DEBU Checking if container support-tools-latest exists
DEBU Inspecting mounts of container support-tools-latest
DEBU Starting container support-tools-latest
DEBU Inspecting entry point of container support-tools-latest
DEBU Entry point PID is a float64
DEBU Entry point of container support-tools-latest is toolbox (PID=0)
Error: invalid entry point PID of container support-tools-latest
[root@kvm-08-guest02 ~]# toolbox enter support-tools-latest
Error: invalid entry point PID of container support-tools-latest
[root@kvm-08-guest02 ~]# toolbox -v enter support-tools-latest
DEBU Running as real user ID 0
DEBU Resolved absolute path to the executable as /usr/bin/toolbox
DEBU Running on a cgroups v2 host
DEBU TOOLBOX_PATH is /usr/bin/toolbox
DEBU Migrating to newer Podman
DEBU Toolbox config directory is /root/.config/toolbox
DEBU Current Podman version is 3.3.1
DEBU Creating runtime directory /run/toolbox
DEBU Old Podman version is 3.3.1
DEBU Migration not needed: Podman version 3.3.1 is unchanged
DEBU Setting up configuration
DEBU Setting up configuration: file /root/.config/containers/toolbox.conf not found
DEBU Resolving image name
DEBU Distribution (CLI): ''
DEBU Image (CLI): ''
DEBU Release (CLI): ''
DEBU Resolved image name
DEBU Image: 'registry.redhat.io/rhel8/support-tools:latest'
DEBU Release: 'latest'
DEBU Resolving container name
DEBU Container: ''
DEBU Image: 'registry.redhat.io/rhel8/support-tools:latest'
DEBU Release: 'latest'
DEBU Resolved container name
DEBU Container: 'support-tools-latest'
DEBU Resolving image name
DEBU Distribution (CLI): ''
DEBU Image (CLI): ''
DEBU Release (CLI): ''
DEBU Resolved image name
DEBU Image: 'registry.redhat.io/rhel8/support-tools:latest'
DEBU Release: 'latest'
DEBU Resolving container name
DEBU Container: 'support-tools-latest'
DEBU Image: 'registry.redhat.io/rhel8/support-tools:latest'
DEBU Release: 'latest'
DEBU Resolved container name
DEBU Container: 'support-tools-latest'
DEBU Checking if container support-tools-latest exists
DEBU Inspecting mounts of container support-tools-latest
DEBU Starting container support-tools-latest
DEBU Inspecting entry point of container support-tools-latest
DEBU Entry point PID is a float64
DEBU Entry point of container support-tools-latest is toolbox (PID=0)
Error: invalid entry point PID of container support-tools-latest

Comment 1 Debarshi Ray 2021-10-01 13:49:30 UTC
I believe this is:
https://github.com/containers/toolbox/issues/821

Could you please paste the logs from:
$ podman start --attach support-tools-latest

... so that we can be sure?

If my guess above is right, then this issue affects running RHEL 8 containers on a RHEL 9 host, because RHEL 9 hosts have glibc-2.34.

In bug 2004563 we changed the default image from registry.redhat.io/rhel8/support-tools:latest to registry.access.redhat.com/ubi9/toolbox:latest ie., in other words, from a RHEL 8 image to a RHEL 9 one. That should reduce the impact of this bug, because only users choosing to use a custom image will encounter it.

We are currently trying to fix this bug for Fedora 35 too. Would it be OK if we fixed this for RHEL 9.0 GA, and not Beta?

Comment 2 Alex Jia 2021-10-02 02:35:46 UTC
(In reply to Debarshi Ray from comment #1)
> I believe this is:
> https://github.com/containers/toolbox/issues/821
> 
> Could you please paste the logs from:
> $ podman start --attach support-tools-latest
> 
> ... so that we can be sure?
> 
> If my guess above is right, then this issue affects running RHEL 8
> containers on a RHEL 9 host, because RHEL 9 hosts have glibc-2.34.

1. try to start support-tools-latest container was created yesterday

[root@kvm-08-guest26 ~]# podman ps -a
CONTAINER ID  IMAGE                                                                        COMMAND               CREATED       STATUS                   PORTS       NAMES
9fc71475c6e6  registry-proxy.engineering.redhat.com/rh-osbs/rhel9-support-tools:9.0.0beta  /usr/bin/bash         3 days ago    Exited (0) 3 days ago                rhel9-support-tools
4e8353196e3a  registry.redhat.io/rhel8/support-tools:latest                                toolbox --log-lev...  22 hours ago  Exited (1) 22 hours ago              support-tools-latest

[root@kvm-08-guest26 ~]# podman start --attach support-tools-latest
Error: unable to start container 4e8353196e3ac745081f95ee97c88dbfa3b6209d42743888e4a4c946961a4ccd: error stat'ing file `/run/toolbox`: No such file or directory: OCI runtime attempted to invoke a command that was not found

NOTE: this error didn't hit when cleaning all of container then try again, please see the following tests.

2. clean all of existing container then try again

[root@kvm-08-guest26 ~]# podman rm -f `podman ps -qa`
9fc71475c6e6b9616fee3e15dd1ed7d00edb209d11ba5bc0cfdecf62fdc270e9
4e8353196e3ac745081f95ee97c88dbfa3b6209d42743888e4a4c946961a4ccd

[root@kvm-08-guest26 ~]# podman ps -a
CONTAINER ID  IMAGE       COMMAND     CREATED     STATUS      PORTS       NAMES

[root@kvm-08-guest26 ~]# toolbox
No toolbox containers found. Create now? [y/N] y
Error: invalid entry point PID of container support-tools-latest

[root@kvm-08-guest26 ~]# podman ps -a
CONTAINER ID  IMAGE                                          COMMAND               CREATED        STATUS                    PORTS       NAMES
baba2b18db66  registry.redhat.io/rhel8/support-tools:latest  toolbox --log-lev...  4 seconds ago  Exited (1) 5 seconds ago              support-tools-latest

[root@kvm-08-guest26 ~]# podman start --attach support-tools-latest
toolbox: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by toolbox)

> 
> In bug 2004563 we changed the default image from
> registry.redhat.io/rhel8/support-tools:latest to
> registry.access.redhat.com/ubi9/toolbox:latest ie., in other words, from a
> RHEL 8 image to a RHEL 9 one. That should reduce the impact of this bug,
> because only users choosing to use a custom image will encounter it.
> 
> We are currently trying to fix this bug for Fedora 35 too. Would it be OK if
> we fixed this for RHEL 9.0 GA, and not Beta?

It should be okay if we could provide a workaround or documentation to end users.

Comment 3 Debarshi Ray 2021-10-04 15:11:34 UTC
(In reply to Alex Jia from comment #2)

> 1. try to start support-tools-latest container was created yesterday
> 
> [root@kvm-08-guest26 ~]# podman ps -a
> CONTAINER ID  IMAGE                                                         
> COMMAND               CREATED       STATUS                   PORTS      
> NAMES
> 9fc71475c6e6 
> registry-proxy.engineering.redhat.com/rh-osbs/rhel9-support-tools:9.0.0beta 
> /usr/bin/bash         3 days ago    Exited (0) 3 days ago               
> rhel9-support-tools
> 4e8353196e3a  registry.redhat.io/rhel8/support-tools:latest                 
> toolbox --log-lev...  22 hours ago  Exited (1) 22 hours ago             
> support-tools-latest
> 
> [root@kvm-08-guest26 ~]# podman start --attach support-tools-latest
> Error: unable to start container
> 4e8353196e3ac745081f95ee97c88dbfa3b6209d42743888e4a4c946961a4ccd: error
> stat'ing file `/run/toolbox`: No such file or directory: OCI runtime
> attempted to invoke a command that was not found

'podman start --attach ...' needs to be invoked on a container that already failed to start with Toolbox. Otherwise it won't reveal the real cause of the problem. :)

> 2. clean all of existing container then try again
> 
> [root@kvm-08-guest26 ~]# podman rm -f `podman ps -qa`
> 9fc71475c6e6b9616fee3e15dd1ed7d00edb209d11ba5bc0cfdecf62fdc270e9
> 4e8353196e3ac745081f95ee97c88dbfa3b6209d42743888e4a4c946961a4ccd
> 
> [root@kvm-08-guest26 ~]# podman ps -a
> CONTAINER ID  IMAGE       COMMAND     CREATED     STATUS      PORTS      
> NAMES
> 
> [root@kvm-08-guest26 ~]# toolbox
> No toolbox containers found. Create now? [y/N] y
> Error: invalid entry point PID of container support-tools-latest
> 
> [root@kvm-08-guest26 ~]# podman ps -a
> CONTAINER ID  IMAGE                                          COMMAND        
> CREATED        STATUS                    PORTS       NAMES
> baba2b18db66  registry.redhat.io/rhel8/support-tools:latest  toolbox
> --log-lev...  4 seconds ago  Exited (1) 5 seconds ago             
> support-tools-latest
> 
> [root@kvm-08-guest26 ~]# podman start --attach support-tools-latest
> toolbox: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by
> toolbox)

This confirms my guess in comment 1

> > In bug 2004563 we changed the default image from
> > registry.redhat.io/rhel8/support-tools:latest to
> > registry.access.redhat.com/ubi9/toolbox:latest ie., in other words, from a
> > RHEL 8 image to a RHEL 9 one. That should reduce the impact of this bug,
> > because only users choosing to use a custom image will encounter it.
> > 
> > We are currently trying to fix this bug for Fedora 35 too. Would it be OK if
> > we fixed this for RHEL 9.0 GA, and not Beta?
> 
> It should be okay if we could provide a workaround or documentation to end
> users.

This root cause of this is quite complex. One workaround I can think of, is to also switch the default image to the RHEL 9 toolbox image (ie., bug 2004563) in RHEL 9 Beta.

Comment 8 Debarshi Ray 2021-12-10 11:12:15 UTC
*** Bug 2016906 has been marked as a duplicate of this bug. ***

Comment 9 Debarshi Ray 2021-12-10 13:33:35 UTC
Pull request: https://gitlab.com/redhat/centos-stream/rpms/toolbox/-/merge_requests/43

Comment 10 Debarshi Ray 2021-12-15 01:46:54 UTC
*** Bug 2031665 has been marked as a duplicate of this bug. ***

Comment 18 errata-xmlrpc 2022-05-17 12:31:44 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new packages: toolbox), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:2331