RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2000807 - /usr/bin/toolbox linked against glibc-2.34 doesn't run on older glibc
Summary: /usr/bin/toolbox linked against glibc-2.34 doesn't run on older glibc
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: toolbox
Version: 9.0
Hardware: x86_64
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: Debarshi Ray
QA Contact: Martin Krajnak
URL:
Whiteboard:
: 2016906 2031665 (view as bug list)
Depends On:
Blocks: 2016901 2031652
TreeView+ depends on / blocked
 
Reported: 2021-09-03 03:31 UTC by Alex Jia
Modified: 2022-05-17 12:45 UTC (History)
3 users (show)

Fixed In Version: toolbox-0.0.99.3-2.el9_0
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-05-17 12:31:44 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github containers toolbox issues 821 0 None open /usr/bin/toolbox linked against glibc-2.34 doesn't run on older glibc 2021-10-01 13:49:29 UTC
Red Hat Issue Tracker RHELPLAN-96064 0 None None None 2021-09-03 03:31:59 UTC
Red Hat Product Errata RHBA-2022:2331 0 None None None 2022-05-17 12:31:52 UTC

Description Alex Jia 2021-09-03 03:31:42 UTC 
Description of problem:
Can't run toolbox command successfully and got error like this "Error: invalid entry point PID of container support-tools-latest".

Version-Release number of selected component (if applicable):

[root@kvm-08-guest02 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux release 9.0 Beta (Plow)

[root@kvm-08-guest02 ~]# rpm -q toolbox podman crun systemd kernel
toolbox-0.0.99.3-3.module+el9.0.0+12467+378c8264.x86_64
podman-3.3.1-6.module+el9.0.0+12467+378c8264.x86_64
crun-1.0-1.module+el9.0.0+12467+378c8264.x86_64
systemd-249-4.el9.x86_64
kernel-5.14.0-0.rc7.54.el9.x86_64

How reproducible:
always

Steps to Reproduce:
1. podman login registry.redhat.io
2. toolbox


Actual results:

[root@kvm-08-guest02 ~]# toolbox
No toolbox containers found. Create now? [y/N] y
Image required to create toolbox container.
Download registry.redhat.io/rhel8/support-tools:latest (500MB)? [y/N]: y
Error: invalid entry point PID of container support-tools-latest

[root@kvm-08-guest02 ~]# podman ps -a
CONTAINER ID  IMAGE                                          COMMAND               CREATED             STATUS                         PORTS       NAMES
d67764882c91  registry.redhat.io/rhel8/support-tools:latest  toolbox --log-lev...  About a minute ago  Exited (1) About a minute ago              support-tools-latest


Expected results:
fix it.

Additional info:

[root@kvm-08-guest02 ~]# toolbox
Error: invalid entry point PID of container support-tools-latest
[root@kvm-08-guest02 ~]# toolbox -v
DEBU Running as real user ID 0
DEBU Resolved absolute path to the executable as /usr/bin/toolbox
DEBU Running on a cgroups v2 host
DEBU TOOLBOX_PATH is /usr/bin/toolbox
DEBU Migrating to newer Podman
DEBU Toolbox config directory is /root/.config/toolbox
DEBU Current Podman version is 3.3.1
DEBU Creating runtime directory /run/toolbox
DEBU Old Podman version is 3.3.1
DEBU Migration not needed: Podman version 3.3.1 is unchanged
DEBU Setting up configuration
DEBU Setting up configuration: file /root/.config/containers/toolbox.conf not found
DEBU Resolving image name
DEBU Distribution (CLI): ''
DEBU Image (CLI): ''
DEBU Release (CLI): ''
DEBU Resolved image name
DEBU Image: 'registry.redhat.io/rhel8/support-tools:latest'
DEBU Release: 'latest'
DEBU Resolving container name
DEBU Container: ''
DEBU Image: 'registry.redhat.io/rhel8/support-tools:latest'
DEBU Release: 'latest'
DEBU Resolved container name
DEBU Container: 'support-tools-latest'
DEBU Resolving image name
DEBU Distribution (CLI): ''
DEBU Image (CLI): ''
DEBU Release (CLI): ''
DEBU Resolved image name
DEBU Image: 'registry.redhat.io/rhel8/support-tools:latest'
DEBU Release: 'latest'
DEBU Resolving container name
DEBU Container: ''
DEBU Image: 'registry.redhat.io/rhel8/support-tools:latest'
DEBU Release: 'latest'
DEBU Resolved container name
DEBU Container: 'support-tools-latest'
DEBU Checking if container support-tools-latest exists
DEBU Inspecting mounts of container support-tools-latest
DEBU Starting container support-tools-latest
DEBU Inspecting entry point of container support-tools-latest
DEBU Entry point PID is a float64
DEBU Entry point of container support-tools-latest is toolbox (PID=0)
Error: invalid entry point PID of container support-tools-latest
[root@kvm-08-guest02 ~]# toolbox enter support-tools-latest
Error: invalid entry point PID of container support-tools-latest
[root@kvm-08-guest02 ~]# toolbox -v enter support-tools-latest
DEBU Running as real user ID 0
DEBU Resolved absolute path to the executable as /usr/bin/toolbox
DEBU Running on a cgroups v2 host
DEBU TOOLBOX_PATH is /usr/bin/toolbox
DEBU Migrating to newer Podman
DEBU Toolbox config directory is /root/.config/toolbox
DEBU Current Podman version is 3.3.1
DEBU Creating runtime directory /run/toolbox
DEBU Old Podman version is 3.3.1
DEBU Migration not needed: Podman version 3.3.1 is unchanged
DEBU Setting up configuration
DEBU Setting up configuration: file /root/.config/containers/toolbox.conf not found
DEBU Resolving image name
DEBU Distribution (CLI): ''
DEBU Image (CLI): ''
DEBU Release (CLI): ''
DEBU Resolved image name
DEBU Image: 'registry.redhat.io/rhel8/support-tools:latest'
DEBU Release: 'latest'
DEBU Resolving container name
DEBU Container: ''
DEBU Image: 'registry.redhat.io/rhel8/support-tools:latest'
DEBU Release: 'latest'
DEBU Resolved container name
DEBU Container: 'support-tools-latest'
DEBU Resolving image name
DEBU Distribution (CLI): ''
DEBU Image (CLI): ''
DEBU Release (CLI): ''
DEBU Resolved image name
DEBU Image: 'registry.redhat.io/rhel8/support-tools:latest'
DEBU Release: 'latest'
DEBU Resolving container name
DEBU Container: 'support-tools-latest'
DEBU Image: 'registry.redhat.io/rhel8/support-tools:latest'
DEBU Release: 'latest'
DEBU Resolved container name
DEBU Container: 'support-tools-latest'
DEBU Checking if container support-tools-latest exists
DEBU Inspecting mounts of container support-tools-latest
DEBU Starting container support-tools-latest
DEBU Inspecting entry point of container support-tools-latest
DEBU Entry point PID is a float64
DEBU Entry point of container support-tools-latest is toolbox (PID=0)
Error: invalid entry point PID of container support-tools-latest
Comment 1 Debarshi Ray 2021-10-01 13:49:30 UTC 
I believe this is:
https://github.com/containers/toolbox/issues/821

Could you please paste the logs from:
$ podman start --attach support-tools-latest

... so that we can be sure?

If my guess above is right, then this issue affects running RHEL 8 containers on a RHEL 9 host, because RHEL 9 hosts have glibc-2.34.

In bug 2004563 we changed the default image from registry.redhat.io/rhel8/support-tools:latest to registry.access.redhat.com/ubi9/toolbox:latest ie., in other words, from a RHEL 8 image to a RHEL 9 one. That should reduce the impact of this bug, because only users choosing to use a custom image will encounter it.

We are currently trying to fix this bug for Fedora 35 too. Would it be OK if we fixed this for RHEL 9.0 GA, and not Beta?
Comment 2 Alex Jia 2021-10-02 02:35:46 UTC 
(In reply to Debarshi Ray from comment #1)
> I believe this is:
> https://github.com/containers/toolbox/issues/821
> 
> Could you please paste the logs from:
> $ podman start --attach support-tools-latest
> 
> ... so that we can be sure?
> 
> If my guess above is right, then this issue affects running RHEL 8
> containers on a RHEL 9 host, because RHEL 9 hosts have glibc-2.34.

1. try to start support-tools-latest container was created yesterday

[root@kvm-08-guest26 ~]# podman ps -a
CONTAINER ID  IMAGE                                                                        COMMAND               CREATED       STATUS                   PORTS       NAMES
9fc71475c6e6  registry-proxy.engineering.redhat.com/rh-osbs/rhel9-support-tools:9.0.0beta  /usr/bin/bash         3 days ago    Exited (0) 3 days ago                rhel9-support-tools
4e8353196e3a  registry.redhat.io/rhel8/support-tools:latest                                toolbox --log-lev...  22 hours ago  Exited (1) 22 hours ago              support-tools-latest

[root@kvm-08-guest26 ~]# podman start --attach support-tools-latest
Error: unable to start container 4e8353196e3ac745081f95ee97c88dbfa3b6209d42743888e4a4c946961a4ccd: error stat'ing file `/run/toolbox`: No such file or directory: OCI runtime attempted to invoke a command that was not found

NOTE: this error didn't hit when cleaning all of container then try again, please see the following tests.

2. clean all of existing container then try again

[root@kvm-08-guest26 ~]# podman rm -f `podman ps -qa`
9fc71475c6e6b9616fee3e15dd1ed7d00edb209d11ba5bc0cfdecf62fdc270e9
4e8353196e3ac745081f95ee97c88dbfa3b6209d42743888e4a4c946961a4ccd

[root@kvm-08-guest26 ~]# podman ps -a
CONTAINER ID  IMAGE       COMMAND     CREATED     STATUS      PORTS       NAMES

[root@kvm-08-guest26 ~]# toolbox
No toolbox containers found. Create now? [y/N] y
Error: invalid entry point PID of container support-tools-latest

[root@kvm-08-guest26 ~]# podman ps -a
CONTAINER ID  IMAGE                                          COMMAND               CREATED        STATUS                    PORTS       NAMES
baba2b18db66  registry.redhat.io/rhel8/support-tools:latest  toolbox --log-lev...  4 seconds ago  Exited (1) 5 seconds ago              support-tools-latest

[root@kvm-08-guest26 ~]# podman start --attach support-tools-latest
toolbox: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by toolbox)

> 
> In bug 2004563 we changed the default image from
> registry.redhat.io/rhel8/support-tools:latest to
> registry.access.redhat.com/ubi9/toolbox:latest ie., in other words, from a
> RHEL 8 image to a RHEL 9 one. That should reduce the impact of this bug,
> because only users choosing to use a custom image will encounter it.
> 
> We are currently trying to fix this bug for Fedora 35 too. Would it be OK if
> we fixed this for RHEL 9.0 GA, and not Beta?

It should be okay if we could provide a workaround or documentation to end users.
Comment 3 Debarshi Ray 2021-10-04 15:11:34 UTC 
(In reply to Alex Jia from comment #2)

> 1. try to start support-tools-latest container was created yesterday
> 
> [root@kvm-08-guest26 ~]# podman ps -a
> CONTAINER ID  IMAGE                                                         
> COMMAND               CREATED       STATUS                   PORTS      
> NAMES
> 9fc71475c6e6 
> registry-proxy.engineering.redhat.com/rh-osbs/rhel9-support-tools:9.0.0beta 
> /usr/bin/bash         3 days ago    Exited (0) 3 days ago               
> rhel9-support-tools
> 4e8353196e3a  registry.redhat.io/rhel8/support-tools:latest                 
> toolbox --log-lev...  22 hours ago  Exited (1) 22 hours ago             
> support-tools-latest
> 
> [root@kvm-08-guest26 ~]# podman start --attach support-tools-latest
> Error: unable to start container
> 4e8353196e3ac745081f95ee97c88dbfa3b6209d42743888e4a4c946961a4ccd: error
> stat'ing file `/run/toolbox`: No such file or directory: OCI runtime
> attempted to invoke a command that was not found

'podman start --attach ...' needs to be invoked on a container that already failed to start with Toolbox. Otherwise it won't reveal the real cause of the problem. :)

> 2. clean all of existing container then try again
> 
> [root@kvm-08-guest26 ~]# podman rm -f `podman ps -qa`
> 9fc71475c6e6b9616fee3e15dd1ed7d00edb209d11ba5bc0cfdecf62fdc270e9
> 4e8353196e3ac745081f95ee97c88dbfa3b6209d42743888e4a4c946961a4ccd
> 
> [root@kvm-08-guest26 ~]# podman ps -a
> CONTAINER ID  IMAGE       COMMAND     CREATED     STATUS      PORTS      
> NAMES
> 
> [root@kvm-08-guest26 ~]# toolbox
> No toolbox containers found. Create now? [y/N] y
> Error: invalid entry point PID of container support-tools-latest
> 
> [root@kvm-08-guest26 ~]# podman ps -a
> CONTAINER ID  IMAGE                                          COMMAND        
> CREATED        STATUS                    PORTS       NAMES
> baba2b18db66  registry.redhat.io/rhel8/support-tools:latest  toolbox
> --log-lev...  4 seconds ago  Exited (1) 5 seconds ago             
> support-tools-latest
> 
> [root@kvm-08-guest26 ~]# podman start --attach support-tools-latest
> toolbox: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by
> toolbox)

This confirms my guess in comment 1

> > In bug 2004563 we changed the default image from
> > registry.redhat.io/rhel8/support-tools:latest to
> > registry.access.redhat.com/ubi9/toolbox:latest ie., in other words, from a
> > RHEL 8 image to a RHEL 9 one. That should reduce the impact of this bug,
> > because only users choosing to use a custom image will encounter it.
> > 
> > We are currently trying to fix this bug for Fedora 35 too. Would it be OK if
> > we fixed this for RHEL 9.0 GA, and not Beta?
> 
> It should be okay if we could provide a workaround or documentation to end
> users.

This root cause of this is quite complex. One workaround I can think of, is to also switch the default image to the RHEL 9 toolbox image (ie., bug 2004563) in RHEL 9 Beta.
Comment 8 Debarshi Ray 2021-12-10 11:12:15 UTC 
*** Bug 2016906 has been marked as a duplicate of this bug. ***
Comment 9 Debarshi Ray 2021-12-10 13:33:35 UTC 
Pull request: https://gitlab.com/redhat/centos-stream/rpms/toolbox/-/merge_requests/43
Comment 10 Debarshi Ray 2021-12-15 01:46:54 UTC 
*** Bug 2031665 has been marked as a duplicate of this bug. ***
Comment 18 errata-xmlrpc 2022-05-17 12:31:44 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new packages: toolbox), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:2331
Note You need to log in before you can comment on or make changes to this bug.