Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1940082

Summary:	toolbox does not support a config file
Product:	Red Hat Enterprise Linux 8	Reporter:	Derrick Ornelas <dornelas>
Component:	toolbox	Assignee:	Debarshi Ray <debarshir>
Status:	CLOSED ERRATA	QA Contact:	Alex Jia <ajia>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	8.4	CC:	debarshir, dornelas, jnovy, miabbott, mjenner, smccarty, tpopela, travier, ypu
Target Milestone:	rc	Keywords:	Triaged
Target Release:	8.4	Flags:	pm-rhel: mirror+
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:	container-tools-rhel8-8050020210728060952.faa19cc5	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-11-09 17:37:14 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Derrick Ornelas 2021-03-17 14:59:51 UTC

Description of problem:

github.com/containers/toolbox does not support a system or user config file to override the default, hard-coded image.  This is a regression in functionality compared to rhcos-toolbox.


Version-Release number of selected component (if applicable):

toolbox-0.0.99-1.module+el8.4.0+10198+36d1d0e3


How reproducible: 100%


Steps to Reproduce:
1.  Install toolbox >= v0.0.99

2.  Create a toolbox container

# toolbox create 


Actual results:

Toolbox defaults to using registry.access.redhat.com/ubi8/ubi:8.4 and it is not possible for RH or users to set a different default, such as registry.redhat.io/rhel8/support-tools


Expected results:

Default registry/image can be overridden by user via a config file.  


Additional info:

rhcos-toolbox supports a $HOME/.toolboxrc config file that Support and customers can use to set a new default image to be used

For github.com/containers/toolbox to ever be useful on RHCOS it will need to use the registry.redhat.io/rhel8/support-tools or other RHEL-based support image.  Either the default in the code will need to be changed (possibly resulting in a downstream branch/fork), or, better yet, it should support a RH-provided config file for overriding the default.

Comment 2 Debarshi Ray 2021-03-18 08:30:45 UTC

Note that so far we've only rebased 'toolbox' to github.com/containers/toolbox in RHEL 8.4, not RHOCP (which I think CoreOS is a part of).

We are aware of this issue, it is tracked upstream at https://github.com/containers/toolbox/issues/689 It should be addressed in one of the following toolbox releases that will be included in one of the next RHEL 8 versions (might be even 8.5).

We understand that this impacts RHCOS. Hence the plan is not to rebase 'toolbox' in RHOCP until issues like this can be sorted out - as agreed with the CoreOS team (CC'ing Micah). Therefore, I don't think this qualifies for the 8.4 exception as it would need weeks to be properly addressed and tested upstream.

Comment 4 Tomas Popela 2021-07-12 12:10:30 UTC

Can we please get the QA ack?

Comment 13 Alex Jia 2021-08-03 08:53:26 UTC

The ~/.toolboxrc is invalid for toolbox-0.0.99.2^1.git660b6970e998-1.module+el8.5.0+12014+438a5746,
and toolbox tries to find configuration under the ~/.config/toolbox and ~/.config/containers/toolbox.conf.

[root@kvm-07-guest24 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.5 Beta (Ootpa)

[root@kvm-07-guest24 ~]# rpm -q toolbox podman runc kernel
toolbox-0.0.99.2^1.git660b6970e998-1.module+el8.5.0+12014+438a5746.x86_64
podman-3.3.0-0.17.module+el8.5.0+12014+438a5746.x86_64
runc-1.0.1-3.module+el8.5.0+12014+438a5746.x86_64
kernel-4.18.0-325.el8.x86_64


[root@kvm-07-guest24 ~]# cat ~/.toolboxrc
[root]
interactive = false
default_image = registry.foo/name/image:tag
credentials = /var/lib/foo.conf

[user]
interactive = true
default_image = registry.foo/name/image_user:tag

[root@kvm-07-guest24 ~]# toolbox -v
DEBU Running as real user ID 0                    
DEBU Resolved absolute path to the executable as /usr/bin/toolbox 
DEBU Running on a cgroups v1 host                 
DEBU TOOLBOX_PATH is /usr/bin/toolbox             
DEBU Migrating to newer Podman                    
DEBU Toolbox config directory is /root/.config/toolbox 
DEBU Current Podman version is 3.3.0-dev          
DEBU Creating runtime directory /run/toolbox      
DEBU Old Podman version is 3.3.0-dev              
DEBU Migration not needed: Podman version 3.3.0-dev is unchanged 
DEBU Setting up configuration                     
DEBU Setting up configuration: file /root/.config/containers/toolbox.conf not found 
DEBU Resolving image name                         
DEBU Distribution (CLI): ''                       
DEBU Image (CLI): ''                              
DEBU Release (CLI): ''                            
DEBU Resolved image name                          
DEBU Image: 'ubi:8.5'                             
DEBU Release: '8.5'                               
DEBU Resolving container name                     
DEBU Container: ''                                
DEBU Image: 'ubi:8.5'                             
DEBU Release: '8.5'                               
DEBU Resolved container name                      
DEBU Container: 'rhel-toolbox-8.5'                
DEBU Resolving image name                         
DEBU Distribution (CLI): ''                       
DEBU Image (CLI): ''                              
DEBU Release (CLI): ''                            
DEBU Resolved image name                          
DEBU Image: 'ubi:8.5'                             
DEBU Release: '8.5'                               
DEBU Resolving container name                     
DEBU Container: ''                                
DEBU Image: 'ubi:8.5'                             
DEBU Release: '8.5'                               
DEBU Resolved container name                      
DEBU Container: 'rhel-toolbox-8.5'                
DEBU Checking if container rhel-toolbox-8.5 exists 
DEBU Container rhel-toolbox-8.5 not found         
DEBU Fetching all containers                      
DEBU Found 0 containers                           
No toolbox containers found. Create now? [y/N] y
DEBU Checking if container rhel-toolbox-8.5 already exists 
DEBU Looking for image localhost/ubi:8.5          
DEBU Resolving fully qualified name for image ubi:8.5 from known registries 
DEBU Resolved image ubi:8.5 to registry.access.redhat.com/ubi8/ubi:8.5 
DEBU Looking for image registry.access.redhat.com/ubi8/ubi:8.5 
Image required to create toolbox container.
Download registry.access.redhat.com/ubi8/ubi:8.5 (500MB)? [y/N]: n
DEBU Inspecting mounts of container rhel-toolbox-8.5 
Error: error inspecting object: no such container rhel-toolbox-8.5
Error: failed to inspect entry point of container rhel-toolbox-8.5

[root@kvm-07-guest24 ~]# ls /root/.config/toolbox
podman-system-migrate  toolbox-welcome-shown

Comment 14 Alex Jia 2021-08-07 02:48:06 UTC

The ~/.toolboxrc is invalid for toolbox-0.0.99.3-1.module+el8.5.0+12157+04f1d6be.x86_64.

[root@kvm-02-guest15 ~]# cat ~/.toolboxrc
[root]
interactive = false
default_image = quay.io/libpod/alpine
credentials = /var/lib/foo.conf

[user]
interactive = true
default_image = quay.io/libpod/alpine
[root@kvm-02-guest15 ~]# toolbox -v
DEBU Running as real user ID 0                    
DEBU Resolved absolute path to the executable as /usr/bin/toolbox 
DEBU Running on a cgroups v2 host                 
DEBU TOOLBOX_PATH is /usr/bin/toolbox             
DEBU Migrating to newer Podman                    
DEBU Toolbox config directory is /root/.config/toolbox 
DEBU Current Podman version is 3.3.0-dev          
DEBU Creating runtime directory /run/toolbox      
DEBU Migration to Podman version 3.3.0-dev was ok 
DEBU Updating Podman version in /root/.config/toolbox/podman-system-migrate 
DEBU Setting up configuration                     
DEBU Setting up configuration: file /root/.config/containers/toolbox.conf not found 
DEBU Resolving image name                         
DEBU Distribution (CLI): ''                       
DEBU Image (CLI): ''                              
DEBU Release (CLI): ''                            
DEBU Resolved image name                          
DEBU Image: 'ubi:8.5'                             
DEBU Release: '8.5'                               
DEBU Resolving container name                     
DEBU Container: ''                                
DEBU Image: 'ubi:8.5'                             
DEBU Release: '8.5'                               
DEBU Resolved container name                      
DEBU Container: 'rhel-toolbox-8.5'                
DEBU Resolving image name                         
DEBU Distribution (CLI): ''                       
DEBU Image (CLI): ''                              
DEBU Release (CLI): ''                            
DEBU Resolved image name                          
DEBU Image: 'ubi:8.5'                             
DEBU Release: '8.5'                               
DEBU Resolving container name                     
DEBU Container: ''                                
DEBU Image: 'ubi:8.5'                             
DEBU Release: '8.5'                               
DEBU Resolved container name                      
DEBU Container: 'rhel-toolbox-8.5'                
DEBU Checking if container rhel-toolbox-8.5 exists 
DEBU Container rhel-toolbox-8.5 not found         
DEBU Fetching all containers                      
DEBU Found 0 containers                           
No toolbox containers found. Create now? [y/N] y
DEBU Checking if container rhel-toolbox-8.5 already exists 
DEBU Looking for image localhost/ubi:8.5          
DEBU Resolving fully qualified name for image ubi:8.5 from known registries 
DEBU Resolved image ubi:8.5 to registry.access.redhat.com/ubi8/ubi:8.5 
DEBU Looking for image registry.access.redhat.com/ubi8/ubi:8.5 
Image required to create toolbox container.
Download registry.access.redhat.com/ubi8/ubi:8.5 (500MB)? [y/N]: y
DEBU Pulling image registry.access.redhat.com/ubi8/ubi:8.5 
Trying to pull registry.access.redhat.com/ubi8/ubi:8.5...
Error: initializing source docker://registry.access.redhat.com/ubi8/ubi:8.5: reading manifest 8.5 in registry.access.redhat.com/ubi8/ubi: StatusCode: 404, Not found

Error: failed to pull image registry.access.redhat.com/ubi8/ubi:8.5
If it was a private image, log in with: podman login registry.access.redhat.com
Use 'toolbox --verbose ...' for further details.

Comment 15 Debarshi Ray 2021-08-10 16:56:13 UTC

Thanks for testing out the new Toolbox implementation from github.com/containers/toolbox !

(In reply to Alex Jia from comment #13)
> The ~/.toolboxrc is invalid for
> toolbox-0.0.99.2^1.git660b6970e998-1.module+el8.5.0+12014+438a5746,
> and toolbox tries to find configuration under the ~/.config/toolbox and
> ~/.config/containers/toolbox.conf.

The new implementation looks for configuration files in these locations:
  * /etc/containers/toolbox.conf
  * $XDG_CONFIG_HOME/containers/toolbox.conf, which is usually ~/.config/containers/toolbox.conf

It no longer looks at ~/.toolboxrc.

The ~/.config/toolbox sub-directory is used for some internal stamp files that need to be persistent across boots. It doesn't have anything that we'd call a configuration file.

> [root@kvm-07-guest24 ~]# cat ~/.toolboxrc
> [root]
> interactive = false
> default_image = registry.foo/name/image:tag
> credentials = /var/lib/foo.conf
> 
> [user]
> interactive = true
> default_image = registry.foo/name/image_user:tag

Umm... this doesn't look like the ~/.toolboxrc that the old implementation supported. Are you sure about this?

Comment 17 Debarshi Ray 2021-08-10 17:04:22 UTC

(In reply to Alex Jia from comment #14)
> DEBU Looking for image registry.access.redhat.com/ubi8/ubi:8.5 
> Image required to create toolbox container.
> Download registry.access.redhat.com/ubi8/ubi:8.5 (500MB)? [y/N]: y
> DEBU Pulling image registry.access.redhat.com/ubi8/ubi:8.5 
> Trying to pull registry.access.redhat.com/ubi8/ubi:8.5...
> Error: initializing source docker://registry.access.redhat.com/ubi8/ubi:8.5:
> reading manifest 8.5 in registry.access.redhat.com/ubi8/ubi: StatusCode:
> 404, Not found
> 
> Error: failed to pull image registry.access.redhat.com/ubi8/ubi:8.5
> If it was a private image, log in with: podman login
> registry.access.redhat.com
> Use 'toolbox --verbose ...' for further details.

The problem is that by, default, Toolbox tries to use the UBI image matching the host version from the public registry. That's registry.access.redhat.com/ubi8/ubi:8.5 if the host is RHEL 8.5.

The problem is that the public image for RHEL 8.5 won't be available until RHEL 8.5 is publicly released.

Until then, I can think of these options:

  * Use a RHEL 8.4 host to test Toolbox.

  * Override the default image using the --image option on the command line or the 'image' option in the configuration file.

Comment 18 Alex Jia 2021-08-17 02:29:06 UTC

(In reply to Debarshi Ray from comment #17)
> The problem is that the public image for RHEL 8.5 won't be available until
> RHEL 8.5 is publicly released.
> 
> Until then, I can think of these options:
> 
>   * Use a RHEL 8.4 host to test Toolbox.
> 
>   * Override the default image using the --image option on the command line
> or the 'image' option in the configuration file.

Yes, I know this, thanks!

Comment 19 Alex Jia 2021-08-17 09:25:47 UTC

(In reply to Debarshi Ray from comment #15)

> The new implementation looks for configuration files in these locations:
>   * /etc/containers/toolbox.conf

It works for me.

1. uncomment image
[root@kvm-02-guest20 ~]# vi /etc/containers/toolbox.conf
[root@kvm-02-guest20 ~]# grep fedora-toolbox /etc/containers/toolbox.conf
image = "registry.fedoraproject.org/fedora-toolbox:34"

[root@kvm-02-guest20 ~]# toolbox
No toolbox containers found. Create now? [y/N] y
Image required to create toolbox container.
Download registry.fedoraproject.org/fedora-toolbox:34 (500MB)? [y/N]: y

Welcome to the Toolbox; a container where you can install and run
all your tools.

 - Use DNF in the usual manner to install command line tools.
 - To create a new tools container, run 'toolbox create'.

For more information, see the documentation.

⬢[root@toolbox ~]# cat /etc/redhat-release 
Fedora release 34 (Thirty Four)
⬢[root@toolbox ~]# exit
logout
[root@kvm-02-guest20 ~]# echo $?
0
[root@kvm-02-guest20 ~]# podman rm -f fedora-toolbox-34
d061c4f6b8e5b517bdcc688797b4c351171b580edc80b6c343325b60b38a1649

2. uncomment distro and release

[root@kvm-02-guest20 ~]# grep "distro =" /etc/containers/toolbox.conf
distro = "fedora"
[root@kvm-02-guest20 ~]# grep "release =" /etc/containers/toolbox.conf
release = "33"

[root@kvm-02-guest20 ~]# toolbox
No toolbox containers found. Create now? [y/N] y
Image required to create toolbox container.
Download registry.fedoraproject.org/fedora-toolbox:33 (500MB)? [y/N]: y
⬢[root@toolbox ~]# cat /etc/redhat-release 
Fedora release 33 (Thirty Three)
⬢[root@toolbox ~]# exit
logout
[root@kvm-02-guest20 ~]# echo $?
0

3. using invalid value for image

[root@kvm-02-guest20 ~]# grep fedora-toolbox /etc/containers/toolbox.conf
image = "doesnotexist:unknown"
[root@kvm-02-guest20 ~]# toolbox
No toolbox containers found. Create now? [y/N] y
Error: image doesnotexist:unknown not found in local storage and known registries
[root@kvm-02-guest20 ~]# echo $?
1

>   * $XDG_CONFIG_HOME/containers/toolbox.conf, which is usually
> ~/.config/containers/toolbox.conf

It doesn't work for me.

[root@kvm-02-guest20 ~]# mkdir ~/.config/containers
[root@kvm-02-guest20 ~]# cat ~/.config/containers/toolbox.conf
image = "registry.access.redhat.com/ubi8"

[root@kvm-02-guest20 ~]# toolbox
No toolbox containers found. Create now? [y/N] y
Image required to create toolbox container.
Download registry.access.redhat.com/ubi8/ubi:8.5 (500MB)? [y/N]:

And also tried to put toolbox under the ~/.config/toolbox  
(DEBU Toolbox config directory is /root/.config/toolbox)

[root@kvm-02-guest20 ~]# cat ~/.config/toolbox/toolbox.conf
image = "registry.access.redhat.com/ubi8"
[root@kvm-02-guest20 ~]# toolbox
No toolbox containers found. Create now? [y/N] y
Image required to create toolbox container.
Download registry.access.redhat.com/ubi8/ubi:8.5 (500MB)? [y/N]:

Comment 20 Debarshi Ray 2021-08-19 17:27:58 UTC

Thanks for testing this, Alex!

(In reply to Alex Jia from comment #19)
> (In reply to Debarshi Ray from comment #15)
> >   * $XDG_CONFIG_HOME/containers/toolbox.conf, which is usually
> > ~/.config/containers/toolbox.conf
> 
> It doesn't work for me.
> 
> [root@kvm-02-guest20 ~]# mkdir ~/.config/containers
> [root@kvm-02-guest20 ~]# cat ~/.config/containers/toolbox.conf
> image = "registry.access.redhat.com/ubi8"
> 
> [root@kvm-02-guest20 ~]# toolbox
> No toolbox containers found. Create now? [y/N] y
> Image required to create toolbox container.
> Download registry.access.redhat.com/ubi8/ubi:8.5 (500MB)? [y/N]:
> 
> And also tried to put toolbox under the ~/.config/toolbox  
> (DEBU Toolbox config directory is /root/.config/toolbox)
> 
> [root@kvm-02-guest20 ~]# cat ~/.config/toolbox/toolbox.conf
> image = "registry.access.redhat.com/ubi8"
> [root@kvm-02-guest20 ~]# toolbox
> No toolbox containers found. Create now? [y/N] y
> Image required to create toolbox container.
> Download registry.access.redhat.com/ubi8/ubi:8.5 (500MB)? [y/N]:

I wonder if the environment on the test machine is set up properly.

I suppose $HOME is set. Does it work if you do:
$ export XDG_CONFIG_HOME="$HOME/.config"
?

For what it's worth, we use Go's os.UserConfigDir() API to read the value of $XDG_CONFIG_HOME:
https://pkg.go.dev/os#UserConfigDir
https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html

Comment 21 Derrick Ornelas 2021-08-19 23:01:41 UTC

(In reply to Alex Jia from comment #19)
> 
> It doesn't work for me.
> 
> [root@kvm-02-guest20 ~]# mkdir ~/.config/containers
> [root@kvm-02-guest20 ~]# cat ~/.config/containers/toolbox.conf
> image = "registry.access.redhat.com/ubi8"
> 

It took me some time to realize it because it's such a small thing, but your toolbox.conf is missing the "[general]" section entry.  It should be similar to:

$ cat ~/.config/containers/toolbox.conf
[general]
image = "registry.redhat.io/rhel8/support-tools"


Then it will work:

$ toolbox -v create
DEBU Running as real user ID 1000                 
DEBU Resolved absolute path to the executable as /usr/bin/toolbox 
DEBU Running on a cgroups v1 host                 
DEBU Checking if /etc/subgid and /etc/subuid have entries for user user1 
DEBU Validating sub-ID file /etc/subuid           
DEBU Validating sub-ID file /etc/subgid           
DEBU TOOLBOX_PATH is /usr/bin/toolbox             
DEBU Migrating to newer Podman                    
DEBU Toolbox config directory is /home/user1/.config/toolbox 
DEBU Current Podman version is 3.2.3              
DEBU Creating runtime directory /run/user/1000/toolbox 
DEBU Old Podman version is 3.2.3                  
DEBU Migration not needed: Podman version 3.2.3 is unchanged 
DEBU Setting up configuration                     
DEBU Resolving image name                         
DEBU Distribution (CLI): ''                       
DEBU Image (CLI): ''                              
DEBU Release (CLI): ''                            
DEBU Resolved image name                          
DEBU Image: 'registry.redhat.io/rhel8/support-tools' 
DEBU Release: '8.4'                               
DEBU Resolving container name                     
DEBU Container: ''                                
DEBU Image: 'registry.redhat.io/rhel8/support-tools' 
DEBU Release: '8.4'                               
DEBU Resolved container name                      
DEBU Container: 'support-tools'                   
DEBU Resolving image name                         
DEBU Distribution (CLI): ''                       
DEBU Image (CLI): ''                              
DEBU Release (CLI): ''                            
DEBU Resolved image name                          
DEBU Image: 'registry.redhat.io/rhel8/support-tools' 
DEBU Release: '8.4'                               
DEBU Resolving container name                     
DEBU Container: ''                                
DEBU Image: 'registry.redhat.io/rhel8/support-tools' 
DEBU Release: '8.4'                               
DEBU Resolved container name                      
DEBU Container: 'support-tools'                   
DEBU Checking if container support-tools already exists 
DEBU Looking for image registry.redhat.io/rhel8/support-tools 
Image required to create toolbox container.
Download registry.redhat.io/rhel8/support-tools (500MB)? [y/N]:

Comment 22 Alex Jia 2021-08-23 08:17:06 UTC

(In reply to Derrick Ornelas from comment #21)

> It took me some time to realize it because it's such a small thing, but your
> toolbox.conf is missing the "[general]" section entry.  It should be similar
> to:
> 
> $ cat ~/.config/containers/toolbox.conf
> [general]
> image = "registry.redhat.io/rhel8/support-tools"

It works well for me, thank you Derrick!


This bug has been verified on toolbox-0.0.99.3-0.1.module+el8.5.0+12234+e4609207.

[root@hpe-dl380pgen8-02-vm-5 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.5 Beta (Ootpa)

[root@hpe-dl380pgen8-02-vm-5 ~]# rpm -q toolbox podman kernel
toolbox-0.0.99.3-0.1.module+el8.5.0+12234+e4609207.x86_64
podman-3.3.0-8.module+el8.5.0+12332+af0a4708.x86_64
kernel-4.18.0-330.el8.x86_64

Comment 23 Debarshi Ray 2021-08-25 12:09:19 UTC

Thanks for catching that, Derrick! :)

Comment 25 errata-xmlrpc 2021-11-09 17:37:14 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: container-tools:rhel8 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:4154