Bug 1917663

Summary: Failed to find the default ubi:8.4 image
Product: Red Hat Enterprise Linux 8 Reporter: Alex Jia <ajia>
Component: toolboxAssignee: Debarshi Ray <debarshir>
Status: CLOSED NOTABUG QA Contact: Alex Jia <ajia>
Severity: high Docs Contact:
Priority: unspecified    
Version: 8.4CC: akaris, debarshir, leiwang, tsweeney
Target Milestone: rcFlags: pm-rhel: mirror+
Target Release: 8.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-03-16 18:26:00 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Alex Jia 2021-01-19 05:41:29 UTC 
Description of problem:
Can't create toolbox container due to failed to pull registry.access.redhat.com/ubi8/ubi:8.4.

Version-Release number of selected component (if applicable):
# rpm -q podman toolbox
podman-3.0.0-0.21.module+el8.4.0+9425+98db097b.x86_64
toolbox-0.0.99-1.module+el8.4.0+9497+5ce1a419.x86_64

How reproducible:
always 

Steps to Reproduce:
1. toolbox create

Actual results:
[root@kvm-08-guest17 ~]# toolbox create
Image required to create toolbox container.
Download registry.access.redhat.com/ubi8/ubi:8.4 (500MB)? [y/N]: y
Error: failed to pull image registry.access.redhat.com/ubi8/ubi:8.4

Expected results:
fix image tag issue and can successfully create toolbox container

Additional info:
[root@kvm-08-guest17 ~]# podman --log-level=debug pull registry.access.redhat.com/ubi8/ubi:8.4
INFO[0000] podman filtering at log level debug
DEBU[0000] Called pull.PersistentPreRunE(podman --log-level=debug pull registry.access.redhat.com/ubi8/ubi:8.4)
XXX
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev,metacopy=on]registry.access.redhat.com/ubi8/ubi:8.4"
Trying to pull registry.access.redhat.com/ubi8/ubi:8.4...
DEBU[0000] Trying to access "registry.access.redhat.com/ubi8/ubi:8.4"
DEBU[0000] Credentials not found
DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration
DEBU[0000]  Using "docker" namespace registry.access.redhat.com
DEBU[0000]   Using https://access.redhat.com/webassets/docker/content/sigstore
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.access.redhat.com
DEBU[0000] GET https://registry.access.redhat.com/v2/
DEBU[0001] Ping https://registry.access.redhat.com/v2/ status 200
DEBU[0001] GET https://registry.access.redhat.com/v2/ubi8/ubi/manifests/8.4
DEBU[0003] Content-Type from manifest GET is ""
DEBU[0003] Accessing "registry.access.redhat.com/ubi8/ubi:8.4" failed: Error reading manifest 8.4 in registry.access.redhat.com/ubi8/ubi: StatusCode: 404, Not found
DEBU[0003] Error pulling image ref //registry.access.redhat.com/ubi8/ubi:8.4: Error initializing source docker://registry.access.redhat.com/ubi8/ubi:8.4: Error reading manifest 8.4 in registry.access.redhat.com/ubi8/ubi: StatusCode: 404, Not found
  StatusCode: 404, Not found
Error: Error initializing source docker://registry.access.redhat.com/ubi8/ubi:8.4: Error reading manifest 8.4 in registry.access.redhat.com/ubi8/ubi: StatusCode: 404, Not found
Comment 1 Jens Petersen 2021-01-19 11:45:43 UTC 
There is no ubi:8.4 yet - you need to specify "--release 8.3" at this time..
Comment 2 Debarshi Ray 2021-01-19 17:16:39 UTC 
By default, we use the UBI image from registry.access.redhat.com/ubi8/ubi, which is publicly available without any authentication, and this registry only contains images for released RHEL versions. So the most recent is 8.3.

There is another registry.redhat.io, which has nightly builds of UBI and has 8.4, but it requires authentication, so we don't use it by default.

If you want to test the code paths where --release isn't specified, then I'd suggest using a RHEL 8.3 host.
Comment 3 Alex Jia 2021-01-20 02:42:51 UTC 
(In reply to Debarshi Ray from comment #2)
> If you want to test the code paths where --release isn't specified, then I'd
> suggest using a RHEL 8.3 host.

Thank you Debarshi, a hard code default image should be okay, but I think 
it should be more friendly if we can provide arguments or a configuration file 
to allow users change it on toolbox cmdline.
Comment 4 Debarshi Ray 2021-01-21 14:30:22 UTC 
Unfortunately, github.com/containers/toolbox doesn't yet have a configuration file. We'd like to offer something, but we don't have anything yet.

The reason is that compared to github.com/coreos/toolbox, github.com/containers/toolbox offers more options and scope for configuration. So we need to come up with something that works well. eg., some people want to extend the configuration of the container by adding more bind mounts and such. That's more complicated than just overriding the values of the command line options.

Moreoever, since github.com/coreos/toolbox is a shell script, having a configuration file that's a list of environment variables is a natural extension of the implementation. However, github.com/containers/toolbox went from being written in POSIX shell to Go, so we didn't want to corner ourselves with a configuration file format that's inherently tied to shell scripts.

Toolbox picks up it's factory defaults from things like /usr/lib/os-release. So, once RHEL 8.4 is released and there's a matching UBI available publicly for it, Toolbox will pull registry.access.redhat.com/ubi8/ubi:8.4. But things are a bit weird at the moment because that's not the case.
Comment 5 Andreas Karis 2021-01-30 17:38:23 UTC 
[root@openshift-master-0 ~]# bash -x /bin/toolbox
+ set -eo pipefail
+ trap cleanup EXIT
+ REGISTRY=registry.redhat.io
+ IMAGE=rhel8/support-tools
++ whoami
+ TOOLBOX_NAME=toolbox-root
+ TOOLBOXRC=/root/.toolboxrc
+ main
+ setup
+ '[' -f /root/.toolboxrc ']'
+ echo '.toolboxrc file detected, overriding defaults...'
.toolboxrc file detected, overriding defaults...
+ source /root/.toolboxrc
++ REGISTRY=docker.io
++ IMAGE=centos:latest
+ TOOLBOX_IMAGE=docker.io/centos:latest
+ [[ '' =~ ^(--help|-h)$ ]]
+ run
+ image_exists
+ sudo podman inspect docker.io/centos:latest
++ image_runlabel
++ sudo podman image inspect docker.io/centos:latest --format '{{.Labels.run}}'
+ local 'runlabel=<no value>'
+ container_exists
+ sudo podman inspect toolbox-root
+ echo 'Spawning a container '\''toolbox-root'\'' with image '\''docker.io/centos:latest'\'''
Spawning a container 'toolbox-root' with image 'docker.io/centos:latest'
+ [[ -z <no value> ]]
+ echo 'Detected RUN label in the container image. Using that as the default...'
Detected RUN label in the container image. Using that as the default...
+ container_runlabel
+ sudo podman container runlabel --name toolbox-root RUN docker.io/centos:latest
panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
panic(0x559e98b920a0, 0xc0005b0860)
	/usr/lib/golang/src/runtime/panic.go:1064 +0x471 fp=0xc0006e1700 sp=0xc0006e1648 pc=0x559e97092cf1
runtime.goPanicIndex(0x0, 0x0)
	/usr/lib/golang/src/runtime/panic.go:88 +0xa5 fp=0xc0006e1748 sp=0xc0006e1700 pc=0x559e97090b05
github.com/containers/libpod/pkg/domain/infra/abi.generateCommand(0x0, 0x0, 0xc0005b0220, 0x1f, 0x7ffc0bf9e786, 0xc, 0x0, 0x0, 0x400559e98a4e4a0, 0xffffffffffffffff, ...)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/pkg/domain/infra/abi/containers_runlabel.go:197 +0x8c2 fp=0xc0006e1820 sp=0xc0006e1748 pc=0x559e981d2c82
github.com/containers/libpod/pkg/domain/infra/abi.generateRunlabelCommand(0x0, 0x0, 0xc00018c240, 0xc0003cca20, 0x0, 0x2, 0x0, 0x0, 0x0, 0x0, ...)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/pkg/domain/infra/abi/containers_runlabel.go:147 +0xcf fp=0xc0006e1978 sp=0xc0006e1820 pc=0x559e981d1e5f
github.com/containers/libpod/pkg/domain/infra/abi.(*ContainerEngine).ContainerRunlabel(0xc000630280, 0x559e98cf4cc0, 0xc000044098, 0x7ffc0bf9e793, 0x3, 0x7ffc0bf9e797, 0x17, 0xc0003cca20, 0x0, 0x2, ...)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/pkg/domain/infra/abi/containers_runlabel.go:34 +0x1b2 fp=0xc0006e1b60 sp=0xc0006e1978 pc=0x559e981d14d2
github.com/containers/libpod/cmd/podman/containers.runlabel(0x559e99aa5940, 0xc0003cca00, 0x2, 0x4, 0x0, 0x0)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/cmd/podman/containers/runlabel.go:80 +0x169 fp=0xc0006e1c88 sp=0xc0006e1b60 pc=0x559e98262179
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).execute(0x559e99aa5940, 0xc00003c0a0, 0x4, 0x4, 0x559e99aa5940, 0xc00003c0a0)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:838 +0x455 fp=0xc0006e1d60 sp=0xc0006e1c88 pc=0x559e97a9a3e5
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x559e99ab8720, 0xc000044098, 0x559e98a5cce0, 0x559e99b61e90)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:943 +0x319 fp=0xc0006e1e38 sp=0xc0006e1d60 pc=0x559e97a9aee9
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).Execute(...)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:883
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).ExecuteContext(...)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:876
main.Execute()
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/cmd/podman/root.go:90 +0xee fp=0xc0006e1eb8 sp=0xc0006e1e38 pc=0x559e9835f4ae
main.main()
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/cmd/podman/main.go:77 +0x18e fp=0xc0006e1f88 sp=0xc0006e1eb8 pc=0x559e9835edee
runtime.main()
	/usr/lib/golang/src/runtime/proc.go:203 +0x202 fp=0xc0006e1fe0 sp=0xc0006e1f88 pc=0x559e97095772
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0006e1fe8 sp=0xc0006e1fe0 pc=0x559e970c4881

goroutine 2 [force gc (idle)]:
runtime.gopark(0x559e98c87980, 0x559e99b2e840, 0x1411, 0x1)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000066fb0 sp=0xc000066f90 pc=0x559e97095b46
runtime.goparkunlock(...)
	/usr/lib/golang/src/runtime/proc.go:310
runtime.forcegchelper()
	/usr/lib/golang/src/runtime/proc.go:253 +0xbb fp=0xc000066fe0 sp=0xc000066fb0 pc=0x559e970959eb
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc000066fe8 sp=0xc000066fe0 pc=0x559e970c4881
created by runtime.init.7
	/usr/lib/golang/src/runtime/proc.go:242 +0x37

goroutine 3 [GC sweep wait]:
runtime.gopark(0x559e98c87980, 0x559e99b30f00, 0x140c, 0x1)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc0000677a8 sp=0xc000067788 pc=0x559e97095b46
runtime.goparkunlock(...)
	/usr/lib/golang/src/runtime/proc.go:310
runtime.bgsweep(0xc00007e000)
	/usr/lib/golang/src/runtime/mgcsweep.go:89 +0x135 fp=0xc0000677d8 sp=0xc0000677a8 pc=0x559e97082075
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0000677e0 sp=0xc0000677d8 pc=0x559e970c4881
created by runtime.gcenable
	/usr/lib/golang/src/runtime/mgc.go:214 +0x5e

goroutine 4 [GC scavenge wait]:
runtime.gopark(0x559e98c87980, 0x559e99b30ec0, 0x140d, 0x1)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000067f78 sp=0xc000067f58 pc=0x559e97095b46
runtime.goparkunlock(...)
	/usr/lib/golang/src/runtime/proc.go:310
runtime.bgscavenge(0xc00007e000)
	/usr/lib/golang/src/runtime/mgcscavenge.go:285 +0x294 fp=0xc000067fd8 sp=0xc000067f78 pc=0x559e970806f4
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc000067fe0 sp=0xc000067fd8 pc=0x559e970c4881
created by runtime.gcenable
	/usr/lib/golang/src/runtime/mgc.go:215 +0x80

goroutine 5 [finalizer wait]:
runtime.gopark(0x559e98c87980, 0x559e99b61dd8, 0xc0001a1410, 0x1)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000066758 sp=0xc000066738 pc=0x559e97095b46
runtime.goparkunlock(...)
	/usr/lib/golang/src/runtime/proc.go:310
runtime.runfinq()
	/usr/lib/golang/src/runtime/mfinal.go:175 +0xa7 fp=0xc0000667e0 sp=0xc000066758 pc=0x559e97077b07
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0000667e8 sp=0xc0000667e0 pc=0x559e970c4881
created by runtime.createfing
	/usr/lib/golang/src/runtime/mfinal.go:156 +0x63

goroutine 7 [GC worker (idle)]:
runtime.gopark(0x559e98c87808, 0xc000487f20, 0x1418, 0x0)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000068760 sp=0xc000068740 pc=0x559e97095b46
runtime.gcBgMarkWorker(0xc00004e000)
	/usr/lib/golang/src/runtime/mgc.go:1873 +0x105 fp=0xc0000687d8 sp=0xc000068760 pc=0x559e9707b505
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0000687e0 sp=0xc0000687d8 pc=0x559e970c4881
created by runtime.gcBgMarkStartWorkers
	/usr/lib/golang/src/runtime/mgc.go:1821 +0x79

goroutine 8 [GC worker (idle)]:
runtime.gopark(0x559e98c87808, 0xc00012c000, 0x1418, 0x0)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000068f60 sp=0xc000068f40 pc=0x559e97095b46
runtime.gcBgMarkWorker(0xc000050800)
	/usr/lib/golang/src/runtime/mgc.go:1873 +0x105 fp=0xc000068fd8 sp=0xc000068f60 pc=0x559e9707b505
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc000068fe0 sp=0xc000068fd8 pc=0x559e970c4881
created by runtime.gcBgMarkStartWorkers
	/usr/lib/golang/src/runtime/mgc.go:1821 +0x79

goroutine 19 [GC worker (idle)]:
runtime.gopark(0x559e98c87808, 0xc00012c010, 0x1418, 0x0)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000062760 sp=0xc000062740 pc=0x559e97095b46
runtime.gcBgMarkWorker(0xc000053000)
	/usr/lib/golang/src/runtime/mgc.go:1873 +0x105 fp=0xc0000627d8 sp=0xc000062760 pc=0x559e9707b505
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0000627e0 sp=0xc0000627d8 pc=0x559e970c4881
created by runtime.gcBgMarkStartWorkers
	/usr/lib/golang/src/runtime/mgc.go:1821 +0x79

goroutine 20 [GC worker (idle)]:
runtime.gopark(0x559e98c87808, 0xc00012c020, 0x1418, 0x0)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000062f60 sp=0xc000062f40 pc=0x559e97095b46
runtime.gcBgMarkWorker(0xc000055800)
	/usr/lib/golang/src/runtime/mgc.go:1873 +0x105 fp=0xc000062fd8 sp=0xc000062f60 pc=0x559e9707b505
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc000062fe0 sp=0xc000062fd8 pc=0x559e970c4881
created by runtime.gcBgMarkStartWorkers
	/usr/lib/golang/src/runtime/mgc.go:1821 +0x79

goroutine 21 [GC worker (idle)]:
runtime.gopark(0x559e98c87808, 0xc00012c030, 0x1418, 0x0)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000063760 sp=0xc000063740 pc=0x559e97095b46
runtime.gcBgMarkWorker(0xc000058000)
	/usr/lib/golang/src/runtime/mgc.go:1873 +0x105 fp=0xc0000637d8 sp=0xc000063760 pc=0x559e9707b505
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0000637e0 sp=0xc0000637d8 pc=0x559e970c4881
created by runtime.gcBgMarkStartWorkers
	/usr/lib/golang/src/runtime/mgc.go:1821 +0x79

goroutine 22 [GC worker (idle)]:
runtime.gopark(0x559e98c87808, 0xc000500000, 0x1418, 0x0)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000063f60 sp=0xc000063f40 pc=0x559e97095b46
runtime.gcBgMarkWorker(0xc00005a800)
	/usr/lib/golang/src/runtime/mgc.go:1873 +0x105 fp=0xc000063fd8 sp=0xc000063f60 pc=0x559e9707b505
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc000063fe0 sp=0xc000063fd8 pc=0x559e970c4881
created by runtime.gcBgMarkStartWorkers
	/usr/lib/golang/src/runtime/mgc.go:1821 +0x79

goroutine 52 [chan receive]:
runtime.gopark(0x559e98c87780, 0xc000546358, 0x7f263e54170e, 0x2)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc0005356d0 sp=0xc0005356b0 pc=0x559e97095b46
runtime.chanrecv(0xc000546300, 0xc0005357b0, 0xc0005a2001, 0xc000546300)
	/usr/lib/golang/src/runtime/chan.go:525 +0x2eb fp=0xc000535760 sp=0xc0005356d0 pc=0x559e97064dbb
runtime.chanrecv2(0xc000546300, 0xc0005357b0, 0x0)
	/usr/lib/golang/src/runtime/chan.go:412 +0x2b fp=0xc000535790 sp=0xc000535760 pc=0x559e97064abb
github.com/containers/libpod/vendor/k8s.io/klog.(*loggingT).flushDaemon(0x559e99b33a80)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/k8s.io/klog/klog.go:1010 +0x8d fp=0xc0005357d8 sp=0xc000535790 pc=0x559e97bf487d
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0005357e0 sp=0xc0005357d8 pc=0x559e970c4881
created by github.com/containers/libpod/vendor/k8s.io/klog.init.0
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/k8s.io/klog/klog.go:411 +0xd8

goroutine 80 [syscall]:
syscall.Syscall6(0xe8, 0xa, 0xc00070fb74, 0x7, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/lib/golang/src/syscall/asm_linux_amd64.s:41 +0x5 fp=0xc00070faa8 sp=0xc00070faa0 pc=0x559e97118de5
github.com/containers/libpod/vendor/golang.org/x/sys/unix.EpollWait(0xa, 0xc00070fb74, 0x7, 0x7, 0xffffffffffffffff, 0x0, 0x0, 0x0)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/golang.org/x/sys/unix/zsyscall_linux_amd64.go:76 +0x74 fp=0xc00070fb18 sp=0xc00070faa8 pc=0x559e973f6b34
github.com/containers/libpod/vendor/github.com/fsnotify/fsnotify.(*fdPoller).wait(0xc0001f2aa0, 0x0, 0x0, 0x0)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/github.com/fsnotify/fsnotify/inotify_poller.go:86 +0x93 fp=0xc00070fbe0 sp=0xc00070fb18 pc=0x559e977dad23
github.com/containers/libpod/vendor/github.com/fsnotify/fsnotify.(*Watcher).readEvents(0xc00016a1e0)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/github.com/fsnotify/fsnotify/inotify.go:192 +0x1fa fp=0xc00071ffd8 sp=0xc00070fbe0 pc=0x559e977d9f1a
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc00071ffe0 sp=0xc00071ffd8 pc=0x559e970c4881
created by github.com/containers/libpod/vendor/github.com/fsnotify/fsnotify.NewWatcher
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/github.com/fsnotify/fsnotify/inotify.go:59 +0x1a7

goroutine 81 [select]:
runtime.gopark(0x559e98c879d0, 0x0, 0x1809, 0x1)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000538d40 sp=0xc000538d20 pc=0x559e97095b46
runtime.selectgo(0xc000538f38, 0xc000538eb4, 0x3, 0x0, 0x0)
	/usr/lib/golang/src/runtime/select.go:316 +0xc79 fp=0xc000538e68 sp=0xc000538d40 pc=0x559e970a6069
github.com/containers/libpod/vendor/github.com/cri-o/ocicni/pkg/ocicni.(*cniNetworkPlugin).monitorConfDir(0xc0005e20c0, 0xc00012caa0)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/github.com/cri-o/ocicni/pkg/ocicni/ocicni.go:151 +0x1a3 fp=0xc000538fd0 sp=0xc000538e68 pc=0x559e977dc073
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc000538fd8 sp=0xc000538fd0 pc=0x559e970c4881
created by github.com/containers/libpod/vendor/github.com/cri-o/ocicni/pkg/ocicni.initCNI
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/github.com/cri-o/ocicni/pkg/ocicni/ocicni.go:252 +0x3cb
/bin/toolbox: line 120: 46544 Aborted                 sudo podman container runlabel --name "$TOOLBOX_NAME" RUN "$TOOLBOX_IMAGE" 2>&1
+ echo '/bin/toolbox: failed to runlabel on image '\''docker.io/centos:latest'\'''
/bin/toolbox: failed to runlabel on image 'docker.io/centos:latest'
+ exit 1
+ cleanup
+ sudo podman stop toolbox-root
[root@openshift-master-0 ~]# 
[root@openshift-master-0 ~]# 
[root@openshift-master-0 ~]# 
[root@openshift-master-0 ~]# sudo podman container runlabel --name toolbox-root RUN docker.io/centos:latest
panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
panic(0x55f455db40a0, 0xc000240ae0)
	/usr/lib/golang/src/runtime/panic.go:1064 +0x471 fp=0xc0005d9700 sp=0xc0005d9648 pc=0x55f4542b4cf1
runtime.goPanicIndex(0x0, 0x0)
	/usr/lib/golang/src/runtime/panic.go:88 +0xa5 fp=0xc0005d9748 sp=0xc0005d9700 pc=0x55f4542b2b05
github.com/containers/libpod/pkg/domain/infra/abi.generateCommand(0x0, 0x0, 0xc000241980, 0x1f, 0x7ffc37ec5786, 0xc, 0x0, 0x0, 0x70055f455c704a0, 0xffffffffffffffff, ...)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/pkg/domain/infra/abi/containers_runlabel.go:197 +0x8c2 fp=0xc0005d9820 sp=0xc0005d9748 pc=0x55f4553f4c82
github.com/containers/libpod/pkg/domain/infra/abi.generateRunlabelCommand(0x0, 0x0, 0xc0002bc240, 0xc000030ee0, 0x0, 0x2, 0x0, 0x0, 0x0, 0x0, ...)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/pkg/domain/infra/abi/containers_runlabel.go:147 +0xcf fp=0xc0005d9978 sp=0xc0005d9820 pc=0x55f4553f3e5f
github.com/containers/libpod/pkg/domain/infra/abi.(*ContainerEngine).ContainerRunlabel(0xc00018c590, 0x55f455f16cc0, 0xc00012e020, 0x7ffc37ec5793, 0x3, 0x7ffc37ec5797, 0x17, 0xc000030ee0, 0x0, 0x2, ...)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/pkg/domain/infra/abi/containers_runlabel.go:34 +0x1b2 fp=0xc0005d9b60 sp=0xc0005d9978 pc=0x55f4553f34d2
github.com/containers/libpod/cmd/podman/containers.runlabel(0x55f456cc7940, 0xc000030ec0, 0x2, 0x4, 0x0, 0x0)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/cmd/podman/containers/runlabel.go:80 +0x169 fp=0xc0005d9c88 sp=0xc0005d9b60 pc=0x55f455484179
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).execute(0x55f456cc7940, 0xc000134030, 0x4, 0x4, 0x55f456cc7940, 0xc000134030)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:838 +0x455 fp=0xc0005d9d60 sp=0xc0005d9c88 pc=0x55f454cbc3e5
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x55f456cda720, 0xc00012e020, 0x55f455c7ece0, 0x55f456d83e90)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:943 +0x319 fp=0xc0005d9e38 sp=0xc0005d9d60 pc=0x55f454cbcee9
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).Execute(...)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:883
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).ExecuteContext(...)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:876
main.Execute()
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/cmd/podman/root.go:90 +0xee fp=0xc0005d9eb8 sp=0xc0005d9e38 pc=0x55f4555814ae
main.main()
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/cmd/podman/main.go:77 +0x18e fp=0xc0005d9f88 sp=0xc0005d9eb8 pc=0x55f455580dee
runtime.main()
	/usr/lib/golang/src/runtime/proc.go:203 +0x202 fp=0xc0005d9fe0 sp=0xc0005d9f88 pc=0x55f4542b7772
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0005d9fe8 sp=0xc0005d9fe0 pc=0x55f4542e6881

goroutine 2 [force gc (idle)]:
runtime.gopark(0x55f455ea9980, 0x55f456d50840, 0x1411, 0x1)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000066fb0 sp=0xc000066f90 pc=0x55f4542b7b46
runtime.goparkunlock(...)
	/usr/lib/golang/src/runtime/proc.go:310
runtime.forcegchelper()
	/usr/lib/golang/src/runtime/proc.go:253 +0xbb fp=0xc000066fe0 sp=0xc000066fb0 pc=0x55f4542b79eb
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc000066fe8 sp=0xc000066fe0 pc=0x55f4542e6881
created by runtime.init.7
	/usr/lib/golang/src/runtime/proc.go:242 +0x37

goroutine 3 [GC sweep wait]:
runtime.gopark(0x55f455ea9980, 0x55f456d52f00, 0x140c, 0x1)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc0000677a8 sp=0xc000067788 pc=0x55f4542b7b46
runtime.goparkunlock(...)
	/usr/lib/golang/src/runtime/proc.go:310
runtime.bgsweep(0xc00007e000)
	/usr/lib/golang/src/runtime/mgcsweep.go:89 +0x135 fp=0xc0000677d8 sp=0xc0000677a8 pc=0x55f4542a4075
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0000677e0 sp=0xc0000677d8 pc=0x55f4542e6881
created by runtime.gcenable
	/usr/lib/golang/src/runtime/mgc.go:214 +0x5e

goroutine 4 [GC scavenge wait]:
runtime.gopark(0x55f455ea9980, 0x55f456d52ec0, 0x140d, 0x1)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000067f78 sp=0xc000067f58 pc=0x55f4542b7b46
runtime.goparkunlock(...)
	/usr/lib/golang/src/runtime/proc.go:310
runtime.bgscavenge(0xc00007e000)
	/usr/lib/golang/src/runtime/mgcscavenge.go:285 +0x294 fp=0xc000067fd8 sp=0xc000067f78 pc=0x55f4542a26f4
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc000067fe0 sp=0xc000067fd8 pc=0x55f4542e6881
created by runtime.gcenable
	/usr/lib/golang/src/runtime/mgc.go:215 +0x80

goroutine 18 [finalizer wait]:
runtime.gopark(0x55f455ea9980, 0x55f456d83dd8, 0xc000191410, 0x1)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000066758 sp=0xc000066738 pc=0x55f4542b7b46
runtime.goparkunlock(...)
	/usr/lib/golang/src/runtime/proc.go:310
runtime.runfinq()
	/usr/lib/golang/src/runtime/mfinal.go:175 +0xa7 fp=0xc0000667e0 sp=0xc000066758 pc=0x55f454299b07
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0000667e8 sp=0xc0000667e0 pc=0x55f4542e6881
created by runtime.createfing
	/usr/lib/golang/src/runtime/mfinal.go:156 +0x63

goroutine 34 [GC worker (idle)]:
runtime.gopark(0x55f455ea9808, 0xc0000440a0, 0x1418, 0x0)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000062760 sp=0xc000062740 pc=0x55f4542b7b46
runtime.gcBgMarkWorker(0xc00004e000)
	/usr/lib/golang/src/runtime/mgc.go:1873 +0x105 fp=0xc0000627d8 sp=0xc000062760 pc=0x55f45429d505
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0000627e0 sp=0xc0000627d8 pc=0x55f4542e6881
created by runtime.gcBgMarkStartWorkers
	/usr/lib/golang/src/runtime/mgc.go:1821 +0x79

goroutine 5 [GC worker (idle)]:
runtime.gopark(0x55f455ea9808, 0xc000515e50, 0x1418, 0x0)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000068760 sp=0xc000068740 pc=0x55f4542b7b46
runtime.gcBgMarkWorker(0xc000050800)
	/usr/lib/golang/src/runtime/mgc.go:1873 +0x105 fp=0xc0000687d8 sp=0xc000068760 pc=0x55f45429d505
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0000687e0 sp=0xc0000687d8 pc=0x55f4542e6881
created by runtime.gcBgMarkStartWorkers
	/usr/lib/golang/src/runtime/mgc.go:1821 +0x79

goroutine 35 [GC worker (idle)]:
runtime.gopark(0x55f455ea9808, 0xc000515e60, 0x1418, 0x0)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc0001a0760 sp=0xc0001a0740 pc=0x55f4542b7b46
runtime.gcBgMarkWorker(0xc000053000)
	/usr/lib/golang/src/runtime/mgc.go:1873 +0x105 fp=0xc0001a07d8 sp=0xc0001a0760 pc=0x55f45429d505
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0001a07e0 sp=0xc0001a07d8 pc=0x55f4542e6881
created by runtime.gcBgMarkStartWorkers
	/usr/lib/golang/src/runtime/mgc.go:1821 +0x79

goroutine 20 [GC worker (idle)]:
runtime.gopark(0x55f455ea9808, 0xc0000440b0, 0x1418, 0x0)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000062f60 sp=0xc000062f40 pc=0x55f4542b7b46
runtime.gcBgMarkWorker(0xc000055800)
	/usr/lib/golang/src/runtime/mgc.go:1873 +0x105 fp=0xc000062fd8 sp=0xc000062f60 pc=0x55f45429d505
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc000062fe0 sp=0xc000062fd8 pc=0x55f4542e6881
created by runtime.gcBgMarkStartWorkers
	/usr/lib/golang/src/runtime/mgc.go:1821 +0x79

goroutine 6 [GC worker (idle)]:
runtime.gopark(0x55f455ea9808, 0xc0001a4000, 0x1418, 0x0)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000068f60 sp=0xc000068f40 pc=0x55f4542b7b46
runtime.gcBgMarkWorker(0xc000058000)
	/usr/lib/golang/src/runtime/mgc.go:1873 +0x105 fp=0xc000068fd8 sp=0xc000068f60 pc=0x55f45429d505
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc000068fe0 sp=0xc000068fd8 pc=0x55f4542e6881
created by runtime.gcBgMarkStartWorkers
	/usr/lib/golang/src/runtime/mgc.go:1821 +0x79

goroutine 36 [GC worker (idle)]:
runtime.gopark(0x55f455ea9808, 0xc000515e70, 0x1418, 0x0)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc0001a0f60 sp=0xc0001a0f40 pc=0x55f4542b7b46
runtime.gcBgMarkWorker(0xc00005a800)
	/usr/lib/golang/src/runtime/mgc.go:1873 +0x105 fp=0xc0001a0fd8 sp=0xc0001a0f60 pc=0x55f45429d505
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0001a0fe0 sp=0xc0001a0fd8 pc=0x55f4542e6881
created by runtime.gcBgMarkStartWorkers
	/usr/lib/golang/src/runtime/mgc.go:1821 +0x79

goroutine 22 [chan receive]:
runtime.gopark(0x55f455ea9780, 0xc0000ce238, 0x7fbbe277170e, 0x2)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc0000696d0 sp=0xc0000696b0 pc=0x55f4542b7b46
runtime.chanrecv(0xc0000ce1e0, 0xc0000697b0, 0xc000180501, 0xc0000ce1e0)
	/usr/lib/golang/src/runtime/chan.go:525 +0x2eb fp=0xc000069760 sp=0xc0000696d0 pc=0x55f454286dbb
runtime.chanrecv2(0xc0000ce1e0, 0xc0000697b0, 0x55f455ea9860)
	/usr/lib/golang/src/runtime/chan.go:412 +0x2b fp=0xc000069790 sp=0xc000069760 pc=0x55f454286abb
github.com/containers/libpod/vendor/k8s.io/klog.(*loggingT).flushDaemon(0x55f456d55a80)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/k8s.io/klog/klog.go:1010 +0x8d fp=0xc0000697d8 sp=0xc000069790 pc=0x55f454e1687d
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0000697e0 sp=0xc0000697d8 pc=0x55f4542e6881
created by github.com/containers/libpod/vendor/k8s.io/klog.init.0
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/k8s.io/klog/klog.go:411 +0xd8

goroutine 80 [syscall]:
syscall.Syscall6(0xe8, 0xa, 0xc00058fb74, 0x7, 0xffffffffffffffff, 0x0, 0x0, 0x383738336439623a, 0x3632613537666339, 0x6431613531393834)
	/usr/lib/golang/src/syscall/asm_linux_amd64.s:41 +0x5 fp=0xc00058faa8 sp=0xc00058faa0 pc=0x55f45433ade5
github.com/containers/libpod/vendor/golang.org/x/sys/unix.EpollWait(0xa, 0xc00058fb74, 0x7, 0x7, 0xffffffffffffffff, 0x646363383a363532, 0x3362353035623738, 0x3364353166653130)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/golang.org/x/sys/unix/zsyscall_linux_amd64.go:76 +0x74 fp=0xc00058fb18 sp=0xc00058faa8 pc=0x55f454618b34
github.com/containers/libpod/vendor/github.com/fsnotify/fsnotify.(*fdPoller).wait(0xc000240f40, 0x622d657275746100, 0x6339383738336439, 0x3834363261353766)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/github.com/fsnotify/fsnotify/inotify_poller.go:86 +0x93 fp=0xc00058fbe0 sp=0xc00058fb18 pc=0x55f4549fcd23
github.com/containers/libpod/vendor/github.com/fsnotify/fsnotify.(*Watcher).readEvents(0xc00039e230)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/github.com/fsnotify/fsnotify/inotify.go:192 +0x1fa fp=0xc00059ffd8 sp=0xc00058fbe0 pc=0x55f4549fbf1a
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc00059ffe0 sp=0xc00059ffd8 pc=0x55f4542e6881
created by github.com/containers/libpod/vendor/github.com/fsnotify/fsnotify.NewWatcher
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/github.com/fsnotify/fsnotify/inotify.go:59 +0x1a7

goroutine 81 [select]:
runtime.gopark(0x55f455ea99d0, 0x0, 0x1809, 0x1)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000485d40 sp=0xc000485d20 pc=0x55f4542b7b46
runtime.selectgo(0xc000485f38, 0xc000485eb4, 0x3, 0x10, 0x7)
	/usr/lib/golang/src/runtime/select.go:316 +0xc79 fp=0xc000485e68 sp=0xc000485d40 pc=0x55f4542c8069
github.com/containers/libpod/vendor/github.com/cri-o/ocicni/pkg/ocicni.(*cniNetworkPlugin).monitorConfDir(0xc00057e0c0, 0xc000045f80)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/github.com/cri-o/ocicni/pkg/ocicni/ocicni.go:151 +0x1a3 fp=0xc000485fd0 sp=0xc000485e68 pc=0x55f4549fe073
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc000485fd8 sp=0xc000485fd0 pc=0x55f4542e6881
created by github.com/containers/libpod/vendor/github.com/cri-o/ocicni/pkg/ocicni.initCNI
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/github.com/cri-o/ocicni/pkg/ocicni/ocicni.go:252 +0x3cb
Aborted
[root@openshift-master-0 ~]# podman ps -a
CONTAINER ID  IMAGE   COMMAND  CREATED  STATUS  PORTS   NAMES
[root@openshift-master-0 ~]# sudo podman container runlabel --name toolbox-root RUN docker.io/centos:latest
panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
panic(0x561874be70a0, 0xc000368ca0)
	/usr/lib/golang/src/runtime/panic.go:1064 +0x471 fp=0xc0003f1700 sp=0xc0003f1648 pc=0x5618730e7cf1
runtime.goPanicIndex(0x0, 0x0)
	/usr/lib/golang/src/runtime/panic.go:88 +0xa5 fp=0xc0003f1748 sp=0xc0003f1700 pc=0x5618730e5b05
github.com/containers/libpod/pkg/domain/infra/abi.generateCommand(0x0, 0x0, 0xc00011c4e0, 0x1f, 0x7ffdef11d786, 0xc, 0x0, 0x0, 0x700561874aa34a0, 0xffffffffffffffff, ...)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/pkg/domain/infra/abi/containers_runlabel.go:197 +0x8c2 fp=0xc0003f1820 sp=0xc0003f1748 pc=0x561874227c82
github.com/containers/libpod/pkg/domain/infra/abi.generateRunlabelCommand(0x0, 0x0, 0xc000126240, 0xc00049f4e0, 0x0, 0x2, 0x0, 0x0, 0x0, 0x0, ...)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/pkg/domain/infra/abi/containers_runlabel.go:147 +0xcf fp=0xc0003f1978 sp=0xc0003f1820 pc=0x561874226e5f
github.com/containers/libpod/pkg/domain/infra/abi.(*ContainerEngine).ContainerRunlabel(0xc000010390, 0x561874d49cc0, 0xc000044098, 0x7ffdef11d793, 0x3, 0x7ffdef11d797, 0x17, 0xc00049f4e0, 0x0, 0x2, ...)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/pkg/domain/infra/abi/containers_runlabel.go:34 +0x1b2 fp=0xc0003f1b60 sp=0xc0003f1978 pc=0x5618742264d2
github.com/containers/libpod/cmd/podman/containers.runlabel(0x561875afa940, 0xc00049f4c0, 0x2, 0x4, 0x0, 0x0)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/cmd/podman/containers/runlabel.go:80 +0x169 fp=0xc0003f1c88 sp=0xc0003f1b60 pc=0x5618742b7179
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).execute(0x561875afa940, 0xc00003c0a0, 0x4, 0x4, 0x561875afa940, 0xc00003c0a0)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:838 +0x455 fp=0xc0003f1d60 sp=0xc0003f1c88 pc=0x561873aef3e5
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x561875b0d720, 0xc000044098, 0x561874ab1ce0, 0x561875bb6e90)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:943 +0x319 fp=0xc0003f1e38 sp=0xc0003f1d60 pc=0x561873aefee9
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).Execute(...)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:883
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).ExecuteContext(...)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:876
main.Execute()
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/cmd/podman/root.go:90 +0xee fp=0xc0003f1eb8 sp=0xc0003f1e38 pc=0x5618743b44ae
main.main()
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/cmd/podman/main.go:77 +0x18e fp=0xc0003f1f88 sp=0xc0003f1eb8 pc=0x5618743b3dee
runtime.main()
	/usr/lib/golang/src/runtime/proc.go:203 +0x202 fp=0xc0003f1fe0 sp=0xc0003f1f88 pc=0x5618730ea772
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0003f1fe8 sp=0xc0003f1fe0 pc=0x561873119881

goroutine 2 [force gc (idle)]:
runtime.gopark(0x561874cdc980, 0x561875b83840, 0x1411, 0x1)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000066fb0 sp=0xc000066f90 pc=0x5618730eab46
runtime.goparkunlock(...)
	/usr/lib/golang/src/runtime/proc.go:310
runtime.forcegchelper()
	/usr/lib/golang/src/runtime/proc.go:253 +0xbb fp=0xc000066fe0 sp=0xc000066fb0 pc=0x5618730ea9eb
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc000066fe8 sp=0xc000066fe0 pc=0x561873119881
created by runtime.init.7
	/usr/lib/golang/src/runtime/proc.go:242 +0x37

goroutine 3 [GC sweep wait]:
runtime.gopark(0x561874cdc980, 0x561875b85f00, 0x140c, 0x1)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc0000677a8 sp=0xc000067788 pc=0x5618730eab46
runtime.goparkunlock(...)
	/usr/lib/golang/src/runtime/proc.go:310
runtime.bgsweep(0xc00007e000)
	/usr/lib/golang/src/runtime/mgcsweep.go:89 +0x135 fp=0xc0000677d8 sp=0xc0000677a8 pc=0x5618730d7075
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0000677e0 sp=0xc0000677d8 pc=0x561873119881
created by runtime.gcenable
	/usr/lib/golang/src/runtime/mgc.go:214 +0x5e

goroutine 4 [GC scavenge wait]:
runtime.gopark(0x561874cdc980, 0x561875b85ec0, 0x140d, 0x1)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000067f78 sp=0xc000067f58 pc=0x5618730eab46
runtime.goparkunlock(...)
	/usr/lib/golang/src/runtime/proc.go:310
runtime.bgscavenge(0xc00007e000)
	/usr/lib/golang/src/runtime/mgcscavenge.go:285 +0x294 fp=0xc000067fd8 sp=0xc000067f78 pc=0x5618730d56f4
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc000067fe0 sp=0xc000067fd8 pc=0x561873119881
created by runtime.gcenable
	/usr/lib/golang/src/runtime/mgc.go:215 +0x80

goroutine 5 [finalizer wait]:
runtime.gopark(0x561874cdc980, 0x561875bb6dd8, 0xc0001f1410, 0x1)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000066758 sp=0xc000066738 pc=0x5618730eab46
runtime.goparkunlock(...)
	/usr/lib/golang/src/runtime/proc.go:310
runtime.runfinq()
	/usr/lib/golang/src/runtime/mfinal.go:175 +0xa7 fp=0xc0000667e0 sp=0xc000066758 pc=0x5618730ccb07
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0000667e8 sp=0xc0000667e0 pc=0x561873119881
created by runtime.createfing
	/usr/lib/golang/src/runtime/mfinal.go:156 +0x63

goroutine 19 [GC worker (idle)]:
runtime.gopark(0x561874cdc808, 0xc00013c030, 0x1418, 0x0)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000068760 sp=0xc000068740 pc=0x5618730eab46
runtime.gcBgMarkWorker(0xc00004e000)
	/usr/lib/golang/src/runtime/mgc.go:1873 +0x105 fp=0xc0000687d8 sp=0xc000068760 pc=0x5618730d0505
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0000687e0 sp=0xc0000687d8 pc=0x561873119881
created by runtime.gcBgMarkStartWorkers
	/usr/lib/golang/src/runtime/mgc.go:1821 +0x79

goroutine 20 [GC worker (idle)]:
runtime.gopark(0x561874cdc808, 0xc00013c040, 0x1418, 0x0)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000062760 sp=0xc000062740 pc=0x5618730eab46
runtime.gcBgMarkWorker(0xc000050800)
	/usr/lib/golang/src/runtime/mgc.go:1873 +0x105 fp=0xc0000627d8 sp=0xc000062760 pc=0x5618730d0505
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0000627e0 sp=0xc0000627d8 pc=0x561873119881
created by runtime.gcBgMarkStartWorkers
	/usr/lib/golang/src/runtime/mgc.go:1821 +0x79

goroutine 21 [GC worker (idle)]:
runtime.gopark(0x561874cdc808, 0xc00013c050, 0x1418, 0x0)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000062f60 sp=0xc000062f40 pc=0x5618730eab46
runtime.gcBgMarkWorker(0xc000053000)
	/usr/lib/golang/src/runtime/mgc.go:1873 +0x105 fp=0xc000062fd8 sp=0xc000062f60 pc=0x5618730d0505
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc000062fe0 sp=0xc000062fd8 pc=0x561873119881
created by runtime.gcBgMarkStartWorkers
	/usr/lib/golang/src/runtime/mgc.go:1821 +0x79

goroutine 22 [GC worker (idle)]:
runtime.gopark(0x561874cdc808, 0xc00013c060, 0x1418, 0x0)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000063760 sp=0xc000063740 pc=0x5618730eab46
runtime.gcBgMarkWorker(0xc000055800)
	/usr/lib/golang/src/runtime/mgc.go:1873 +0x105 fp=0xc0000637d8 sp=0xc000063760 pc=0x5618730d0505
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0000637e0 sp=0xc0000637d8 pc=0x561873119881
created by runtime.gcBgMarkStartWorkers
	/usr/lib/golang/src/runtime/mgc.go:1821 +0x79

goroutine 23 [GC worker (idle)]:
runtime.gopark(0x561874cdc808, 0xc00013c070, 0x1418, 0x0)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000063f60 sp=0xc000063f40 pc=0x5618730eab46
runtime.gcBgMarkWorker(0xc000058000)
	/usr/lib/golang/src/runtime/mgc.go:1873 +0x105 fp=0xc000063fd8 sp=0xc000063f60 pc=0x5618730d0505
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc000063fe0 sp=0xc000063fd8 pc=0x561873119881
created by runtime.gcBgMarkStartWorkers
	/usr/lib/golang/src/runtime/mgc.go:1821 +0x79

goroutine 24 [GC worker (idle)]:
runtime.gopark(0x561874cdc808, 0xc00050e000, 0x1418, 0x0)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000064760 sp=0xc000064740 pc=0x5618730eab46
runtime.gcBgMarkWorker(0xc00005a800)
	/usr/lib/golang/src/runtime/mgc.go:1873 +0x105 fp=0xc0000647d8 sp=0xc000064760 pc=0x5618730d0505
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0000647e0 sp=0xc0000647d8 pc=0x561873119881
created by runtime.gcBgMarkStartWorkers
	/usr/lib/golang/src/runtime/mgc.go:1821 +0x79

goroutine 38 [chan receive]:
runtime.gopark(0x561874cdc780, 0xc00062ed78, 0x7f4f9458170e, 0x2)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc0005a06d0 sp=0xc0005a06b0 pc=0x5618730eab46
runtime.chanrecv(0xc00062ed20, 0xc0005a07b0, 0xc00012a001, 0xc00062ed20)
	/usr/lib/golang/src/runtime/chan.go:525 +0x2eb fp=0xc0005a0760 sp=0xc0005a06d0 pc=0x5618730b9dbb
runtime.chanrecv2(0xc00062ed20, 0xc0005a07b0, 0x561874cdc860)
	/usr/lib/golang/src/runtime/chan.go:412 +0x2b fp=0xc0005a0790 sp=0xc0005a0760 pc=0x5618730b9abb
github.com/containers/libpod/vendor/k8s.io/klog.(*loggingT).flushDaemon(0x561875b88a80)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/k8s.io/klog/klog.go:1010 +0x8d fp=0xc0005a07d8 sp=0xc0005a0790 pc=0x561873c4987d
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0005a07e0 sp=0xc0005a07d8 pc=0x561873119881
created by github.com/containers/libpod/vendor/k8s.io/klog.init.0
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/k8s.io/klog/klog.go:411 +0xd8

goroutine 67 [syscall]:
syscall.Syscall6(0xe8, 0xa, 0xc0005ddb74, 0x7, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/lib/golang/src/syscall/asm_linux_amd64.s:41 +0x5 fp=0xc0005ddaa8 sp=0xc0005ddaa0 pc=0x56187316dde5
github.com/containers/libpod/vendor/golang.org/x/sys/unix.EpollWait(0xa, 0xc0005ddb74, 0x7, 0x7, 0xffffffffffffffff, 0x0, 0x0, 0x0)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/golang.org/x/sys/unix/zsyscall_linux_amd64.go:76 +0x74 fp=0xc0005ddb18 sp=0xc0005ddaa8 pc=0x56187344bb34
github.com/containers/libpod/vendor/github.com/fsnotify/fsnotify.(*fdPoller).wait(0xc00039b0a0, 0x0, 0x0, 0x0)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/github.com/fsnotify/fsnotify/inotify_poller.go:86 +0x93 fp=0xc0005ddbe0 sp=0xc0005ddb18 pc=0x56187382fd23
github.com/containers/libpod/vendor/github.com/fsnotify/fsnotify.(*Watcher).readEvents(0xc0001601e0)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/github.com/fsnotify/fsnotify/inotify.go:192 +0x1fa fp=0xc0005edfd8 sp=0xc0005ddbe0 pc=0x56187382ef1a
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0005edfe0 sp=0xc0005edfd8 pc=0x561873119881
created by github.com/containers/libpod/vendor/github.com/fsnotify/fsnotify.NewWatcher
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/github.com/fsnotify/fsnotify/inotify.go:59 +0x1a7

goroutine 68 [select]:
runtime.gopark(0x561874cdc9d0, 0x0, 0x1809, 0x1)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000214540 sp=0xc000214520 pc=0x5618730eab46
runtime.selectgo(0xc000214738, 0xc0002146b4, 0x3, 0x6f772f3435643139, 0x6b72)
	/usr/lib/golang/src/runtime/select.go:316 +0xc79 fp=0xc000214668 sp=0xc000214540 pc=0x5618730fb069
github.com/containers/libpod/vendor/github.com/cri-o/ocicni/pkg/ocicni.(*cniNetworkPlugin).monitorConfDir(0xc0005b60c0, 0xc00013caa0)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/github.com/cri-o/ocicni/pkg/ocicni/ocicni.go:151 +0x1a3 fp=0xc0002147d0 sp=0xc000214668 pc=0x561873831073
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0002147d8 sp=0xc0002147d0 pc=0x561873119881
created by github.com/containers/libpod/vendor/github.com/cri-o/ocicni/pkg/ocicni.initCNI
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/github.com/cri-o/ocicni/pkg/ocicni/ocicni.go:252 +0x3cb
Aborted
[root@openshift-master-0 ~]# 
[root@openshift-master-0 ~]# 
[root@openshift-master-0 ~]# 
[root@openshift-master-0 ~]# sudo podman container runlabel --name toolbox-root docker.io/centos:latest
Error: requires at least 2 arg(s), only received 1
[root@openshift-master-0 ~]# sudo podman container runlabel --name toolbox-root docker.io/centos:latest /bin/bash
Error: invalid reference format
[root@openshift-master-0 ~]# sudo podman run --name toolbox-root docker.io/centos:latest /bin/bash
[root@openshift-master-0 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux CoreOS release 4.7
[root@openshift-master-0 ~]# podman ps
CONTAINER ID  IMAGE   COMMAND  CREATED  STATUS  PORTS   NAMES
[root@openshift-master-0 ~]# podman ps -a
CONTAINER ID  IMAGE                            COMMAND    CREATED         STATUS                     PORTS   NAMES
b25e64938e20  docker.io/library/centos:latest  /bin/bash  16 seconds ago  Exited (0) 16 seconds ago          toolbox-root
[root@openshift-master-0 ~]# sudo podman run -it --name toolbox-root docker.io/centos:latest /bin/bash
Error: error creating container storage: the container name "toolbox-root" is already in use by "b25e64938e20521b6b94a16d9d6d0a87df955930f163551e5ff620c707f505d2". You have to remove that container to be able to reuse that name.: that name is already in use
[root@openshift-master-0 ~]# podman rm toolbox-root
b25e64938e20521b6b94a16d9d6d0a87df955930f163551e5ff620c707f505d2
[root@openshift-master-0 ~]# 
[root@openshift-master-0 ~]# 
[root@openshift-master-0 ~]# 
[root@openshift-master-0 ~]# 
[root@openshift-master-0 ~]# 
[root@openshift-master-0 ~]# 
[root@openshift-master-0 ~]# 
[root@openshift-master-0 ~]# 
[root@openshift-master-0 ~]# 
[root@openshift-master-0 ~]# 
[root@openshift-master-0 ~]# 
[root@openshift-master-0 ~]# 
[root@openshift-master-0 ~]# 
[root@openshift-master-0 ~]# 
[root@openshift-master-0 ~]# 
[root@openshift-master-0 ~]# 
[root@openshift-master-0 ~]# 
[root@openshift-master-0 ~]# 
[root@openshift-master-0 ~]# 
[root@openshift-master-0 ~]# 
[root@openshift-master-0 ~]# 
[root@openshift-master-0 ~]# bash -x /bin/toolbox
+ set -eo pipefail
+ trap cleanup EXIT
+ REGISTRY=registry.redhat.io
+ IMAGE=rhel8/support-tools
++ whoami
+ TOOLBOX_NAME=toolbox-root
+ TOOLBOXRC=/root/.toolboxrc
+ main
+ setup
+ '[' -f /root/.toolboxrc ']'
+ echo '.toolboxrc file detected, overriding defaults...'
.toolboxrc file detected, overriding defaults...
+ source /root/.toolboxrc
++ REGISTRY=docker.io
++ IMAGE=centos:latest
+ TOOLBOX_IMAGE=docker.io/centos:latest
+ [[ '' =~ ^(--help|-h)$ ]]
+ run
+ image_exists
+ sudo podman inspect docker.io/centos:latest
++ image_runlabel
++ sudo podman image inspect docker.io/centos:latest --format '{{.Labels.run}}'
+ local 'runlabel=<no value>'
+ container_exists
+ sudo podman inspect toolbox-root
+ echo 'Spawning a container '\''toolbox-root'\'' with image '\''docker.io/centos:latest'\'''
Spawning a container 'toolbox-root' with image 'docker.io/centos:latest'
+ [[ -z <no value> ]]
+ echo 'Detected RUN label in the container image. Using that as the default...'
Detected RUN label in the container image. Using that as the default...
+ container_runlabel
+ sudo podman container runlabel --name toolbox-root RUN docker.io/centos:latest
panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
panic(0x55b53c0950a0, 0xc000369000)
	/usr/lib/golang/src/runtime/panic.go:1064 +0x471 fp=0xc000407700 sp=0xc000407648 pc=0x55b53a595cf1
runtime.goPanicIndex(0x0, 0x0)
	/usr/lib/golang/src/runtime/panic.go:88 +0xa5 fp=0xc000407748 sp=0xc000407700 pc=0x55b53a593b05
github.com/containers/libpod/pkg/domain/infra/abi.generateCommand(0x0, 0x0, 0xc0001ac5a0, 0x1f, 0x7ffee4479786, 0xc, 0x0, 0x0, 0x40055b53bf514a0, 0xffffffffffffffff, ...)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/pkg/domain/infra/abi/containers_runlabel.go:197 +0x8c2 fp=0xc000407820 sp=0xc000407748 pc=0x55b53b6d5c82
github.com/containers/libpod/pkg/domain/infra/abi.generateRunlabelCommand(0x0, 0x0, 0xc0000fc240, 0xc000022c60, 0x0, 0x2, 0x0, 0x0, 0x0, 0x0, ...)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/pkg/domain/infra/abi/containers_runlabel.go:147 +0xcf fp=0xc000407978 sp=0xc000407820 pc=0x55b53b6d4e5f
github.com/containers/libpod/pkg/domain/infra/abi.(*ContainerEngine).ContainerRunlabel(0xc0000104f8, 0x55b53c1f7cc0, 0xc000044098, 0x7ffee4479793, 0x3, 0x7ffee4479797, 0x17, 0xc000022c60, 0x0, 0x2, ...)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/pkg/domain/infra/abi/containers_runlabel.go:34 +0x1b2 fp=0xc000407b60 sp=0xc000407978 pc=0x55b53b6d44d2
github.com/containers/libpod/cmd/podman/containers.runlabel(0x55b53cfa8940, 0xc000022c40, 0x2, 0x4, 0x0, 0x0)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/cmd/podman/containers/runlabel.go:80 +0x169 fp=0xc000407c88 sp=0xc000407b60 pc=0x55b53b765179
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).execute(0x55b53cfa8940, 0xc00003c0a0, 0x4, 0x4, 0x55b53cfa8940, 0xc00003c0a0)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:838 +0x455 fp=0xc000407d60 sp=0xc000407c88 pc=0x55b53af9d3e5
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x55b53cfbb720, 0xc000044098, 0x55b53bf5fce0, 0x55b53d064e90)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:943 +0x319 fp=0xc000407e38 sp=0xc000407d60 pc=0x55b53af9dee9
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).Execute(...)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:883
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).ExecuteContext(...)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:876
main.Execute()
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/cmd/podman/root.go:90 +0xee fp=0xc000407eb8 sp=0xc000407e38 pc=0x55b53b8624ae
main.main()
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/cmd/podman/main.go:77 +0x18e fp=0xc000407f88 sp=0xc000407eb8 pc=0x55b53b861dee
runtime.main()
	/usr/lib/golang/src/runtime/proc.go:203 +0x202 fp=0xc000407fe0 sp=0xc000407f88 pc=0x55b53a598772
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc000407fe8 sp=0xc000407fe0 pc=0x55b53a5c7881

goroutine 2 [force gc (idle)]:
runtime.gopark(0x55b53c18a980, 0x55b53d031840, 0x1411, 0x1)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000066fb0 sp=0xc000066f90 pc=0x55b53a598b46
runtime.goparkunlock(...)
	/usr/lib/golang/src/runtime/proc.go:310
runtime.forcegchelper()
	/usr/lib/golang/src/runtime/proc.go:253 +0xbb fp=0xc000066fe0 sp=0xc000066fb0 pc=0x55b53a5989eb
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc000066fe8 sp=0xc000066fe0 pc=0x55b53a5c7881
created by runtime.init.7
	/usr/lib/golang/src/runtime/proc.go:242 +0x37

goroutine 3 [GC sweep wait]:
runtime.gopark(0x55b53c18a980, 0x55b53d033f00, 0x140c, 0x1)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc0000677a8 sp=0xc000067788 pc=0x55b53a598b46
runtime.goparkunlock(...)
	/usr/lib/golang/src/runtime/proc.go:310
runtime.bgsweep(0xc00007e000)
	/usr/lib/golang/src/runtime/mgcsweep.go:89 +0x135 fp=0xc0000677d8 sp=0xc0000677a8 pc=0x55b53a585075
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0000677e0 sp=0xc0000677d8 pc=0x55b53a5c7881
created by runtime.gcenable
	/usr/lib/golang/src/runtime/mgc.go:214 +0x5e

goroutine 4 [GC scavenge wait]:
runtime.gopark(0x55b53c18a980, 0x55b53d033ec0, 0x140d, 0x1)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000067f78 sp=0xc000067f58 pc=0x55b53a598b46
runtime.goparkunlock(...)
	/usr/lib/golang/src/runtime/proc.go:310
runtime.bgscavenge(0xc00007e000)
	/usr/lib/golang/src/runtime/mgcscavenge.go:285 +0x294 fp=0xc000067fd8 sp=0xc000067f78 pc=0x55b53a5836f4
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc000067fe0 sp=0xc000067fd8 pc=0x55b53a5c7881
created by runtime.gcenable
	/usr/lib/golang/src/runtime/mgc.go:215 +0x80

goroutine 5 [finalizer wait]:
runtime.gopark(0x55b53c18a980, 0x55b53d064dd8, 0xc000391410, 0x1)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000066758 sp=0xc000066738 pc=0x55b53a598b46
runtime.goparkunlock(...)
	/usr/lib/golang/src/runtime/proc.go:310
runtime.runfinq()
	/usr/lib/golang/src/runtime/mfinal.go:175 +0xa7 fp=0xc0000667e0 sp=0xc000066758 pc=0x55b53a57ab07
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0000667e8 sp=0xc0000667e0 pc=0x55b53a5c7881
created by runtime.createfing
	/usr/lib/golang/src/runtime/mfinal.go:156 +0x63

goroutine 34 [GC worker (idle)]:
runtime.gopark(0x55b53c18a808, 0xc000487ef0, 0x1418, 0x0)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000068760 sp=0xc000068740 pc=0x55b53a598b46
runtime.gcBgMarkWorker(0xc00004e000)
	/usr/lib/golang/src/runtime/mgc.go:1873 +0x105 fp=0xc0000687d8 sp=0xc000068760 pc=0x55b53a57e505
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0000687e0 sp=0xc0000687d8 pc=0x55b53a5c7881
created by runtime.gcBgMarkStartWorkers
	/usr/lib/golang/src/runtime/mgc.go:1821 +0x79

goroutine 7 [GC worker (idle)]:
runtime.gopark(0x55b53c18a808, 0xc00012c000, 0x1418, 0x0)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000068f60 sp=0xc000068f40 pc=0x55b53a598b46
runtime.gcBgMarkWorker(0xc000050800)
	/usr/lib/golang/src/runtime/mgc.go:1873 +0x105 fp=0xc000068fd8 sp=0xc000068f60 pc=0x55b53a57e505
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc000068fe0 sp=0xc000068fd8 pc=0x55b53a5c7881
created by runtime.gcBgMarkStartWorkers
	/usr/lib/golang/src/runtime/mgc.go:1821 +0x79

goroutine 19 [GC worker (idle)]:
runtime.gopark(0x55b53c18a808, 0xc000518000, 0x1418, 0x0)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000062760 sp=0xc000062740 pc=0x55b53a598b46
runtime.gcBgMarkWorker(0xc000053000)
	/usr/lib/golang/src/runtime/mgc.go:1873 +0x105 fp=0xc0000627d8 sp=0xc000062760 pc=0x55b53a57e505
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0000627e0 sp=0xc0000627d8 pc=0x55b53a5c7881
created by runtime.gcBgMarkStartWorkers
	/usr/lib/golang/src/runtime/mgc.go:1821 +0x79

goroutine 35 [GC worker (idle)]:
runtime.gopark(0x55b53c18a808, 0xc000487f00, 0x1418, 0x0)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000520760 sp=0xc000520740 pc=0x55b53a598b46
runtime.gcBgMarkWorker(0xc000055800)
	/usr/lib/golang/src/runtime/mgc.go:1873 +0x105 fp=0xc0005207d8 sp=0xc000520760 pc=0x55b53a57e505
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0005207e0 sp=0xc0005207d8 pc=0x55b53a5c7881
created by runtime.gcBgMarkStartWorkers
	/usr/lib/golang/src/runtime/mgc.go:1821 +0x79

goroutine 20 [GC worker (idle)]:
runtime.gopark(0x55b53c18a808, 0xc000487f10, 0x1418, 0x0)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000062f60 sp=0xc000062f40 pc=0x55b53a598b46
runtime.gcBgMarkWorker(0xc000058000)
	/usr/lib/golang/src/runtime/mgc.go:1873 +0x105 fp=0xc000062fd8 sp=0xc000062f60 pc=0x55b53a57e505
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc000062fe0 sp=0xc000062fd8 pc=0x55b53a5c7881
created by runtime.gcBgMarkStartWorkers
	/usr/lib/golang/src/runtime/mgc.go:1821 +0x79

goroutine 8 [GC worker (idle)]:
runtime.gopark(0x55b53c18a808, 0xc000518010, 0x1418, 0x0)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000069760 sp=0xc000069740 pc=0x55b53a598b46
runtime.gcBgMarkWorker(0xc00005a800)
	/usr/lib/golang/src/runtime/mgc.go:1873 +0x105 fp=0xc0000697d8 sp=0xc000069760 pc=0x55b53a57e505
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0000697e0 sp=0xc0000697d8 pc=0x55b53a5c7881
created by runtime.gcBgMarkStartWorkers
	/usr/lib/golang/src/runtime/mgc.go:1821 +0x79

goroutine 39 [chan receive]:
runtime.gopark(0x55b53c18a780, 0xc000129a38, 0x7fe1a09b170e, 0x2)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000523ed0 sp=0xc000523eb0 pc=0x55b53a598b46
runtime.chanrecv(0xc0001299e0, 0xc000523fb0, 0xc00012a001, 0xc0001299e0)
	/usr/lib/golang/src/runtime/chan.go:525 +0x2eb fp=0xc000523f60 sp=0xc000523ed0 pc=0x55b53a567dbb
runtime.chanrecv2(0xc0001299e0, 0xc000523fb0, 0x55b53c18a860)
	/usr/lib/golang/src/runtime/chan.go:412 +0x2b fp=0xc000523f90 sp=0xc000523f60 pc=0x55b53a567abb
github.com/containers/libpod/vendor/k8s.io/klog.(*loggingT).flushDaemon(0x55b53d036a80)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/k8s.io/klog/klog.go:1010 +0x8d fp=0xc000523fd8 sp=0xc000523f90 pc=0x55b53b0f787d
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc000523fe0 sp=0xc000523fd8 pc=0x55b53a5c7881
created by github.com/containers/libpod/vendor/k8s.io/klog.init.0
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/k8s.io/klog/klog.go:411 +0xd8

goroutine 82 [syscall]:
syscall.Syscall6(0xe8, 0xa, 0xc00068fb74, 0x7, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/lib/golang/src/syscall/asm_linux_amd64.s:41 +0x5 fp=0xc00068faa8 sp=0xc00068faa0 pc=0x55b53a61bde5
github.com/containers/libpod/vendor/golang.org/x/sys/unix.EpollWait(0xa, 0xc00068fb74, 0x7, 0x7, 0xffffffffffffffff, 0x0, 0x0, 0x0)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/golang.org/x/sys/unix/zsyscall_linux_amd64.go:76 +0x74 fp=0xc00068fb18 sp=0xc00068faa8 pc=0x55b53a8f9b34
github.com/containers/libpod/vendor/github.com/fsnotify/fsnotify.(*fdPoller).wait(0xc000369420, 0x0, 0x0, 0x0)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/github.com/fsnotify/fsnotify/inotify_poller.go:86 +0x93 fp=0xc00068fbe0 sp=0xc00068fb18 pc=0x55b53acddd23
github.com/containers/libpod/vendor/github.com/fsnotify/fsnotify.(*Watcher).readEvents(0xc0002321e0)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/github.com/fsnotify/fsnotify/inotify.go:192 +0x1fa fp=0xc00069ffd8 sp=0xc00068fbe0 pc=0x55b53acdcf1a
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc00069ffe0 sp=0xc00069ffd8 pc=0x55b53a5c7881
created by github.com/containers/libpod/vendor/github.com/fsnotify/fsnotify.NewWatcher
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/github.com/fsnotify/fsnotify/inotify.go:59 +0x1a7

goroutine 83 [select]:
runtime.gopark(0x55b53c18a9d0, 0x0, 0x1809, 0x1)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000149d40 sp=0xc000149d20 pc=0x55b53a598b46
runtime.selectgo(0xc000149f38, 0xc000149eb4, 0x3, 0x0, 0x0)
	/usr/lib/golang/src/runtime/select.go:316 +0xc79 fp=0xc000149e68 sp=0xc000149d40 pc=0x55b53a5a9069
github.com/containers/libpod/vendor/github.com/cri-o/ocicni/pkg/ocicni.(*cniNetworkPlugin).monitorConfDir(0xc0005480c0, 0xc00039f130)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/github.com/cri-o/ocicni/pkg/ocicni/ocicni.go:151 +0x1a3 fp=0xc000149fd0 sp=0xc000149e68 pc=0x55b53acdf073
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc000149fd8 sp=0xc000149fd0 pc=0x55b53a5c7881
created by github.com/containers/libpod/vendor/github.com/cri-o/ocicni/pkg/ocicni.initCNI
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/github.com/cri-o/ocicni/pkg/ocicni/ocicni.go:252 +0x3cb
/bin/toolbox: line 120: 49720 Aborted                 sudo podman container runlabel --name "$TOOLBOX_NAME" RUN "$TOOLBOX_IMAGE" 2>&1
+ echo '/bin/toolbox: failed to runlabel on image '\''docker.io/centos:latest'\'''
/bin/toolbox: failed to runlabel on image 'docker.io/centos:latest'
+ exit 1
+ cleanup
+ sudo podman stop toolbox-root
Comment 6 Alex Jia 2021-02-01 14:17:39 UTC 
(In reply to Andreas Karis from comment #5)
> [root@openshift-master-0 ~]# bash -x /bin/toolbox

The latest toolbox-0.0.99-1 is implemented by golang, and it doens't support
.toolboxrc configuration any more, I think we have different upstream toolbox 
implementation between RHEL(https://github.com/containers/toolbox) and 
CoreOS(https://github.com/coreos/toolbox), please correct me if I'm wrong.

[root@kvm-03-guest13 ~]# bash -x `which toolbox`
/usr/bin/toolbox: /usr/bin/toolbox: cannot execute binary file

[root@kvm-03-guest13 ~]# file `which toolbox`
/usr/bin/toolbox: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=530b0d021644e562f8514ab07144f0a8cefb7d42, stripped

[root@kvm-03-guest13 ~]# rpm -q toolbox podman
toolbox-0.0.99-1.module+el8.4.0+9742+44abad1f.x86_64
podman-3.0.0-0.33rc2.module+el8.4.0+9742+44abad1f.x86_64

[root@kvm-03-guest13 ~]# cat ~/.toolboxrc
IMAGE=libpod/alpine
REGISTRY=quay.io

[root@kvm-03-guest13 ~]# toolbox create
Image required to create toolbox container.
Download registry.access.redhat.com/ubi8/ubi:8.4 (500MB)? [y/N]: y
Error: failed to pull image registry.access.redhat.com/ubi8/ubi:8.4

> + set -eo pipefail
> + trap cleanup EXIT
> + REGISTRY=registry.redhat.io
> + IMAGE=rhel8/support-tools
> ++ whoami
> + TOOLBOX_NAME=toolbox-root
> + TOOLBOXRC=/root/.toolboxrc
> + main
> + setup
> + '[' -f /root/.toolboxrc ']'
> + echo '.toolboxrc file detected, overriding defaults...'
> .toolboxrc file detected, overriding defaults...
> + source /root/.toolboxrc
> ++ REGISTRY=docker.io
> ++ IMAGE=centos:latest
> + TOOLBOX_IMAGE=docker.io/centos:latest
> + [[ '' =~ ^(--help|-h)$ ]]
> + run
> + image_exists
> + sudo podman inspect docker.io/centos:latest
> ++ image_runlabel
> ++ sudo podman image inspect docker.io/centos:latest --format
> '{{.Labels.run}}'
> + local 'runlabel=<no value>'
> + container_exists
> + sudo podman inspect toolbox-root
> + echo 'Spawning a container '\''toolbox-root'\'' with image
> '\''docker.io/centos:latest'\'''
> Spawning a container 'toolbox-root' with image 'docker.io/centos:latest'
> + [[ -z <no value> ]]
> + echo 'Detected RUN label in the container image. Using that as the
> default...'
> Detected RUN label in the container image. Using that as the default...
> + container_runlabel
> + sudo podman container runlabel --name toolbox-root RUN
> docker.io/centos:latest
> panic: runtime error: index out of range [0] with length 0
> 

It should be a known bug 1867987.
Comment 7 Andreas Karis 2021-02-02 12:34:55 UTC 
@Alex sorry, I think I posted that output to the wrong bug. Thanks for your answer!
Comment 8 Debarshi Ray 2021-02-03 12:20:31 UTC 
Umm... can't this bug be closed now?
Comment 9 Alex Jia 2021-02-09 03:44:47 UTC 
(In reply to Debarshi Ray from comment #8)
> Umm... can't this bug be closed now?

The default ubi8/ubi:8.4 image doesn't exist on the registry.access.redhat.com.

[root@ibm-x3650m4-01-vm-05 ~]# toolbox create
Image required to create toolbox container.
Download registry.access.redhat.com/ubi8/ubi:8.4 (500MB)? [y/N]: y
Error: failed to pull image registry.access.redhat.com/ubi8/ubi:8.4
Comment 10 Alex Jia 2021-02-09 04:26:23 UTC 
Please also see bug 1926567.
Comment 11 Debarshi Ray 2021-02-10 16:53:43 UTC 
(In reply to Alex Jia from comment #9)
> (In reply to Debarshi Ray from comment #8)
> > Umm... can't this bug be closed now?
> 
> The default ubi8/ubi:8.4 image doesn't exist on the
> registry.access.redhat.com.

That's because the public UBI image at registry.access.redhat.com/ubi8/ubi doesn't contain an image for RHEL 8.4. It should appear once RHEL 8.4 is released. See comment 2.

I don't know what to do about this, other than ask Release Engineering whether they have some special options for us.
Comment 12 Alex Jia 2021-02-22 10:52:40 UTC 
(In reply to Debarshi Ray from comment #11)

> That's because the public UBI image at registry.access.redhat.com/ubi8/ubi
> doesn't contain an image for RHEL 8.4. It should appear once RHEL 8.4 is
> released. See comment 2.

I gave sanity tests for toolbox-0.0.99-1.module+el8.3.1+9940+bdcdabd6
on RHEL 8.3, it worked well for me. In addition, I also gave tests for
toolbox-0.0.99-1.module+el8.4.0+10073+30e5ea69.x86_64 w/ specified image
on RHEL 8.4, it also worked for me.

1. on RHEL 8.3

[root@ibm-x3650m4-01-vm-09 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.3 (Ootpa)

[root@ibm-x3650m4-01-vm-09 ~]# rpm -q toolbox podman runc crun
toolbox-0.0.99-1.module+el8.3.1+9940+bdcdabd6.x86_64
podman-3.0.0-1.module+el8.3.1+9940+bdcdabd6.x86_64
runc-1.0.0-71.rc92.module+el8.3.1+9940+bdcdabd6.x86_64
crun-0.17-1.module+el8.3.1+9940+bdcdabd6.x86_64

[root@ibm-x3650m4-01-vm-09 ~]# toolbox list
[root@ibm-x3650m4-01-vm-09 ~]# toolbox create
Image required to create toolbox container.
Download registry.access.redhat.com/ubi8/ubi:8.3 (500MB)? [y/N]: y
Created container: rhel-toolbox-8.3
Enter with: toolbox enter

[root@ibm-x3650m4-01-vm-09 ~]# toolbox list
CONTAINER ID  CONTAINER NAME    CREATED                 STATUS      IMAGE NAME
534878de7088  rhel-toolbox-8.3  Less than a second ago  configured  registry.access.redhat.com/ubi8/ubi:8.3

[root@ibm-x3650m4-01-vm-09 ~]# toolbox list --containers
CONTAINER ID  CONTAINER NAME    CREATED                 STATUS      IMAGE NAME
534878de7088  rhel-toolbox-8.3  Less than a second ago  configured  registry.access.redhat.com/ubi8/ubi:8.3

[root@ibm-x3650m4-01-vm-09 ~]# toolbox list -c
CONTAINER ID  CONTAINER NAME    CREATED                 STATUS      IMAGE NAME
534878de7088  rhel-toolbox-8.3  Less than a second ago  configured  registry.access.redhat.com/ubi8/ubi:8.3

[root@ibm-x3650m4-01-vm-09 ~]# toolbox list --images
[root@ibm-x3650m4-01-vm-09 ~]# toolbox list -i
[root@ibm-x3650m4-01-vm-09 ~]# podman ps
CONTAINER ID  IMAGE   COMMAND  CREATED  STATUS  PORTS   NAMES
[root@ibm-x3650m4-01-vm-09 ~]# podman ps -a
CONTAINER ID  IMAGE                                    COMMAND               CREATED         STATUS   PORTS   NAMES
534878de7088  registry.access.redhat.com/ubi8/ubi:8.3  toolbox --verbose...  53 seconds ago  Created          rhel-toolbox-8.3
[root@ibm-x3650m4-01-vm-09 ~]# toolbox enter

Welcome to the Toolbox; a container where you can install and run
all your tools.

 - Use DNF in the usual manner to install command line tools.
 - To create a new tools container, run 'toolbox create'.

For more information, see the documentation.

⬢[root@toolbox ~]# dnf install -y wget
Failed to set locale, defaulting to C.UTF-8
Updating Subscription Management repositories.
Unable to read consumer identity
Subscription Manager is operating in container mode.

This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.

Red Hat Universal Base Image 8 (RPMs) - BaseOS                                                                                                                                                                592 kB/s | 775 kB     00:01    
Red Hat Universal Base Image 8 (RPMs) - AppStream                                                                                                                                                             3.8 MB/s | 5.1 MB     00:01    
Red Hat Universal Base Image 8 (RPMs) - CodeReady Builder                                                                                                                                                      18 kB/s |  13 kB     00:00    
Dependencies resolved.
==============================================================================================================================================================================================================================================
 Package                                             Architecture                                          Version                                                       Repository                                                      Size
==============================================================================================================================================================================================================================================
Installing:
 wget                                                x86_64                                                1.19.5-10.el8                                                 ubi-8-appstream                                                734 k

Transaction Summary
==============================================================================================================================================================================================================================================
Install  1 Package

Total download size: 734 k
Installed size: 2.8 M
Downloading Packages:
wget-1.19.5-10.el8.x86_64.rpm                                                                                                                                                                                 953 kB/s | 734 kB     00:00    
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                         948 kB/s | 734 kB     00:00     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                                                      1/1 
  Installing       : wget-1.19.5-10.el8.x86_64                                                                                                                                                                                            1/1 
  Running scriptlet: wget-1.19.5-10.el8.x86_64                                                                                                                                                                                            1/1 
  Verifying        : wget-1.19.5-10.el8.x86_64                                                                                                                                                                                            1/1 
Installed products updated.

Installed:
  wget-1.19.5-10.el8.x86_64                                                                                                                                                                                                                   

Complete!
⬢[root@toolbox ~]# rpm -q wget
wget-1.19.5-10.el8.x86_64
⬢[root@toolbox ~]# exit
logout

[root@ibm-x3650m4-01-vm-09 ~]# toolbox list
CONTAINER ID  CONTAINER NAME    CREATED                 STATUS   IMAGE NAME
534878de7088  rhel-toolbox-8.3  Less than a second ago  running  registry.access.redhat.com/ubi8/ubi:8.3

[root@ibm-x3650m4-01-vm-09 ~]# podman ps
CONTAINER ID  IMAGE                                    COMMAND               CREATED        STATUS                 PORTS   NAMES
534878de7088  registry.access.redhat.com/ubi8/ubi:8.3  toolbox --verbose...  2 minutes ago  Up About a minute ago          rhel-toolbox-8.3

2. on RHEL 8.4

[root@ibm-x3650m4-01-vm-02 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.4 Beta (Ootpa)

[root@ibm-x3650m4-01-vm-02 ~]# rpm -q toolbox podman crun runc 
toolbox-0.0.99-1.module+el8.4.0+10073+30e5ea69.x86_64
podman-3.0.1-1.module+el8.4.0+10073+30e5ea69.x86_64
crun-0.18-1.module+el8.4.0+10073+30e5ea69.x86_64
runc-1.0.0-70.rc92.module+el8.4.0+10073+30e5ea69.x86_64

[root@ibm-x3650m4-01-vm-02 ~]# toolbox create --image registry.redhat.io/rhel8/support-tools
Image required to create toolbox container.
Download registry.redhat.io/rhel8/support-tools (500MB)? [y/N]: y
Created container: support-tools
Enter with: toolbox enter support-tools
[root@ibm-x3650m4-01-vm-02 ~]# toolbox list
CONTAINER ID  CONTAINER NAME    CREATED         STATUS      IMAGE NAME
f38ef038f74c  support-tools     25 seconds ago  configured  registry.redhat.io/rhel8/support-tools

[root@ibm-x3650m4-01-vm-02 ~]# toolbox enter support-tools
⬢[root@toolbox ~]# rpm -q telnet sysstat mtr strace
telnet-0.17-74.el8_3.1.x86_64
package sysstat is not installed
mtr-0.92-3.el8.x86_64
strace-5.1-1.el8.x86_64

⬢[root@toolbox ~]# exit
logout
[root@ibm-x3650m4-01-vm-02 ~]# toolbox list
CONTAINER ID  CONTAINER NAME    CREATED         STATUS   IMAGE NAME
f38ef038f74c  support-tools     4 minutes ago   running  registry.redhat.io/rhel8/support-tools
Comment 13 Debarshi Ray 2021-02-24 21:55:37 UTC 
That looks like the tests passed, right?
Comment 14 Debarshi Ray 2021-02-24 21:56:57 UTC 
That looks like the tests passed, right?
Comment 16 Alex Jia 2021-03-16 18:26:00 UTC 
(In reply to Debarshi Ray from comment #13)
> That looks like the tests passed, right?

Verified codes logic of 'toolbox create' on RHEL 8.3, the default ubi:8.3 image
can be pulled successfully, I assume it's also okay for ubi:8.4 image on RHEL 8.4,
and I will give another try after the ubi:8.4 image is released. so I tend to 
close this bug as NOTABUG.