RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1867987 - panic: runtime error: index out of range [0] with length 0
Summary: panic: runtime error: index out of range [0] with length 0
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: toolbox
Version: 8.3
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: rc
: 8.0
Assignee: Jindrich Novy
QA Contact: Alex Jia
URL:
Whiteboard:
Depends On:
Blocks: 1924020
TreeView+ depends on / blocked
 
Reported: 2020-08-11 10:37 UTC by Alex Jia
Modified: 2021-02-05 07:05 UTC (History)
4 users (show)

Fixed In Version: toolbox-0.0.99-1.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1924020 (view as bug list)
Environment:
Last Closed: 2021-02-05 07:05:10 UTC
Type: Bug
Target Upstream Version:
Embargoed:
pm-rhel: mirror+

Attachments (Terms of Use)

Description Alex Jia 2020-08-11 10:37:36 UTC 
Description of problem:
Got error like this 'panic: runtime error: index out of range [0] with length 0' when ran toolbox w/ .toolboxrc configuration.

Version-Release number of selected component (if applicable):

[root@hpe-dl380pgen8-02-vm-3 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux release 8.3 Beta (Ootpa)

[root@hpe-dl380pgen8-02-vm-3 ~]# rpm -q toolbox
toolbox-0.0.8-1.module+el8.3.0+7529+55b23fb9.noarch

How reproducible:
always

Steps to Reproduce:
1. cat << EOF > ~/.toolboxrc
IMAGE=fedora:latest
REGISTRY=docker.io
EOF
2. toolbox

Actual results:

[root@hpe-dl380pgen8-02-vm-3 ~]# toolbox
.toolboxrc file detected, overriding defaults...
panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
panic(0x55c3e236b4e0, 0xc0001ac7a0)
        /usr/lib/golang/src/runtime/panic.go:1064 +0x471 fp=0xc000549700 sp=0xc000549648 pc=0x55c3e087c381
runtime.goPanicIndex(0x0, 0x0)
        /usr/lib/golang/src/runtime/panic.go:88 +0xa5 fp=0xc000549748 sp=0xc000549700 pc=0x55c3e087a195
github.com/containers/libpod/pkg/domain/infra/abi.generateCommand(0x0, 0x0, 0xc00036cb80, 0x1f, 0xc0001ac792, 0x6, 0x0, 0x0, 0x55c3e22284e0, 0xffffffffffffffff, ...)
        /builddir/build/BUILD/podman-2.0.4/_build/src/github.com/containers/libpod/pkg/domain/infra/abi/containers_runlabel.go:197 +0x8c2 fp=0xc000549820 sp=0xc000549748 pc=0x55c3e19b6ed2
github.com/containers/libpod/pkg/domain/infra/abi.generateRunlabelCommand(0x0, 0x0, 0xc0000c5680, 0xc000443880, 0x0, 0x1, 0x0, 0x0, 0x0, 0x0, ...)
        /builddir/build/BUILD/podman-2.0.4/_build/src/github.com/containers/libpod/pkg/domain/infra/abi/containers_runlabel.go:147 +0xcf fp=0xc000549978 sp=0xc000549820 pc=0x55c3e19b60af
github.com/containers/libpod/pkg/domain/infra/abi.(*ContainerEngine).ContainerRunlabel(0xc000010878, 0x55c3e24cda40, 0xc000128020, 0x7ffd25b8f77d, 0x3, 0x7ffd25b8f781, 0x17, 0xc000443880, 0x0, 0x1, ...)
        /builddir/build/BUILD/podman-2.0.4/_build/src/github.com/containers/libpod/pkg/domain/infra/abi/containers_runlabel.go:34 +0x1b2 fp=0xc000549b60 sp=0xc000549978 pc=0x55c3e19b5722
github.com/containers/libpod/cmd/podman/containers.runlabel(0x55c3e32668c0, 0xc000443860, 0x2, 0x3, 0x0, 0x0)
        /builddir/build/BUILD/podman-2.0.4/_build/src/github.com/containers/libpod/cmd/podman/containers/runlabel.go:80 +0x169 fp=0xc000549c88 sp=0xc000549b60 pc=0x55c3e1a46629
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).execute(0x55c3e32668c0, 0xc00012c150, 0x3, 0x3, 0x55c3e32668c0, 0xc00012c150)
        /builddir/build/BUILD/podman-2.0.4/_build/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:838 +0x455 fp=0xc000549d60 sp=0xc000549c88 pc=0x55c3e1283085
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x55c3e32786e0, 0xc000128020, 0x55c3e2236d20, 0x55c3e3321cb0)
        /builddir/build/BUILD/podman-2.0.4/_build/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:943 +0x319 fp=0xc000549e38 sp=0xc000549d60 pc=0x55c3e1283b89
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).Execute(...)
        /builddir/build/BUILD/podman-2.0.4/_build/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:883
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).ExecuteContext(...)
        /builddir/build/BUILD/podman-2.0.4/_build/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:876
main.Execute()
        /builddir/build/BUILD/podman-2.0.4/_build/src/github.com/containers/libpod/cmd/podman/root.go:91 +0xee fp=0xc000549eb8 sp=0xc000549e38 pc=0x55c3e1b3e70e
main.main()
        /builddir/build/BUILD/podman-2.0.4/_build/src/github.com/containers/libpod/cmd/podman/main.go:68 +0x18e fp=0xc000549f88 sp=0xc000549eb8 pc=0x55c3e1b3e12e
runtime.main()
        /usr/lib/golang/src/runtime/proc.go:203 +0x202 fp=0xc000549fe0 sp=0xc000549f88 pc=0x55c3e087ee02
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc000549fe8 sp=0xc000549fe0 pc=0x55c3e08adf11

goroutine 2 [force gc (idle)]:
runtime.gopark(0x55c3e24609f0, 0x55c3e32ee690, 0x1411, 0x1)
        /usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc00005efb0 sp=0xc00005ef90 pc=0x55c3e087f1d6
runtime.goparkunlock(...)
        /usr/lib/golang/src/runtime/proc.go:310
runtime.forcegchelper()
        /usr/lib/golang/src/runtime/proc.go:253 +0xbb fp=0xc00005efe0 sp=0xc00005efb0 pc=0x55c3e087f07b
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc00005efe8 sp=0xc00005efe0 pc=0x55c3e08adf11
created by runtime.init.7
        /usr/lib/golang/src/runtime/proc.go:242 +0x37

goroutine 3 [GC sweep wait]:
runtime.gopark(0x55c3e24609f0, 0x55c3e32f0d60, 0x140c, 0x1)
        /usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc00005f7a8 sp=0xc00005f788 pc=0x55c3e087f1d6
runtime.goparkunlock(...)
        /usr/lib/golang/src/runtime/proc.go:310
runtime.bgsweep(0xc00007c000)
        /usr/lib/golang/src/runtime/mgcsweep.go:89 +0x135 fp=0xc00005f7d8 sp=0xc00005f7a8 pc=0x55c3e086b705
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc00005f7e0 sp=0xc00005f7d8 pc=0x55c3e08adf11
created by runtime.gcenable
        /usr/lib/golang/src/runtime/mgc.go:214 +0x5e

goroutine 4 [sleep]:
runtime.gopark(0x55c3e24609f0, 0x55c3e32f0d20, 0x1313, 0x2)
        /usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc00005ff28 sp=0xc00005ff08 pc=0x55c3e087f1d6
runtime.goparkunlock(...)
        /usr/lib/golang/src/runtime/proc.go:310
runtime.scavengeSleep(0xb48e4, 0x212eb2)
        /usr/lib/golang/src/runtime/mgcscavenge.go:214 +0xc1 fp=0xc00005ff78 sp=0xc00005ff28 pc=0x55c3e0869ac1
runtime.bgscavenge(0xc00007c000)
        /usr/lib/golang/src/runtime/mgcscavenge.go:337 +0x1df fp=0xc00005ffd8 sp=0xc00005ff78 pc=0x55c3e0869ccf
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc00005ffe0 sp=0xc00005ffd8 pc=0x55c3e08adf11
created by runtime.gcenable
        /usr/lib/golang/src/runtime/mgc.go:215 +0x80

goroutine 18 [finalizer wait]:
runtime.gopark(0x55c3e24609f0, 0x55c3e3321bf8, 0x1410, 0x1)
        /usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc00005e758 sp=0xc00005e738 pc=0x55c3e087f1d6
runtime.goparkunlock(...)
        /usr/lib/golang/src/runtime/proc.go:310
runtime.runfinq()
        /usr/lib/golang/src/runtime/mfinal.go:175 +0xa7 fp=0xc00005e7e0 sp=0xc00005e758 pc=0x55c3e0861197
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc00005e7e8 sp=0xc00005e7e0 pc=0x55c3e08adf11
created by runtime.createfing
        /usr/lib/golang/src/runtime/mfinal.go:156 +0x63

goroutine 21 [GC worker (idle)]:
runtime.gopark(0x55c3e2460878, 0xc00048be80, 0x1418, 0x0)
        /usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc00005a760 sp=0xc00005a740 pc=0x55c3e087f1d6
runtime.gcBgMarkWorker(0xc000050000)
        /usr/lib/golang/src/runtime/mgc.go:1873 +0x105 fp=0xc00005a7d8 sp=0xc00005a760 pc=0x55c3e0864b95
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc00005a7e0 sp=0xc00005a7d8 pc=0x55c3e08adf11
created by runtime.gcBgMarkStartWorkers
        /usr/lib/golang/src/runtime/mgc.go:1821 +0x79

goroutine 22 [GC worker (idle)]:
runtime.gopark(0x55c3e2460878, 0xc00048be90, 0x1418, 0x0)
        /usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc00005af60 sp=0xc00005af40 pc=0x55c3e087f1d6
runtime.gcBgMarkWorker(0xc000052800)
        /usr/lib/golang/src/runtime/mgc.go:1873 +0x105 fp=0xc00005afd8 sp=0xc00005af60 pc=0x55c3e0864b95
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc00005afe0 sp=0xc00005afd8 pc=0x55c3e08adf11
created by runtime.gcBgMarkStartWorkers
        /usr/lib/golang/src/runtime/mgc.go:1821 +0x79

goroutine 10 [chan receive]:
runtime.gopark(0x55c3e24607f0, 0xc0005380b8, 0x7fa62809170e, 0x2)
        /usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000061ed0 sp=0xc000061eb0 pc=0x55c3e087f1d6
runtime.chanrecv(0xc000538060, 0xc000061fb0, 0xc0004ce101, 0xc000538060)
        /usr/lib/golang/src/runtime/chan.go:525 +0x2eb fp=0xc000061f60 sp=0xc000061ed0 pc=0x55c3e084e44b
runtime.chanrecv2(0xc000538060, 0xc000061fb0, 0x55c3e24608d0)
        /usr/lib/golang/src/runtime/chan.go:412 +0x2b fp=0xc000061f90 sp=0xc000061f60 pc=0x55c3e084e14b
github.com/containers/libpod/vendor/k8s.io/klog.(*loggingT).flushDaemon(0x55c3e32f38a0)
        /builddir/build/BUILD/podman-2.0.4/_build/src/github.com/containers/libpod/vendor/k8s.io/klog/klog.go:1010 +0x8d fp=0xc000061fd8 sp=0xc000061f90 pc=0x55c3e13dc62d
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc000061fe0 sp=0xc000061fd8 pc=0x55c3e08adf11
created by github.com/containers/libpod/vendor/k8s.io/klog.init.0
        /builddir/build/BUILD/podman-2.0.4/_build/src/github.com/containers/libpod/vendor/k8s.io/klog/klog.go:411 +0xd8

goroutine 41 [syscall]:
syscall.Syscall6(0xe8, 0xa, 0xc00058fb74, 0x7, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/lib/golang/src/syscall/asm_linux_amd64.s:41 +0x5 fp=0xc00058faa8 sp=0xc00058faa0 pc=0x55c3e0902365
github.com/containers/libpod/vendor/golang.org/x/sys/unix.EpollWait(0xa, 0xc00058fb74, 0x7, 0x7, 0xffffffffffffffff, 0x0, 0x0, 0x0)
        /builddir/build/BUILD/podman-2.0.4/_build/src/github.com/containers/libpod/vendor/golang.org/x/sys/unix/zsyscall_linux_amd64.go:76 +0x74 fp=0xc00058fb18 sp=0xc00058faa8 pc=0x55c3e0bdfdc4
github.com/containers/libpod/vendor/github.com/fsnotify/fsnotify.(*fdPoller).wait(0xc00039ddc0, 0x0, 0xc0000d37a0, 0x97)
        /builddir/build/BUILD/podman-2.0.4/_build/src/github.com/containers/libpod/vendor/github.com/fsnotify/fsnotify/inotify_poller.go:86 +0x93 fp=0xc00058fbe0 sp=0xc00058fb18 pc=0x55c3e0fc38f3
github.com/containers/libpod/vendor/github.com/fsnotify/fsnotify.(*Watcher).readEvents(0xc000246f50)
        /builddir/build/BUILD/podman-2.0.4/_build/src/github.com/containers/libpod/vendor/github.com/fsnotify/fsnotify/inotify.go:192 +0x1fa fp=0xc00059ffd8 sp=0xc00058fbe0 pc=0x55c3e0fc2aea
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc00059ffe0 sp=0xc00059ffd8 pc=0x55c3e08adf11
created by github.com/containers/libpod/vendor/github.com/fsnotify/fsnotify.NewWatcher
        /builddir/build/BUILD/podman-2.0.4/_build/src/github.com/containers/libpod/vendor/github.com/fsnotify/fsnotify/inotify.go:59 +0x1a7

goroutine 42 [select]:
runtime.gopark(0x55c3e2460a40, 0x0, 0x1809, 0x1)
        /usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc0000bdd40 sp=0xc0000bdd20 pc=0x55c3e087f1d6
runtime.selectgo(0xc0000bdf38, 0xc0000bdeb4, 0x3, 0x0, 0x0)
        /usr/lib/golang/src/runtime/select.go:316 +0xc79 fp=0xc0000bde68 sp=0xc0000bdd40 pc=0x55c3e088f6f9
github.com/containers/libpod/vendor/github.com/cri-o/ocicni/pkg/ocicni.(*cniNetworkPlugin).monitorConfDir(0xc0000c20c0, 0xc000508c60)
        /builddir/build/BUILD/podman-2.0.4/_build/src/github.com/containers/libpod/vendor/github.com/cri-o/ocicni/pkg/ocicni/ocicni.go:151 +0x1a3 fp=0xc0000bdfd0 sp=0xc0000bde68 pc=0x55c3e0fc4c43
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0000bdfd8 sp=0xc0000bdfd0 pc=0x55c3e08adf11
created by github.com/containers/libpod/vendor/github.com/cri-o/ocicni/pkg/ocicni.initCNI
        /builddir/build/BUILD/podman-2.0.4/_build/src/github.com/containers/libpod/vendor/github.com/cri-o/ocicni/pkg/ocicni/ocicni.go:252 +0x3cb
Spawning a container 'toolbox-root' with image 'docker.io/fedora:latest'
[root@toolbox /]# ls
bin  boot  dev  etc  home  host  lib  lib64  lost+found  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var

Expected results:
fix it

Additional info:
Comment 5 Alex Jia 2021-01-19 08:23:10 UTC 
Please also see https://bugzilla.redhat.com/show_bug.cgi?id=1917663
Comment 6 Andreas Karis 2021-01-30 17:34:17 UTC 
I hit the same in RHCOS 4.7:

[root@openshift-master-0 ~]# rpm -qa | grep toolbox
toolbox-0.0.8-2.rhaos4.7.el8.noarch

[root@openshift-master-0 ~]# cat ~/.toolboxrc 
REGISTRY=docker.io
IMAGE=centos:latest

[root@openshift-master-0 ~]# toolbox
.toolboxrc file detected, overriding defaults...
Spawning a container 'toolbox-root' with image 'docker.io/centos:latest'
Detected RUN label in the container image. Using that as the default...
panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
panic(0x56048dd710a0, 0xc00028ebe0)
	/usr/lib/golang/src/runtime/panic.go:1064 +0x471 fp=0xc0002ad700 sp=0xc0002ad648 pc=0x56048c271cf1
runtime.goPanicIndex(0x0, 0x0)
	/usr/lib/golang/src/runtime/panic.go:88 +0xa5 fp=0xc0002ad748 sp=0xc0002ad700 pc=0x56048c26fb05
github.com/containers/libpod/pkg/domain/infra/abi.generateCommand(0x0, 0x0, 0xc00028f3e0, 0x1f, 0x7ffdecd47786, 0xc, 0x0, 0x0, 0x40056048dc2d4a0, 0xffffffffffffffff, ...)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/pkg/domain/infra/abi/containers_runlabel.go:197 +0x8c2 fp=0xc0002ad820 sp=0xc0002ad748 pc=0x56048d3b1c82
github.com/containers/libpod/pkg/domain/infra/abi.generateRunlabelCommand(0x0, 0x0, 0xc0001ce000, 0xc0005d7420, 0x0, 0x2, 0x0, 0x0, 0x0, 0x0, ...)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/pkg/domain/infra/abi/containers_runlabel.go:147 +0xcf fp=0xc0002ad978 sp=0xc0002ad820 pc=0x56048d3b0e5f
github.com/containers/libpod/pkg/domain/infra/abi.(*ContainerEngine).ContainerRunlabel(0xc0005b6780, 0x56048ded3cc0, 0xc000044098, 0x7ffdecd47793, 0x3, 0x7ffdecd47797, 0x17, 0xc0005d7420, 0x0, 0x2, ...)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/pkg/domain/infra/abi/containers_runlabel.go:34 +0x1b2 fp=0xc0002adb60 sp=0xc0002ad978 pc=0x56048d3b04d2
github.com/containers/libpod/cmd/podman/containers.runlabel(0x56048ec84940, 0xc0005d7400, 0x2, 0x4, 0x0, 0x0)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/cmd/podman/containers/runlabel.go:80 +0x169 fp=0xc0002adc88 sp=0xc0002adb60 pc=0x56048d441179
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).execute(0x56048ec84940, 0xc00003c0a0, 0x4, 0x4, 0x56048ec84940, 0xc00003c0a0)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:838 +0x455 fp=0xc0002add60 sp=0xc0002adc88 pc=0x56048cc793e5
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x56048ec97720, 0xc000044098, 0x56048dc3bce0, 0x56048ed40e90)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:943 +0x319 fp=0xc0002ade38 sp=0xc0002add60 pc=0x56048cc79ee9
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).Execute(...)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:883
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).ExecuteContext(...)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:876
main.Execute()
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/cmd/podman/root.go:90 +0xee fp=0xc0002adeb8 sp=0xc0002ade38 pc=0x56048d53e4ae
main.main()
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/cmd/podman/main.go:77 +0x18e fp=0xc0002adf88 sp=0xc0002adeb8 pc=0x56048d53ddee
runtime.main()
	/usr/lib/golang/src/runtime/proc.go:203 +0x202 fp=0xc0002adfe0 sp=0xc0002adf88 pc=0x56048c274772
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0002adfe8 sp=0xc0002adfe0 pc=0x56048c2a3881

goroutine 2 [force gc (idle)]:
runtime.gopark(0x56048de66980, 0x56048ed0d840, 0x1411, 0x1)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000066fb0 sp=0xc000066f90 pc=0x56048c274b46
runtime.goparkunlock(...)
	/usr/lib/golang/src/runtime/proc.go:310
runtime.forcegchelper()
	/usr/lib/golang/src/runtime/proc.go:253 +0xbb fp=0xc000066fe0 sp=0xc000066fb0 pc=0x56048c2749eb
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc000066fe8 sp=0xc000066fe0 pc=0x56048c2a3881
created by runtime.init.7
	/usr/lib/golang/src/runtime/proc.go:242 +0x37

goroutine 3 [GC sweep wait]:
runtime.gopark(0x56048de66980, 0x56048ed0ff00, 0x140c, 0x1)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc0000677a8 sp=0xc000067788 pc=0x56048c274b46
runtime.goparkunlock(...)
	/usr/lib/golang/src/runtime/proc.go:310
runtime.bgsweep(0xc00007e000)
	/usr/lib/golang/src/runtime/mgcsweep.go:89 +0x135 fp=0xc0000677d8 sp=0xc0000677a8 pc=0x56048c261075
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0000677e0 sp=0xc0000677d8 pc=0x56048c2a3881
created by runtime.gcenable
	/usr/lib/golang/src/runtime/mgc.go:214 +0x5e

goroutine 4 [GC scavenge wait]:
runtime.gopark(0x56048de66980, 0x56048ed0fec0, 0x140d, 0x1)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000067f78 sp=0xc000067f58 pc=0x56048c274b46
runtime.goparkunlock(...)
	/usr/lib/golang/src/runtime/proc.go:310
runtime.bgscavenge(0xc00007e000)
	/usr/lib/golang/src/runtime/mgcscavenge.go:285 +0x294 fp=0xc000067fd8 sp=0xc000067f78 pc=0x56048c25f6f4
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc000067fe0 sp=0xc000067fd8 pc=0x56048c2a3881
created by runtime.gcenable
	/usr/lib/golang/src/runtime/mgc.go:215 +0x80

goroutine 5 [finalizer wait]:
runtime.gopark(0x56048de66980, 0x56048ed40dd8, 0xc0001b1410, 0x1)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000066758 sp=0xc000066738 pc=0x56048c274b46
runtime.goparkunlock(...)
	/usr/lib/golang/src/runtime/proc.go:310
runtime.runfinq()
	/usr/lib/golang/src/runtime/mfinal.go:175 +0xa7 fp=0xc0000667e0 sp=0xc000066758 pc=0x56048c256b07
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0000667e8 sp=0xc0000667e0 pc=0x56048c2a3881
created by runtime.createfing
	/usr/lib/golang/src/runtime/mfinal.go:156 +0x63

goroutine 18 [GC worker (idle)]:
runtime.gopark(0x56048de66808, 0xc000487ef0, 0x1418, 0x0)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000068760 sp=0xc000068740 pc=0x56048c274b46
runtime.gcBgMarkWorker(0xc00004e000)
	/usr/lib/golang/src/runtime/mgc.go:1873 +0x105 fp=0xc0000687d8 sp=0xc000068760 pc=0x56048c25a505
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0000687e0 sp=0xc0000687d8 pc=0x56048c2a3881
created by runtime.gcBgMarkStartWorkers
	/usr/lib/golang/src/runtime/mgc.go:1821 +0x79

goroutine 8 [GC worker (idle)]:
runtime.gopark(0x56048de66808, 0xc00011a000, 0x1418, 0x0)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000068f60 sp=0xc000068f40 pc=0x56048c274b46
runtime.gcBgMarkWorker(0xc000050800)
	/usr/lib/golang/src/runtime/mgc.go:1873 +0x105 fp=0xc000068fd8 sp=0xc000068f60 pc=0x56048c25a505
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc000068fe0 sp=0xc000068fd8 pc=0x56048c2a3881
created by runtime.gcBgMarkStartWorkers
	/usr/lib/golang/src/runtime/mgc.go:1821 +0x79

goroutine 34 [GC worker (idle)]:
runtime.gopark(0x56048de66808, 0xc000520010, 0x1418, 0x0)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000062760 sp=0xc000062740 pc=0x56048c274b46
runtime.gcBgMarkWorker(0xc000053000)
	/usr/lib/golang/src/runtime/mgc.go:1873 +0x105 fp=0xc0000627d8 sp=0xc000062760 pc=0x56048c25a505
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0000627e0 sp=0xc0000627d8 pc=0x56048c2a3881
created by runtime.gcBgMarkStartWorkers
	/usr/lib/golang/src/runtime/mgc.go:1821 +0x79

goroutine 9 [GC worker (idle)]:
runtime.gopark(0x56048de66808, 0xc000520020, 0x1418, 0x0)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000069760 sp=0xc000069740 pc=0x56048c274b46
runtime.gcBgMarkWorker(0xc000055800)
	/usr/lib/golang/src/runtime/mgc.go:1873 +0x105 fp=0xc0000697d8 sp=0xc000069760 pc=0x56048c25a505
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0000697e0 sp=0xc0000697d8 pc=0x56048c2a3881
created by runtime.gcBgMarkStartWorkers
	/usr/lib/golang/src/runtime/mgc.go:1821 +0x79

goroutine 19 [GC worker (idle)]:
runtime.gopark(0x56048de66808, 0xc00011a010, 0x1418, 0x0)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000528760 sp=0xc000528740 pc=0x56048c274b46
runtime.gcBgMarkWorker(0xc000058000)
	/usr/lib/golang/src/runtime/mgc.go:1873 +0x105 fp=0xc0005287d8 sp=0xc000528760 pc=0x56048c25a505
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0005287e0 sp=0xc0005287d8 pc=0x56048c2a3881
created by runtime.gcBgMarkStartWorkers
	/usr/lib/golang/src/runtime/mgc.go:1821 +0x79

goroutine 10 [GC worker (idle)]:
runtime.gopark(0x56048de66808, 0xc00011a020, 0x1418, 0x0)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000069f60 sp=0xc000069f40 pc=0x56048c274b46
runtime.gcBgMarkWorker(0xc00005a800)
	/usr/lib/golang/src/runtime/mgc.go:1873 +0x105 fp=0xc000069fd8 sp=0xc000069f60 pc=0x56048c25a505
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc000069fe0 sp=0xc000069fd8 pc=0x56048c2a3881
created by runtime.gcBgMarkStartWorkers
	/usr/lib/golang/src/runtime/mgc.go:1821 +0x79

goroutine 55 [chan receive]:
runtime.gopark(0x56048de66780, 0xc000297df8, 0x7f2a162b170e, 0x2)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc0005cfed0 sp=0xc0005cfeb0 pc=0x56048c274b46
runtime.chanrecv(0xc000297da0, 0xc0005cffb0, 0xc00051c001, 0xc000297da0)
	/usr/lib/golang/src/runtime/chan.go:525 +0x2eb fp=0xc0005cff60 sp=0xc0005cfed0 pc=0x56048c243dbb
runtime.chanrecv2(0xc000297da0, 0xc0005cffb0, 0x0)
	/usr/lib/golang/src/runtime/chan.go:412 +0x2b fp=0xc0005cff90 sp=0xc0005cff60 pc=0x56048c243abb
github.com/containers/libpod/vendor/k8s.io/klog.(*loggingT).flushDaemon(0x56048ed12a80)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/k8s.io/klog/klog.go:1010 +0x8d fp=0xc0005cffd8 sp=0xc0005cff90 pc=0x56048cdd387d
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0005cffe0 sp=0xc0005cffd8 pc=0x56048c2a3881
created by github.com/containers/libpod/vendor/k8s.io/klog.init.0
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/k8s.io/klog/klog.go:411 +0xd8

goroutine 84 [syscall]:
syscall.Syscall6(0xe8, 0xa, 0xc00063fb74, 0x7, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/lib/golang/src/syscall/asm_linux_amd64.s:41 +0x5 fp=0xc00063faa8 sp=0xc00063faa0 pc=0x56048c2f7de5
github.com/containers/libpod/vendor/golang.org/x/sys/unix.EpollWait(0xa, 0xc00063fb74, 0x7, 0x7, 0xffffffffffffffff, 0x0, 0x0, 0x0)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/golang.org/x/sys/unix/zsyscall_linux_amd64.go:76 +0x74 fp=0xc00063fb18 sp=0xc00063faa8 pc=0x56048c5d5b34
github.com/containers/libpod/vendor/github.com/fsnotify/fsnotify.(*fdPoller).wait(0xc00028ec80, 0x0, 0x0, 0x0)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/github.com/fsnotify/fsnotify/inotify_poller.go:86 +0x93 fp=0xc00063fbe0 sp=0xc00063fb18 pc=0x56048c9b9d23
github.com/containers/libpod/vendor/github.com/fsnotify/fsnotify.(*Watcher).readEvents(0xc0001a01e0)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/github.com/fsnotify/fsnotify/inotify.go:192 +0x1fa fp=0xc00064ffd8 sp=0xc00063fbe0 pc=0x56048c9b8f1a
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc00064ffe0 sp=0xc00064ffd8 pc=0x56048c2a3881
created by github.com/containers/libpod/vendor/github.com/fsnotify/fsnotify.NewWatcher
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/github.com/fsnotify/fsnotify/inotify.go:59 +0x1a7

goroutine 85 [select]:
runtime.gopark(0x56048de669d0, 0x0, 0x1809, 0x1)
	/usr/lib/golang/src/runtime/proc.go:304 +0xe6 fp=0xc000525d40 sp=0xc000525d20 pc=0x56048c274b46
runtime.selectgo(0xc000525f38, 0xc000525eb4, 0x3, 0x0, 0x0)
	/usr/lib/golang/src/runtime/select.go:316 +0xc79 fp=0xc000525e68 sp=0xc000525d40 pc=0x56048c285069
github.com/containers/libpod/vendor/github.com/cri-o/ocicni/pkg/ocicni.(*cniNetworkPlugin).monitorConfDir(0xc0005e60c0, 0xc0004f6910)
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/github.com/cri-o/ocicni/pkg/ocicni/ocicni.go:151 +0x1a3 fp=0xc000525fd0 sp=0xc000525e68 pc=0x56048c9bb073
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc000525fd8 sp=0xc000525fd0 pc=0x56048c2a3881
created by github.com/containers/libpod/vendor/github.com/cri-o/ocicni/pkg/ocicni.initCNI
	/builddir/build/BUILD/podman-f5c92373ec5e9d118cf6c098d1ae1b770e5055ae/_build/src/github.com/containers/libpod/vendor/github.com/cri-o/ocicni/pkg/ocicni/ocicni.go:252 +0x3cb
/bin/toolbox: line 120: 45278 Aborted                 sudo podman container runlabel --name "$TOOLBOX_NAME" RUN "$TOOLBOX_IMAGE" 2>&1
/bin/toolbox: failed to runlabel on image 'docker.io/centos:latest'
[root@openshift-master-0 ~]#
Comment 7 Andreas Karis 2021-01-30 17:38:05 UTC 
And that looks like: https://github.com/containers/podman/issues/8038
Comment 8 Andreas Karis 2021-01-30 17:50:09 UTC 
The problem is here:
https://github.com/coreos/toolbox/blame/f61c747c84ea4ea18b853aaab5e6318b2cd74162/rhcos-toolbox#L29

Podman returns "<no value>" and not "":
~~~
[root@openshift-master-0 ~]# sudo podman image inspect "docker.io/library/fedora" --format "{{.Labels.run}}"
<no value>
~~~
     26     local runlabel=$(image_runlabel)
     27     if ! container_exists; then
     28         echo "Spawning a container '$TOOLBOX_NAME' with image '$TOOLBOX_IMAGE'"
     29         if [[ -z "$runlabel" ]]; then
     30             container_run
     31             return
     32         else
     33             echo "Detected RUN label in the container image. Using that as the default..."
     34             container_runlabel
     35             return
     36         fi
     37     else
     38         echo "Container '$TOOLBOX_NAME' already exists. Trying to start..."
     39         echo "(To remove the container and start with a fresh toolbox, run: sudo podman rm '$TOOLBOX_NAME')"
     40     fi
~~~

With that change, it works:
~~~
cp /bin/toolbox /root/toolbox
vi /root/toolbox
~~~

~~~
[root@openshift-master-0 ~]# diff -i /bin/toolbox /root/toolbox 
29c29
<         if [[ -z "$runlabel" ]]; then
---
>         if [[ -z "$runlabel" ]] || [ "$runlabel" == "<no value>" ]; then
~~~

Without custom container:
~~~
[root@openshift-master-0 ~]# bash -x toolbox 
+ set -eo pipefail
+ trap cleanup EXIT
+ REGISTRY=registry.redhat.io
+ IMAGE=rhel8/support-tools
++ whoami
+ TOOLBOX_NAME=toolbox-root
+ TOOLBOXRC=/root/.toolboxrc
+ main
+ setup
+ '[' -f /root/.toolboxrc ']'
+ TOOLBOX_IMAGE=registry.redhat.io/rhel8/support-tools
+ [[ '' =~ ^(--help|-h)$ ]]
+ run
+ image_exists
+ sudo podman inspect registry.redhat.io/rhel8/support-tools
++ image_runlabel
++ sudo podman image inspect registry.redhat.io/rhel8/support-tools --format '{{.Labels.run}}'
+ local 'runlabel=podman run -it --name NAME --privileged --ipc=host --net=host --pid=host -e HOST=/host -e NAME=NAME -e IMAGE=IMAGE -v /run:/run -v /var/log:/var/log -v /etc/machine-id:/etc/machine-id -v /etc/localtime:/etc/localtime -v /:/host IMAGE'
+ container_exists
+ sudo podman inspect toolbox-root
+ echo 'Spawning a container '\''toolbox-root'\'' with image '\''registry.redhat.io/rhel8/support-tools'\'''
Spawning a container 'toolbox-root' with image 'registry.redhat.io/rhel8/support-tools'
+ [[ -z podman run -it --name NAME --privileged --ipc=host --net=host --pid=host -e HOST=/host -e NAME=NAME -e IMAGE=IMAGE -v /run:/run -v /var/log:/var/log -v /etc/machine-id:/etc/machine-id -v /etc/localtime:/etc/localtime -v /:/host IMAGE ]]
+ '[' 'podman run -it --name NAME --privileged --ipc=host --net=host --pid=host -e HOST=/host -e NAME=NAME -e IMAGE=IMAGE -v /run:/run -v /var/log:/var/log -v /etc/machine-id:/etc/machine-id -v /etc/localtime:/etc/localtime -v /:/host IMAGE' == '<no value>' ']'
+ echo 'Detected RUN label in the container image. Using that as the default...'
Detected RUN label in the container image. Using that as the default...
+ container_runlabel
+ sudo podman container runlabel --name toolbox-root RUN registry.redhat.io/rhel8/support-tools
~~~

With custom container:
~~~
[root@openshift-master-0 ~]# cat .toolboxrc 
REGISTRY=docker.io
IMAGE=fedora:latest
[root@openshift-master-0 ~]# bash -x toolbox 
+ set -eo pipefail
+ trap cleanup EXIT
+ REGISTRY=registry.redhat.io
+ IMAGE=rhel8/support-tools
++ whoami
+ TOOLBOX_NAME=toolbox-root
+ TOOLBOXRC=/root/.toolboxrc
+ main
+ setup
+ '[' -f /root/.toolboxrc ']'
+ echo '.toolboxrc file detected, overriding defaults...'
.toolboxrc file detected, overriding defaults...
+ source /root/.toolboxrc
++ REGISTRY=docker.io
++ IMAGE=fedora:latest
+ TOOLBOX_IMAGE=docker.io/fedora:latest
+ [[ '' =~ ^(--help|-h)$ ]]
+ run
+ image_exists
+ sudo podman inspect docker.io/fedora:latest
++ image_runlabel
++ sudo podman image inspect docker.io/fedora:latest --format '{{.Labels.run}}'
+ local 'runlabel=<no value>'
+ container_exists
+ sudo podman inspect toolbox-root
+ echo 'Spawning a container '\''toolbox-root'\'' with image '\''docker.io/fedora:latest'\'''
Spawning a container 'toolbox-root' with image 'docker.io/fedora:latest'
+ [[ -z <no value> ]]
+ '[' '<no value>' == '<no value>' ']'
+ container_run
+ sudo podman run --hostname toolbox --name toolbox-root --privileged --net=host --pid=host --ipc=host --tty --interactive -e HOST=/host -e NAME=toolbox-root -e IMAGE=fedora:latest --security-opt label=disable --volume /run:/run --volume /var/log:/var/log --volume /etc/machine-id:/etc/machine-id --volume /etc/localtime:/etc/localtime --volume /:/host docker.io/fedora:latest
[root@toolbox /]# cat /etc/redhat-release 
Fedora release 33 (Thirty Three)
~~~
Comment 9 Andreas Karis 2021-01-30 17:52:53 UTC 
Are we going to upgrade to github.com/containers/toolbox in RHEL 8.3 / CoreOS 4.7?  If so, will we be able to use custom containers? Otherwise, this can be fixed really easily, see my earlier comment.
Comment 11 Alex Jia 2021-02-01 13:57:32 UTC 
The latest toolbox is re-factored by golang, and it doesn't support .toolboxrc any more,
so need I close this bug as WONTFIX or something like that?

[root@kvm-03-guest13 ~]# cat ~/.toolboxrc
IMAGE=libpod/alpine
REGISTRY=quay.io

[root@kvm-03-guest13 ~]# toolbox create
Image required to create toolbox container.
Download registry.access.redhat.com/ubi8/ubi:8.4 (500MB)? [y/N]:

NOTE, BTW, at present, the ubi8/ubi:8.4 doesn't exist, it's known bug 1917663
Comment 12 Andreas Karis 2021-02-02 12:36:45 UTC 
Thanks, I cloned this https://bugzilla.redhat.com/show_bug.cgi?id=1924020

Just in case that the solution for RHCOS 4.7 would be to upgrade to the golang toolbox ... Does the latest golang toolbox have a way to specify custom images and registries? 

Thanks!
Comment 13 Debarshi Ray 2021-02-03 12:23:29 UTC 
> Does the latest golang toolbox have a way to specify
> custom images and registries? 


With github.com/containers/toolbox, you can do:
$ toolbox create --image <full-url-to-image>
Comment 15 Jindrich Novy 2021-02-05 07:05:10 UTC 
This bug doesn't apply to container-tools in RHEL as rhcos-toolbox was replaced by new toolbox to which this bug doesn't apply.
Note You need to log in before you can comment on or make changes to this bug.