Description of problem: The 4.5 builds in this search are crashing with the following dump. https://search.svc.ci.openshift.org/?search=not+found%3A+ID+does+not+exist&maxAge=336h&context=-1&type=bug%2Bjunit&name=&maxMatches=1&maxBytes=20971520 Apr 08 16:05:41.108912 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: 2020-04-08T16:05:41Z [verbose] Add: openshift-image-registry:node-ca-9pqw9:unknownUID:(openshift-sdn):eth0 {"cniVersion":"0.3.1","interfaces":[{"name":"eth0","sandbox":"/proc/41456/ns/net"}],"ips":[{"version":"4","interface":0,"address":"10.130.0.37/23"}],"routes":[{"dst":"0.0.0.0/0","gw":"10.130.0.1"},{"dst":"224.0.0.0/4"},{"dst":"10.128.0.0/14"}],"dns":{}} Apr 08 16:05:41.111411 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: panic: runtime error: invalid memory address or nil pointer dereference Apr 08 16:05:41.111411 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xa6cdf5] Apr 08 16:05:41.111411 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: goroutine 1 [running]: Apr 08 16:05:41.111411 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: panic(0x122f0a0, 0x20b9a30) Apr 08 16:05:41.111669 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/panic.go:565 +0x2c5 fp=0xc000255330 sp=0xc0002552a0 pc=0x432105 Apr 08 16:05:41.111669 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.panicmem(...) Apr 08 16:05:41.111669 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/panic.go:82 Apr 08 16:05:41.111669 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.sigpanic() Apr 08 16:05:41.111669 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/signal_unix.go:390 +0x411 fp=0xc000255360 sp=0xc000255330 pc=0x446d61 Apr 08 16:05:41.111669 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: github.com/intel/multus-cni/vendor/k8s.io/api/core/v1.(*Pod).GetObjectKind(0x0, 0x4, 0xc000410000) Apr 08 16:05:41.111669 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: <autogenerated>:1 +0x5 fp=0xc000255368 sp=0xc000255360 pc=0xa6cdf5 Apr 08 16:05:41.111669 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: github.com/intel/multus-cni/vendor/k8s.io/client-go/tools/reference.GetReference(0xc0001920e0, 0x1562aa0, 0x0, 0x4d7468, 0xc0003ce180, 0x11a0ee0) Apr 08 16:05:41.111669 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/src/multus-cni/gopath/src/github.com/intel/multus-cni/vendor/k8s.io/client-go/tools/reference/ref.go:50 +0x73 fp=0xc0002554e8 sp=0xc000255368 pc=0xf59213 Apr 08 16:05:41.112120 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: github.com/intel/multus-cni/vendor/k8s.io/client-go/tools/record.(*recorderImpl).generateEvent(0xc0002dc5c0, 0x1562aa0, 0x0, 0x0, 0xbf9b9b75468128dc, 0x75ac3416f, 0x20d0940, 0x138ef96, 0x6, 0x1397216, ...) Apr 08 16:05:41.112120 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/src/multus-cni/gopath/src/github.com/intel/multus-cni/vendor/k8s.io/client-go/tools/record/event.go:257 +0x5d fp=0xc0002555d0 sp=0xc0002554e8 pc=0xfde62d Apr 08 16:05:41.112120 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: github.com/intel/multus-cni/vendor/k8s.io/client-go/tools/record.(*recorderImpl).Event(0xc0002dc5c0, 0x1562aa0, 0x0, 0x138ef96, 0x6, 0x1397216, 0xe, 0xc0003d0420, 0x19) Apr 08 16:05:41.112120 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/src/multus-cni/gopath/src/github.com/intel/multus-cni/vendor/k8s.io/client-go/tools/record/event.go:287 +0xc3 fp=0xc000255648 sp=0xc0002555d0 pc=0xfdeaa3 Apr 08 16:05:41.112120 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: github.com/intel/multus-cni/vendor/k8s.io/client-go/tools/record.(*recorderImpl).Eventf(0xc0002dc5c0, 0x1562aa0, 0x0, 0x138ef96, 0x6, 0x1397216, 0xe, 0x1392566, 0x9, 0xc0000f06e0, ...) Apr 08 16:05:41.112120 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/src/multus-cni/gopath/src/github.com/intel/multus-cni/vendor/k8s.io/client-go/tools/record/event.go:291 +0xca fp=0xc0002556a0 sp=0xc000255648 pc=0xfdeb8a Apr 08 16:05:41.112120 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: github.com/intel/multus-cni/k8sclient.(*ClientInfo).Eventf(...) Apr 08 16:05:41.112120 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/src/multus-cni/gopath/src/github.com/intel/multus-cni/k8sclient/k8sclient.go:89 Apr 08 16:05:41.112120 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: main.delegateAdd(0x0, 0x0, 0xc0002dc600, 0x0, 0xc00004000b, 0x4, 0xc00025c000, 0xc000364120, 0xc0002565b0, 0xf, ...) Apr 08 16:05:41.112120 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/src/multus-cni/gopath/src/github.com/intel/multus-cni/multus/multus.go:362 +0xef2 fp=0xc0002559b8 sp=0xc0002556a0 pc=0x1109d02 Apr 08 16:05:41.112120 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: main.cmdAdd(0xc00023a9a0, 0x0, 0x0, 0xc0002dc600, 0xc0000be000, 0x5, 0x5, 0x0) Apr 08 16:05:41.112120 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/src/multus-cni/gopath/src/github.com/intel/multus-cni/multus/multus.go:522 +0x82a fp=0xc000255c80 sp=0xc0002559b8 pc=0x110c8aa Apr 08 16:05:41.112120 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: main.main.func1(0xc00023a9a0, 0xc000041ad8, 0x5) Apr 08 16:05:41.112120 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/src/multus-cni/gopath/src/github.com/intel/multus-cni/multus/multus.go:756 +0x40 fp=0xc000255cd0 sp=0xc000255c80 pc=0x110f040 Apr 08 16:05:41.113137 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: github.com/intel/multus-cni/vendor/github.com/containernetworking/cni/pkg/skel.(*dispatcher).checkVersionAndCall(0xc0000cde90, 0xc00023a9a0, 0x155f020, 0xc0000aa4e0, 0x141d218, 0x0, 0xc000222740) Apr 08 16:05:41.113137 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/src/multus-cni/gopath/src/github.com/intel/multus-cni/vendor/github.com/containernetworking/cni/pkg/skel/skel.go:185 +0x259 fp=0xc000255d60 sp=0xc000255cd0 pc=0x57e929 Apr 08 16:05:41.113137 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: github.com/intel/multus-cni/vendor/github.com/containernetworking/cni/pkg/skel.(*dispatcher).pluginMain(0xc0000cde90, 0x141d218, 0x141d220, 0x141d228, 0x155f020, 0xc0000aa4e0, 0x13bf813, 0x2f, 0x20d18c0) Apr 08 16:05:41.113137 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/src/multus-cni/gopath/src/github.com/intel/multus-cni/vendor/github.com/containernetworking/cni/pkg/skel/skel.go:221 +0x549 fp=0xc000255e28 sp=0xc000255d60 pc=0x57f0e9 Apr 08 16:05:41.113137 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: github.com/intel/multus-cni/vendor/github.com/containernetworking/cni/pkg/skel.PluginMainWithError(...) Apr 08 16:05:41.113137 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/src/multus-cni/gopath/src/github.com/intel/multus-cni/vendor/github.com/containernetworking/cni/pkg/skel/skel.go:286 Apr 08 16:05:41.113137 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: github.com/intel/multus-cni/vendor/github.com/containernetworking/cni/pkg/skel.PluginMain(0x141d218, 0x141d220, 0x141d228, 0x155f020, 0xc0000aa4e0, 0x13bf813, 0x2f) Apr 08 16:05:41.113137 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/src/multus-cni/gopath/src/github.com/intel/multus-cni/vendor/github.com/containernetworking/cni/pkg/skel/skel.go:301 +0x129 fp=0xc000255ee0 sp=0xc000255e28 pc=0x57f999 Apr 08 16:05:41.113137 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: main.main() Apr 08 16:05:41.113137 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/src/multus-cni/gopath/src/github.com/intel/multus-cni/multus/multus.go:754 +0x37e fp=0xc000255f98 sp=0xc000255ee0 pc=0x110ed6e Apr 08 16:05:41.113137 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.main() Apr 08 16:05:41.113137 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/proc.go:200 +0x20c fp=0xc000255fe0 sp=0xc000255f98 pc=0x433e2c Apr 08 16:05:41.113137 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.goexit() Apr 08 16:05:41.113137 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc000255fe8 sp=0xc000255fe0 pc=0x45e5e1 Apr 08 16:05:41.113137 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: goroutine 2 [force gc (idle)]: Apr 08 16:05:41.113137 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.gopark(0x141d8f0, 0x20cf8d0, 0x1410, 0x1) Apr 08 16:05:41.113137 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/proc.go:301 +0xef fp=0xc00005efb0 sp=0xc00005ef90 pc=0x43421f Apr 08 16:05:41.113137 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.goparkunlock(...) Apr 08 16:05:41.113137 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/proc.go:307 Apr 08 16:05:41.113137 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.forcegchelper() Apr 08 16:05:41.113137 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/proc.go:250 +0xb7 fp=0xc00005efe0 sp=0xc00005efb0 pc=0x4340c7 Apr 08 16:05:41.114211 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.goexit() Apr 08 16:05:41.114211 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc00005efe8 sp=0xc00005efe0 pc=0x45e5e1 Apr 08 16:05:41.114211 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: created by runtime.init.5 Apr 08 16:05:41.114211 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/proc.go:239 +0x35 Apr 08 16:05:41.114211 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: goroutine 3 [GC sweep wait]: Apr 08 16:05:41.114211 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.gopark(0x141d8f0, 0x20d0000, 0x140c, 0x1) Apr 08 16:05:41.114211 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/proc.go:301 +0xef fp=0xc00005f7a8 sp=0xc00005f788 pc=0x43421f Apr 08 16:05:41.114475 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.goparkunlock(...) Apr 08 16:05:41.114475 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/proc.go:307 Apr 08 16:05:41.114475 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.bgsweep(0xc000086000) Apr 08 16:05:41.114475 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/mgcsweep.go:70 +0x9c fp=0xc00005f7d8 sp=0xc00005f7a8 pc=0x42769c Apr 08 16:05:41.114475 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.goexit() Apr 08 16:05:41.114475 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc00005f7e0 sp=0xc00005f7d8 pc=0x45e5e1 Apr 08 16:05:41.114475 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: created by runtime.gcenable Apr 08 16:05:41.114475 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/mgc.go:208 +0x58 Apr 08 16:05:41.114475 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: goroutine 4 [finalizer wait]: Apr 08 16:05:41.114475 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.gopark(0x141d8f0, 0x20edfc8, 0x140f, 0x1) Apr 08 16:05:41.114475 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/proc.go:301 +0xef fp=0xc00005ff58 sp=0xc00005ff38 pc=0x43421f Apr 08 16:05:41.114475 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.goparkunlock(...) Apr 08 16:05:41.114475 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/proc.go:307 Apr 08 16:05:41.114475 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.runfinq() Apr 08 16:05:41.114475 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/mfinal.go:175 +0xa6 fp=0xc00005ffe0 sp=0xc00005ff58 pc=0x41e366 Apr 08 16:05:41.114475 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.goexit() Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc00005ffe8 sp=0xc00005ffe0 pc=0x45e5e1 Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: created by runtime.createfing Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/mfinal.go:156 +0x61 Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: goroutine 5 [chan receive]: Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.gopark(0x141d8f0, 0xc0000ee058, 0x170d, 0x3) Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/proc.go:301 +0xef fp=0xc00005e6d0 sp=0xc00005e6b0 pc=0x43421f Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.goparkunlock(...) Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/proc.go:307 Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.chanrecv(0xc0000ee000, 0xc00005e7b0, 0x20d0c01, 0x20d0cb8) Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/chan.go:524 +0x2ea fp=0xc00005e760 sp=0xc00005e6d0 pc=0x40c17a Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.chanrecv2(0xc0000ee000, 0xc00005e7b0, 0x1) Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/chan.go:411 +0x2b fp=0xc00005e790 sp=0xc00005e760 pc=0x40be7b Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: github.com/intel/multus-cni/vendor/k8s.io/klog.(*loggingT).flushDaemon(0x20d0ca0) Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/src/multus-cni/gopath/src/github.com/intel/multus-cni/vendor/k8s.io/klog/klog.go:1010 +0x8b fp=0xc00005e7d8 sp=0xc00005e790 pc=0x6711eb Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.goexit() Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc00005e7e0 sp=0xc00005e7d8 pc=0x45e5e1 Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: created by github.com/intel/multus-cni/vendor/k8s.io/klog.init.0 Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/src/multus-cni/gopath/src/github.com/intel/multus-cni/vendor/k8s.io/klog/klog.go:411 +0xd6 Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: goroutine 18 [syscall]: Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.notetsleepg(0x20d56a0, 0x12a03ec60, 0x0) Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/lock_futex.go:227 +0x34 fp=0xc00005a760 sp=0xc00005a730 pc=0x410bd4 Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.timerproc(0x20d5680) Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/time.go:311 +0x2ea fp=0xc00005a7d8 sp=0xc00005a760 pc=0x44fe0a Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.goexit() Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc00005a7e0 sp=0xc00005a7d8 pc=0x45e5e1 Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: created by runtime.(*timersBucket).addtimerLocked Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/time.go:169 +0x10e Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: goroutine 6 [chan receive]: Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.gopark(0x141d8f0, 0xc0002ccbf8, 0xc00006170d, 0x3) Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/proc.go:301 +0xef fp=0xc000060698 sp=0xc000060678 pc=0x43421f Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.goparkunlock(...) Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/proc.go:307 Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.chanrecv(0xc0002ccba0, 0xc0000607a8, 0xc0002bd701, 0xc0002e89c0) Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/chan.go:524 +0x2ea fp=0xc000060728 sp=0xc000060698 pc=0x40c17a Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.chanrecv2(0xc0002ccba0, 0xc0000607a8, 0x14) Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/chan.go:411 +0x2b fp=0xc000060758 sp=0xc000060728 pc=0x40be7b Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: github.com/intel/multus-cni/vendor/k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop(0xc0002dc540) Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/src/multus-cni/gopath/src/github.com/intel/multus-cni/vendor/k8s.io/apimachinery/pkg/watch/mux.go:207 +0x66 fp=0xc0000607d8 sp=0xc000060758 pc=0x85c5e6 Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.goexit() Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0000607e0 sp=0xc0000607d8 pc=0x45e5e1 Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: created by github.com/intel/multus-cni/vendor/k8s.io/apimachinery/pkg/watch.NewBroadcaster Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/src/multus-cni/gopath/src/github.com/intel/multus-cni/vendor/k8s.io/apimachinery/pkg/watch/mux.go:75 +0xcc Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: goroutine 7 [chan receive]: Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.gopark(0x141d8f0, 0xc0002ccc58, 0x170d, 0x3) Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/proc.go:301 +0xef fp=0xc000060ea8 sp=0xc000060e88 pc=0x43421f Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.goparkunlock(...) Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/proc.go:307 Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.chanrecv(0xc0002ccc00, 0xc000060f98, 0x1, 0xc000060f70) Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/chan.go:524 +0x2ea fp=0xc000060f38 sp=0xc000060ea8 pc=0x40c17a Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.chanrecv2(0xc0002ccc00, 0xc000060f98, 0x0) Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/chan.go:411 +0x2b fp=0xc000060f68 sp=0xc000060f38 pc=0x40be7b Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: github.com/intel/multus-cni/vendor/k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1(0x15649a0, 0xc0002c9710, 0xc0002f60f0) Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/src/multus-cni/gopath/src/github.com/intel/multus-cni/vendor/k8s.io/client-go/tools/record/event.go:231 +0xa4 fp=0xc000060fc8 sp=0xc000060f68 pc=0xfe1914 Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.goexit() Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc000060fd0 sp=0xc000060fc8 pc=0x45e5e1 Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: created by github.com/intel/multus-cni/vendor/k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/src/multus-cni/gopath/src/github.com/intel/multus-cni/vendor/k8s.io/client-go/tools/record/event.go:229 +0x6e Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: goroutine 8 [chan receive]: Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.gopark(0x141d8f0, 0xc0002ccd18, 0x170d, 0x3) Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/proc.go:301 +0xef fp=0xc0000616a8 sp=0xc000061688 pc=0x43421f Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.goparkunlock(...) Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/proc.go:307 Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.chanrecv(0xc0002cccc0, 0xc000061798, 0x1, 0xc000061770) Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/chan.go:524 +0x2ea fp=0xc000061738 sp=0xc0000616a8 pc=0x40c17a Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.chanrecv2(0xc0002cccc0, 0xc000061798, 0x0) Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/chan.go:411 +0x2b fp=0xc000061768 sp=0xc000061738 pc=0x40be7b Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: github.com/intel/multus-cni/vendor/k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1(0x15649a0, 0xc0002c98f0, 0xc0002c98c0) Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/src/multus-cni/gopath/src/github.com/intel/multus-cni/vendor/k8s.io/client-go/tools/record/event.go:231 +0xa4 fp=0xc0000617c8 sp=0xc000061768 pc=0xfe1914 Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.goexit() Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0000617d0 sp=0xc0000617c8 pc=0x45e5e1 Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: created by github.com/intel/multus-cni/vendor/k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/src/multus-cni/gopath/src/github.com/intel/multus-cni/vendor/k8s.io/client-go/tools/record/event.go:229 +0x6e Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: goroutine 10 [timer goroutine (idle)]: Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.gopark(0x141d8f0, 0x20d5600, 0x1414, 0x1) Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/proc.go:301 +0xef fp=0xc000061f60 sp=0xc000061f40 pc=0x43421f Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.goparkunlock(...) Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/proc.go:307 Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.timerproc(0x20d5600) Apr 08 16:05:41.118840 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/time.go:303 +0x273 fp=0xc000061fd8 sp=0xc000061f60 pc=0x44fd93 Apr 08 16:05:41.118840 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.goexit() Apr 08 16:05:41.118840 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc000061fe0 sp=0xc000061fd8 pc=0x45e5e1 Apr 08 16:05:41.118840 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: created by runtime.(*timersBucket).addtimerLocked Apr 08 16:05:41.118840 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/time.go:169 +0x10e Apr 08 16:05:41.118840 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: goroutine 34 [timer goroutine (idle)]: Apr 08 16:05:41.118840 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.gopark(0x141d8f0, 0x20d5700, 0x1414, 0x1) Apr 08 16:05:41.118840 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/proc.go:301 +0xef fp=0xc000312760 sp=0xc000312740 pc=0x43421f Apr 08 16:05:41.118840 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.goparkunlock(...) Apr 08 16:05:41.119468 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/proc.go:307 Apr 08 16:05:41.119468 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.timerproc(0x20d5700) Apr 08 16:05:41.119468 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/time.go:303 +0x273 fp=0xc0003127d8 sp=0xc000312760 pc=0x44fd93 Apr 08 16:05:41.119468 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.goexit() Apr 08 16:05:41.119468 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0003127e0 sp=0xc0003127d8 pc=0x45e5e1 Apr 08 16:05:41.119468 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: created by runtime.(*timersBucket).addtimerLocked Apr 08 16:05:41.119468 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: /usr/lib/golang/src/runtime/time.go:169 +0x10e Apr 08 16:05:41.143967 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal systemd[1]: Started Process Core Dump (PID 43593/UID 0). Apr 08 16:05:41.145908 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal hyperkube[1488]: I0408 16:05:41.145871 1488 factory.go:173] Using factory "raw" for container "/system.slice/system-systemd\\x2dcoredump.slice/systemd-coredump" Apr 08 16:05:41.146399 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal hyperkube[1488]: I0408 16:05:41.146371 1488 manager.go:950] Added container: "/system.slice/system-systemd\\x2dcoredump.slice/systemd-coredump" (aliases: [], namespace: "") Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Here is the PR upstream : https://github.com/kubernetes/kubernetes/pull/90034
Here is related code from multus.go ``` // send kubernetes events if delegate.Name != "" { kubeClient.Eventf(pod, v1.EventTypeNormal, "AddedInterface", "Add %s %v from %s", rt.IfName, ips, delegate.Name) } else { kubeClient.Eventf(pod, v1.EventTypeNormal, "AddedInterface", "Add %s %v", rt.IfName, ips) ``` I am not familiar with multus - the above should be examined. PR #90034 would be good to have - k8s shouldn't panic even if GetReference() is unable to proceed.
I tried the following addition to staging/src/k8s.io/client-go/tools/reference/ref_test.go but was unable to reproduce the panic: func TestGetReferencePod(t *testing.T) { tests := []struct { name string input *v1.Pod groupVersion schema.GroupVersion expectedRefVersion string }{ { name: "Pod", input: &v1.Pod{ Status: v1.PodStatus{ ContainerStatuses: []v1.ContainerStatus{ { Name: "foo", }, }, }, }, groupVersion: schema.GroupVersion{Group: "", Version: "v1"}, }, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { scheme := runtime.NewScheme() scheme.AddKnownTypes(test.groupVersion, &v1.Pod{}) if _, err := GetReference(scheme, test.input); err != nil { } if test.expectedRefVersion != "v1" { t.Errorf("expected %q, got %q", test.expectedRefVersion, "v1") } }) } }
Stepping into the GetReference() call, I saw <autogenerated>:2 when trying to step into v1.(*Pod).GetObjectKind() - though GoLand didn't show me the generated code. I noted that multus seems to be using go 1.12 while k8s master uses go 1.13
I went over vendor/k8s.io/api/core/v1/generated.pb.go under multus There were Pod methods such as Pod#ProtoMessage() but I didn't find Pod#GetObjectKind
Looking at staging/src/k8s.io/client-go/tools/reference/ref.go from k8s, the following code: ``` var listMeta metav1.Common objectMeta, err := meta.Accessor(obj) if err != nil { listMeta, err = meta.CommonAccessor(obj) ``` is not in vendor/k8s.io/client-go/tools/reference/ref.go It seems k8s dependency in multus should be upgraded. However the golang version difference seems to be a hurdle.
Issue upstream: https://github.com/intel/multus-cni/issues/481
Apparently the downstream build should be golang 1.13 Luke Meyer from ART advises: > ART OCP builds ignore what's in your dockerfile > and overwrite it with golang-builder:1.13 > so downstream builds should already be 1.13 Will look towards updating the dockerfile with a golang 1.13.
After chatting with Tomofumi-san: Proceed with: https://github.com/intel/multus-cni/pull/482 Tomofumi-san will add more log in another PR to give us more clue if the crash happens again.
There is no way to verify this bz because we still do not know how to reproduce it. QE verify this bug now and will open it again if CI testing fail in the future.
If the crash is no longer seen in master log, can this be resolved ?
I believe so. So let me check once again someday next week.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2409