Bug 1822803 - multus-cni crash on 4.5 with nil reference in client-go
Summary: multus-cni crash on 4.5 with nil reference in client-go
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.5
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: ---
: 4.5.0
Assignee: Tomofumi Hayashi
QA Contact: Weibin Liang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-04-09 21:44 UTC by Ryan Phillips
Modified: 2020-07-13 17:27 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-07-13 17:26:42 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift multus-cni pull 59 0 None closed Bug 1822803: Add nil check for pod in case of Eventf() 2020-06-23 09:56:29 UTC
Red Hat Product Errata RHBA-2020:2409 0 None None None 2020-07-13 17:27:10 UTC

Description Ryan Phillips 2020-04-09 21:44:50 UTC
Description of problem:

The 4.5 builds in this search are crashing with the following dump.

https://search.svc.ci.openshift.org/?search=not+found%3A+ID+does+not+exist&maxAge=336h&context=-1&type=bug%2Bjunit&name=&maxMatches=1&maxBytes=20971520

Apr 08 16:05:41.108912 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: 2020-04-08T16:05:41Z [verbose] Add: openshift-image-registry:node-ca-9pqw9:unknownUID:(openshift-sdn):eth0 {"cniVersion":"0.3.1","interfaces":[{"name":"eth0","sandbox":"/proc/41456/ns/net"}],"ips":[{"version":"4","interface":0,"address":"10.130.0.37/23"}],"routes":[{"dst":"0.0.0.0/0","gw":"10.130.0.1"},{"dst":"224.0.0.0/4"},{"dst":"10.128.0.0/14"}],"dns":{}}
Apr 08 16:05:41.111411 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: panic: runtime error: invalid memory address or nil pointer dereference
Apr 08 16:05:41.111411 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xa6cdf5]
Apr 08 16:05:41.111411 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: goroutine 1 [running]:
Apr 08 16:05:41.111411 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: panic(0x122f0a0, 0x20b9a30)
Apr 08 16:05:41.111669 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/panic.go:565 +0x2c5 fp=0xc000255330 sp=0xc0002552a0 pc=0x432105
Apr 08 16:05:41.111669 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.panicmem(...)
Apr 08 16:05:41.111669 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/panic.go:82
Apr 08 16:05:41.111669 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.sigpanic()
Apr 08 16:05:41.111669 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/signal_unix.go:390 +0x411 fp=0xc000255360 sp=0xc000255330 pc=0x446d61
Apr 08 16:05:41.111669 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: github.com/intel/multus-cni/vendor/k8s.io/api/core/v1.(*Pod).GetObjectKind(0x0, 0x4, 0xc000410000)
Apr 08 16:05:41.111669 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         <autogenerated>:1 +0x5 fp=0xc000255368 sp=0xc000255360 pc=0xa6cdf5
Apr 08 16:05:41.111669 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: github.com/intel/multus-cni/vendor/k8s.io/client-go/tools/reference.GetReference(0xc0001920e0, 0x1562aa0, 0x0, 0x4d7468, 0xc0003ce180, 0x11a0ee0)
Apr 08 16:05:41.111669 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/src/multus-cni/gopath/src/github.com/intel/multus-cni/vendor/k8s.io/client-go/tools/reference/ref.go:50 +0x73 fp=0xc0002554e8 sp=0xc000255368 pc=0xf59213
Apr 08 16:05:41.112120 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: github.com/intel/multus-cni/vendor/k8s.io/client-go/tools/record.(*recorderImpl).generateEvent(0xc0002dc5c0, 0x1562aa0, 0x0, 0x0, 0xbf9b9b75468128dc, 0x75ac3416f, 0x20d0940, 0x138ef96, 0x6, 0x1397216, ...)
Apr 08 16:05:41.112120 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/src/multus-cni/gopath/src/github.com/intel/multus-cni/vendor/k8s.io/client-go/tools/record/event.go:257 +0x5d fp=0xc0002555d0 sp=0xc0002554e8 pc=0xfde62d
Apr 08 16:05:41.112120 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: github.com/intel/multus-cni/vendor/k8s.io/client-go/tools/record.(*recorderImpl).Event(0xc0002dc5c0, 0x1562aa0, 0x0, 0x138ef96, 0x6, 0x1397216, 0xe, 0xc0003d0420, 0x19)
Apr 08 16:05:41.112120 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/src/multus-cni/gopath/src/github.com/intel/multus-cni/vendor/k8s.io/client-go/tools/record/event.go:287 +0xc3 fp=0xc000255648 sp=0xc0002555d0 pc=0xfdeaa3
Apr 08 16:05:41.112120 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: github.com/intel/multus-cni/vendor/k8s.io/client-go/tools/record.(*recorderImpl).Eventf(0xc0002dc5c0, 0x1562aa0, 0x0, 0x138ef96, 0x6, 0x1397216, 0xe, 0x1392566, 0x9, 0xc0000f06e0, ...)
Apr 08 16:05:41.112120 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/src/multus-cni/gopath/src/github.com/intel/multus-cni/vendor/k8s.io/client-go/tools/record/event.go:291 +0xca fp=0xc0002556a0 sp=0xc000255648 pc=0xfdeb8a
Apr 08 16:05:41.112120 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: github.com/intel/multus-cni/k8sclient.(*ClientInfo).Eventf(...)
Apr 08 16:05:41.112120 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/src/multus-cni/gopath/src/github.com/intel/multus-cni/k8sclient/k8sclient.go:89
Apr 08 16:05:41.112120 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: main.delegateAdd(0x0, 0x0, 0xc0002dc600, 0x0, 0xc00004000b, 0x4, 0xc00025c000, 0xc000364120, 0xc0002565b0, 0xf, ...)
Apr 08 16:05:41.112120 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/src/multus-cni/gopath/src/github.com/intel/multus-cni/multus/multus.go:362 +0xef2 fp=0xc0002559b8 sp=0xc0002556a0 pc=0x1109d02
Apr 08 16:05:41.112120 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: main.cmdAdd(0xc00023a9a0, 0x0, 0x0, 0xc0002dc600, 0xc0000be000, 0x5, 0x5, 0x0)
Apr 08 16:05:41.112120 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/src/multus-cni/gopath/src/github.com/intel/multus-cni/multus/multus.go:522 +0x82a fp=0xc000255c80 sp=0xc0002559b8 pc=0x110c8aa
Apr 08 16:05:41.112120 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: main.main.func1(0xc00023a9a0, 0xc000041ad8, 0x5)
Apr 08 16:05:41.112120 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/src/multus-cni/gopath/src/github.com/intel/multus-cni/multus/multus.go:756 +0x40 fp=0xc000255cd0 sp=0xc000255c80 pc=0x110f040
Apr 08 16:05:41.113137 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: github.com/intel/multus-cni/vendor/github.com/containernetworking/cni/pkg/skel.(*dispatcher).checkVersionAndCall(0xc0000cde90, 0xc00023a9a0, 0x155f020, 0xc0000aa4e0, 0x141d218, 0x0, 0xc000222740)
Apr 08 16:05:41.113137 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/src/multus-cni/gopath/src/github.com/intel/multus-cni/vendor/github.com/containernetworking/cni/pkg/skel/skel.go:185 +0x259 fp=0xc000255d60 sp=0xc000255cd0 pc=0x57e929
Apr 08 16:05:41.113137 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: github.com/intel/multus-cni/vendor/github.com/containernetworking/cni/pkg/skel.(*dispatcher).pluginMain(0xc0000cde90, 0x141d218, 0x141d220, 0x141d228, 0x155f020, 0xc0000aa4e0, 0x13bf813, 0x2f, 0x20d18c0)
Apr 08 16:05:41.113137 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/src/multus-cni/gopath/src/github.com/intel/multus-cni/vendor/github.com/containernetworking/cni/pkg/skel/skel.go:221 +0x549 fp=0xc000255e28 sp=0xc000255d60 pc=0x57f0e9
Apr 08 16:05:41.113137 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: github.com/intel/multus-cni/vendor/github.com/containernetworking/cni/pkg/skel.PluginMainWithError(...)
Apr 08 16:05:41.113137 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/src/multus-cni/gopath/src/github.com/intel/multus-cni/vendor/github.com/containernetworking/cni/pkg/skel/skel.go:286
Apr 08 16:05:41.113137 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: github.com/intel/multus-cni/vendor/github.com/containernetworking/cni/pkg/skel.PluginMain(0x141d218, 0x141d220, 0x141d228, 0x155f020, 0xc0000aa4e0, 0x13bf813, 0x2f)
Apr 08 16:05:41.113137 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/src/multus-cni/gopath/src/github.com/intel/multus-cni/vendor/github.com/containernetworking/cni/pkg/skel/skel.go:301 +0x129 fp=0xc000255ee0 sp=0xc000255e28 pc=0x57f999
Apr 08 16:05:41.113137 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: main.main()
Apr 08 16:05:41.113137 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/src/multus-cni/gopath/src/github.com/intel/multus-cni/multus/multus.go:754 +0x37e fp=0xc000255f98 sp=0xc000255ee0 pc=0x110ed6e
Apr 08 16:05:41.113137 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.main()
Apr 08 16:05:41.113137 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/proc.go:200 +0x20c fp=0xc000255fe0 sp=0xc000255f98 pc=0x433e2c
Apr 08 16:05:41.113137 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.goexit()
Apr 08 16:05:41.113137 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc000255fe8 sp=0xc000255fe0 pc=0x45e5e1
Apr 08 16:05:41.113137 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: goroutine 2 [force gc (idle)]:
Apr 08 16:05:41.113137 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.gopark(0x141d8f0, 0x20cf8d0, 0x1410, 0x1)
Apr 08 16:05:41.113137 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/proc.go:301 +0xef fp=0xc00005efb0 sp=0xc00005ef90 pc=0x43421f
Apr 08 16:05:41.113137 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.goparkunlock(...)
Apr 08 16:05:41.113137 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/proc.go:307
Apr 08 16:05:41.113137 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.forcegchelper()
Apr 08 16:05:41.113137 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/proc.go:250 +0xb7 fp=0xc00005efe0 sp=0xc00005efb0 pc=0x4340c7
Apr 08 16:05:41.114211 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.goexit()
Apr 08 16:05:41.114211 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc00005efe8 sp=0xc00005efe0 pc=0x45e5e1
Apr 08 16:05:41.114211 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: created by runtime.init.5
Apr 08 16:05:41.114211 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/proc.go:239 +0x35
Apr 08 16:05:41.114211 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: goroutine 3 [GC sweep wait]:
Apr 08 16:05:41.114211 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.gopark(0x141d8f0, 0x20d0000, 0x140c, 0x1)
Apr 08 16:05:41.114211 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/proc.go:301 +0xef fp=0xc00005f7a8 sp=0xc00005f788 pc=0x43421f
Apr 08 16:05:41.114475 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.goparkunlock(...)
Apr 08 16:05:41.114475 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/proc.go:307
Apr 08 16:05:41.114475 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.bgsweep(0xc000086000)
Apr 08 16:05:41.114475 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/mgcsweep.go:70 +0x9c fp=0xc00005f7d8 sp=0xc00005f7a8 pc=0x42769c
Apr 08 16:05:41.114475 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.goexit()
Apr 08 16:05:41.114475 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc00005f7e0 sp=0xc00005f7d8 pc=0x45e5e1
Apr 08 16:05:41.114475 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: created by runtime.gcenable
Apr 08 16:05:41.114475 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/mgc.go:208 +0x58
Apr 08 16:05:41.114475 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: goroutine 4 [finalizer wait]:
Apr 08 16:05:41.114475 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.gopark(0x141d8f0, 0x20edfc8, 0x140f, 0x1)
Apr 08 16:05:41.114475 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/proc.go:301 +0xef fp=0xc00005ff58 sp=0xc00005ff38 pc=0x43421f
Apr 08 16:05:41.114475 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.goparkunlock(...)
Apr 08 16:05:41.114475 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/proc.go:307
Apr 08 16:05:41.114475 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.runfinq()
Apr 08 16:05:41.114475 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/mfinal.go:175 +0xa6 fp=0xc00005ffe0 sp=0xc00005ff58 pc=0x41e366
Apr 08 16:05:41.114475 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.goexit()
Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc00005ffe8 sp=0xc00005ffe0 pc=0x45e5e1
Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: created by runtime.createfing
Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/mfinal.go:156 +0x61
Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: goroutine 5 [chan receive]:
Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.gopark(0x141d8f0, 0xc0000ee058, 0x170d, 0x3)
Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/proc.go:301 +0xef fp=0xc00005e6d0 sp=0xc00005e6b0 pc=0x43421f
Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.goparkunlock(...)
Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/proc.go:307
Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.chanrecv(0xc0000ee000, 0xc00005e7b0, 0x20d0c01, 0x20d0cb8)
Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/chan.go:524 +0x2ea fp=0xc00005e760 sp=0xc00005e6d0 pc=0x40c17a
Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.chanrecv2(0xc0000ee000, 0xc00005e7b0, 0x1)
Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/chan.go:411 +0x2b fp=0xc00005e790 sp=0xc00005e760 pc=0x40be7b
Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: github.com/intel/multus-cni/vendor/k8s.io/klog.(*loggingT).flushDaemon(0x20d0ca0)
Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/src/multus-cni/gopath/src/github.com/intel/multus-cni/vendor/k8s.io/klog/klog.go:1010 +0x8b fp=0xc00005e7d8 sp=0xc00005e790 pc=0x6711eb
Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.goexit()
Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc00005e7e0 sp=0xc00005e7d8 pc=0x45e5e1
Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: created by github.com/intel/multus-cni/vendor/k8s.io/klog.init.0
Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/src/multus-cni/gopath/src/github.com/intel/multus-cni/vendor/k8s.io/klog/klog.go:411 +0xd6
Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: goroutine 18 [syscall]:
Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.notetsleepg(0x20d56a0, 0x12a03ec60, 0x0)
Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/lock_futex.go:227 +0x34 fp=0xc00005a760 sp=0xc00005a730 pc=0x410bd4
Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.timerproc(0x20d5680)
Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/time.go:311 +0x2ea fp=0xc00005a7d8 sp=0xc00005a760 pc=0x44fe0a
Apr 08 16:05:41.115166 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.goexit()
Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc00005a7e0 sp=0xc00005a7d8 pc=0x45e5e1
Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: created by runtime.(*timersBucket).addtimerLocked
Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/time.go:169 +0x10e
Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: goroutine 6 [chan receive]:
Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.gopark(0x141d8f0, 0xc0002ccbf8, 0xc00006170d, 0x3)
Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/proc.go:301 +0xef fp=0xc000060698 sp=0xc000060678 pc=0x43421f
Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.goparkunlock(...)
Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/proc.go:307
Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.chanrecv(0xc0002ccba0, 0xc0000607a8, 0xc0002bd701, 0xc0002e89c0)
Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/chan.go:524 +0x2ea fp=0xc000060728 sp=0xc000060698 pc=0x40c17a
Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.chanrecv2(0xc0002ccba0, 0xc0000607a8, 0x14)
Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/chan.go:411 +0x2b fp=0xc000060758 sp=0xc000060728 pc=0x40be7b
Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: github.com/intel/multus-cni/vendor/k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop(0xc0002dc540)
Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/src/multus-cni/gopath/src/github.com/intel/multus-cni/vendor/k8s.io/apimachinery/pkg/watch/mux.go:207 +0x66 fp=0xc0000607d8 sp=0xc000060758 pc=0x85c5e6
Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.goexit()
Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0000607e0 sp=0xc0000607d8 pc=0x45e5e1
Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: created by github.com/intel/multus-cni/vendor/k8s.io/apimachinery/pkg/watch.NewBroadcaster
Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/src/multus-cni/gopath/src/github.com/intel/multus-cni/vendor/k8s.io/apimachinery/pkg/watch/mux.go:75 +0xcc
Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: goroutine 7 [chan receive]:
Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.gopark(0x141d8f0, 0xc0002ccc58, 0x170d, 0x3)
Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/proc.go:301 +0xef fp=0xc000060ea8 sp=0xc000060e88 pc=0x43421f
Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.goparkunlock(...)
Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/proc.go:307
Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.chanrecv(0xc0002ccc00, 0xc000060f98, 0x1, 0xc000060f70)
Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/chan.go:524 +0x2ea fp=0xc000060f38 sp=0xc000060ea8 pc=0x40c17a
Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.chanrecv2(0xc0002ccc00, 0xc000060f98, 0x0)
Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/chan.go:411 +0x2b fp=0xc000060f68 sp=0xc000060f38 pc=0x40be7b
Apr 08 16:05:41.116344 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: github.com/intel/multus-cni/vendor/k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1(0x15649a0, 0xc0002c9710, 0xc0002f60f0)
Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/src/multus-cni/gopath/src/github.com/intel/multus-cni/vendor/k8s.io/client-go/tools/record/event.go:231 +0xa4 fp=0xc000060fc8 sp=0xc000060f68 pc=0xfe1914
Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.goexit()
Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc000060fd0 sp=0xc000060fc8 pc=0x45e5e1
Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: created by github.com/intel/multus-cni/vendor/k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/src/multus-cni/gopath/src/github.com/intel/multus-cni/vendor/k8s.io/client-go/tools/record/event.go:229 +0x6e
Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: goroutine 8 [chan receive]:
Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.gopark(0x141d8f0, 0xc0002ccd18, 0x170d, 0x3)
Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/proc.go:301 +0xef fp=0xc0000616a8 sp=0xc000061688 pc=0x43421f
Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.goparkunlock(...)
Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/proc.go:307
Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.chanrecv(0xc0002cccc0, 0xc000061798, 0x1, 0xc000061770)
Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/chan.go:524 +0x2ea fp=0xc000061738 sp=0xc0000616a8 pc=0x40c17a
Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.chanrecv2(0xc0002cccc0, 0xc000061798, 0x0)
Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/chan.go:411 +0x2b fp=0xc000061768 sp=0xc000061738 pc=0x40be7b
Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: github.com/intel/multus-cni/vendor/k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1(0x15649a0, 0xc0002c98f0, 0xc0002c98c0)
Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/src/multus-cni/gopath/src/github.com/intel/multus-cni/vendor/k8s.io/client-go/tools/record/event.go:231 +0xa4 fp=0xc0000617c8 sp=0xc000061768 pc=0xfe1914
Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.goexit()
Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0000617d0 sp=0xc0000617c8 pc=0x45e5e1
Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: created by github.com/intel/multus-cni/vendor/k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/src/multus-cni/gopath/src/github.com/intel/multus-cni/vendor/k8s.io/client-go/tools/record/event.go:229 +0x6e
Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: goroutine 10 [timer goroutine (idle)]:
Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.gopark(0x141d8f0, 0x20d5600, 0x1414, 0x1)
Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/proc.go:301 +0xef fp=0xc000061f60 sp=0xc000061f40 pc=0x43421f
Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.goparkunlock(...)
Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/proc.go:307
Apr 08 16:05:41.117745 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.timerproc(0x20d5600)
Apr 08 16:05:41.118840 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/time.go:303 +0x273 fp=0xc000061fd8 sp=0xc000061f60 pc=0x44fd93
Apr 08 16:05:41.118840 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.goexit()
Apr 08 16:05:41.118840 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc000061fe0 sp=0xc000061fd8 pc=0x45e5e1
Apr 08 16:05:41.118840 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: created by runtime.(*timersBucket).addtimerLocked
Apr 08 16:05:41.118840 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/time.go:169 +0x10e
Apr 08 16:05:41.118840 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: goroutine 34 [timer goroutine (idle)]:
Apr 08 16:05:41.118840 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.gopark(0x141d8f0, 0x20d5700, 0x1414, 0x1)
Apr 08 16:05:41.118840 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/proc.go:301 +0xef fp=0xc000312760 sp=0xc000312740 pc=0x43421f
Apr 08 16:05:41.118840 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.goparkunlock(...)
Apr 08 16:05:41.119468 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/proc.go:307
Apr 08 16:05:41.119468 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.timerproc(0x20d5700)
Apr 08 16:05:41.119468 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/time.go:303 +0x273 fp=0xc0003127d8 sp=0xc000312760 pc=0x44fd93
Apr 08 16:05:41.119468 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: runtime.goexit()
Apr 08 16:05:41.119468 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0003127e0 sp=0xc0003127d8 pc=0x45e5e1
Apr 08 16:05:41.119468 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]: created by runtime.(*timersBucket).addtimerLocked
Apr 08 16:05:41.119468 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal crio[1396]:         /usr/lib/golang/src/runtime/time.go:169 +0x10e
Apr 08 16:05:41.143967 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal systemd[1]: Started Process Core Dump (PID 43593/UID 0).
Apr 08 16:05:41.145908 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal hyperkube[1488]: I0408 16:05:41.145871    1488 factory.go:173] Using factory "raw" for container "/system.slice/system-systemd\\x2dcoredump.slice/systemd-coredump"
Apr 08 16:05:41.146399 ci-op-kf7v5-m-0.c.openshift-gce-devel-ci.internal hyperkube[1488]: I0408 16:05:41.146371    1488 manager.go:950] Added container: "/system.slice/system-systemd\\x2dcoredump.slice/systemd-coredump" (aliases: [], namespace: "")



Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Ted Yu 2020-04-09 22:01:15 UTC
Here is the PR upstream :

https://github.com/kubernetes/kubernetes/pull/90034

Comment 2 Ted Yu 2020-04-09 23:11:39 UTC
Here is related code from multus.go
```
        // send kubernetes events
        if delegate.Name != "" {
                kubeClient.Eventf(pod, v1.EventTypeNormal, "AddedInterface", "Add %s %v from %s", rt.IfName, ips, delegate.Name)
        } else {
                kubeClient.Eventf(pod, v1.EventTypeNormal, "AddedInterface", "Add %s %v", rt.IfName, ips)
```
I am not familiar with multus - the above should be examined.

PR #90034 would be good to have - k8s shouldn't panic even if GetReference() is unable to proceed.

Comment 3 Ted Yu 2020-04-10 00:53:33 UTC
I tried the following addition to staging/src/k8s.io/client-go/tools/reference/ref_test.go but was unable to reproduce the panic:

func TestGetReferencePod(t *testing.T) {
	tests := []struct {
		name               string
		input              *v1.Pod
		groupVersion       schema.GroupVersion
		expectedRefVersion string
	}{
		{
			name: "Pod",
			input: &v1.Pod{
				Status: v1.PodStatus{
					ContainerStatuses: []v1.ContainerStatus{
						{
							Name: "foo",
						},
					},
				},
			},
			groupVersion:       schema.GroupVersion{Group: "", Version: "v1"},
		},
	}
	for _, test := range tests {
		t.Run(test.name, func(t *testing.T) {
			scheme := runtime.NewScheme()
			scheme.AddKnownTypes(test.groupVersion, &v1.Pod{})
			if _, err := GetReference(scheme, test.input); err != nil {
			}
			if test.expectedRefVersion != "v1" {
				t.Errorf("expected %q, got %q", test.expectedRefVersion, "v1")
			}
		})
	}
}

Comment 4 Ted Yu 2020-04-10 02:36:06 UTC
Stepping into the GetReference() call, I saw <autogenerated>:2 when trying to step into v1.(*Pod).GetObjectKind() - though GoLand didn't show me the generated code.

I noted that multus seems to be using go 1.12 while k8s master uses go 1.13

Comment 5 Ted Yu 2020-04-12 02:33:29 UTC
I went over vendor/k8s.io/api/core/v1/generated.pb.go under multus
There were Pod methods such as Pod#ProtoMessage() but I didn't find Pod#GetObjectKind

Comment 6 Ted Yu 2020-04-17 02:33:18 UTC
Looking at staging/src/k8s.io/client-go/tools/reference/ref.go from k8s, the following code:
```
        var listMeta metav1.Common
        objectMeta, err := meta.Accessor(obj)
        if err != nil {
                listMeta, err = meta.CommonAccessor(obj)
```
is not in vendor/k8s.io/client-go/tools/reference/ref.go

It seems k8s dependency in multus should be upgraded. However the golang version difference seems to be a hurdle.

Comment 7 Ted Yu 2020-04-17 13:59:47 UTC
Issue upstream:

https://github.com/intel/multus-cni/issues/481

Comment 8 Douglas Smith 2020-04-17 18:29:23 UTC
Apparently the downstream build should be golang 1.13 

Luke Meyer from ART advises:

> ART OCP builds ignore what's in your dockerfile 
> and overwrite it with golang-builder:1.13 
> so downstream builds should already be 1.13

Will look towards updating the dockerfile with a golang 1.13.

Comment 9 Ted Yu 2020-04-20 15:04:30 UTC
After chatting with Tomofumi-san:

Proceed with:
https://github.com/intel/multus-cni/pull/482

Tomofumi-san will add more log in another PR to give us more clue if the crash happens again.

Comment 12 Weibin Liang 2020-04-23 14:34:00 UTC
There is no way to verify this bz because we still do not know how to reproduce it.
QE verify this bug now and will open it again if CI testing fail in the future.

Comment 13 Ted Yu 2020-05-01 16:03:37 UTC
If the crash is no longer seen in master log, can this be resolved ?

Comment 14 Tomofumi Hayashi 2020-05-03 06:49:02 UTC
I believe so. So let me check once again someday next week.

Comment 15 errata-xmlrpc 2020-07-13 17:26:42 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409


Note You need to log in before you can comment on or make changes to this bug.