Bug 1445504

Summary: sentry: nil panic in pkg/sdn/plugin/pod.go", line 228, in UpdateLocalMulticastRules
Product: OpenShift Container Platform Reporter: Ben Bennett <bbennett>
Component: NetworkingAssignee: Dan Williams <dcbw>
Status: CLOSED ERRATA QA Contact: Meng Bo <bmeng>
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.5.0CC: aos-bugs, hongli, smunilla
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: The initialization order of the SDN plugin sets the event handler up too late. Consequence: Early events have no handler so the SDN panics Fix: Re-order the SDN initialization so the event handler is in place before it can ever be called Result: No more panics
 Story Points: ---
Clone Of:
: 1445505 (view as bug list) Environment:
Last Closed: 2017-08-10 05:21:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1445505    

Description Ben Bennett 2017-04-25 20:04:48 UTC 
Description of problem:

Seen in the sentry logs:
runtime.errorString: runtime error: invalid memory address or nil pointer dereference
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/cmd/util/serviceability/panic.go", line 28, in CapturePanic-fm
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/runtime/runtime.go", line 51, in HandleCrash
  File "/usr/lib/golang/src/runtime/asm_amd64.s", line 479, in call32
  File "/usr/lib/golang/src/runtime/panic.go", line 458, in gopanic
  File "/usr/lib/golang/src/runtime/panic.go", line 62, in panicmem
  File "/usr/lib/golang/src/runtime/sigpanic_unix.go", line 24, in sigpanic
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/pod.go", line 228, in UpdateLocalMulticastRules
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/multitenant.go", line 100, in updatePodNetwork
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/multitenant.go", line 105, in AddNetNamespace
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/vnids_node.go", line 194, in func1
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/eventqueue.go", line 108, in func1
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/cache/delta_fifo.go", line 420, in Pop
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/eventqueue.go", line 114, in Pop
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/common.go", line 146, in RunEventQueue
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/vnids_node.go", line 203, in watchNetNamespaces
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/vnids_node.go", line 173, in watchNetNamespaces)-fm
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go", line 96, in func1
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go", line 97, in JitterUntil
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go", line 52, in Until
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go", line 43, in Forever
  File "/usr/lib/golang/src/runtime/asm_amd64.s", line 2086, in goexit

Version-Release number of selected component (if applicable):

3.5.0 ?

How reproducible:

Unknown


Steps to Reproduce:
1. Unknown
Comment 1 Ben Bennett 2017-04-25 20:06:38 UTC 
Fixed by https://github.com/openshift/origin/pull/13766
Comment 3 Hongan Li 2017-06-05 09:40:46 UTC 
do regression testing on atomic-openshift-3.6.94-1.git.0.ba4aad2.el7.x86_64 and no issue found.

and the node log at startup as below:
Jun 05 05:32:13 atomic-openshift-node[2657]: I0605 05:32:13.575439    2657 node.go:236] Waiting for kubelet network plugin initialization
Jun 05 05:32:14 atomic-openshift-node[2657]: I0605 05:32:14.121589    2657 node.go:269] Starting openshift-sdn pod manager
Jun 05 05:32:14 atomic-openshift-node[2657]: I0605 05:32:14.149582    2657 node.go:292] openshift-sdn network plugin ready
Comment 5 errata-xmlrpc 2017-08-10 05:21:25 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1716