Bug 1445504 - sentry: nil panic in pkg/sdn/plugin/pod.go", line 228, in UpdateLocalMulticastRules
Summary: sentry: nil panic in pkg/sdn/plugin/pod.go", line 228, in UpdateLocalMulticas...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 3.5.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: ---
Assignee: Dan Williams
QA Contact: Meng Bo
URL:
Whiteboard:
Depends On:
Blocks: 1445505
TreeView+ depends on / blocked
 
Reported: 2017-04-25 20:04 UTC by Ben Bennett
Modified: 2017-08-16 19:51 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: The initialization order of the SDN plugin sets the event handler up too late. Consequence: Early events have no handler so the SDN panics Fix: Re-order the SDN initialization so the event handler is in place before it can ever be called Result: No more panics
Clone Of:
: 1445505 (view as bug list)
Environment:
Last Closed: 2017-08-10 05:21:25 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Origin (Github) 13742 0 None None None 2017-04-25 20:04:47 UTC
Origin (Github) 13766 0 None None None 2017-04-25 20:06:24 UTC
Red Hat Product Errata RHEA-2017:1716 0 normal SHIPPED_LIVE Red Hat OpenShift Container Platform 3.6 RPM Release Advisory 2017-08-10 09:02:50 UTC

Description Ben Bennett 2017-04-25 20:04:48 UTC 
Description of problem:

Seen in the sentry logs:
runtime.errorString: runtime error: invalid memory address or nil pointer dereference
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/cmd/util/serviceability/panic.go", line 28, in CapturePanic-fm
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/runtime/runtime.go", line 51, in HandleCrash
  File "/usr/lib/golang/src/runtime/asm_amd64.s", line 479, in call32
  File "/usr/lib/golang/src/runtime/panic.go", line 458, in gopanic
  File "/usr/lib/golang/src/runtime/panic.go", line 62, in panicmem
  File "/usr/lib/golang/src/runtime/sigpanic_unix.go", line 24, in sigpanic
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/pod.go", line 228, in UpdateLocalMulticastRules
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/multitenant.go", line 100, in updatePodNetwork
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/multitenant.go", line 105, in AddNetNamespace
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/vnids_node.go", line 194, in func1
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/eventqueue.go", line 108, in func1
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/cache/delta_fifo.go", line 420, in Pop
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/eventqueue.go", line 114, in Pop
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/common.go", line 146, in RunEventQueue
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/vnids_node.go", line 203, in watchNetNamespaces
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/vnids_node.go", line 173, in watchNetNamespaces)-fm
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go", line 96, in func1
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go", line 97, in JitterUntil
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go", line 52, in Until
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go", line 43, in Forever
  File "/usr/lib/golang/src/runtime/asm_amd64.s", line 2086, in goexit

Version-Release number of selected component (if applicable):

3.5.0 ?

How reproducible:

Unknown


Steps to Reproduce:
1. Unknown
Comment 1 Ben Bennett 2017-04-25 20:06:38 UTC 
Fixed by https://github.com/openshift/origin/pull/13766
Comment 3 Hongan Li 2017-06-05 09:40:46 UTC 
do regression testing on atomic-openshift-3.6.94-1.git.0.ba4aad2.el7.x86_64 and no issue found.

and the node log at startup as below:
Jun 05 05:32:13 atomic-openshift-node[2657]: I0605 05:32:13.575439    2657 node.go:236] Waiting for kubelet network plugin initialization
Jun 05 05:32:14 atomic-openshift-node[2657]: I0605 05:32:14.121589    2657 node.go:269] Starting openshift-sdn pod manager
Jun 05 05:32:14 atomic-openshift-node[2657]: I0605 05:32:14.149582    2657 node.go:292] openshift-sdn network plugin ready
Comment 5 errata-xmlrpc 2017-08-10 05:21:25 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1716
Note You need to log in before you can comment on or make changes to this bug.