Bug 1445505 - sentry: nil panic in pkg/sdn/plugin/pod.go", line 228, in UpdateLocalMulticastRules
Summary: sentry: nil panic in pkg/sdn/plugin/pod.go", line 228, in UpdateLocalMulticas...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 3.5.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: 3.5.z
Assignee: Ben Bennett
QA Contact: Meng Bo
URL:
Whiteboard:
Depends On: 1445504
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-04-25 20:07 UTC by Ben Bennett
Modified: 2017-05-18 09:28 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: The initialization order of the SDN plugin sets the event handler up too late. Consequence: Early events have no handler so the SDN panics Fix: Re-order the SDN initialization so the event handler is in place before it can ever be called Result: No more panics
Clone Of: 1445504
Environment:
Last Closed: 2017-05-18 09:28:53 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift ose pull 722 0 None None None 2017-04-25 20:07:48 UTC
Origin (Github) 13742 0 None None None 2017-04-25 20:07:48 UTC
Red Hat Product Errata RHBA-2017:1235 0 normal SHIPPED_LIVE OpenShift Container Platform 3.5, 3.4, 3.3, and 3.1 bug fix update 2017-05-18 13:15:52 UTC

Description Ben Bennett 2017-04-25 20:07:49 UTC 
+++ This bug was initially created as a clone of Bug #1445504 +++

Description of problem:

Seen in the sentry logs:
runtime.errorString: runtime error: invalid memory address or nil pointer dereference
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/cmd/util/serviceability/panic.go", line 28, in CapturePanic-fm
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/runtime/runtime.go", line 51, in HandleCrash
  File "/usr/lib/golang/src/runtime/asm_amd64.s", line 479, in call32
  File "/usr/lib/golang/src/runtime/panic.go", line 458, in gopanic
  File "/usr/lib/golang/src/runtime/panic.go", line 62, in panicmem
  File "/usr/lib/golang/src/runtime/sigpanic_unix.go", line 24, in sigpanic
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/pod.go", line 228, in UpdateLocalMulticastRules
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/multitenant.go", line 100, in updatePodNetwork
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/multitenant.go", line 105, in AddNetNamespace
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/vnids_node.go", line 194, in func1
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/eventqueue.go", line 108, in func1
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/cache/delta_fifo.go", line 420, in Pop
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/eventqueue.go", line 114, in Pop
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/common.go", line 146, in RunEventQueue
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/vnids_node.go", line 203, in watchNetNamespaces
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/vnids_node.go", line 173, in watchNetNamespaces)-fm
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go", line 96, in func1
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go", line 97, in JitterUntil
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go", line 52, in Until
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go", line 43, in Forever
  File "/usr/lib/golang/src/runtime/asm_amd64.s", line 2086, in goexit

Version-Release number of selected component (if applicable):

3.5.0 ?

How reproducible:

Unknown


Steps to Reproduce:
1. Unknown

--- Additional comment from Ben Bennett on 2017-04-25 16:06:38 EDT ---

Fixed by https://github.com/openshift/origin/pull/13766
Comment 2 Meng Bo 2017-05-03 11:23:20 UTC 
Do regression testing on ocp build v3.5.5.12 no issue found.

And the node starting log has been changed to following:

May 03 07:20:33 ip-172-18-10-93.ec2.internal atomic-openshift-node[32671]: I0503 07:20:33.300345   32671 node.go:233] Waiting for kubelet network plugin initialization
May 03 07:20:33 ip-172-18-10-93.ec2.internal atomic-openshift-node[32671]: I0503 07:20:33.302424   32671 node.go:347] Watch Sync event for Service "docker-registry"
May 03 07:20:33 ip-172-18-10-93.ec2.internal atomic-openshift-node[32671]: I0503 07:20:33.317274   32671 node.go:347] Watch Sync event for Service "kubernetes"
May 03 07:20:33 ip-172-18-10-93.ec2.internal atomic-openshift-node[32671]: I0503 07:20:33.365122   32671 node.go:347] Watch Sync event for Service "registry-console"
May 03 07:20:33 ip-172-18-10-93.ec2.internal atomic-openshift-node[32671]: I0503 07:20:33.369829   32671 node.go:347] Watch Sync event for Service "router"
May 03 07:20:33 ip-172-18-10-93.ec2.internal atomic-openshift-node[32671]: I0503 07:20:33.839026   32671 node.go:244] Starting openshift-sdn pod manager
May 03 07:20:33 ip-172-18-10-93.ec2.internal atomic-openshift-node[32671]: I0503 07:20:33.842518   32671 node.go:267] openshift-sdn network plugin ready
Comment 4 errata-xmlrpc 2017-05-18 09:28:53 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1235
Note You need to log in before you can comment on or make changes to this bug.