Bug 1445505

Summary: sentry: nil panic in pkg/sdn/plugin/pod.go", line 228, in UpdateLocalMulticastRules
Product: OpenShift Container Platform Reporter: Ben Bennett <bbennett>
Component: NetworkingAssignee: Ben Bennett <bbennett>
Status: CLOSED ERRATA QA Contact: Meng Bo <bmeng>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 3.5.0CC: aos-bugs, bmeng, dcbw, smunilla
Target Milestone: ---   
Target Release: 3.5.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: The initialization order of the SDN plugin sets the event handler up too late. Consequence: Early events have no handler so the SDN panics Fix: Re-order the SDN initialization so the event handler is in place before it can ever be called Result: No more panics
 Story Points: ---
Clone Of: 1445504 Environment:
Last Closed: 2017-05-18 09:28:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1445504    
Bug Blocks:    

Description Ben Bennett 2017-04-25 20:07:49 UTC 
+++ This bug was initially created as a clone of Bug #1445504 +++

Description of problem:

Seen in the sentry logs:
runtime.errorString: runtime error: invalid memory address or nil pointer dereference
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/cmd/util/serviceability/panic.go", line 28, in CapturePanic-fm
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/runtime/runtime.go", line 51, in HandleCrash
  File "/usr/lib/golang/src/runtime/asm_amd64.s", line 479, in call32
  File "/usr/lib/golang/src/runtime/panic.go", line 458, in gopanic
  File "/usr/lib/golang/src/runtime/panic.go", line 62, in panicmem
  File "/usr/lib/golang/src/runtime/sigpanic_unix.go", line 24, in sigpanic
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/pod.go", line 228, in UpdateLocalMulticastRules
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/multitenant.go", line 100, in updatePodNetwork
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/multitenant.go", line 105, in AddNetNamespace
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/vnids_node.go", line 194, in func1
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/eventqueue.go", line 108, in func1
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/cache/delta_fifo.go", line 420, in Pop
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/eventqueue.go", line 114, in Pop
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/common.go", line 146, in RunEventQueue
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/vnids_node.go", line 203, in watchNetNamespaces
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/vnids_node.go", line 173, in watchNetNamespaces)-fm
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go", line 96, in func1
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go", line 97, in JitterUntil
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go", line 52, in Until
  File "/builddir/build/BUILD/atomic-openshift-git-0.2acc0f0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go", line 43, in Forever
  File "/usr/lib/golang/src/runtime/asm_amd64.s", line 2086, in goexit

Version-Release number of selected component (if applicable):

3.5.0 ?

How reproducible:

Unknown


Steps to Reproduce:
1. Unknown

--- Additional comment from Ben Bennett on 2017-04-25 16:06:38 EDT ---

Fixed by https://github.com/openshift/origin/pull/13766
Comment 2 Meng Bo 2017-05-03 11:23:20 UTC 
Do regression testing on ocp build v3.5.5.12 no issue found.

And the node starting log has been changed to following:

May 03 07:20:33 ip-172-18-10-93.ec2.internal atomic-openshift-node[32671]: I0503 07:20:33.300345   32671 node.go:233] Waiting for kubelet network plugin initialization
May 03 07:20:33 ip-172-18-10-93.ec2.internal atomic-openshift-node[32671]: I0503 07:20:33.302424   32671 node.go:347] Watch Sync event for Service "docker-registry"
May 03 07:20:33 ip-172-18-10-93.ec2.internal atomic-openshift-node[32671]: I0503 07:20:33.317274   32671 node.go:347] Watch Sync event for Service "kubernetes"
May 03 07:20:33 ip-172-18-10-93.ec2.internal atomic-openshift-node[32671]: I0503 07:20:33.365122   32671 node.go:347] Watch Sync event for Service "registry-console"
May 03 07:20:33 ip-172-18-10-93.ec2.internal atomic-openshift-node[32671]: I0503 07:20:33.369829   32671 node.go:347] Watch Sync event for Service "router"
May 03 07:20:33 ip-172-18-10-93.ec2.internal atomic-openshift-node[32671]: I0503 07:20:33.839026   32671 node.go:244] Starting openshift-sdn pod manager
May 03 07:20:33 ip-172-18-10-93.ec2.internal atomic-openshift-node[32671]: I0503 07:20:33.842518   32671 node.go:267] openshift-sdn network plugin ready
Comment 4 errata-xmlrpc 2017-05-18 09:28:53 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1235