Bug 2036622 - sdn-controller crashes when restarted while a previous egress IP assignment exists
Summary: sdn-controller crashes when restarted while a previous egress IP assignment e...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.10
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.10.0
Assignee: Ben Bennett
QA Contact: huirwang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-01-03 12:32 UTC by Alexander Constantinescu
Modified: 2022-08-25 08:28 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-03-12 04:40:32 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift sdn pull 386 0 None open Bug 2036622: Fix CloudPrivateIPConfig sync on sdn-controller restart 2022-01-03 12:36:58 UTC
Red Hat Product Errata RHSA-2022:0056 0 None None None 2022-03-12 04:40:42 UTC

Description Alexander Constantinescu 2022-01-03 12:32:42 UTC
Description of problem:

Restarting sdn-controller with existing egress IP assignments has the pod crashing with the following stack trace:

I0103 11:54:57.180079       1 egressip.go:314] CloudPrivateIPConfig: 10.0.150.25 is being moved and still exists, enqueuing its creation
E0103 11:54:57.180152       1 runtime.go:78] Observed a panic: "assignment to entry in nil map" (assignment to entry in nil map)
goroutine 111 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x1756ce0, 0x1b987f0})
        k8s.io/apimachinery.0-rc.0/pkg/util/runtime/runtime.go:74 +0x85
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc0001a80c0})
        k8s.io/apimachinery.0-rc.0/pkg/util/runtime/runtime.go:48 +0x75
panic({0x1756ce0, 0x1b987f0})
        runtime/panic.go:1038 +0x215
github.com/openshift/sdn/pkg/network/master.(*egressIPManager).ClaimEgressIP(0xc000118000, 0x2, {0xc0006b2910, 0xb}, {0xc0006b2920, 0x0}, {0x0, 0x0})
        github.com/openshift/sdn/pkg/network/master/egressip.go:315 +0x3bc
github.com/openshift/sdn/pkg/network/common.(*EgressIPTracker).syncEgressNodeState(0xc000560000, 0xc0005a61c0, 0x18)
        github.com/openshift/sdn/pkg/network/common/egressip.go:625 +0x471
github.com/openshift/sdn/pkg/network/common.(*EgressIPTracker).syncEgressIPs(0xc000560000)
        github.com/openshift/sdn/pkg/network/common/egressip.go:600 +0xeb
github.com/openshift/sdn/pkg/network/common.(*EgressIPTracker).UpdateNetNamespaceEgress(0xc000560000, 0xc000612f68)
        github.com/openshift/sdn/pkg/network/common/egressip.go:560 +0x426
github.com/openshift/sdn/pkg/network/common.(*EgressIPTracker).handleAddOrUpdateNetNamespace(0x0, {0x195b860, 0xc000612f68}, {0x100000000000000, 0x0}, {0x19841b8, 0x5})
        github.com/openshift/sdn/pkg/network/common/egressip.go:512 +0x1cc
github.com/openshift/sdn/pkg/network/common.InformerFuncs.func1({0x195b860, 0xc000612f68})
        github.com/openshift/sdn/pkg/network/common/informers.go:19 +0x39
k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd(...)
        k8s.io/client-go.0-rc.0/tools/cache/controller.go:231
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
        k8s.io/client-go.0-rc.0/tools/cache/shared_informer.go:777 +0x9f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x7fdc2c166ba8)
        k8s.io/apimachinery.0-rc.0/pkg/util/wait/wait.go:155 +0x67
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00066d738, {0x1ba1cc0, 0xc00051c030}, 0x1, 0xc0001822a0)
        k8s.io/apimachinery.0-rc.0/pkg/util/wait/wait.go:156 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0, 0x3b9aca00, 0x0, 0x0, 0x0)
        k8s.io/apimachinery.0-rc.0/pkg/util/wait/wait.go:133 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
        k8s.io/apimachinery.0-rc.0/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*processorListener).run(0xc000178380)
        k8s.io/client-go.0-rc.0/tools/cache/shared_informer.go:771 +0x6b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        k8s.io/apimachinery.0-rc.0/pkg/util/wait/wait.go:73 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        k8s.io/apimachinery.0-rc.0/pkg/util/wait/wait.go:71 +0x88
panic: assignment to entry in nil map [recovered]
        panic: assignment to entry in nil map

goroutine 111 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc0001a80c0})
        k8s.io/apimachinery.0-rc.0/pkg/util/runtime/runtime.go:55 +0xd8
panic({0x1756ce0, 0x1b987f0})
        runtime/panic.go:1038 +0x215
github.com/openshift/sdn/pkg/network/master.(*egressIPManager).ClaimEgressIP(0xc000118000, 0x2, {0xc0006b2910, 0xb}, {0xc0006b2920, 0x0}, {0x0, 0x0})
        github.com/openshift/sdn/pkg/network/master/egressip.go:315 +0x3bc
github.com/openshift/sdn/pkg/network/common.(*EgressIPTracker).syncEgressNodeState(0xc000560000, 0xc0005a61c0, 0x18)
        github.com/openshift/sdn/pkg/network/common/egressip.go:625 +0x471
github.com/openshift/sdn/pkg/network/common.(*EgressIPTracker).syncEgressIPs(0xc000560000)
        github.com/openshift/sdn/pkg/network/common/egressip.go:600 +0xeb
github.com/openshift/sdn/pkg/network/common.(*EgressIPTracker).UpdateNetNamespaceEgress(0xc000560000, 0xc000612f68)
        github.com/openshift/sdn/pkg/network/common/egressip.go:560 +0x426
github.com/openshift/sdn/pkg/network/common.(*EgressIPTracker).handleAddOrUpdateNetNamespace(0x0, {0x195b860, 0xc000612f68}, {0x100000000000000, 0x0}, {0x19841b8, 0x5})
        github.com/openshift/sdn/pkg/network/common/egressip.go:512 +0x1cc
github.com/openshift/sdn/pkg/network/common.InformerFuncs.func1({0x195b860, 0xc000612f68})
        github.com/openshift/sdn/pkg/network/common/informers.go:19 +0x39
k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd(...)
        k8s.io/client-go.0-rc.0/tools/cache/controller.go:231
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
        k8s.io/client-go.0-rc.0/tools/cache/shared_informer.go:777 +0x9f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x7fdc2c166ba8)
        k8s.io/apimachinery.0-rc.0/pkg/util/wait/wait.go:155 +0x67
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00066d738, {0x1ba1cc0, 0xc00051c030}, 0x1, 0xc0001822a0)
        k8s.io/apimachinery.0-rc.0/pkg/util/wait/wait.go:156 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0, 0x3b9aca00, 0x0, 0x0, 0x0)
        k8s.io/apimachinery.0-rc.0/pkg/util/wait/wait.go:133 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
        k8s.io/apimachinery.0-rc.0/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*processorListener).run(0xc000178380)
        k8s.io/client-go.0-rc.0/tools/cache/shared_informer.go:771 +0x6b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        k8s.io/apimachinery.0-rc.0/pkg/util/wait/wait.go:73 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        k8s.io/apimachinery.0-rc.0/pkg/util/wait/wait.go:71 +0x88  


Version-Release number of selected component (if applicable):


How reproducible:

Always


Steps to Reproduce:
1. Perform egress IP assignments on a public cloud (ex: AWS)
2. Restart the leading sdn-controller pod (ex: by deleting it) 
3.

Actual results:

Crashloops

Expected results:

Not crashlooping 

Additional info:

Comment 7 errata-xmlrpc 2022-03-12 04:40:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056

Comment 8 anglinajolie 2022-08-25 06:47:07 UTC Comment hidden (spam)

Note You need to log in before you can comment on or make changes to this bug.