Bug 1886935 - panic: assignment to entry in nil map
Summary: panic: assignment to entry in nil map
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cluster Version Operator
Version: 4.5
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.6.0
Assignee: W. Trevor King
QA Contact: Yang Yang
URL:
Whiteboard:
Depends On: 1886600
Blocks: 1886939
TreeView+ depends on / blocked
 
Reported: 2020-10-09 18:55 UTC by W. Trevor King
Modified: 2020-10-27 16:48 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of: 1886600
: 1886939 (view as bug list)
Environment:
Undiagnosed panic detected in pod
Last Closed: 2020-10-27 16:47:52 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github kubernetes kubernetes pull 87821 0 None closed Check Annotations map against nil for ConfigMapLock#Update() 2020-12-04 18:51:27 UTC
Github openshift cluster-version-operator pull 406 0 None closed Bump client-go and library-go to current 4.6 tips 2020-12-04 18:51:28 UTC
Github openshift cluster-version-operator pull 420 0 None closed vendor: Bump client-go and library-go to current 4.6 tips (2020-07-24) 2020-12-04 18:51:28 UTC
Red Hat Product Errata RHBA-2020:4196 0 None None None 2020-10-27 16:48:11 UTC

Description W. Trevor King 2020-10-09 18:55:56 UTC
+++ This bug was initially created as a clone of Bug #1886600 +++

https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-azure-4.5/1314281190638751744
https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_ovn-kubernetes/227/pull-ci-openshift-ovn-kubernetes-release-4.5-e2e-gcp-ovn-upgrade/1314289613564547072

panic: assignment to entry in nil map

goroutine 80 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0x105
panic(0x163d780, 0x1a61ab0)
	/usr/local/go/src/runtime/panic.go:679 +0x1b2
k8s.io/client-go/tools/leaderelection/resourcelock.(*ConfigMapLock).Update(0xc0000f6000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/client-go/tools/leaderelection/resourcelock/configmaplock.go:90 +0x129
k8s.io/client-go/tools/leaderelection.(*LeaderElector).release(0xc000f56b40, 0x6fc23ac00)
	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:306 +0xc9
k8s.io/client-go/tools/leaderelection.(*LeaderElector).renew(0xc000f56b40, 0x1abc1a0, 0xc0004a4180)
	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:294 +0x166
k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run(0xc000f56b40, 0x1abc1a0, 0xc0004a4140)
	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:208 +0x15d
k8s.io/client-go/tools/leaderelection.RunOrDie(0x1abc1a0, 0xc00030de00, 0x1acbcc0, 0xc0000f6000, 0x14f46b0400, 0xa7a358200, 0x6fc23ac00, 0xc001060f30, 0xc0010baa10, 0x0, ...)
	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:221 +0x96
github.com/openshift/cluster-version-operator/pkg/start.(*Options).run.func4(0x1abc1a0, 0xc00030de00, 0xc0000f6000, 0xc000f3ab40, 0xc000099d80, 0x1abc1a0, 0xc00030dd80, 0xc00094b500, 0xc0010ba9d0)
	/go/src/github.com/openshift/cluster-version-operator/pkg/start/start.go:179 +0x1b2
created by github.com/openshift/cluster-version-operator/pkg/start.(*Options).run
	/go/src/github.com/openshift/cluster-version-operator/pkg/start/start.go:177 +0x3f9

--- Additional comment from Scott Dodson on 2020-10-09 00:09:02 UTC ---

Both of these are in 4.5 where we recently merged https://github.com/openshift/cluster-version-operator/pull/446 which is relevant to leader election, this is probably not a 4.6 blocker but we'll evaluate deeper tomorrow, leaving target release unassigned for now.

--- Additional comment from W. Trevor King on 2020-10-09 04:19:58 UTC ---

Looks like we need an upstream client-go fix backported.  I'll try to figure out what our process is for that...

--- Additional comment from W. Trevor King on 2020-10-09 05:51:19 UTC ---

I've filed an upstream PR requesting a backport to 1.17.

--- Additional comment from Scott Dodson on 2020-10-09 13:51:13 UTC ---

This is 4.5 specific but the bot will revert my changes if i set target release to 4.5 directly, so setting it 4.7 now to get off the blocker list.

--- Additional comment from W. Trevor King on 2020-10-09 18:51:56 UTC ---

4.6+ use 1.19 clients which have this patched, so setting this modified to unblock a new-in-4.5 patch.

Comment 3 Yang Yang 2020-10-10 10:06:50 UTC
I'm verifying it on 4.6.0-0.nightly-2020-10-10-041109. 

Steps of verification are as below:
1. Fresh install an Azure cluster -- passed
2. Set up an OVN cluster on GCP with 4.5.14, and then do an upgrade to 4.6.0-0.nightly-2020-10-10-041109 - passed

There's no panic found in CVO, hence moving it to verified state.

Comment 6 errata-xmlrpc 2020-10-27 16:47:52 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196


Note You need to log in before you can comment on or make changes to this bug.