Bug 1886600 - panic: assignment to entry in nil map
Summary: panic: assignment to entry in nil map
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cluster Version Operator
Version: 4.5
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.7.0
Assignee: W. Trevor King
QA Contact: Yang Yang
URL:
Whiteboard:
Depends On:
Blocks: 1886935
TreeView+ depends on / blocked
 
Reported: 2020-10-08 21:08 UTC by Benjamin Gilbert
Modified: 2022-12-14 08:15 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
: 1886935 (view as bug list)
Environment:
Undiagnosed panic detected in pod
Last Closed: 2021-02-24 15:24:26 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github kubernetes kubernetes pull 87821 0 None closed Check Annotations map against nil for ConfigMapLock#Update() 2021-01-15 04:33:42 UTC
Github kubernetes kubernetes pull 95430 0 None closed [1.17 cherrypick] Check Annotations map against nil for ConfigMapLock#Update() 2021-01-15 04:33:42 UTC
Github openshift cluster-version-operator pull 406 0 None closed Bump client-go and library-go to current 4.6 tips 2021-01-15 04:33:43 UTC
Github openshift cluster-version-operator pull 420 0 None closed vendor: Bump client-go and library-go to current 4.6 tips (2020-07-24) 2021-01-15 04:33:43 UTC
Red Hat Product Errata RHSA-2020:5633 0 None None None 2021-02-24 15:24:54 UTC

Description Benjamin Gilbert 2020-10-08 21:08:09 UTC
https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-azure-4.5/1314281190638751744
https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_ovn-kubernetes/227/pull-ci-openshift-ovn-kubernetes-release-4.5-e2e-gcp-ovn-upgrade/1314289613564547072

panic: assignment to entry in nil map

goroutine 80 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0x105
panic(0x163d780, 0x1a61ab0)
	/usr/local/go/src/runtime/panic.go:679 +0x1b2
k8s.io/client-go/tools/leaderelection/resourcelock.(*ConfigMapLock).Update(0xc0000f6000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/client-go/tools/leaderelection/resourcelock/configmaplock.go:90 +0x129
k8s.io/client-go/tools/leaderelection.(*LeaderElector).release(0xc000f56b40, 0x6fc23ac00)
	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:306 +0xc9
k8s.io/client-go/tools/leaderelection.(*LeaderElector).renew(0xc000f56b40, 0x1abc1a0, 0xc0004a4180)
	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:294 +0x166
k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run(0xc000f56b40, 0x1abc1a0, 0xc0004a4140)
	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:208 +0x15d
k8s.io/client-go/tools/leaderelection.RunOrDie(0x1abc1a0, 0xc00030de00, 0x1acbcc0, 0xc0000f6000, 0x14f46b0400, 0xa7a358200, 0x6fc23ac00, 0xc001060f30, 0xc0010baa10, 0x0, ...)
	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:221 +0x96
github.com/openshift/cluster-version-operator/pkg/start.(*Options).run.func4(0x1abc1a0, 0xc00030de00, 0xc0000f6000, 0xc000f3ab40, 0xc000099d80, 0x1abc1a0, 0xc00030dd80, 0xc00094b500, 0xc0010ba9d0)
	/go/src/github.com/openshift/cluster-version-operator/pkg/start/start.go:179 +0x1b2
created by github.com/openshift/cluster-version-operator/pkg/start.(*Options).run
	/go/src/github.com/openshift/cluster-version-operator/pkg/start/start.go:177 +0x3f9

Comment 1 Scott Dodson 2020-10-09 00:09:02 UTC
Both of these are in 4.5 where we recently merged https://github.com/openshift/cluster-version-operator/pull/446 which is relevant to leader election, this is probably not a 4.6 blocker but we'll evaluate deeper tomorrow, leaving target release unassigned for now.

Comment 2 W. Trevor King 2020-10-09 04:19:58 UTC
Looks like we need an upstream client-go fix backported.  I'll try to figure out what our process is for that...

Comment 3 W. Trevor King 2020-10-09 05:51:19 UTC
I've filed an upstream PR requesting a backport to 1.17.

Comment 4 Scott Dodson 2020-10-09 13:51:13 UTC
This is 4.5 specific but the bot will revert my changes if i set target release to 4.5 directly, so setting it 4.7 now to get off the blocker list.

Comment 5 W. Trevor King 2020-10-09 18:51:56 UTC
4.6+ use 1.19 clients which have this patched, so setting this modified to unblock a new-in-4.5 patch.

Comment 7 Yang Yang 2020-10-27 08:18:09 UTC
I'm trying to verify it on 4.7.0-0.nightly-2020-10-24-155529, but fresh installation on Azure faces Bug1891702, and GCP OVN cluster upgrade faces Bug1891716. I'll verify it once those bugs get fixed.

Comment 8 Yang Yang 2020-11-20 10:10:38 UTC
I'm verifying it on 4.7.0-0.nightly-2020-11-18-203317. 

Steps of verification are as below:
1. Fresh install an Azure cluster -- passed
2. Set up an OVN cluster on GCP with 4.6.5, and then do an upgrade to 4.7.0-0.nightly-2020-11-18-203317 - passed

There's no panic found in CVO, hence moving it to verified state.

Comment 11 errata-xmlrpc 2021-02-24 15:24:26 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633

Comment 12 anglinajolie 2022-08-25 06:42:10 UTC Comment hidden (spam)
Comment 13 Allenwood 2022-12-14 08:15:04 UTC Comment hidden (spam)

Note You need to log in before you can comment on or make changes to this bug.