Bug 2028030

Summary:	Panic detected in cluster-image-registry-operator pod
Product:	OpenShift Container Platform	Reporter:	Oleg Bulatov <obulatov>
Component:	Image Registry	Assignee:	Oleg Bulatov <obulatov>
Status:	CLOSED ERRATA	QA Contact:	XiuJuan Wang <xiuwang>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	4.6	CC:	aos-bugs
Target Milestone:	---
Target Release:	4.10.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	Cause: the operator was modifying objects from the informer Consequence: these objects could be concurrently modified by the informer and cause race conditions Fix: copy objects before modifying them Result: controllers and informers have different copies of the object and don't have race conditions	Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-03-10 16:31:07 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Oleg Bulatov 2021-12-01 10:54:40 UTC

Description of problem:

The image registry operator sometimes panics.

Version-Release number of selected component (if applicable):

4.x

How reproducible:

~0.1% of CI failures

Steps to Reproduce:
1. only in CI

Actual results:

https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.10-e2e-vsphere-upi-serial/1465808428298407936

E1130 23:39:58.467231       1 runtime.go:78] Observed a panic: runtime.boundsError{x:9, y:9, signed:true, code:0x0} (runtime error: index out of range [9] with length 9)
goroutine 709 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x2755c20, 0xc000ec7638})
	/go/src/github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0x85
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x2de70a0})
	/go/src/github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x75
panic({0x2755c20, 0xc000ec7638})
	/usr/lib/golang/src/runtime/panic.go:1038 +0x215
github.com/openshift/api/operator/v1.(*OperatorStatus).DeepCopyInto(0xc001b75ed8, 0xc000e5b0d8)
	/go/src/github.com/openshift/cluster-image-registry-operator/vendor/github.com/openshift/api/operator/v1/zz_generated.deepcopy.go:2838 +0x25d
github.com/openshift/api/imageregistry/v1.(*ImageRegistryStatus).DeepCopyInto(0xc001b75ed8, 0xc000e5b0d8)
	/go/src/github.com/openshift/cluster-image-registry-operator/vendor/github.com/openshift/api/imageregistry/v1/zz_generated.deepcopy.go:536 +0x74
github.com/openshift/api/imageregistry/v1.(*Config).DeepCopyInto(0xc001b75c00, 0xc000e5ae00)
	/go/src/github.com/openshift/cluster-image-registry-operator/vendor/github.com/openshift/api/imageregistry/v1/zz_generated.deepcopy.go:22 +0x105
github.com/openshift/api/imageregistry/v1.(*Config).DeepCopy(...)
	/go/src/github.com/openshift/cluster-image-registry-operator/vendor/github.com/openshift/api/imageregistry/v1/zz_generated.deepcopy.go:32
github.com/openshift/cluster-image-registry-operator/pkg/operator.(*Controller).sync(0xc0007c3770)
	/go/src/github.com/openshift/cluster-image-registry-operator/pkg/operator/controller.go:251 +0x9e
github.com/openshift/cluster-image-registry-operator/pkg/operator.(*Controller).eventProcessor.func1(0xc0007c3770, {0x23f98a0, 0x2de70a0})
	/go/src/github.com/openshift/cluster-image-registry-operator/pkg/operator/controller.go:367 +0xde
github.com/openshift/cluster-image-registry-operator/pkg/operator.(*Controller).eventProcessor(0xc0007c3770)
	/go/src/github.com/openshift/cluster-image-registry-operator/pkg/operator/controller.go:374 +0x47
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x7f25c92778a0)
	/go/src/github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x67
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc001814f60, {0x2e0ade0, 0xc000c75140}, 0x1, 0xc0009ea660)
	/go/src/github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00173e360, 0x3b9aca00, 0x0, 0x8, 0x445705)
	/go/src/github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(0x7d8f2a, 0xc00195f080, 0xc000e45f78)
	/go/src/github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x25
created by github.com/openshift/cluster-image-registry-operator/pkg/operator.(*Controller).Run
	/go/src/github.com/openshift/cluster-image-registry-operator/pkg/operator/controller.go:445 +0x1ef
panic: runtime error: index out of range [9] with length 9 [recovered]
	panic: runtime error: index out of range [9] with length 9

goroutine 709 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x2de70a0})
	/go/src/github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0xd8
panic({0x2755c20, 0xc000ec7638})
	/usr/lib/golang/src/runtime/panic.go:1038 +0x215
github.com/openshift/api/operator/v1.(*OperatorStatus).DeepCopyInto(0xc001b75ed8, 0xc000e5b0d8)
	/go/src/github.com/openshift/cluster-image-registry-operator/vendor/github.com/openshift/api/operator/v1/zz_generated.deepcopy.go:2838 +0x25d
github.com/openshift/api/imageregistry/v1.(*ImageRegistryStatus).DeepCopyInto(0xc001b75ed8, 0xc000e5b0d8)
	/go/src/github.com/openshift/cluster-image-registry-operator/vendor/github.com/openshift/api/imageregistry/v1/zz_generated.deepcopy.go:536 +0x74
github.com/openshift/api/imageregistry/v1.(*Config).DeepCopyInto(0xc001b75c00, 0xc000e5ae00)
	/go/src/github.com/openshift/cluster-image-registry-operator/vendor/github.com/openshift/api/imageregistry/v1/zz_generated.deepcopy.go:22 +0x105
github.com/openshift/api/imageregistry/v1.(*Config).DeepCopy(...)
	/go/src/github.com/openshift/cluster-image-registry-operator/vendor/github.com/openshift/api/imageregistry/v1/zz_generated.deepcopy.go:32
github.com/openshift/cluster-image-registry-operator/pkg/operator.(*Controller).sync(0xc0007c3770)
	/go/src/github.com/openshift/cluster-image-registry-operator/pkg/operator/controller.go:251 +0x9e
github.com/openshift/cluster-image-registry-operator/pkg/operator.(*Controller).eventProcessor.func1(0xc0007c3770, {0x23f98a0, 0x2de70a0})
	/go/src/github.com/openshift/cluster-image-registry-operator/pkg/operator/controller.go:367 +0xde
github.com/openshift/cluster-image-registry-operator/pkg/operator.(*Controller).eventProcessor(0xc0007c3770)
	/go/src/github.com/openshift/cluster-image-registry-operator/pkg/operator/controller.go:374 +0x47
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x7f25c92778a0)
	/go/src/github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x67
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc001814f60, {0x2e0ade0, 0xc000c75140}, 0x1, 0xc0009ea660)
	/go/src/github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00173e360, 0x3b9aca00, 0x0, 0x8, 0x445705)
	/go/src/github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(0x7d8f2a, 0xc00195f080, 0xc000e45f78)
	/go/src/github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x25
created by github.com/openshift/cluster-image-registry-operator/pkg/operator.(*Controller).Run
	/go/src/github.com/openshift/cluster-image-registry-operator/pkg/operator/controller.go:445 +0x1ef

Expected results:

No panics.

Additional info:

More recent failures can be found at https://search.ci.openshift.org/?search=pods%2Fopenshift-image-registry_cluster-image-registry-operator.*+Observed+a+panic%3A&maxAge=336h&context=1&type=bug%2Bjunit&name=&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job

Comment 3 XiuJuan Wang 2021-12-03 07:59:16 UTC

After the pr merged, prow ci doesn't show this failure in 4.10. Mark this bug to verified.
https://search.ci.openshift.org/?search=pods%2Fopenshift-image-registry_cluster-image-registry-operator.*+Observed+a+panic%3A&maxAge=48h&context=1&type=bug%2Bjunit&name=&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job

Comment 6 errata-xmlrpc 2022-03-10 16:31:07 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056