Bug 2003683

Summary:	Samples operator is panicking in CI
Product:	OpenShift Container Platform	Reporter:	Stephen Benjamin <stbenjam>
Component:	Samples	Assignee:	Gabe Montero <gmontero>
Status:	CLOSED ERRATA	QA Contact:	Jitendar Singh <jitsingh>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	4.10	CC:	aos-bugs, gmontero, sippy
Target Milestone:	---
Target Release:	4.10.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:	Symptom Detection.Undiagnosed panic detected in pod
Last Closed:	2022-03-10 16:10:01 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Stephen Benjamin 2021-09-13 12:32:54 UTC

Symptom Detection.Undiagnosed panic detected in pod is failing in CI, due to samples operator panics.

It is coming from this line:

https://github.com/openshift/cluster-samples-operator/blob/6fef0d252c7b70a542b646d7132a668eb4e044d3/pkg/stub/imagestreams.go#L179

imagestreamInCluster is guaranteed to be nil, since you're checking for that on 170.




Example job:
https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.10-e2e-gcp-techpreview/1436466436968026112


Full stack trace:

2021-09-11T08:34:44.227264225Z E0911 08:34:44.227190       8 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
2021-09-11T08:34:44.227264225Z goroutine 169 [running]:
2021-09-11T08:34:44.227264225Z k8s.io/apimachinery/pkg/util/runtime.logPanic(0x1cd2660, 0x3012c70)
2021-09-11T08:34:44.227264225Z  /go/src/github.com/openshift/cluster-samples-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0x95
2021-09-11T08:34:44.227264225Z k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
2021-09-11T08:34:44.227264225Z  /go/src/github.com/openshift/cluster-samples-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x86
2021-09-11T08:34:44.227264225Z panic(0x1cd2660, 0x3012c70)
2021-09-11T08:34:44.227264225Z  /usr/lib/golang/src/runtime/panic.go:965 +0x1b9
2021-09-11T08:34:44.227264225Z github.com/openshift/cluster-samples-operator/pkg/stub.(*Handler).upsertImageStream(0xc0003a5a20, 0xc000911980, 0x0, 0xc000b99b00, 0x221ac60, 0xc00065dcc0)
2021-09-11T08:34:44.227264225Z  /go/src/github.com/openshift/cluster-samples-operator/pkg/stub/imagestreams.go:179 +0xc2f
2021-09-11T08:34:44.227264225Z github.com/openshift/cluster-samples-operator/pkg/stub.(*Handler).createSamples(0xc0003a5a20, 0xc000b99b00, 0xc0005e0001, 0xc000829050, 0xc000829020, 0x0, 0x5, 0x0)
2021-09-11T08:34:44.227264225Z  /go/src/github.com/openshift/cluster-samples-operator/pkg/stub/handler.go:1042 +0x495
2021-09-11T08:34:44.227264225Z github.com/openshift/cluster-samples-operator/pkg/stub.(*Handler).Handle(0xc0003a5a20, 0x22269b0, 0xc0004a0fc0, 0x0, 0x0, 0x0)
2021-09-11T08:34:44.227264225Z  /go/src/github.com/openshift/cluster-samples-operator/pkg/stub/handler.go:901 +0x1f65
2021-09-11T08:34:44.227264225Z github.com/openshift/cluster-samples-operator/pkg/operator.(*Controller).handleWork(0xc0004d2ea0, 0x22192a0, 0x306f380, 0x1bfa400, 0xc0008beb20, 0xc000833d00, 0x4167eb)
2021-09-11T08:34:44.227264225Z  /go/src/github.com/openshift/cluster-samples-operator/pkg/operator/controller.go:292 +0x17b
2021-09-11T08:34:44.227264225Z github.com/openshift/cluster-samples-operator/pkg/operator.(*queueWorker).workqueueProcessor.func1(0xc00058c450, 0x1bfa400, 0xc0008beb20)
2021-09-11T08:34:44.227264225Z  /go/src/github.com/openshift/cluster-samples-operator/pkg/operator/controller.go:368 +0x13e
2021-09-11T08:34:44.227264225Z github.com/openshift/cluster-samples-operator/pkg/operator.(*queueWorker).workqueueProcessor(0xc00058c450)
2021-09-11T08:34:44.227264225Z  /go/src/github.com/openshift/cluster-samples-operator/pkg/operator/controller.go:375 +0xad
2021-09-11T08:34:44.227264225Z k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00074c280)
2021-09-11T08:34:44.227264225Z  /go/src/github.com/openshift/cluster-samples-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
2021-09-11T08:34:44.227264225Z k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00074c280, 0x221ae00, 0xc0007395c0, 0x1, 0xc000264120)
2021-09-11T08:34:44.227264225Z  /go/src/github.com/openshift/cluster-samples-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0x9b
2021-09-11T08:34:44.227264225Z k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00074c280, 0x3b9aca00, 0x0, 0x2091101, 0xc000264120)
2021-09-11T08:34:44.227264225Z  /go/src/github.com/openshift/cluster-samples-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
2021-09-11T08:34:44.227264225Z k8s.io/apimachinery/pkg/util/wait.Until(0xc00074c280, 0x3b9aca00, 0xc000264120)
2021-09-11T08:34:44.227264225Z  /go/src/github.com/openshift/cluster-samples-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x4d
2021-09-11T08:34:44.227264225Z created by github.com/openshift/cluster-samples-operator/pkg/operator.(*Controller).Run
2021-09-11T08:34:44.227264225Z  /go/src/github.com/openshift/cluster-samples-operator/pkg/operator/controller.go:185 +0x5d4
2021-09-11T08:34:44.229362815Z panic: runtime error: invalid memory address or nil pointer dereference [recovered]
2021-09-11T08:34:44.229362815Z  panic: runtime error: invalid memory address or nil pointer dereference

Comment 1 Gabe Montero 2021-09-13 15:50:53 UTC

thanks for the heads up

fixing the release.... that commit is only in 4.10

Comment 3 Gabe Montero 2021-09-14 11:39:43 UTC

pretty straight forward on verify ...

Comment 6 errata-xmlrpc 2022-03-10 16:10:01 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056