Bug 2003683 - Samples operator is panicking in CI
Summary: Samples operator is panicking in CI
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Samples
Version: 4.10
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 4.10.0
Assignee: Gabe Montero
QA Contact: Jitendar Singh
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-09-13 12:32 UTC by Stephen Benjamin
Modified: 2022-03-10 16:10 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Symptom Detection.Undiagnosed panic detected in pod
Last Closed: 2022-03-10 16:10:01 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-samples-operator pull 395 0 None open Bug 2003683: fix recently introduced panic when conflict errors occur 2021-09-13 18:42:22 UTC
Red Hat Product Errata RHSA-2022:0056 0 None None None 2022-03-10 16:10:28 UTC

Description Stephen Benjamin 2021-09-13 12:32:54 UTC
Symptom Detection.Undiagnosed panic detected in pod is failing in CI, due to samples operator panics.

It is coming from this line:

https://github.com/openshift/cluster-samples-operator/blob/6fef0d252c7b70a542b646d7132a668eb4e044d3/pkg/stub/imagestreams.go#L179

imagestreamInCluster is guaranteed to be nil, since you're checking for that on 170.




Example job:
https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.10-e2e-gcp-techpreview/1436466436968026112


Full stack trace:

2021-09-11T08:34:44.227264225Z E0911 08:34:44.227190       8 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
2021-09-11T08:34:44.227264225Z goroutine 169 [running]:
2021-09-11T08:34:44.227264225Z k8s.io/apimachinery/pkg/util/runtime.logPanic(0x1cd2660, 0x3012c70)
2021-09-11T08:34:44.227264225Z  /go/src/github.com/openshift/cluster-samples-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0x95
2021-09-11T08:34:44.227264225Z k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
2021-09-11T08:34:44.227264225Z  /go/src/github.com/openshift/cluster-samples-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x86
2021-09-11T08:34:44.227264225Z panic(0x1cd2660, 0x3012c70)
2021-09-11T08:34:44.227264225Z  /usr/lib/golang/src/runtime/panic.go:965 +0x1b9
2021-09-11T08:34:44.227264225Z github.com/openshift/cluster-samples-operator/pkg/stub.(*Handler).upsertImageStream(0xc0003a5a20, 0xc000911980, 0x0, 0xc000b99b00, 0x221ac60, 0xc00065dcc0)
2021-09-11T08:34:44.227264225Z  /go/src/github.com/openshift/cluster-samples-operator/pkg/stub/imagestreams.go:179 +0xc2f
2021-09-11T08:34:44.227264225Z github.com/openshift/cluster-samples-operator/pkg/stub.(*Handler).createSamples(0xc0003a5a20, 0xc000b99b00, 0xc0005e0001, 0xc000829050, 0xc000829020, 0x0, 0x5, 0x0)
2021-09-11T08:34:44.227264225Z  /go/src/github.com/openshift/cluster-samples-operator/pkg/stub/handler.go:1042 +0x495
2021-09-11T08:34:44.227264225Z github.com/openshift/cluster-samples-operator/pkg/stub.(*Handler).Handle(0xc0003a5a20, 0x22269b0, 0xc0004a0fc0, 0x0, 0x0, 0x0)
2021-09-11T08:34:44.227264225Z  /go/src/github.com/openshift/cluster-samples-operator/pkg/stub/handler.go:901 +0x1f65
2021-09-11T08:34:44.227264225Z github.com/openshift/cluster-samples-operator/pkg/operator.(*Controller).handleWork(0xc0004d2ea0, 0x22192a0, 0x306f380, 0x1bfa400, 0xc0008beb20, 0xc000833d00, 0x4167eb)
2021-09-11T08:34:44.227264225Z  /go/src/github.com/openshift/cluster-samples-operator/pkg/operator/controller.go:292 +0x17b
2021-09-11T08:34:44.227264225Z github.com/openshift/cluster-samples-operator/pkg/operator.(*queueWorker).workqueueProcessor.func1(0xc00058c450, 0x1bfa400, 0xc0008beb20)
2021-09-11T08:34:44.227264225Z  /go/src/github.com/openshift/cluster-samples-operator/pkg/operator/controller.go:368 +0x13e
2021-09-11T08:34:44.227264225Z github.com/openshift/cluster-samples-operator/pkg/operator.(*queueWorker).workqueueProcessor(0xc00058c450)
2021-09-11T08:34:44.227264225Z  /go/src/github.com/openshift/cluster-samples-operator/pkg/operator/controller.go:375 +0xad
2021-09-11T08:34:44.227264225Z k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00074c280)
2021-09-11T08:34:44.227264225Z  /go/src/github.com/openshift/cluster-samples-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
2021-09-11T08:34:44.227264225Z k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00074c280, 0x221ae00, 0xc0007395c0, 0x1, 0xc000264120)
2021-09-11T08:34:44.227264225Z  /go/src/github.com/openshift/cluster-samples-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0x9b
2021-09-11T08:34:44.227264225Z k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00074c280, 0x3b9aca00, 0x0, 0x2091101, 0xc000264120)
2021-09-11T08:34:44.227264225Z  /go/src/github.com/openshift/cluster-samples-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
2021-09-11T08:34:44.227264225Z k8s.io/apimachinery/pkg/util/wait.Until(0xc00074c280, 0x3b9aca00, 0xc000264120)
2021-09-11T08:34:44.227264225Z  /go/src/github.com/openshift/cluster-samples-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x4d
2021-09-11T08:34:44.227264225Z created by github.com/openshift/cluster-samples-operator/pkg/operator.(*Controller).Run
2021-09-11T08:34:44.227264225Z  /go/src/github.com/openshift/cluster-samples-operator/pkg/operator/controller.go:185 +0x5d4
2021-09-11T08:34:44.229362815Z panic: runtime error: invalid memory address or nil pointer dereference [recovered]
2021-09-11T08:34:44.229362815Z  panic: runtime error: invalid memory address or nil pointer dereference

Comment 1 Gabe Montero 2021-09-13 15:50:53 UTC
thanks for the heads up

fixing the release.... that commit is only in 4.10

Comment 3 Gabe Montero 2021-09-14 11:39:43 UTC
pretty straight forward on verify ...

Comment 6 errata-xmlrpc 2022-03-10 16:10:01 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056


Note You need to log in before you can comment on or make changes to this bug.