Bug 1952684 - cluster-etcd-operator: metrics controller panics on invalid response from client
Summary: cluster-etcd-operator: metrics controller panics on invalid response from client
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Etcd
Version: 4.5
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 4.8.0
Assignee: Sam Batschelet
QA Contact: ge liu
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-04-22 20:50 UTC by Sam Batschelet
Modified: 2021-07-27 23:03 UTC (History)
0 users

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-07-27 23:03:04 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-etcd-operator pull 573 0 None open Bug 1952684: pkg/operator/metriccontroller: dont panic on invalid response from prom client 2021-04-22 20:51:49 UTC
Github openshift cluster-etcd-operator pull 584 0 None open Bug 1952684: pkg/operator/metriccontroller: check vector length 2021-05-03 13:01:05 UTC
Red Hat Product Errata RHSA-2021:2438 0 None None None 2021-07-27 23:03:27 UTC

Description Sam Batschelet 2021-04-22 20:50:05 UTC
Description of problem: panic observed in CI [1],[2]
```
E0415 15:58:49.986605       1 runtime.go:78] Observed a panic: runtime.boundsError{x:0, y:0, signed:true, code:0x0} (runtime error: index out of range [0] with length 0)
goroutine 252 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x2527900, 0xc002a0c258)
	k8s.io/apimachinery.0-rc.0/pkg/util/runtime/runtime.go:74 +0x95
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0xc002c91d58, 0x1, 0x1)
	k8s.io/apimachinery.0-rc.0/pkg/util/runtime/runtime.go:48 +0x86
panic(0x2527900, 0xc002a0c258)
	runtime/panic.go:965 +0x1b9
github.com/openshift/cluster-etcd-operator/pkg/operator/metriccontroller.(*FSyncController).sync(0xc000795820, 0x2a477b8, 0xc000ea0000, 0x2a42390, 0xc0009f43c0, 0x0, 0x0)
	github.com/openshift/cluster-etcd-operator/pkg/operator/metriccontroller/fsync_controller.go:56 +0xecc
github.com/openshift/library-go/pkg/controller/factory.(*baseController).reconcile(0xc000e877a0, 0x2a477b8, 0xc000ea0000, 0x2a42390, 0xc0009f43c0, 0x0, 0x0)
	github.com/openshift/library-go.0-20210414082648-6e767630a0dc/pkg/controller/factory/base_controller.go:204 +0x56
github.com/openshift/library-go/pkg/controller/factory.(*baseController).processNextWorkItem(0xc000e877a0, 0x2a477b8, 0xc000ea0000)
	github.com/openshift/library-go.0-20210414082648-6e767630a0dc/pkg/controller/factory/base_controller.go:258 +0x253
github.com/openshift/library-go/pkg/controller/factory.(*baseController).runWorker.func1(0x2a477b8, 0xc000ea0000)
	github.com/openshift/library-go.0-20210414082648-6e767630a0dc/pkg/controller/factory/base_controller.go:195 +0xb9
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1()
	k8s.io/apimachinery.0-rc.0/pkg/util/wait/wait.go:185 +0x37
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00107bef8)
	k8s.io/apimachinery.0-rc.0/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc002c91ef8, 0x29eef80, 0xc000a337d0, 0xc000ea0001, 0xc000cdacc0)
	k8s.io/apimachinery.0-rc.0/pkg/util/wait/wait.go:156 +0x9b
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00107bef8, 0x3b9aca00, 0x0, 0x1, 0xc000cdacc0)
	k8s.io/apimachinery.0-rc.0/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext(0x2a477b8, 0xc000ea0000, 0xc00107bf58, 0x3b9aca00, 0x0, 0x1)
	k8s.io/apimachinery.0-rc.0/pkg/util/wait/wait.go:185 +0xa6
k8s.io/apimachinery/pkg/util/wait.UntilWithContext(...)
	k8s.io/apimachinery.0-rc.0/pkg/util/wait/wait.go:99
github.com/openshift/library-go/pkg/controller/factory.(*baseController).runWorker(0xc000e877a0, 0x2a477b8, 0xc000ea0000)
	github.com/openshift/library-go.0-20210414082648-6e767630a0dc/pkg/controller/factory/base_controller.go:186 +0x74
github.com/openshift/library-go/pkg/controller/factory.(*baseController).Run.func2(0xc000e877a0, 0xc000056d50, 0x2a477b8, 0xc000ea0000)
	github.com/openshift/library-go.0-20210414082648-6e767630a0dc/pkg/controller/factory/base_controller.go:116 +0x75
created by github.com/openshift/library-go/pkg/controller/factory.(*baseController).Run
	github.com/openshift/library-go.0-20210414082648-6e767630a0dc/pkg/controller/factory/base_controller.go:111 +0x2f6
I0415 15:58:49.987493       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"9c5c2e31-fbd5-421f-a288-247e6571b4c2", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'Openshift-Cluster-Etcd-OperatorPanic' Panic observed: runtime error: index out of range [0] with length 0
```

[1] https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_clus[…]tcd-operator-7897fff8c9-njwzl_etcd-operator.log

[2] https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_cluster-net[…]rk-operator-master-e2e-gcp-ovn-upgrade/1382687306791522304

Version-Release number of selected component (if applicable):


How reproducible: unknown


Steps to Reproduce:
1.
2.
3.

Actual results: panic index out of range [0] with length 0 


Expected results: response length verified greater than 0 before accessing first index of array.


Additional info:

Comment 3 ge liu 2021-05-24 09:38:02 UTC
This problem have not be hit in ci log again.

Comment 7 errata-xmlrpc 2021-07-27 23:03:04 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438


Note You need to log in before you can comment on or make changes to this bug.