Bug 1691023 - Node panics due to cadvisor index out-of-range
Summary: Node panics due to cadvisor index out-of-range
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Node
Version: 3.11.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 3.11.z
Assignee: Seth Jennings
QA Contact: Weinan Liu
Depends On:
TreeView+ depends on / blocked
Reported: 2019-03-20 16:29 UTC by Pablo Alonso Rodriguez
Modified: 2019-10-28 09:51 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Fixes an issue where the node process panics in cadvisor with an out-of-range error
Clone Of:
Last Closed: 2019-06-26 09:07:55 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:1605 0 None None None 2019-06-26 09:08:04 UTC

Description Pablo Alonso Rodriguez 2019-03-20 16:29:35 UTC
Description of problem:

atomic-openshift-node had the following panic:

panic: runtime error: index out of range
goroutine 256178939 [running]:
github.com/openshift/origin/vendor/github.com/google/cadvisor/metrics.(*PrometheusCollector).collectContainersInfo(0xc420421500, 0xc42d4a9320)
/builddir/build/BUILD/atomic-openshift-git-0.647ac05/_output/local/go/src/github.com/openshift/origin/vendor/github.com/google/cadvisor/metrics/prometheus.go:892 +0x1339
github.com/openshift/origin/vendor/github.com/google/cadvisor/metrics.(*PrometheusCollector).Collect(0xc420421500, 0xc42d4a9320)
/builddir/build/BUILD/atomic-openshift-git-0.647ac05/_output/local/go/src/github.com/openshift/origin/vendor/github.com/google/cadvisor/metrics/prometheus.go:808 +0x89
github.com/openshift/origin/vendor/github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func2(0xc42c4defa0, 0xc42d4a9320, 0x63f5840, 0xc420421500)
/builddir/build/BUILD/atomic-openshift-git-0.647ac05/_output/local/go/src/github.com/openshift/origin/vendor/github.com/prometheus/client_golang/prometheus/registry.go:382 +0x61
created by github.com/openshift/origin/vendor/github.com/prometheus/client_golang/prometheus.(*Registry).Gather
/builddir/build/BUILD/atomic-openshift-git-0.647ac05/_output/local/go/src/github.com/openshift/origin/vendor/github.com/prometheus/client_golang/prometheus/registry.go:380 +0x302

The source of this panic seems to be an already fixed bug in cadvisor:


We would need to have a cadvisor version with the fix vendored in OpenShift 3.11 so that the panic does not reproduce

Version-Release number of selected component (if applicable):

3.11.43 and 3.11.88

How reproducible:

Sometimes, randomly

Steps to Reproduce:
1. It just reproduces randomly

Actual results:

Panics when a container has no stats in the mentioned line

Expected results:

It should not panic.

Additional info:

Comment 7 errata-xmlrpc 2019-06-26 09:07:55 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.