Bug 1691023

Summary: Node panics due to cadvisor index out-of-range
Product: OpenShift Container Platform Reporter: Pablo Alonso Rodriguez <palonsor>
Component: NodeAssignee: Seth Jennings <sjenning>
Status: CLOSED ERRATA QA Contact: Weinan Liu <weinliu>
Severity: high Docs Contact:
Priority: high    
Version: 3.11.0CC: aos-bugs, erich, jack.ottofaro, jhou, jokerman, mmccomas, sjenning, vlaad, xtian
Target Milestone: ---   
Target Release: 3.11.z   
Hardware: Unspecified   
OS: Unspecified   
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Fixes an issue where the node process panics in cadvisor with an out-of-range error
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-06-26 09:07:55 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Pablo Alonso Rodriguez 2019-03-20 16:29:35 UTC
Description of problem:

atomic-openshift-node had the following panic:

panic: runtime error: index out of range
goroutine 256178939 [running]:
github.com/openshift/origin/vendor/github.com/google/cadvisor/metrics.(*PrometheusCollector).collectContainersInfo(0xc420421500, 0xc42d4a9320)
/builddir/build/BUILD/atomic-openshift-git-0.647ac05/_output/local/go/src/github.com/openshift/origin/vendor/github.com/google/cadvisor/metrics/prometheus.go:892 +0x1339
github.com/openshift/origin/vendor/github.com/google/cadvisor/metrics.(*PrometheusCollector).Collect(0xc420421500, 0xc42d4a9320)
/builddir/build/BUILD/atomic-openshift-git-0.647ac05/_output/local/go/src/github.com/openshift/origin/vendor/github.com/google/cadvisor/metrics/prometheus.go:808 +0x89
github.com/openshift/origin/vendor/github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func2(0xc42c4defa0, 0xc42d4a9320, 0x63f5840, 0xc420421500)
/builddir/build/BUILD/atomic-openshift-git-0.647ac05/_output/local/go/src/github.com/openshift/origin/vendor/github.com/prometheus/client_golang/prometheus/registry.go:382 +0x61
created by github.com/openshift/origin/vendor/github.com/prometheus/client_golang/prometheus.(*Registry).Gather
/builddir/build/BUILD/atomic-openshift-git-0.647ac05/_output/local/go/src/github.com/openshift/origin/vendor/github.com/prometheus/client_golang/prometheus/registry.go:380 +0x302

The source of this panic seems to be an already fixed bug in cadvisor:


We would need to have a cadvisor version with the fix vendored in OpenShift 3.11 so that the panic does not reproduce

Version-Release number of selected component (if applicable):

3.11.43 and 3.11.88

How reproducible:

Sometimes, randomly

Steps to Reproduce:
1. It just reproduces randomly

Actual results:

Panics when a container has no stats in the mentioned line

Expected results:

It should not panic.

Additional info:

Comment 7 errata-xmlrpc 2019-06-26 09:07:55 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.