Bug 1691023

Summary:	Node panics due to cadvisor index out-of-range
Product:	OpenShift Container Platform	Reporter:	Pablo Alonso Rodriguez <palonsor>
Component:	Node	Assignee:	Seth Jennings <sjenning>
Status:	CLOSED ERRATA	QA Contact:	Weinan Liu <weinliu>
Severity:	high	Docs Contact:
Priority:	high
Version:	3.11.0	CC:	aos-bugs, erich, jack.ottofaro, jhou, jokerman, mmccomas, sjenning, vlaad, xtian
Target Milestone:	---
Target Release:	3.11.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	Fixes an issue where the node process panics in cadvisor with an out-of-range error	Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-06-26 09:07:55 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Pablo Alonso Rodriguez 2019-03-20 16:29:35 UTC

Description of problem:

atomic-openshift-node had the following panic:

panic: runtime error: index out of range
goroutine 256178939 [running]:
github.com/openshift/origin/vendor/github.com/google/cadvisor/metrics.(*PrometheusCollector).collectContainersInfo(0xc420421500, 0xc42d4a9320)
/builddir/build/BUILD/atomic-openshift-git-0.647ac05/_output/local/go/src/github.com/openshift/origin/vendor/github.com/google/cadvisor/metrics/prometheus.go:892 +0x1339
github.com/openshift/origin/vendor/github.com/google/cadvisor/metrics.(*PrometheusCollector).Collect(0xc420421500, 0xc42d4a9320)
/builddir/build/BUILD/atomic-openshift-git-0.647ac05/_output/local/go/src/github.com/openshift/origin/vendor/github.com/google/cadvisor/metrics/prometheus.go:808 +0x89
github.com/openshift/origin/vendor/github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func2(0xc42c4defa0, 0xc42d4a9320, 0x63f5840, 0xc420421500)
/builddir/build/BUILD/atomic-openshift-git-0.647ac05/_output/local/go/src/github.com/openshift/origin/vendor/github.com/prometheus/client_golang/prometheus/registry.go:382 +0x61
created by github.com/openshift/origin/vendor/github.com/prometheus/client_golang/prometheus.(*Registry).Gather
/builddir/build/BUILD/atomic-openshift-git-0.647ac05/_output/local/go/src/github.com/openshift/origin/vendor/github.com/prometheus/client_golang/prometheus/registry.go:380 +0x302

The source of this panic seems to be an already fixed bug in cadvisor:

https://github.com/google/cadvisor/pull/2004

We would need to have a cadvisor version with the fix vendored in OpenShift 3.11 so that the panic does not reproduce

Version-Release number of selected component (if applicable):

3.11.43 and 3.11.88

How reproducible:

Sometimes, randomly

Steps to Reproduce:
1. It just reproduces randomly

Actual results:

Panics when a container has no stats in the mentioned line

Expected results:

It should not panic.

Additional info:

Comment 7 errata-xmlrpc 2019-06-26 09:07:55 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1605