Bug 1679612 - extended.test panic: runtime error: invalid memory address or nil pointer dereference
Summary: extended.test panic: runtime error: invalid memory address or nil pointer der...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Node
Version: 3.11.0
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
: 3.11.z
Assignee: Ryan Phillips
QA Contact: Weinan Liu
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-02-21 14:56 UTC by Jeff Young
Modified: 2019-07-23 19:56 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-07-23 19:56:23 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:1753 0 None None None 2019-07-23 19:56:35 UTC

Description Jeff Young 2019-02-21 14:56:11 UTC
Description of problem:
Installing and running atomic-openshift-tests fails with 
panic: runtime error: invalid memory address or nil pointer dereference

Version-Release number of selected component (if applicable):
3.11.82

How reproducible:
100%

Steps to Reproduce:
1. Install openshift 
2. Install atomic-openshift-tests-<version_matching_ocp_install>.rpm
3. run:
KUBECONFIG=/etc/origin/master/admin.kubeconfig /usr/libexec/atomic-openshift/extended.test --ginkgo.v=true --ginkgo.focus='\[k8s.io\]'

Actual results:
[k8s.io] [sig-node] Kubelet [Serial] [Slow] [k8s.io] [sig-node] experimental resource usage tracking [Feature:ExperimentalResourceUsageTracking] 
  resource tracking for 100 pods per node [Suite:k8s]
  /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/node/kubelet_perf.go:281
[BeforeEach] [Top Level]
  /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/test/extended/util/test.go:51
[BeforeEach] [k8s.io] [sig-node] Kubelet [Serial] [Slow]
  /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:141
STEP: Creating a kubernetes client
Feb 21 02:31:06.684: INFO: >>> kubeConfig: /etc/origin/master/admin.kubeconfig
STEP: Building a namespace api object
Feb 21 02:31:06.937: INFO: About to run a Kube e2e test, ensuring namespace is privileged
STEP: Waiting for a default service account to be provisioned in namespace
[BeforeEach] [k8s.io] [sig-node] Kubelet [Serial] [Slow]
  /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/node/kubelet_perf.go:201
Feb 21 02:31:07.145: INFO: Waiting for pods to enter Success, but no pods in "kube-system" match label map[name:e2e-image-puller]
W0221 02:31:07.159617   96627 metrics_grabber.go:81] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled.
[It] resource tracking for 100 pods per node [Suite:k8s]
  /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/node/kubelet_perf.go:281
STEP: Creating a RC of 100 pods and wait until all pods of this RC are running
STEP: creating replication controller resource100-a6e4964c-35aa-11e9-87f2-3440b58f2653 in namespace e2e-tests-kubelet-perf-whvvk
I0221 02:31:07.225126   96627 runners.go:177] Created replication controller with name: resource100-a6e4964c-35aa-11e9-87f2-3440b58f2653, namespace: e2e-tests-kubelet-perf-whvvk, replica count: 100
Feb 21 02:31:07.876: INFO: Missing info/stats for container "kubelet" on node "ibm-x3250m4-06.rhts.eng.bos.redhat.com"
I0221 02:31:17.275430   96627 runners.go:177] resource100-a6e4964c-35aa-11e9-87f2-3440b58f2653 Pods: 100 out of 100 created, 0 running, 100 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runni
ngButNotReady 
E0221 02:31:18.595895   96627 runtime.go:66] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
/builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72
/builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/asm_amd64.s:573
/opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/panic.go:502
/opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/panic.go:63
/opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/signal_unix.go:388
/builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/kubelet_stats.go:593
/builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/kubelet_stats.go:566
/builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
/builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
/builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
/opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/asm_amd64.s:2361
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x2c11ab7]

goroutine 140736 [running]:
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
        /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x107
panic(0x3d4a0e0, 0x7024d70)
        /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/panic.go:502 +0x229
github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework.(*resourceCollector).collectStats(0xc4212afd50, 0xc42244c0c0)
        /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/kubelet_stats.go:593 +0x237
github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework.(*resourceCollector).Start.func1()
        /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/kubelet_stats.go:566 +0x33
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc4231113c0)
        /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x54
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc4231113c0, 0x2540be400, 0x0, 0x1, 0xc421063b60)
        /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xbd
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc4231113c0, 0x2540be400, 0xc421063b60)
        /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d
created by github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework.(*resourceCollector).Start
        /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/kubelet_stats.go:566 +0xdc

Expected results:
passing tests

Additional info:
I've tried this a few times, and it will fail in other places.

Also --gingko-skip="" will skip all tests (no matter what the regex is)

Comment 5 Ryan Phillips 2019-06-15 01:36:51 UTC
Tracked this down to the kubelet's calls to get the latest stats. The test case is assuming that .CPU is instantiated when it can be nil.

https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/stats/helper.go#L36-L40
https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/stats/helper.go#L77-L84

PR: https://github.com/kubernetes/kubernetes/pull/79057

Comment 6 Ryan Phillips 2019-07-08 14:03:00 UTC
Origin PR: https://github.com/openshift/origin/pull/23332

Comment 7 Ryan Phillips 2019-07-08 17:31:07 UTC
3.11 Cherrypick: https://github.com/openshift/origin/pull/23335

Comment 8 Ryan Phillips 2019-07-08 19:01:14 UTC
Both PR's merged.

Comment 14 errata-xmlrpc 2019-07-23 19:56:23 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1753


Note You need to log in before you can comment on or make changes to this bug.