Bug 1679612
| Summary: | extended.test panic: runtime error: invalid memory address or nil pointer dereference | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Jeff Young <jeyoung> |
| Component: | Node | Assignee: | Ryan Phillips <rphillips> |
| Status: | CLOSED ERRATA | QA Contact: | Weinan Liu <weinliu> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 3.11.0 | CC: | aos-bugs, eparis, jokerman, mmccomas, schoudha, vlaad |
| Target Milestone: | --- | ||
| Target Release: | 3.11.z | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-07-23 19:56:23 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Tracked this down to the kubelet's calls to get the latest stats. The test case is assuming that .CPU is instantiated when it can be nil. https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/stats/helper.go#L36-L40 https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/stats/helper.go#L77-L84 PR: https://github.com/kubernetes/kubernetes/pull/79057 3.11 Cherrypick: https://github.com/openshift/origin/pull/23335 Both PR's merged. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:1753 |
Description of problem: Installing and running atomic-openshift-tests fails with panic: runtime error: invalid memory address or nil pointer dereference Version-Release number of selected component (if applicable): 3.11.82 How reproducible: 100% Steps to Reproduce: 1. Install openshift 2. Install atomic-openshift-tests-<version_matching_ocp_install>.rpm 3. run: KUBECONFIG=/etc/origin/master/admin.kubeconfig /usr/libexec/atomic-openshift/extended.test --ginkgo.v=true --ginkgo.focus='\[k8s.io\]' Actual results: [k8s.io] [sig-node] Kubelet [Serial] [Slow] [k8s.io] [sig-node] experimental resource usage tracking [Feature:ExperimentalResourceUsageTracking] resource tracking for 100 pods per node [Suite:k8s] /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/node/kubelet_perf.go:281 [BeforeEach] [Top Level] /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/test/extended/util/test.go:51 [BeforeEach] [k8s.io] [sig-node] Kubelet [Serial] [Slow] /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:141 STEP: Creating a kubernetes client Feb 21 02:31:06.684: INFO: >>> kubeConfig: /etc/origin/master/admin.kubeconfig STEP: Building a namespace api object Feb 21 02:31:06.937: INFO: About to run a Kube e2e test, ensuring namespace is privileged STEP: Waiting for a default service account to be provisioned in namespace [BeforeEach] [k8s.io] [sig-node] Kubelet [Serial] [Slow] /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/node/kubelet_perf.go:201 Feb 21 02:31:07.145: INFO: Waiting for pods to enter Success, but no pods in "kube-system" match label map[name:e2e-image-puller] W0221 02:31:07.159617 96627 metrics_grabber.go:81] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. [It] resource tracking for 100 pods per node [Suite:k8s] /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/node/kubelet_perf.go:281 STEP: Creating a RC of 100 pods and wait until all pods of this RC are running STEP: creating replication controller resource100-a6e4964c-35aa-11e9-87f2-3440b58f2653 in namespace e2e-tests-kubelet-perf-whvvk I0221 02:31:07.225126 96627 runners.go:177] Created replication controller with name: resource100-a6e4964c-35aa-11e9-87f2-3440b58f2653, namespace: e2e-tests-kubelet-perf-whvvk, replica count: 100 Feb 21 02:31:07.876: INFO: Missing info/stats for container "kubelet" on node "ibm-x3250m4-06.rhts.eng.bos.redhat.com" I0221 02:31:17.275430 96627 runners.go:177] resource100-a6e4964c-35aa-11e9-87f2-3440b58f2653 Pods: 100 out of 100 created, 0 running, 100 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runni ngButNotReady E0221 02:31:18.595895 96627 runtime.go:66] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference) /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72 /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65 /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51 /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/asm_amd64.s:573 /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/panic.go:502 /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/panic.go:63 /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/signal_unix.go:388 /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/kubelet_stats.go:593 /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/kubelet_stats.go:566 /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/asm_amd64.s:2361 panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x2c11ab7] goroutine 140736 [running]: github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0) /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x107 panic(0x3d4a0e0, 0x7024d70) /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/panic.go:502 +0x229 github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework.(*resourceCollector).collectStats(0xc4212afd50, 0xc42244c0c0) /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/kubelet_stats.go:593 +0x237 github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework.(*resourceCollector).Start.func1() /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/kubelet_stats.go:566 +0x33 github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc4231113c0) /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x54 github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc4231113c0, 0x2540be400, 0x0, 0x1, 0xc421063b60) /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xbd github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc4231113c0, 0x2540be400, 0xc421063b60) /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d created by github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework.(*resourceCollector).Start /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/kubelet_stats.go:566 +0xdc Expected results: passing tests Additional info: I've tried this a few times, and it will fail in other places. Also --gingko-skip="" will skip all tests (no matter what the regex is)