Hide Forgot
Description of problem: Installing and running atomic-openshift-tests fails with panic: runtime error: invalid memory address or nil pointer dereference Version-Release number of selected component (if applicable): 3.11.82 How reproducible: 100% Steps to Reproduce: 1. Install openshift 2. Install atomic-openshift-tests-<version_matching_ocp_install>.rpm 3. run: KUBECONFIG=/etc/origin/master/admin.kubeconfig /usr/libexec/atomic-openshift/extended.test --ginkgo.v=true --ginkgo.focus='\[k8s.io\]' Actual results: [k8s.io] [sig-node] Kubelet [Serial] [Slow] [k8s.io] [sig-node] experimental resource usage tracking [Feature:ExperimentalResourceUsageTracking] resource tracking for 100 pods per node [Suite:k8s] /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/node/kubelet_perf.go:281 [BeforeEach] [Top Level] /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/test/extended/util/test.go:51 [BeforeEach] [k8s.io] [sig-node] Kubelet [Serial] [Slow] /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:141 STEP: Creating a kubernetes client Feb 21 02:31:06.684: INFO: >>> kubeConfig: /etc/origin/master/admin.kubeconfig STEP: Building a namespace api object Feb 21 02:31:06.937: INFO: About to run a Kube e2e test, ensuring namespace is privileged STEP: Waiting for a default service account to be provisioned in namespace [BeforeEach] [k8s.io] [sig-node] Kubelet [Serial] [Slow] /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/node/kubelet_perf.go:201 Feb 21 02:31:07.145: INFO: Waiting for pods to enter Success, but no pods in "kube-system" match label map[name:e2e-image-puller] W0221 02:31:07.159617 96627 metrics_grabber.go:81] Master node is not registered. Grabbing metrics from Scheduler, ControllerManager and ClusterAutoscaler is disabled. [It] resource tracking for 100 pods per node [Suite:k8s] /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/node/kubelet_perf.go:281 STEP: Creating a RC of 100 pods and wait until all pods of this RC are running STEP: creating replication controller resource100-a6e4964c-35aa-11e9-87f2-3440b58f2653 in namespace e2e-tests-kubelet-perf-whvvk I0221 02:31:07.225126 96627 runners.go:177] Created replication controller with name: resource100-a6e4964c-35aa-11e9-87f2-3440b58f2653, namespace: e2e-tests-kubelet-perf-whvvk, replica count: 100 Feb 21 02:31:07.876: INFO: Missing info/stats for container "kubelet" on node "ibm-x3250m4-06.rhts.eng.bos.redhat.com" I0221 02:31:17.275430 96627 runners.go:177] resource100-a6e4964c-35aa-11e9-87f2-3440b58f2653 Pods: 100 out of 100 created, 0 running, 100 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runni ngButNotReady E0221 02:31:18.595895 96627 runtime.go:66] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference) /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72 /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65 /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51 /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/asm_amd64.s:573 /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/panic.go:502 /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/panic.go:63 /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/signal_unix.go:388 /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/kubelet_stats.go:593 /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/kubelet_stats.go:566 /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/asm_amd64.s:2361 panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x2c11ab7] goroutine 140736 [running]: github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0) /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x107 panic(0x3d4a0e0, 0x7024d70) /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/panic.go:502 +0x229 github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework.(*resourceCollector).collectStats(0xc4212afd50, 0xc42244c0c0) /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/kubelet_stats.go:593 +0x237 github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework.(*resourceCollector).Start.func1() /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/kubelet_stats.go:566 +0x33 github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc4231113c0) /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x54 github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc4231113c0, 0x2540be400, 0x0, 0x1, 0xc421063b60) /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xbd github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc4231113c0, 0x2540be400, 0xc421063b60) /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d created by github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework.(*resourceCollector).Start /builddir/build/BUILD/atomic-openshift-git-0.08bc31b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/kubelet_stats.go:566 +0xdc Expected results: passing tests Additional info: I've tried this a few times, and it will fail in other places. Also --gingko-skip="" will skip all tests (no matter what the regex is)
Tracked this down to the kubelet's calls to get the latest stats. The test case is assuming that .CPU is instantiated when it can be nil. https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/stats/helper.go#L36-L40 https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/stats/helper.go#L77-L84 PR: https://github.com/kubernetes/kubernetes/pull/79057
Origin PR: https://github.com/openshift/origin/pull/23332
3.11 Cherrypick: https://github.com/openshift/origin/pull/23335
Both PR's merged.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:1753