Hide Forgot
Created attachment 1520148 [details] openshift-cluster-kube-scheduler pod logs Description of problem: My openshift-cluster-kube-scheduler-operator pod logs covering about 8 hours has 126 instances of this panic: I0111 16:41:50.567169 1 shared_informer.go:123] caches populated I0111 16:41:50.567461 1 config_observer_controller.go:95] decode of existing config failed with error: EOF E0111 16:41:50.568134 1 runtime.go:66] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference) /go/src/github.com/openshift/cluster-kube-scheduler-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72 /go/src/github.com/openshift/cluster-kube-scheduler-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65 /go/src/github.com/openshift/cluster-kube-scheduler-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51 /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/asm_amd64.s:573 /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/panic.go:502 /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/panic.go:63 /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/signal_unix.go:388 /go/src/github.com/openshift/cluster-kube-scheduler-operator/vendor/github.com/openshift/library-go/pkg/operator/configobserver/config_observer_controller.go:102 /go/src/github.com/openshift/cluster-kube-scheduler-operator/vendor/github.com/openshift/library-go/pkg/operator/configobserver/config_observer_controller.go:189 /go/src/github.com/openshift/cluster-kube-scheduler-operator/vendor/github.com/openshift/library-go/pkg/operator/configobserver/config_observer_controller.go:175 /go/src/github.com/openshift/cluster-kube-scheduler-operator/vendor/github.com/openshift/library-go/pkg/operator/configobserver/config_observer_controller.go:169 /go/src/github.com/openshift/cluster-kube-scheduler-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 /go/src/github.com/openshift/cluster-kube-scheduler-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 /go/src/github.com/openshift/cluster-kube-scheduler-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/asm_amd64.s:2361 No pod restarts though. Version-Release number of selected component (if applicable): 4.0.0-0.nightly-2019-01-10-165754 How reproducible: Always in this build Steps to Reproduce: 1. Install typical 3 master/3 worker cluster on AWS with nextgen installer using OCP build 4.0.0-0.nightly-2019-01-10-165754 2. oc logs -n openshift-cluster-kube-scheduler <pod> Actual results: Repeated instances of the subject panic Additional info: Full pod log attached.
We already have fix for this. Thanks to Dan for identifying it earlier and providing fix. https://github.com/openshift/cluster-kube-scheduler-operator/pull/65
The above PR merged.
Verified on 4.0.0-0.nightly-2019-03-06-074438
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0758