Bug 1569349
| Summary: | Got go routine panic when access /healthz in master api static pod | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | weiwei jiang <wjiang> |
| Component: | kube-apiserver | Assignee: | Stefan Schimanski <sttts> |
| Status: | CLOSED DEFERRED | QA Contact: | Wang Haoran <haowang> |
| Severity: | low | Docs Contact: | |
| Priority: | low | ||
| Version: | 3.10.0 | CC: | aos-bugs, jokerman, mfojtik, mmccomas, wjiang, wmeng |
| Target Milestone: | --- | ||
| Target Release: | 3.10.z | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-11-20 18:56:03 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Does the apiserver process actually panic? I don't see any panic in the stacktrace. The stacktrace is only printing out because /healthz is returning a 500 error. https://github.com/openshift/origin/blob/master/vendor/k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/server/httplog/httplog.go#L68-L70 This should be a transient situation and is not panicing the apiserver process. It is log spam though. Would be better if it didn't do this. (In reply to Seth Jennings from comment #1) > Does the apiserver process actually panic? I don't see any panic in the > stacktrace. Apiserver is not down for this panic. And 500 is not a kind output even in log. Agree, this should be transient until the API server stabilizes. Lowering the priority. OCP 3.6-3.10 is no longer on full support [1]. Marking CLOSED DEFERRED. If you have a customer case with a support exception or have reproduced on 3.11+, please reopen and include those details. When reopening, please set the Target Release to the appropriate version where needed. [1]: https://access.redhat.com/support/policy/updates/openshift |
Description of problem: Found the following logs in master api static pod: I0419 05:34:30.961158 1 wrap.go:42] GET /healthz: (18.643692ms) 500 goroutine 3255 [running]: github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).recordStatus(0xc420f443f0, 0x1f4) /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:207 +0xdd github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader(0xc420f443f0, 0x1f4) /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:186 +0x35 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.(*baseTimeoutWriter).WriteHeader(0xc430f9f420, 0x1f4) /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:188 +0xac net/http.Error(0x7f795b2054a0, 0xc4326fac48, 0xc4226c2a00, 0x4d5, 0x1f4) /usr/lib/golang/src/net/http/server.go:1930 +0xda github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/healthz.handleRootHealthz.func1(0x7f795b2054a0, 0xc4326fac48, 0xc4343ba700) /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/healthz/healthz.go:121 +0x508 net/http.HandlerFunc.ServeHTTP(0xc429718f40, 0x7f795b2054a0, 0xc4326fac48, 0xc4343ba700) /usr/lib/golang/src/net/http/server.go:1918 +0x44 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc4324566c0, 0x7f795b2054a0, 0xc4326fac48, 0xc4343ba700) /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:241 +0x55a github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc426f0d5e0, 0x7f795b2054a0, 0xc4326fac48, 0xc4343ba700) /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x5a3cd68, 0xf, 0xc42f0b43f0, 0xc426f0d5e0, 0x7f795b2054a0, 0xc4326fac48, 0xc4343ba700) /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/handler.go:160 +0x6ad github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc42bb55c60, 0x7f795b2054a0, 0xc4326fac48, 0xc4343ba700) <autogenerated>:1 +0x75 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x7f795b2054a0, 0xc4326fac48, 0xc4343ba700) /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:52 +0x37d net/http.HandlerFunc.ServeHTTP(0xc429ca2190, 0x7f795b2054a0, 0xc4326fac48, 0xc4343ba700) /usr/lib/golang/src/net/http/server.go:1918 +0x44 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1(0x7f795b2054a0, 0xc4326fac48, 0xc4343ba700) /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:165 +0x42a net/http.HandlerFunc.ServeHTTP(0xc42aa0e100, 0x7f795b2054a0, 0xc4326fac48, 0xc4343ba700) /usr/lib/golang/src/net/http/server.go:1918 +0x44 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1(0x7f795b2054a0, 0xc4326fac48, 0xc4343ba700) /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:49 +0x203a net/http.HandlerFunc.ServeHTTP(0xc429ca21e0, 0x7f795b2054a0, 0xc4326fac48, 0xc4343ba700) /usr/lib/golang/src/net/http/server.go:1918 +0x44 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1(0x7f795b2054a0, 0xc4326fac48, 0xc4343ba700) /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:78 +0x2b1 net/http.HandlerFunc.ServeHTTP(0xc429ca2280, 0x7f795b2054a0, 0xc4326fac48, 0xc4343ba700) /usr/lib/golang/src/net/http/server.go:1918 +0x44 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/request.WithRequestContext.func1(0x7f795b2054a0, 0xc4326fac48, 0xc4343ba700) /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110 +0xcb net/http.HandlerFunc.ServeHTTP(0xc42bb55d40, 0x7f795b2054a0, 0xc4326fac48, 0xc4343ba700) /usr/lib/golang/src/net/http/server.go:1918 +0x44 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.WithCORS.func1(0x7f795b2054a0, 0xc4326fac48, 0xc4343ba700) /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/cors.go:75 +0x17b net/http.HandlerFunc.ServeHTTP(0xc42bab5c20, 0x7f795b2054a0, 0xc4326fac48, 0xc4343ba700) /usr/lib/golang/src/net/http/server.go:1918 +0x44 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1(0xc42aa101a0, 0xffdba40, 0xc4326fac48, 0xc4343ba700, 0xc431551800) /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:93 +0x8d created by github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:92 +0x1ab logging error output: "[+]ping ok\n[+]etcd ok\n[+]poststarthook/generic-apiserver-start-informers ok\n[+]poststarthook/start-apiextensions-informers ok\n[+]poststarthook/start-apiextensions-controllers ok\n[+]poststarthook/project.openshift.io-projectcache ok\n[+]poststarthook/project.openshift.io-projectauthorizationcache ok\n[-]poststarthook/security.openshift.io-bootstrapscc failed: reason withheld\n[-]poststarthook/bootstrap-controller failed: reason withheld\n[-]poststarthook/ca-registration failed: reason withheld\n[+]poststarthook/start-kube-aggregator-informers ok\n[+]poststarthook/apiservice-registration-controller ok\n[+]poststarthook/apiservice-status-available-controller ok\n[+]poststarthook/apiservice-openapi-controller ok\n[+]poststarthook/kube-apiserver-autoregistration ok\n[+]autoregister-completion ok\n[-]poststarthook/authorization.openshift.io-bootstrapclusterroles failed: reason withheld\n[-]poststarthook/authorization.openshift.io-ensureopenshift-infra failed: reason withheld\n[+]poststarthook/quota.openshift.io-clusterquotamapping ok\n[-]poststarthook/openshift.io-AdmissionInit failed: reason withheld\n[+]poststarthook/openshift.io-StartInformers ok\n[+]poststarthook/oauth.openshift.io-StartOAuthClientsBootstrapping ok\nhealthz check failed\n" [[openshift/v1.10.0+b81c8f8 (linux/amd64) kubernetes/b81c8f8] 172.18.13.147:54368] Version-Release number of selected component (if applicable): # oc version oc v3.10.0-0.22.0 kubernetes v1.10.0+b81c8f8 features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://ip-172-18-13-147.ec2.internal:8443 openshift v3.10.0-0.22.0 kubernetes v1.10.0+b81c8f8 # uname -a Linux ip-172-18-13-147.ec2.internal 3.10.0-862.el7.x86_64 #1 SMP Wed Mar 21 18:14:51 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux # cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.5 (Maipo) How reproducible: Neally Always Steps to Reproduce: 1. Setup OCP with bootstrap mode 2. Check the master api pod log. 3. Actual results: I0419 05:57:09.841112 1 wrap.go:42] GET /healthz: (8.447866ms) 500 goroutine 3578 [running]: github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).recordStatus(0xc425de05b0, 0x1f4) /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:207 +0xdd github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader(0xc425de05b0, 0x1f4) /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:186 +0x35 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.(*baseTimeoutWriter).WriteHeader(0xc42ee05c80, 0x1f4) /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:188 +0xac net/http.Error(0x7f1545ccda60, 0xc4322bf018, 0xc428361400, 0x4c0, 0x1f4) /usr/lib/golang/src/net/http/server.go:1930 +0xda github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/healthz.handleRootHealthz.func1(0x7f1545ccda60, 0xc4322bf018, 0xc42416fa00) /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/healthz/healthz.go:121 +0x508 net/http.HandlerFunc.ServeHTTP(0xc432e39160, 0x7f1545ccda60, 0xc4322bf018, 0xc42416fa00) /usr/lib/golang/src/net/http/server.go:1918 +0x44 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc42b9afa40, 0x7f1545ccda60, 0xc4322bf018, 0xc42416fa00) /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:241 +0x55a github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc429448e70, 0x7f1545ccda60, 0xc4322bf018, 0xc42416fa00) /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x5a3cd68, 0xf, 0xc4262e61b0, 0xc429448e70, 0x7f1545ccda60, 0xc4322bf018, 0xc42416fa00) /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/handler.go:160 +0x6ad github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc42ce46c40, 0x7f1545ccda60, 0xc4322bf018, 0xc42416fa00) <autogenerated>:1 +0x75 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x7f1545ccda60, 0xc4322bf018, 0xc42416fa00) /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:52 +0x37d net/http.HandlerFunc.ServeHTTP(0xc4288ce320, 0x7f1545ccda60, 0xc4322bf018, 0xc42416fa00) /usr/lib/golang/src/net/http/server.go:1918 +0x44 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1(0x7f1545ccda60, 0xc4322bf018, 0xc42416fa00) /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:165 +0x42a net/http.HandlerFunc.ServeHTTP(0xc42a84af00, 0x7f1545ccda60, 0xc4322bf018, 0xc42416fa00) /usr/lib/golang/src/net/http/server.go:1918 +0x44 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1(0x7f1545ccda60, 0xc4322bf018, 0xc42416fa00) /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:49 +0x203a net/http.HandlerFunc.ServeHTTP(0xc4288ce370, 0x7f1545ccda60, 0xc4322bf018, 0xc42416fa00) /usr/lib/golang/src/net/http/server.go:1918 +0x44 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1(0x7f1545ccda60, 0xc4322bf018, 0xc42416fa00) /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:78 +0x2b1 net/http.HandlerFunc.ServeHTTP(0xc4288ce3c0, 0x7f1545ccda60, 0xc4322bf018, 0xc42416fa00) /usr/lib/golang/src/net/http/server.go:1918 +0x44 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/request.WithRequestContext.func1(0x7f1545ccda60, 0xc4322bf018, 0xc42416fa00) /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110 +0xcb net/http.HandlerFunc.ServeHTTP(0xc42ce46d40, 0x7f1545ccda60, 0xc4322bf018, 0xc42416fa00) /usr/lib/golang/src/net/http/server.go:1918 +0x44 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.WithCORS.func1(0x7f1545ccda60, 0xc4322bf018, 0xc42416fa00) /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/cors.go:75 +0x17b net/http.HandlerFunc.ServeHTTP(0xc4254a3560, 0x7f1545ccda60, 0xc4322bf018, 0xc42416fa00) /usr/lib/golang/src/net/http/server.go:1918 +0x44 github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1(0xc42ce473a0, 0xffdba40, 0xc4322bf018, 0xc42416fa00, 0xc42863b140) /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:93 +0x8d created by github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:92 +0x1ab logging error output: "[+]ping ok\n[+]etcd ok\n[+]poststarthook/generic-apiserver-start-informers ok\n[+]poststarthook/start-apiextensions-informers ok\n[+]poststarthook/start-apiextensions-controllers ok\n[+]poststarthook/project.openshift.io-projectcache ok\n[+]poststarthook/project.openshift.io-projectauthorizationcache ok\n[-]poststarthook/security.openshift.io-bootstrapscc failed: reason withheld\n[-]poststarthook/bootstrap-controller failed: reason withheld\n[-]poststarthook/ca-registration failed: reason withheld\n[+]poststarthook/start-kube-aggregator-informers ok\n[+]poststarthook/apiservice-registration-controller ok\n[+]poststarthook/apiservice-status-available-controller ok\n[+]poststarthook/apiservice-openapi-controller ok\n[+]poststarthook/kube-apiserver-autoregistration ok\n[+]autoregister-completion ok\n[-]poststarthook/authorization.openshift.io-bootstrapclusterroles failed: reason withheld\n[-]poststarthook/authorization.openshift.io-ensureopenshift-infra failed: reason withheld\n[+]poststarthook/quota.openshift.io-clusterquotamapping ok\n[+]poststarthook/openshift.io-AdmissionInit ok\n[+]poststarthook/openshift.io-StartInformers ok\n[+]poststarthook/oauth.openshift.io-StartOAuthClientsBootstrapping ok\nhealthz check failed\n" [[openshift/v1.10.0+b81c8f8 (linux/amd64) kubernetes/b81c8f8] 172.18.13.147:59786] Expected results: Should not have panic Additional info: