Bug 1569349 - Got go routine panic when access /healthz in master api static pod
Summary: Got go routine panic when access /healthz in master api static pod
Keywords:
Status: CLOSED DEFERRED
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: kube-apiserver
Version: 3.10.0
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: 3.10.z
Assignee: Stefan Schimanski
QA Contact: Wang Haoran
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-19 06:29 UTC by weiwei jiang
Modified: 2019-11-20 18:56 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-11-20 18:56:03 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description weiwei jiang 2018-04-19 06:29:11 UTC
Description of problem:

Found the following logs in master api static pod:

I0419 05:34:30.961158       1 wrap.go:42] GET /healthz: (18.643692ms) 500
goroutine 3255 [running]:
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).recordStatus(0xc420f443f0, 0x1f4)
        /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:207 +0xdd
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader(0xc420f443f0, 0x1f4)
        /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:186 +0x35
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.(*baseTimeoutWriter).WriteHeader(0xc430f9f420, 0x1f4)
        /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:188 +0xac
net/http.Error(0x7f795b2054a0, 0xc4326fac48, 0xc4226c2a00, 0x4d5, 0x1f4)
        /usr/lib/golang/src/net/http/server.go:1930 +0xda
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/healthz.handleRootHealthz.func1(0x7f795b2054a0, 0xc4326fac48, 0xc4343ba700)
        /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/healthz/healthz.go:121 +0x508
net/http.HandlerFunc.ServeHTTP(0xc429718f40, 0x7f795b2054a0, 0xc4326fac48, 0xc4343ba700)
        /usr/lib/golang/src/net/http/server.go:1918 +0x44
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc4324566c0, 0x7f795b2054a0, 0xc4326fac48, 0xc4343ba700)
        /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:241 +0x55a
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc426f0d5e0, 0x7f795b2054a0, 0xc4326fac48, 0xc4343ba700)
        /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x5a3cd68, 0xf, 0xc42f0b43f0, 0xc426f0d5e0, 0x7f795b2054a0, 0xc4326fac48, 0xc4343ba700)
        /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/handler.go:160 +0x6ad
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc42bb55c60, 0x7f795b2054a0, 0xc4326fac48, 0xc4343ba700)
        <autogenerated>:1 +0x75
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x7f795b2054a0, 0xc4326fac48, 0xc4343ba700)
        /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:52 +0x37d
net/http.HandlerFunc.ServeHTTP(0xc429ca2190, 0x7f795b2054a0, 0xc4326fac48, 0xc4343ba700)
        /usr/lib/golang/src/net/http/server.go:1918 +0x44
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1(0x7f795b2054a0, 0xc4326fac48, 0xc4343ba700)
        /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:165 +0x42a
net/http.HandlerFunc.ServeHTTP(0xc42aa0e100, 0x7f795b2054a0, 0xc4326fac48, 0xc4343ba700)
        /usr/lib/golang/src/net/http/server.go:1918 +0x44
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1(0x7f795b2054a0, 0xc4326fac48, 0xc4343ba700)
        /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:49 +0x203a
net/http.HandlerFunc.ServeHTTP(0xc429ca21e0, 0x7f795b2054a0, 0xc4326fac48, 0xc4343ba700)
        /usr/lib/golang/src/net/http/server.go:1918 +0x44
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1(0x7f795b2054a0, 0xc4326fac48, 0xc4343ba700)
        /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:78 +0x2b1
net/http.HandlerFunc.ServeHTTP(0xc429ca2280, 0x7f795b2054a0, 0xc4326fac48, 0xc4343ba700)
        /usr/lib/golang/src/net/http/server.go:1918 +0x44
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/request.WithRequestContext.func1(0x7f795b2054a0, 0xc4326fac48, 0xc4343ba700)
        /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110 +0xcb
net/http.HandlerFunc.ServeHTTP(0xc42bb55d40, 0x7f795b2054a0, 0xc4326fac48, 0xc4343ba700)
        /usr/lib/golang/src/net/http/server.go:1918 +0x44
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.WithCORS.func1(0x7f795b2054a0, 0xc4326fac48, 0xc4343ba700)
        /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/cors.go:75 +0x17b
net/http.HandlerFunc.ServeHTTP(0xc42bab5c20, 0x7f795b2054a0, 0xc4326fac48, 0xc4343ba700)
        /usr/lib/golang/src/net/http/server.go:1918 +0x44
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1(0xc42aa101a0, 0xffdba40, 0xc4326fac48, 0xc4343ba700, 0xc431551800)
        /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:93 +0x8d
created by github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP
        /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:92 +0x1ab

logging error output: "[+]ping ok\n[+]etcd ok\n[+]poststarthook/generic-apiserver-start-informers ok\n[+]poststarthook/start-apiextensions-informers ok\n[+]poststarthook/start-apiextensions-controllers ok\n[+]poststarthook/project.openshift.io-projectcache ok\n[+]poststarthook/project.openshift.io-projectauthorizationcache ok\n[-]poststarthook/security.openshift.io-bootstrapscc failed: reason withheld\n[-]poststarthook/bootstrap-controller failed: reason withheld\n[-]poststarthook/ca-registration failed: reason withheld\n[+]poststarthook/start-kube-aggregator-informers ok\n[+]poststarthook/apiservice-registration-controller ok\n[+]poststarthook/apiservice-status-available-controller ok\n[+]poststarthook/apiservice-openapi-controller ok\n[+]poststarthook/kube-apiserver-autoregistration ok\n[+]autoregister-completion ok\n[-]poststarthook/authorization.openshift.io-bootstrapclusterroles failed: reason withheld\n[-]poststarthook/authorization.openshift.io-ensureopenshift-infra failed: reason withheld\n[+]poststarthook/quota.openshift.io-clusterquotamapping ok\n[-]poststarthook/openshift.io-AdmissionInit failed: reason withheld\n[+]poststarthook/openshift.io-StartInformers ok\n[+]poststarthook/oauth.openshift.io-StartOAuthClientsBootstrapping ok\nhealthz check failed\n"
 [[openshift/v1.10.0+b81c8f8 (linux/amd64) kubernetes/b81c8f8] 172.18.13.147:54368]


Version-Release number of selected component (if applicable):
# oc version 
oc v3.10.0-0.22.0
kubernetes v1.10.0+b81c8f8
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://ip-172-18-13-147.ec2.internal:8443
openshift v3.10.0-0.22.0
kubernetes v1.10.0+b81c8f8
# uname -a 
Linux ip-172-18-13-147.ec2.internal 3.10.0-862.el7.x86_64 #1 SMP Wed Mar 21 18:14:51 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.5 (Maipo)


How reproducible:
Neally Always

Steps to Reproduce:
1. Setup OCP with bootstrap mode
2. Check the master api pod log.
3.

Actual results:
I0419 05:57:09.841112       1 wrap.go:42] GET /healthz: (8.447866ms) 500
goroutine 3578 [running]:
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).recordStatus(0xc425de05b0, 0x1f4)
        /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:207 +0xdd
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader(0xc425de05b0, 0x1f4)
        /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:186 +0x35
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.(*baseTimeoutWriter).WriteHeader(0xc42ee05c80, 0x1f4)
        /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:188 +0xac
net/http.Error(0x7f1545ccda60, 0xc4322bf018, 0xc428361400, 0x4c0, 0x1f4)
        /usr/lib/golang/src/net/http/server.go:1930 +0xda
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/healthz.handleRootHealthz.func1(0x7f1545ccda60, 0xc4322bf018, 0xc42416fa00)
        /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/healthz/healthz.go:121 +0x508
net/http.HandlerFunc.ServeHTTP(0xc432e39160, 0x7f1545ccda60, 0xc4322bf018, 0xc42416fa00)
        /usr/lib/golang/src/net/http/server.go:1918 +0x44
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc42b9afa40, 0x7f1545ccda60, 0xc4322bf018, 0xc42416fa00)
        /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:241 +0x55a
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc429448e70, 0x7f1545ccda60, 0xc4322bf018, 0xc42416fa00)
        /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x5a3cd68, 0xf, 0xc4262e61b0, 0xc429448e70, 0x7f1545ccda60, 0xc4322bf018, 0xc42416fa00)
        /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/handler.go:160 +0x6ad
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc42ce46c40, 0x7f1545ccda60, 0xc4322bf018, 0xc42416fa00)
        <autogenerated>:1 +0x75
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x7f1545ccda60, 0xc4322bf018, 0xc42416fa00)
        /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:52 +0x37d
net/http.HandlerFunc.ServeHTTP(0xc4288ce320, 0x7f1545ccda60, 0xc4322bf018, 0xc42416fa00)
        /usr/lib/golang/src/net/http/server.go:1918 +0x44
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1(0x7f1545ccda60, 0xc4322bf018, 0xc42416fa00)
        /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:165 +0x42a
net/http.HandlerFunc.ServeHTTP(0xc42a84af00, 0x7f1545ccda60, 0xc4322bf018, 0xc42416fa00)
        /usr/lib/golang/src/net/http/server.go:1918 +0x44
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1(0x7f1545ccda60, 0xc4322bf018, 0xc42416fa00)
        /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:49 +0x203a
net/http.HandlerFunc.ServeHTTP(0xc4288ce370, 0x7f1545ccda60, 0xc4322bf018, 0xc42416fa00)
        /usr/lib/golang/src/net/http/server.go:1918 +0x44
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1(0x7f1545ccda60, 0xc4322bf018, 0xc42416fa00)
        /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:78 +0x2b1
net/http.HandlerFunc.ServeHTTP(0xc4288ce3c0, 0x7f1545ccda60, 0xc4322bf018, 0xc42416fa00)
        /usr/lib/golang/src/net/http/server.go:1918 +0x44
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/request.WithRequestContext.func1(0x7f1545ccda60, 0xc4322bf018, 0xc42416fa00)
        /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110 +0xcb
net/http.HandlerFunc.ServeHTTP(0xc42ce46d40, 0x7f1545ccda60, 0xc4322bf018, 0xc42416fa00)
        /usr/lib/golang/src/net/http/server.go:1918 +0x44
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.WithCORS.func1(0x7f1545ccda60, 0xc4322bf018, 0xc42416fa00)
        /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/cors.go:75 +0x17b
net/http.HandlerFunc.ServeHTTP(0xc4254a3560, 0x7f1545ccda60, 0xc4322bf018, 0xc42416fa00)
        /usr/lib/golang/src/net/http/server.go:1918 +0x44
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1(0xc42ce473a0, 0xffdba40, 0xc4322bf018, 0xc42416fa00, 0xc42863b140)
        /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:93 +0x8d
created by github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP
        /builddir/build/BUILD/atomic-openshift-git-0.cbe986b/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:92 +0x1ab

logging error output: "[+]ping ok\n[+]etcd ok\n[+]poststarthook/generic-apiserver-start-informers ok\n[+]poststarthook/start-apiextensions-informers ok\n[+]poststarthook/start-apiextensions-controllers ok\n[+]poststarthook/project.openshift.io-projectcache ok\n[+]poststarthook/project.openshift.io-projectauthorizationcache ok\n[-]poststarthook/security.openshift.io-bootstrapscc failed: reason withheld\n[-]poststarthook/bootstrap-controller failed: reason withheld\n[-]poststarthook/ca-registration failed: reason withheld\n[+]poststarthook/start-kube-aggregator-informers ok\n[+]poststarthook/apiservice-registration-controller ok\n[+]poststarthook/apiservice-status-available-controller ok\n[+]poststarthook/apiservice-openapi-controller ok\n[+]poststarthook/kube-apiserver-autoregistration ok\n[+]autoregister-completion ok\n[-]poststarthook/authorization.openshift.io-bootstrapclusterroles failed: reason withheld\n[-]poststarthook/authorization.openshift.io-ensureopenshift-infra failed: reason withheld\n[+]poststarthook/quota.openshift.io-clusterquotamapping ok\n[+]poststarthook/openshift.io-AdmissionInit ok\n[+]poststarthook/openshift.io-StartInformers ok\n[+]poststarthook/oauth.openshift.io-StartOAuthClientsBootstrapping ok\nhealthz check failed\n"
 [[openshift/v1.10.0+b81c8f8 (linux/amd64) kubernetes/b81c8f8] 172.18.13.147:59786]


Expected results:
Should not have panic

Additional info:

Comment 1 Seth Jennings 2018-04-19 17:03:42 UTC
Does the apiserver process actually panic?  I don't see any panic in the stacktrace.

Comment 2 Seth Jennings 2018-04-19 17:10:24 UTC
The stacktrace is only printing out because /healthz is returning a 500 error.

https://github.com/openshift/origin/blob/master/vendor/k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/server/httplog/httplog.go#L68-L70

This should be a transient situation and is not panicing the apiserver process.

It is log spam though.  Would be better if it didn't do this.

Comment 3 weiwei jiang 2018-04-20 08:28:56 UTC
(In reply to Seth Jennings from comment #1)
> Does the apiserver process actually panic?  I don't see any panic in the
> stacktrace.

Apiserver is not down for this panic.

And 500 is not a kind output even in log.

Comment 4 Michal Fojtik 2018-04-23 10:26:37 UTC
Agree, this should be transient until the API server stabilizes. Lowering the priority.

Comment 5 Stefan Schimanski 2019-06-14 11:24:31 UTC
https://github.com/openshift/origin/pull/23172

Comment 6 Stephen Cuppett 2019-11-20 18:56:03 UTC
OCP 3.6-3.10 is no longer on full support [1]. Marking CLOSED DEFERRED. If you have a customer case with a support exception or have reproduced on 3.11+, please reopen and include those details. When reopening, please set the Target Release to the appropriate version where needed.

[1]: https://access.redhat.com/support/policy/updates/openshift


Note You need to log in before you can comment on or make changes to this bug.