Created attachment 1343394 [details] master logs Description of problem: APIServer panic'd on GET /api/v1/namespaces/extended-test-cli-deployment-59v3j-tb8s9: multiple NewLogged calls! Running a subset of conformance (https://github.com/openshift/svt/tree/master/conformance) against 3.7.0-0.178.0 the master-api logs started showing panics like the one above. Eventually the conformance tests hung with taking a long time to terminate. Lots of watch failures in the log as well Version-Release number of selected component (if applicable): 3.7.0-0.178.0 How reproducible: Unknown - I will clean the cluster up and try again Steps to Reproduce: 1. 1 master, 1 infra, 2 compute - all m4.xlarge on AWS 2. Run the svt_conformance.sh script linked above. There were failures due to a missing wildfly imagestream, so I addded that and ran the script a second time. Actual results: master-api panics and lots of broken watches. Search on "APIServer panic'd" in the logs Many conformance tests failed as well due this. Additional info:
Clayton FYI since according to git blame you touched this code in kube ;-)
WithPanicRecovery wraps with a http logger When WithMaxInFlightLimit reaches its limit and calls tooManyRequests, that also tries to wrap, which fails.
fixed in https://github.com/kubernetes/kubernetes/pull/48813 upstream
picked in https://github.com/openshift/origin/pull/17048
Verification blocked by bug 1508061
Verified on 3.7.0-0.190.0. Ran conformance multiple times without seeing this error.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2017:3188