Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.
Bug 1506375 - API server panics while running conformance: APIServer panic'd on GET /api/v1/namespaces/extended-test-cli-deployment-59v3j-tb8s9: multiple NewLogged calls!
API server panics while running conformance: APIServer panic'd on GET /api/...
Status: CLOSED ERRATA
Product: OpenShift Container Platform
Classification: Red Hat
Component: Master (Show other bugs)
3.7.0
x86_64 Linux
unspecified Severity medium
: ---
: 3.7.0
Assigned To: Jordan Liggitt
Mike Fiedler
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-10-25 15:55 EDT by Mike Fiedler
Modified: 2017-11-28 17:19 EST (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-11-28 17:19:38 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
master logs (800.46 KB, application/x-gzip)
2017-10-25 15:55 EDT, Mike Fiedler
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2017:3188 normal SHIPPED_LIVE Moderate: Red Hat OpenShift Container Platform 3.7 security, bug, and enhancement update 2017-11-28 21:34:54 EST

  None (edit)
Description Mike Fiedler 2017-10-25 15:55:05 EDT
Created attachment 1343394 [details]
master logs

Description of problem:

 APIServer panic'd on GET /api/v1/namespaces/extended-test-cli-deployment-59v3j-tb8s9: multiple NewLogged calls!

Running a subset of conformance (https://github.com/openshift/svt/tree/master/conformance) against 3.7.0-0.178.0 the master-api logs started showing panics like the one above.

Eventually the conformance tests hung with taking a long time to terminate.   Lots of watch failures in the log as well


Version-Release number of selected component (if applicable): 3.7.0-0.178.0


How reproducible: Unknown - I will clean the cluster up and try again


Steps to Reproduce:
1.  1 master, 1 infra, 2 compute - all m4.xlarge on AWS
2.  Run the svt_conformance.sh script linked above.   There were failures due to a missing wildfly imagestream, so I addded that and ran the script a second time.


Actual results:

master-api panics and lots of broken watches.   Search on "APIServer panic'd" in the logs

Many conformance tests failed as well due this.




Additional info:
Comment 1 Michal Fojtik 2017-10-26 09:11:09 EDT
Clayton FYI since according to git blame you touched this code in kube ;-)
Comment 2 Jordan Liggitt 2017-10-26 09:25:22 EDT
WithPanicRecovery wraps with a http logger

When WithMaxInFlightLimit reaches its limit and calls tooManyRequests, that also tries to wrap, which fails.
Comment 3 Jordan Liggitt 2017-10-26 09:26:28 EDT
fixed in https://github.com/kubernetes/kubernetes/pull/48813 upstream
Comment 4 Jordan Liggitt 2017-10-26 09:30:02 EDT
picked in https://github.com/openshift/origin/pull/17048
Comment 5 Mike Fiedler 2017-10-31 15:01:08 EDT
Verification blocked by bug 1508061
Comment 7 Mike Fiedler 2017-11-07 18:07:36 EST
Verified on 3.7.0-0.190.0.   Ran conformance multiple times without seeing this error.
Comment 10 errata-xmlrpc 2017-11-28 17:19:38 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3188

Note You need to log in before you can comment on or make changes to this bug.