Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.
First Last Prev Next    This bug is not in your last search results.
Bug 1506375 - API server panics while running conformance: APIServer panic'd on GET /api/v1/namespaces/extended-test-cli-deployment-59v3j-tb8s9: multiple NewLogged calls!
API server panics while running conformance: APIServer panic'd on GET /api/...
Status: CLOSED ERRATA
Product: OpenShift Container Platform
Classification: Red Hat
Component: Master (Show other bugs)
3.7.0
x86_64 Linux
unspecified Severity medium
: ---
: 3.7.0
Assigned To: Jordan Liggitt
Mike Fiedler
:
Depends On:
Blocks:
  Show dependency tree / graph
 
Reported: 2017-10-25 15:55 EDT by Mike Fiedler
Modified: 2017-11-28 17:19 EST (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-11-28 17:19:38 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
master logs (800.46 KB, application/x-gzip)
2017-10-25 15:55 EDT, Mike Fiedler 		no flags Details
Add an attachment (proposed patch, testcase, etc.)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2017:3188 normal SHIPPED_LIVE Moderate: Red Hat OpenShift Container Platform 3.7 security, bug, and enhancement update 2017-11-28 21:34:54 EST

  None (edit)
Description Mike Fiedler 2017-10-25 15:55:05 EDT 
Created attachment 1343394 [details]
master logs

Description of problem:

 APIServer panic'd on GET /api/v1/namespaces/extended-test-cli-deployment-59v3j-tb8s9: multiple NewLogged calls!

Running a subset of conformance (https://github.com/openshift/svt/tree/master/conformance) against 3.7.0-0.178.0 the master-api logs started showing panics like the one above.

Eventually the conformance tests hung with taking a long time to terminate.   Lots of watch failures in the log as well


Version-Release number of selected component (if applicable): 3.7.0-0.178.0


How reproducible: Unknown - I will clean the cluster up and try again


Steps to Reproduce:
1.  1 master, 1 infra, 2 compute - all m4.xlarge on AWS
2.  Run the svt_conformance.sh script linked above.   There were failures due to a missing wildfly imagestream, so I addded that and ran the script a second time.


Actual results:

master-api panics and lots of broken watches.   Search on "APIServer panic'd" in the logs

Many conformance tests failed as well due this.




Additional info:
Comment 1 Michal Fojtik 2017-10-26 09:11:09 EDT 
Clayton FYI since according to git blame you touched this code in kube ;-)
Comment 2 Jordan Liggitt 2017-10-26 09:25:22 EDT 
WithPanicRecovery wraps with a http logger

When WithMaxInFlightLimit reaches its limit and calls tooManyRequests, that also tries to wrap, which fails.
Comment 3 Jordan Liggitt 2017-10-26 09:26:28 EDT 
fixed in https://github.com/kubernetes/kubernetes/pull/48813 upstream
Comment 4 Jordan Liggitt 2017-10-26 09:30:02 EDT 
picked in https://github.com/openshift/origin/pull/17048
Comment 5 Mike Fiedler 2017-10-31 15:01:08 EDT 
Verification blocked by bug 1508061
Comment 7 Mike Fiedler 2017-11-07 18:07:36 EST 
Verified on 3.7.0-0.190.0.   Ran conformance multiple times without seeing this error.
Comment 10 errata-xmlrpc 2017-11-28 17:19:38 EST 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3188
Note You need to log in before you can comment on or make changes to this bug.
First Last Prev Next    This bug is not in your last search results.