Bug 1546188 – api server get stuck after exhausting all grpc connections

Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.

Bug 1546188 - api server get stuck after exhausting all grpc connections

Summary:	api server get stuck after exhausting all grpc connections

Status:	CLOSED ERRATA

Aliases:	None

Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Master (Show other bugs)
Sub Component:
Version:	3.9.0
Hardware:	Unspecified Unspecified

Priority	urgent Severity urgent
Target Milestone:	---
Target Release:	3.9.0
Assigned To:	Stefan Schimanski
QA Contact:	Wang Haoran
Docs Contact:

URL:
Whiteboard:
Keywords:	OnlineStarter

Depends On:
Blocks:
	Show dependency tree / graph

Reported:	2018-02-16 09:36 EST by Michal Fojtik
Modified:	2018-03-28 10:29 EDT (History)
CC List:	9 users (show)

See Also:
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Story Points:	---
Clone Of:
Environment:
Last Closed:	2018-03-28 10:29:21 EDT
Type:	Bug
Regression:	---
Mount Type:	---
Documentation:	---
CRM:
Verified Versions:
Category:	---
oVirt Team:	---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---

Attachments	(Terms of Use)
/deb/pprof/goroutine?debug=1 (286.26 KB, text/plain) 2018-02-16 10:09 EST, Eric Paris	no flags	Details
/metrics snapshot (6.06 MB, text/plain) 2018-02-16 10:22 EST, Eric Paris	no flags	Details
Add an attachment (proposed patch, testcase, etc.)

External Trackers
Tracker	ID	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2018:0489	None	None	None	2018-03-28 10:29 EDT

Groups: None (edit)

Description Michal Fojtik 2018-02-16 09:36:18 EST

Description of problem:

Discovered during 3.9 upgrade in ca-centra-1 cluster. Seems like the gprc connections are not freed which will cause the api server to be stuck and eventually
have to be killed and restarted.

David made a monitoring tool that will detect this situation and automatically
panic the api server which force it to restart.

We need to figure out what is the root cause of this. It sounds like a race condition in gprc library.

Comment 5 Eric Paris 2018-02-16 10:09 EST

Created attachment 1397069 [details]
/deb/pprof/goroutine?debug=1

Comment 6 Eric Paris 2018-02-16 10:22 EST

Created attachment 1397085 [details]
/metrics snapshot

Comment 9 Derek Carr 2018-02-28 08:07:03 EST

linked pr has merged.

Comment 17 errata-xmlrpc 2018-03-28 10:29:21 EDT

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0489

Note You need to log in before you can comment on or make changes to this bug.