Bug 1546188

Summary: api server get stuck after exhausting all grpc connections
Product: OpenShift Container Platform Reporter: Michal Fojtik <mfojtik>
Component: MasterAssignee: Stefan Schimanski <sttts>
Status: CLOSED ERRATA QA Contact: Wang Haoran <haowang>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 3.9.0CC: aos-bugs, deads, decarr, eparis, jokerman, mfojtik, mmccomas, sjenning, wmeng
Target Milestone: ---Keywords: OnlineStarter
Target Release: 3.9.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-03-28 14:29:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
/deb/pprof/goroutine?debug=1
none
/metrics snapshot none

Description Michal Fojtik 2018-02-16 14:36:18 UTC 
Description of problem:

Discovered during 3.9 upgrade in ca-centra-1 cluster. Seems like the gprc connections are not freed which will cause the api server to be stuck and eventually
have to be killed and restarted.

David made a monitoring tool that will detect this situation and automatically
panic the api server which force it to restart.

We need to figure out what is the root cause of this. It sounds like a race condition in gprc library.
Comment 5 Eric Paris 2018-02-16 15:09:37 UTC 
Created attachment 1397069 [details]
/deb/pprof/goroutine?debug=1
Comment 6 Eric Paris 2018-02-16 15:22:19 UTC 
Created attachment 1397085 [details]
/metrics snapshot
Comment 9 Derek Carr 2018-02-28 13:07:03 UTC 
linked pr has merged.
Comment 17 errata-xmlrpc 2018-03-28 14:29:21 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0489