Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1791780

Summary: OpenShift API is suddenly not available (EOF error)
Product: OpenShift Container Platform Reporter: palshure
Component: kube-apiserverAssignee: Michal Fojtik <mfojtik>
Status: CLOSED DEFERRED QA Contact: Xingxing Xia <xxia>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.2.0CC: aarapov, amcdermo, aos-bugs, bleanhar, deads, dmoessne, jnordell, mfojtik, nagrawal, openshift-bugs-escalate, palonsor, palshure, pehunt, pweil, rchanter, rkshirsa, sbatsche, scuppett, sdodson, srengan, sttts, ykheddac
Target Milestone: ---   
Target Release: 4.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1793694 (view as bug list) Environment:
Last Closed: 2020-02-19 13:22:43 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Comment 1 David Eads 2020-01-16 16:16:28 UTC
You need at least two masters for etcd quorum.  Without etcd quorum REST calls won't succeed.  You'll need to get information from all masters, not just one.  the connection refused messages from etcd here suggest that etcd is in a fatal state.

Comment 2 David Eads 2020-01-16 16:39:00 UTC
if only one master is running, the near total failure of the kube-apiserver is expected because etcd will not have a quorum.  Only one set of master-node pod logs was included.

next steps
1. ensure that at least two, preferably three physical (or virtual) master nodes (the machines themselves) are running
2. wait for 5 minutes
3. confirm you still have failures
4. collect pod logs, kubelet logs, and crio logs from all the masters

With at least two master nodes running, we can reasonably expect the cluster to have an available kube-apiserver.  Without that, there isn't much to do.