Bug 1791780 - OpenShift API is suddenly not available (EOF error)
Summary: OpenShift API is suddenly not available (EOF error)
Keywords:
Status: CLOSED DEFERRED
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: kube-apiserver
Version: 4.2.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 4.5.0
Assignee: Michal Fojtik
QA Contact: Xingxing Xia
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-01-16 13:14 UTC by palshure
Modified: 2023-03-24 16:44 UTC (History)
22 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1793694 (view as bug list)
Environment:
Last Closed: 2020-02-19 13:22:43 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Comment 1 David Eads 2020-01-16 16:16:28 UTC
You need at least two masters for etcd quorum.  Without etcd quorum REST calls won't succeed.  You'll need to get information from all masters, not just one.  the connection refused messages from etcd here suggest that etcd is in a fatal state.

Comment 2 David Eads 2020-01-16 16:39:00 UTC
if only one master is running, the near total failure of the kube-apiserver is expected because etcd will not have a quorum.  Only one set of master-node pod logs was included.

next steps
1. ensure that at least two, preferably three physical (or virtual) master nodes (the machines themselves) are running
2. wait for 5 minutes
3. confirm you still have failures
4. collect pod logs, kubelet logs, and crio logs from all the masters

With at least two master nodes running, we can reasonably expect the cluster to have an available kube-apiserver.  Without that, there isn't much to do.


Note You need to log in before you can comment on or make changes to this bug.