Bug 1490428 - [3.5.z] Stopping etcd leader can cause authentication failures due to OpenShift Master caching and using etcd leader IP address.
Summary: [3.5.z] Stopping etcd leader can cause authentication failures due to OpenShi...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Master
Version: 3.5.1
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 3.5.z
Assignee: Robert Rati
QA Contact: Wang Haoran
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-09-11 14:11 UTC by Eric Rich
Modified: 2021-03-11 15:45 UTC (History)
19 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: In some failure cases the etcd client used by OpenShift won't rotate through all the available etcd cluster members. The client will end up repeatedly trying the same server, and if that server is down then requests will fail for an extended time until the client finds the server invalid. Consequence: If the etcd leader goes away when it is attempted to be contacted for something like authentication then the authentication will fail and the etcd client will be stuck trying to talk to the etcd member that doesn't exist. User authentication would fail for an extended period of time. Fix: The etcd client now rotates to other cluster members even on failure. Result: If the etcd leader goes away, the worst that should happen is a failure of that one authentication attempt. The next attempt will succeed because a different etcd member will be used.
Clone Of: 1475184
Environment:
Last Closed: 2017-10-25 13:06:40 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github coreos etcd issues 8515 0 None None None 2017-09-11 14:11:34 UTC
Github coreos etcd pull 8519 0 None None None 2017-09-11 14:11:34 UTC
Red Hat Product Errata RHBA-2017:3049 0 normal SHIPPED_LIVE OpenShift Container Platform 3.6, 3.5, and 3.4 bug fix and enhancement update 2017-10-25 15:57:15 UTC

Comment 3 ge liu 2017-09-27 08:56:56 UTC 
Verified with ocp version:

openshift v3.5.5.31.29
kubernetes v1.5.2+43a9be4
etcd 3.1.0

Steps:

1). Create HA env:
3 master with 3 etcd, 4 node, 2 router

2). Run oc login command in loop:

3). turn off master service randomly

4). login command still works well without interruption

5). turn off master service on another master randomly

6). login command still works well without interruption

7). turn off master service on last living master

8). login fail as expected

Unable to connect to the server: EOF
error: EOF
Unable to connect to the server: EOF
error: EOF

9). start master on master randomly

10). login command works well again.
Comment 5 errata-xmlrpc 2017-10-25 13:06:40 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:3049
Note You need to log in before you can comment on or make changes to this bug.