Bug 1288211

Summary:	Native HA deployment cannot find leader
Product:	OpenShift Container Platform	Reporter:	Eric Jones <erjones>
Component:	Node	Assignee:	Andrew Butcher <abutcher>
Status:	CLOSED INSUFFICIENT_DATA	QA Contact:	Jianwei Hou <jhou>
Severity:	high	Docs Contact:
Priority:	high
Version:	3.1.0	CC:	aos-bugs, bperkins, erich, esauer, jdetiber, jokerman, mmccomas
Target Milestone:	---	Keywords:	Unconfirmed
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-01-04 20:01:47 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Eric Jones 2015-12-03 20:47:25 UTC

Description of problem:
Attempted to restart atomic-openshift-master-api.service and atomix-openshift-master-controllers.service in order to confirm changes to the master-config.yaml, once the restart finished
Version-Release number of selected component (if applicable):
OSE 3.1, brand new installation as of 2015/12/2

How reproducible:
Seems to be 100%

Steps to Reproduce:
1. Edit master-config.yaml
2. Restart atomic-openshift-master-api.service and atomix-openshift-master-controllers.service 

Actual results:
the masters seem to have no "leader" for the system. The logs for the  atomic-openshift-master-controllers.service all indicate
"Attempting to acquire controller lease as master-##thisid##
 Lease openshift.io/leases/controllers owned by master-##differentid## at ...."

The owner id did not match any of the ids with which the masters are trying to use to become the leader. All three masters continue to attempt this until they either timeout or give up. 

The logs continue with
"...unable to wait for lease expiration openshift.io/leases/controllers: 401: The event in requested index is outdated and cleared (the requested history has been cleared"

Expected results:
Everything to return to normal with the changes in the master-config.yaml made.

Additional info:
3 Masters
The change to the master-config.yaml was to the identity provider. Originally setup using htpasswd but changed to LDAP.