Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.
Bug 1359900 - With multiple API servers they race to bootstrap policy
With multiple API servers they race to bootstrap policy
Status: CLOSED ERRATA
Product: OpenShift Container Platform
Classification: Red Hat
Component: Auth (Show other bugs)
3.3.0
All Linux
medium Severity high
: ---
: ---
Assigned To: Jordan Liggitt
weiwei jiang
:
: 1361313 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-07-25 12:46 EDT by Jeremy Eder
Modified: 2017-03-08 13 EST (History)
13 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Multiple API servers starting simultaneously with an empty etcd datastore would race to populate the default system policy. Consequence: A partially created policy could result, leaving a new cluster with a policy that would forbid system components from making some API calls. Fix: The policy APIs were updated to perform the same resourceVersion checking as other APIs, and fault-tolerant logic was added to the initial policy population step. Result: New clusters populate default policy as expected.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-09-27 05:41:34 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
/etc/origin, master log and ansible inventory (880.06 KB, application/x-gzip)
2016-07-27 10:49 EDT, Mike Fiedler
no flags Details
api server logs (7.22 KB, application/x-gzip)
2016-07-28 11:19 EDT, Scott Dodson
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:1933 normal SHIPPED_LIVE Red Hat OpenShift Container Platform 3.3 Release Advisory 2016-09-27 09:24:36 EDT

  None (edit)
Comment 1 Scott Dodson 2016-07-26 14:55:25 EDT
I've been able to reproduce this with a node that has its inventory name set to an ip address. ie: the second node below fails, the first works.

[nodes]
ose3-master.example.com openshift_node_labels="{'region':'infra','zone':'default'}" openshift_schedulable=true
192.168.122.102 openshift_node_labels="{'region':'primary','zone':'east'}"
Comment 2 Scott Dodson 2016-07-27 10:30:44 EDT
My test environment was bad. Now that I've re-provisioned the environment I can no longer re-produce this when specifying the inventory name as an ip.
Comment 3 Mike Fiedler 2016-07-27 10:49 EDT
Created attachment 1184683 [details]
/etc/origin, master log and ansible inventory

/etc/origin, master log and ansible inventory attached
Comment 4 Scott Dodson 2016-07-27 17:00:17 EDT
`oadm policy reconcile-cluster-role-bindings` fixed the issue, existing nodes immediately registered themselves. Now as to why that's necessary, we're still not sure.
Comment 5 Scott Dodson 2016-07-28 11:13:25 EDT
This seems to be the result of 3 API servers starting for the first time at the same time. We can work around this in the installer but it'd be nice if the product itself prevented that from being a problem via some sort of locking mechanism. I'll attach logs.

Ansible work-around https://github.com/openshift/openshift-ansible/pull/2233
Comment 6 Scott Dodson 2016-07-28 11:19 EDT
Created attachment 1185156 [details]
api server logs
Comment 7 Scott Dodson 2016-07-29 09:29:05 EDT
*** Bug 1361313 has been marked as a duplicate of this bug. ***
Comment 8 Jordan Liggitt 2016-08-02 10:58:49 EDT
Fixed upstream in https://github.com/openshift/origin/pull/10099
Comment 9 Troy Dawson 2016-08-03 15:00:17 EDT
This has been merged and is in OSE v3.3.0.14 or newer.
Comment 11 Gan Huang 2016-08-04 02:32:39 EDT
Verified with openshift v3.3.0.14

Successed to install HA env and make S2I build.
Comment 13 errata-xmlrpc 2016-09-27 05:41:34 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1933

Note You need to log in before you can comment on or make changes to this bug.