Bug 1571088 - [service catalog HA]The leader change events are missing
Summary: [service catalog HA]The leader change events are missing
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Service Catalog
Version: 3.9.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 3.10.0
Assignee: Jay Boyd
QA Contact: Jian Zhang
Depends On:
Blocks: 1554623
TreeView+ depends on / blocked
Reported: 2018-04-24 05:49 UTC by Jian Zhang
Modified: 2018-07-30 19:14 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: The necessary types were not registered with the Events Scheme. Consequence: The Service Catalog configmap wasn't showing leadership change events and some warnings were in the logs about LeaderElection changes. Fix: Register the necessary types with the events scheme Result: service-catalog-controller-manager configmap now shows leadership change events and the logs are free from these warnings.
Clone Of:
Last Closed: 2018-07-30 19:13:42 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:1816 0 None None None 2018-07-30 19:14:17 UTC

Comment 1 Paul Morie 2018-04-24 15:34:20 UTC
Jay, would you take a look at this? Relevent code is in client-go/tools/leaderelection/resourcelock

Call chain to when RecordEvent should be called:

-> LeaderElector.acquire() or LeaderElector.renew()
   -> Interface.RecordEvent()

Comment 2 Jay Boyd 2018-04-25 13:23:11 UTC
The Scheme we were using with the event recorder was not properly initialized with the required types.  This was previously reported upstream as https://github.com/kubernetes-incubator/service-catalog/issues/1628 and is being fixed by https://github.com/kubernetes-incubator/service-catalog/pull/1981

Comment 3 Jay Boyd 2018-04-27 20:44:04 UTC
The fix is in the upstream release v0.1.15.  Should be picked up by origin next week.

Comment 4 Jay Boyd 2018-05-04 19:41:59 UTC
Delivered today with https://github.com/openshift/openshift-ansible/pull/8205

Comment 5 Jay Boyd 2018-05-04 20:00:59 UTC
only fixed in master, still needs to be ported to 3.9.

Comment 6 Jay Boyd 2018-05-15 16:07:08 UTC
reviewed with Paul, we'll only fix this in 3.10, we don't expect this to be that big of an issue with serviceability to warrant fixing it in 3.9.

Note again this was fixed in 3.10 with https://github.com/openshift/openshift-ansible/pull/8205

Comment 7 Jian Zhang 2018-06-04 06:58:49 UTC
Change status to "ON_QA" since the corresponding image is ready.

Comment 8 Jian Zhang 2018-06-04 07:07:22 UTC
Verify success.

Service catalog version: 

The event info as below after changed leader pods:

[root@shared58r5-master-etcd-2 ~]# oc describe cm service-catalog-controller-manager -n kube-service-catalog
Name:         service-catalog-controller-manager
Namespace:    kube-service-catalog
Labels:       <none>
Annotations:  control-plane.alpha.kubernetes.io/leader={"holderIdentity":"controller-manager-m85x9-external-service-catalog-controller","leaseDurationSeconds":15,"acquireTime":"2018-06-04T06:44:07Z","renewTime":"20...

  Type    Reason          Age   From                                Message
  ----    ------          ----  ----                                -------
  Normal  LeaderElection  3h    service-catalog-controller-manager  controller-manager-qzrfj-external-service-catalog-controller became leader
  Normal  LeaderElection  3h    service-catalog-controller-manager  controller-manager-fcw54-external-service-catalog-controller became leader
  Normal  LeaderElection  2h    service-catalog-controller-manager  controller-manager-d8dm7-external-service-catalog-controller became leader
  Normal  LeaderElection  1m    service-catalog-controller-manager  controller-manager-85pqb-external-service-catalog-controller became leader
  Normal  LeaderElection  36s   service-catalog-controller-manager  controller-manager-m85x9-external-service-catalog-controller became leader

Comment 10 errata-xmlrpc 2018-07-30 19:13:42 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.