Bug 1571088

Summary: [service catalog HA]The leader change events are missing
Product: OpenShift Container Platform Reporter: Jian Zhang <jiazha>
Component: Service CatalogAssignee: Jay Boyd <jaboyd>
Status: CLOSED ERRATA QA Contact: Jian Zhang <jiazha>
Severity: medium Docs Contact:
Priority: high    
Version: 3.9.0CC: chezhang, jaboyd, zhsun, zitang
Target Milestone: ---Keywords: Reopened
Target Release: 3.10.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: The necessary types were not registered with the Events Scheme. Consequence: The Service Catalog configmap wasn't showing leadership change events and some warnings were in the logs about LeaderElection changes. Fix: Register the necessary types with the events scheme Result: service-catalog-controller-manager configmap now shows leadership change events and the logs are free from these warnings.
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-07-30 19:13:42 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1554623    

Comment 1 Paul Morie 2018-04-24 15:34:20 UTC
Jay, would you take a look at this? Relevent code is in client-go/tools/leaderelection/resourcelock

Call chain to when RecordEvent should be called:


LeaderElector.Run()
-> LeaderElector.acquire() or LeaderElector.renew()
   -> Interface.RecordEvent()

Comment 2 Jay Boyd 2018-04-25 13:23:11 UTC
The Scheme we were using with the event recorder was not properly initialized with the required types.  This was previously reported upstream as https://github.com/kubernetes-incubator/service-catalog/issues/1628 and is being fixed by https://github.com/kubernetes-incubator/service-catalog/pull/1981

Comment 3 Jay Boyd 2018-04-27 20:44:04 UTC
The fix is in the upstream release v0.1.15.  Should be picked up by origin next week.

Comment 4 Jay Boyd 2018-05-04 19:41:59 UTC
Delivered today with https://github.com/openshift/openshift-ansible/pull/8205

Comment 5 Jay Boyd 2018-05-04 20:00:59 UTC
only fixed in master, still needs to be ported to 3.9.

Comment 6 Jay Boyd 2018-05-15 16:07:08 UTC
reviewed with Paul, we'll only fix this in 3.10, we don't expect this to be that big of an issue with serviceability to warrant fixing it in 3.9.

Note again this was fixed in 3.10 with https://github.com/openshift/openshift-ansible/pull/8205

Comment 7 Jian Zhang 2018-06-04 06:58:49 UTC
Change status to "ON_QA" since the corresponding image is ready.

Comment 8 Jian Zhang 2018-06-04 07:07:22 UTC
Verify success.

Service catalog version: 
v3.10.0-0.58.0;Upstream:v0.1.19

The event info as below after changed leader pods:

[root@shared58r5-master-etcd-2 ~]# oc describe cm service-catalog-controller-manager -n kube-service-catalog
Name:         service-catalog-controller-manager
Namespace:    kube-service-catalog
Labels:       <none>
Annotations:  control-plane.alpha.kubernetes.io/leader={"holderIdentity":"controller-manager-m85x9-external-service-catalog-controller","leaseDurationSeconds":15,"acquireTime":"2018-06-04T06:44:07Z","renewTime":"20...

Data
====
Events:
  Type    Reason          Age   From                                Message
  ----    ------          ----  ----                                -------
  Normal  LeaderElection  3h    service-catalog-controller-manager  controller-manager-qzrfj-external-service-catalog-controller became leader
  Normal  LeaderElection  3h    service-catalog-controller-manager  controller-manager-fcw54-external-service-catalog-controller became leader
  Normal  LeaderElection  2h    service-catalog-controller-manager  controller-manager-d8dm7-external-service-catalog-controller became leader
  Normal  LeaderElection  1m    service-catalog-controller-manager  controller-manager-85pqb-external-service-catalog-controller became leader
  Normal  LeaderElection  36s   service-catalog-controller-manager  controller-manager-m85x9-external-service-catalog-controller became leader

Comment 10 errata-xmlrpc 2018-07-30 19:13:42 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1816