Description of problem: In the kube-service-catalog project, both "ds/apiserver" and "ds/controller-manager" has a nodeSelector of "openshift-infra: apiserver". However, this label is added to only one master by default. We have /etc/origin/master/master.etcd-ca.crt available on all the master which is required for this. If we add the label "openshift-infra: apiserver" on other masters, the pods came up fine. My questions is, does this service require to only be a single instance? This means that, this service is a single point of failure. If the first master goes down, it will break the apiserver. Can we add this label to all the masters as well so that the pods can land up on each master?
I discussed with Paul and he agrees: yes, put the label on all masters. The controller-manager has leader election code that will ensure only one is actively handling load.
As I known, we have a relate PR for controller: enable leader election https://github.com/kubernetes-incubator/service-catalog/pull/647
Unmerged PR: https://github.com/openshift/openshift-ansible/pull/7530
*** Bug 1566039 has been marked as a duplicate of this bug. ***
@Jeff, I want to double confirm if you still want to provide fix in 3.9.z. Thanks.
Yes, I do.
Base on current test result, I think service catalog HA is blocking by issue "Service Catalog does not refresh ClusterServicePlan after removing from catalog". Original Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1548122#c20 It was fixed in 3.10, but not in 3.9.z We have to merge relate fix in 3.9.z I opened another bug with target release "3.9.z" to trace: https://bugzilla.redhat.com/show_bug.cgi?id=1568815
I'm changing status to "Modified" and adding depend on "BZ 1568815", please correct me if I have mistake. Thanks.
I'm really not sure how cluster service plans sticking around would influence multiple catalog pods running properly. But to be more specific, this bug is only about running multiple catalog pods versus just one. I think any weird interactions with deleting the current leader should be tracked separately. As far as endpoints versus configmaps go, I regret not merging https://github.com/openshift/openshift-ansible/pull/7530 with https://github.com/openshift/openshift-ansible/pull/8001, which only happened 5 hours ago. Once a build is tested with the latter PR, endpoints aren't used for anything related to HA any more.
To be more clear, I new a bug 1569397 to trace the reboot issue(step1 comment15). And now, we encounter the bug 1568815 more frequently regardless killing the leader pod or rebooting the node that the leader pod is running on.
Jay, will you take a look at this bug; there seems to be an entanglement with the bug for removed plans.
As noted in bug 1568815 the fix for the RemovedFromBrokerCatalog problem is in builds ose-service-catalog:v3.9.21-1 and newer (this build is too old). Are there any remaining problems for development in this issue?
Verified bug 1568815 with ose-service-catalog:v3.9.21-1 LGTM, we will double confirm SC HA function with v3.9.21-1 or newer version.
Jeff, Test it with the 3.9.24 cluster, the "Endpoints" type for leader election has been removed. [root@qe-jiazha-3920-gcemaster-etcd-1 ~]# oc get endpoints NAME ENDPOINTS AGE apiserver 10.2.0.3:6443,10.2.4.3:6443,10.2.6.3:6443 17m (1) Get the leader pod by checking the ConfigMap, it is pod "controller-manager-vx2hs" that running "qe-jiazha-3920-gcemaster-etcd-2". [root@qe-jiazha-3920-gcemaster-etcd-1 ~]# oc get cm openshift-master-controllers -o yaml -n kube-system apiVersion: v1 kind: ConfigMap metadata: annotations: control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"qe-jiazha-3920-gcemaster-etcd-2","leaseDurationSeconds":15,"acquireTime":"2018-04-20T08:14:08Z","renewTime":"2018-04-20T08:53:55Z","leaderTransitions":0}' creationTimestamp: 2018-04-20T08:14:08Z name: openshift-master-controllers namespace: kube-system resourceVersion: "49085" selfLink: /api/v1/namespaces/kube-system/configmaps/openshift-master-controllers uid: cc64bf03-4472-11e8-b925-42010af0002d [root@qe-jiazha-3920-gcemaster-etcd-1 ~]# oc get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE apiserver-9f75n 1/1 Running 0 17m 10.2.0.3 qe-jiazha-3920-gcemaster-etcd-2 apiserver-pm27l 1/1 Running 0 17m 10.2.6.3 qe-jiazha-3920-gcemaster-etcd-1 apiserver-zhvhb 1/1 Running 0 17m 10.2.4.3 qe-jiazha-3920-gcemaster-etcd-3 controller-manager-lmp26 1/1 Running 0 17m 10.2.4.4 qe-jiazha-3920-gcemaster-etcd-3 controller-manager-ssnfg 1/1 Running 2 17m 10.2.6.4 qe-jiazha-3920-gcemaster-etcd-1 controller-manager-vx2hs 1/1 Running 0 17m 10.2.0.4 qe-jiazha-3920-gcemaster-etcd-2 (2) But, the real lead pod is "controller-manager-ssnfg" running on node "qe-jiazha-3920-gcemaster-etcd-1", I think this is a problem, changed status to "ASSIGNED", correct me if I'm wrong. [root@qe-jiazha-3920-gcemaster-etcd-1 ~]# oc logs -f controller-manager-ssnfg I0420 08:34:32.323932 1 feature_gate.go:184] feature gates: map[OriginatingIdentity:true] I0420 08:34:32.374433 1 hyperkube.go:188] Service Catalog version v3.9.24 (built 2018-04-18T18:07:34Z) I0420 08:34:32.408271 1 leaderelection.go:174] attempting to acquire leader lease... E0420 08:34:32.500860 1 event.go:260] Could not construct reference to: '&v1.ConfigMap{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"service-catalog-controller-manager", GenerateName:"", Namespace:"kube-service-catalog", SelfLink:"/api/v1/namespaces/kube-service-catalog/configmaps/service-catalog-controller-manager", UID:"99cdcde5-4475-11e8-a149-42010af0002d", ResourceVersion:"44897", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63659810051, loc:(*time.Location)(0x31c6c00)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string{"control-plane.alpha.kubernetes.io/leader":"{\"holderIdentity\":\"controller-manager-ssnfg-external-service-catalog-controller\",\"leaseDurationSeconds\":15,\"acquireTime\":\"2018-04-20T08:34:11Z\",\"renewTime\":\"2018-04-20T08:34:32Z\",\"leaderTransitions\":0}"}, OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Data:map[string]string(nil)}' due to: 'no kind is registered for the type v1.ConfigMap'. Will not report event: 'Normal' 'LeaderElection' 'controller-manager-ssnfg-external-service-catalog-controller became leader' I0420 08:34:32.500984 1 leaderelection.go:184] successfully acquired lease kube-service-catalog/service-catalog-controller-manager
I believe the command you're using to find the leader for catalog is incorrect. I'm guessing there's a configmap in the catalog namespace that is used, not in kube-system. (I don't know exactly what it is - the command I gave earlier was only an example.)
Jeff, Oh, maybe my misunderstood, the ConfigMap "openshift-master-controllers" doesn't matter with the election of the "service-catalog", right? I'm sorry for this. Now, we can see the ConfigMap "service-catalog-controller-manager" in the kube-service-catalog namespace. Like below: [root@qe-jiazha-3923-gcemaster-etcd-1 ~]# oc get cm service-catalog-controller-manager -o yaml apiVersion: v1 kind: ConfigMap metadata: annotations: control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"controller-manager-d6pvk-external-service-catalog-controller","leaseDurationSeconds":15,"acquireTime":"2018-04-23T02:39:57Z","renewTime":"2018-04-23T02:49:19Z","leaderTransitions":1}' creationTimestamp: 2018-04-23T02:04:56Z name: service-catalog-controller-manager namespace: kube-service-catalog resourceVersion: "52499" selfLink: /api/v1/namespaces/kube-service-catalog/configmaps/service-catalog-controller-manager uid: b8571c0b-469a-11e8-8685-42010af00019 It works well and we can see the "leaderTransitions" changed to 1 after one leader change. But, the "Events" is none, is it as we expect? Why did not display the leader change notice in the "Events" field? Or am I missing something? [root@qe-jiazha-3923-gcemaster-etcd-1 ~]# oc describe service-catalog-controller-manager the server doesn't have a resource type "service-catalog-controller-manager" [root@qe-jiazha-3923-gcemaster-etcd-1 ~]# oc describe cm service-catalog-controller-manager Name: service-catalog-controller-manager Namespace: kube-service-catalog Labels: <none> Annotations: control-plane.alpha.kubernetes.io/leader={"holderIdentity":"controller-manager-d6pvk-external-service-catalog-controller","leaseDurationSeconds":15,"acquireTime":"2018-04-23T02:39:57Z","renewTime":"20... Data ==== Events: <none>
I don't know why the events are missing. The configmap resource lock implements RecordEvent just like the endpoint resource lock.
Jeff, I think we should fix this issue, so I open a new bug 1571088 for tracing this event missing issue.
Remove the "NeedsTestCase" since test case has already covered it.
Since bug 1571088 only fixed in 3.10, so I don't think it's a good choice to ship this HA feature of service catalog in 3.9.z. Below is the latest test on 3.9.z, bug 1571088 issue(event missing) still exists. [root@qe-jiazha-39master-etcd-1 ~]# oc get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE apiserver-77w6w 1/1 Running 0 2h 10.2.2.4 qe-jiazha-39master-etcd-1 apiserver-78l5b 1/1 Running 0 2h 10.2.0.4 qe-jiazha-39master-etcd-3 apiserver-wpxql 1/1 Running 0 2h 10.2.4.3 qe-jiazha-39master-etcd-2 controller-manager-5x2rc 1/1 Running 0 8s 10.2.2.7 qe-jiazha-39master-etcd-1 controller-manager-dpgmb 1/1 Running 0 8m 10.2.4.7 qe-jiazha-39master-etcd-2 controller-manager-rdc7j 1/1 Running 0 2h 10.2.0.5 qe-jiazha-39master-etcd-3 [root@qe-jiazha-39master-etcd-1 ~]# oc describe cm service-catalog-controller-manager Name: service-catalog-controller-manager Namespace: kube-service-catalog Labels: <none> Annotations: control-plane.alpha.kubernetes.io/leader={"holderIdentity":"controller-manager-k2tpk-external-service-catalog-controller","leaseDurationSeconds":15,"acquireTime":"2018-07-26T05:16:46Z","renewTime":"20... Data ==== Events: <none> [root@qe-jiazha-39master-etcd-1 ~]# oc version oc v3.9.38 kubernetes v1.9.1+a0ce1bc657 features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://qe-jiazha-39lb-1.0725-tms.qe.rhcloud.com openshift v3.9.38 kubernetes v1.9.1+a0ce1bc657
bug 1571088 is limited to setting the events on the configmap when a leadership change occurs. The change information is still logged in the controller logs when a new pod takes leadership. This comes down to a "serviceablity" issue and easily being able to understand leadership changes. The entries above indicate HA is working properly with the exception of the configmap events. The bottom line is that HA is working for the controller manager - leadership properly changes when the current leader is made unavailable. We agree it would be ideal to fix all bugs in all releases, but we have to determine where to focus our limited development resources. This decision was not made lightly and much discussion was had about if 1571088 should be back ported to 3.9.z. The end decision was "no". I will make sure to include the test in future discussions around if specific bugs will be backported. I'm going to put this back to ON_QA as the core issue in this bug has been addressed in 3.9.z.
Jay, Thanks for your patient explanation! Verify it.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:2335