Bug 1470624

Summary: Can't delete broker resource after setup env by openshift-ansible
Product: OpenShift Container Platform Reporter: DeShuai Ma <dma>
Component: InstallerAssignee: ewolinet
Status: CLOSED ERRATA QA Contact: DeShuai Ma <dma>
Severity: high Docs Contact:
Priority: high    
Version: 3.6.0CC: aos-bugs, decarr, dma, ewolinet, jokerman, jpeeler, mmccomas, pmorie
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-08-10 05:31:01 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
apiserver.log
none
controller-manager.log none

Description DeShuai Ma 2017-07-13 10:28:34 UTC
Description of problem:
Sometime can't delete the broker resource, this more happen the the situation, If controller-manger can't access the broker server.

Version-Release number of selected component (if applicable):
openshift v3.6.143
kubernetes v1.6.1+5115d708d7
etcd 3.2.1

How reproducible:
Sometime

Steps to Reproduce:
1. Get the broker and delete the broker
[root@host-8-174-87 dma]# oc get broker
NAME                      KIND
ansible-service-broker    Broker.v1alpha1.servicecatalog.k8s.io
template-service-broker   Broker.v1alpha1.servicecatalog.k8s.io
[root@host-8-174-87 dma]# oc delete broker template-service-broker
broker "template-service-broker" deleted
[root@host-8-174-87 dma]# oc get broker
NAME                      KIND
ansible-service-broker    Broker.v1alpha1.servicecatalog.k8s.io
template-service-broker   Broker.v1alpha1.servicecatalog.k8s.io
[root@host-8-174-87 dma]# oc delete broker template-service-broker
broker "template-service-broker" deleted
[root@host-8-174-87 dma]# oc delete broker template-service-broker
broker "template-service-broker" deleted
[root@host-8-174-87 dma]# oc get broker
NAME                      KIND
ansible-service-broker    Broker.v1alpha1.servicecatalog.k8s.io
template-service-broker   Broker.v1alpha1.servicecatalog.k8s.io
2.
3.

Actual results:
1. Should delete the broker successfully

Expected results:
1. Should delete the broker successfully

Additional info:
log of apiserver, controller-manager is attached

Comment 1 Jeff Peeler 2017-07-14 15:46:06 UTC
The logs didn't seem to make it.

Comment 2 DeShuai Ma 2017-07-14 16:58:01 UTC
Created attachment 1298508 [details]
apiserver.log

Comment 3 DeShuai Ma 2017-07-14 16:58:54 UTC
Created attachment 1298510 [details]
controller-manager.log

Comment 4 DeShuai Ma 2017-07-17 03:06:51 UTC
Try to reproduce and debug. I find some logs:

W0717 02:45:41.134178       1 controller_broker.go:276] Error deleting ServiceClass "mediawiki-apb" (Broker "ansible-service-broker"): User "system:serviceaccount:kube-service-catalog:service-catalog-controller" cannot delete serviceclasses.servicecatalog.k8s.io at the cluster scope
I0717 02:45:41.134216       1 controller_broker.go:391] Found status change for Broker "ansible-service-broker" condition "Ready": "False" -> "Unknown"; setting lastTransitionTime to 2017-07-17 02:45:41.134207935 +0000 UTC
I0717 02:45:41.134245       1 controller_broker.go:403] Updating ready condition for Broker ansible-service-broker to Unknown
E0717 02:45:41.151083       1 controller_broker.go:406] Error updating ready condition for Broker ansible-service-broker: Operation cannot be fulfilled on brokers "ansible-service-broker": the object has been modified; please apply your changes to the latest version and try again
I0717 02:45:41.151153       1 controller.go:195] Error syncing Broker ansible-service-broker: User "system:serviceaccount:kube-service-catalog:service-catalog-controller" cannot delete serviceclasses.servicecatalog.k8s.io at the cluster scope
I0717 02:45:41.151380       1 event.go:217] Event(v1.ObjectReference{Kind:"Broker", Namespace:"", Name:"ansible-service-broker", UID:"8306dbc0-6a98-11e7-84c7-0a580a810003", APIVersion:"servicecatalog.k8s.io", ResourceVersion:"5038", FieldPath:""}): type: 'Warning' reason: 'ErrorDeletingServiceClass' Error deleting service class. Error deleting ServiceClass "mediawiki-apb" (Broker "ansible-service-broker"): User "system:serviceaccount:kube-service-catalog:service-catalog-controller" cannot delete serviceclasses.servicecatalog.k8s.io at the cluster scope
I0717 02:45:41.162144       1 controller_broker.go:161] Processing Broker ansible-service-broker
I0717 02:45:41.162174       1 controller_broker.go:182] Creating client for Broker ansible-service-broker, URL: http://asb.openshift-ansible-service-broker.svc:1338
I0717 02:45:41.162190       1 controller_broker.go:249] Finalizing Broker ansible-service-broker


Version to reproduce:
openshift-ansible-3.6.151-1.git.0.a82f0c2.el7.noarch.rpm
openshift v3.6.151
kubernetes v1.6.1+5115d708d7
etcd 3.2.1

As I install the env by openshift-ansible. move to installer.

Comment 5 Derek Carr 2017-07-17 14:36:25 UTC
This is target release 3.7 as catalog is tech preview in 3.6, and therefore, we will not block a release for any bugs associated.

Jeff - can you look at the logs further to isolate the actual problem?

Comment 6 Jeff Peeler 2017-07-17 15:44:27 UTC
My understanding is that if the controller is not available it's expected behavior for deletions to not be processed. But I will confirm later with additional investigation.

Comment 7 DeShuai Ma 2017-07-19 10:11:02 UTC
My describe the issue clear again:
After enable serivce-catalog by openshift-ansible. Then delete broker I met this error.
Why I need delete broker? Because there are two bug about auto update serviceclass:
https://bugzilla.redhat.com/show_bug.cgi?id=1468173
https://bugzilla.redhat.com/show_bug.cgi?id=1469448
I must delete broker then recreate. Otherwise serviceclass is empty. User can't see any apb service in console.

From the controller-manage log: 
I0719 10:05:00.959306       1 controller.go:200] Dropping Broker "ansible-service-broker" out of the queue: User "system:serviceaccount:kube-service-catalog:service-catalog-controller" cannot delete serviceclasses.servicecatalog.k8s.io at the cluster scope

So I think we have set wrong permission for the sa in installer, So I think we need fix for the installer in ocp36. aggregate that?
If I'm wrong, please correct me, thanks.

Comment 8 DeShuai Ma 2017-07-19 10:21:32 UTC
After grant permission (actually we don't need so large permission) I can delete broker successfully
[root@host-8-175-72 ~]# oadm policy add-cluster-role-to-user cluster-admin system:serviceaccount:kube-service-catalog:service-catalog-controller
cluster role "cluster-admin" added: "system:serviceaccount:kube-service-catalog:service-catalog-controller"
[root@host-8-175-72 ~]# oc get broker
NAME                     KIND
ansible-service-broker   Broker.v1alpha1.servicecatalog.k8s.io
[root@host-8-175-72 ~]# oc delete broker ansible-service-broker 
broker "ansible-service-broker" deleted
[root@host-8-175-72 ~]# oc get broker
No resources found.

Comment 9 Paul Morie 2017-07-19 21:01:03 UTC
Seems like a problem with the rbac setup created by the installer - reassigning to eric.

Comment 10 ewolinet 2017-07-19 22:03:10 UTC
I'd like to make sure we can get all missing permissions at once... should the service-catalog-controller be able to perform any other operations (update/patch/list/watch) other than create/delete on serviceclasses?

Comment 14 DeShuai Ma 2017-07-24 06:43:23 UTC
Verify on openshift-ansible-3.6.162-1.git.0.50e29bd.el7.noarch.rpm.
Now can delete broker successfully after install.

Comment 16 errata-xmlrpc 2017-08-10 05:31:01 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1716