Bug 1470624 - Can't delete broker resource after setup env by openshift-ansible
Can't delete broker resource after setup env by openshift-ansible
Status: CLOSED ERRATA
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer (Show other bugs)
3.6.0
Unspecified Unspecified
high Severity high
: ---
: ---
Assigned To: ewolinet
DeShuai Ma
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-07-13 06:28 EDT by DeShuai Ma
Modified: 2017-08-16 15 EDT (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-08-10 01:31:01 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
apiserver.log (148.86 KB, text/plain)
2017-07-14 12:58 EDT, DeShuai Ma
no flags Details
controller-manager.log (117.46 KB, text/plain)
2017-07-14 12:58 EDT, DeShuai Ma
no flags Details

  None (edit)
Description DeShuai Ma 2017-07-13 06:28:34 EDT
Description of problem:
Sometime can't delete the broker resource, this more happen the the situation, If controller-manger can't access the broker server.

Version-Release number of selected component (if applicable):
openshift v3.6.143
kubernetes v1.6.1+5115d708d7
etcd 3.2.1

How reproducible:
Sometime

Steps to Reproduce:
1. Get the broker and delete the broker
[root@host-8-174-87 dma]# oc get broker
NAME                      KIND
ansible-service-broker    Broker.v1alpha1.servicecatalog.k8s.io
template-service-broker   Broker.v1alpha1.servicecatalog.k8s.io
[root@host-8-174-87 dma]# oc delete broker template-service-broker
broker "template-service-broker" deleted
[root@host-8-174-87 dma]# oc get broker
NAME                      KIND
ansible-service-broker    Broker.v1alpha1.servicecatalog.k8s.io
template-service-broker   Broker.v1alpha1.servicecatalog.k8s.io
[root@host-8-174-87 dma]# oc delete broker template-service-broker
broker "template-service-broker" deleted
[root@host-8-174-87 dma]# oc delete broker template-service-broker
broker "template-service-broker" deleted
[root@host-8-174-87 dma]# oc get broker
NAME                      KIND
ansible-service-broker    Broker.v1alpha1.servicecatalog.k8s.io
template-service-broker   Broker.v1alpha1.servicecatalog.k8s.io
2.
3.

Actual results:
1. Should delete the broker successfully

Expected results:
1. Should delete the broker successfully

Additional info:
log of apiserver, controller-manager is attached
Comment 1 Jeff Peeler 2017-07-14 11:46:06 EDT
The logs didn't seem to make it.
Comment 2 DeShuai Ma 2017-07-14 12:58 EDT
Created attachment 1298508 [details]
apiserver.log
Comment 3 DeShuai Ma 2017-07-14 12:58 EDT
Created attachment 1298510 [details]
controller-manager.log
Comment 4 DeShuai Ma 2017-07-16 23:06:51 EDT
Try to reproduce and debug. I find some logs:

W0717 02:45:41.134178       1 controller_broker.go:276] Error deleting ServiceClass "mediawiki-apb" (Broker "ansible-service-broker"): User "system:serviceaccount:kube-service-catalog:service-catalog-controller" cannot delete serviceclasses.servicecatalog.k8s.io at the cluster scope
I0717 02:45:41.134216       1 controller_broker.go:391] Found status change for Broker "ansible-service-broker" condition "Ready": "False" -> "Unknown"; setting lastTransitionTime to 2017-07-17 02:45:41.134207935 +0000 UTC
I0717 02:45:41.134245       1 controller_broker.go:403] Updating ready condition for Broker ansible-service-broker to Unknown
E0717 02:45:41.151083       1 controller_broker.go:406] Error updating ready condition for Broker ansible-service-broker: Operation cannot be fulfilled on brokers "ansible-service-broker": the object has been modified; please apply your changes to the latest version and try again
I0717 02:45:41.151153       1 controller.go:195] Error syncing Broker ansible-service-broker: User "system:serviceaccount:kube-service-catalog:service-catalog-controller" cannot delete serviceclasses.servicecatalog.k8s.io at the cluster scope
I0717 02:45:41.151380       1 event.go:217] Event(v1.ObjectReference{Kind:"Broker", Namespace:"", Name:"ansible-service-broker", UID:"8306dbc0-6a98-11e7-84c7-0a580a810003", APIVersion:"servicecatalog.k8s.io", ResourceVersion:"5038", FieldPath:""}): type: 'Warning' reason: 'ErrorDeletingServiceClass' Error deleting service class. Error deleting ServiceClass "mediawiki-apb" (Broker "ansible-service-broker"): User "system:serviceaccount:kube-service-catalog:service-catalog-controller" cannot delete serviceclasses.servicecatalog.k8s.io at the cluster scope
I0717 02:45:41.162144       1 controller_broker.go:161] Processing Broker ansible-service-broker
I0717 02:45:41.162174       1 controller_broker.go:182] Creating client for Broker ansible-service-broker, URL: http://asb.openshift-ansible-service-broker.svc:1338
I0717 02:45:41.162190       1 controller_broker.go:249] Finalizing Broker ansible-service-broker


Version to reproduce:
openshift-ansible-3.6.151-1.git.0.a82f0c2.el7.noarch.rpm
openshift v3.6.151
kubernetes v1.6.1+5115d708d7
etcd 3.2.1

As I install the env by openshift-ansible. move to installer.
Comment 5 Derek Carr 2017-07-17 10:36:25 EDT
This is target release 3.7 as catalog is tech preview in 3.6, and therefore, we will not block a release for any bugs associated.

Jeff - can you look at the logs further to isolate the actual problem?
Comment 6 Jeff Peeler 2017-07-17 11:44:27 EDT
My understanding is that if the controller is not available it's expected behavior for deletions to not be processed. But I will confirm later with additional investigation.
Comment 7 DeShuai Ma 2017-07-19 06:11:02 EDT
My describe the issue clear again:
After enable serivce-catalog by openshift-ansible. Then delete broker I met this error.
Why I need delete broker? Because there are two bug about auto update serviceclass:
https://bugzilla.redhat.com/show_bug.cgi?id=1468173
https://bugzilla.redhat.com/show_bug.cgi?id=1469448
I must delete broker then recreate. Otherwise serviceclass is empty. User can't see any apb service in console.

From the controller-manage log: 
I0719 10:05:00.959306       1 controller.go:200] Dropping Broker "ansible-service-broker" out of the queue: User "system:serviceaccount:kube-service-catalog:service-catalog-controller" cannot delete serviceclasses.servicecatalog.k8s.io at the cluster scope

So I think we have set wrong permission for the sa in installer, So I think we need fix for the installer in ocp36. aggregate that?
If I'm wrong, please correct me, thanks.
Comment 8 DeShuai Ma 2017-07-19 06:21:32 EDT
After grant permission (actually we don't need so large permission) I can delete broker successfully
[root@host-8-175-72 ~]# oadm policy add-cluster-role-to-user cluster-admin system:serviceaccount:kube-service-catalog:service-catalog-controller
cluster role "cluster-admin" added: "system:serviceaccount:kube-service-catalog:service-catalog-controller"
[root@host-8-175-72 ~]# oc get broker
NAME                     KIND
ansible-service-broker   Broker.v1alpha1.servicecatalog.k8s.io
[root@host-8-175-72 ~]# oc delete broker ansible-service-broker 
broker "ansible-service-broker" deleted
[root@host-8-175-72 ~]# oc get broker
No resources found.
Comment 9 Paul Morie 2017-07-19 17:01:03 EDT
Seems like a problem with the rbac setup created by the installer - reassigning to eric.
Comment 10 ewolinet 2017-07-19 18:03:10 EDT
I'd like to make sure we can get all missing permissions at once... should the service-catalog-controller be able to perform any other operations (update/patch/list/watch) other than create/delete on serviceclasses?
Comment 14 DeShuai Ma 2017-07-24 02:43:23 EDT
Verify on openshift-ansible-3.6.162-1.git.0.50e29bd.el7.noarch.rpm.
Now can delete broker successfully after install.
Comment 16 errata-xmlrpc 2017-08-10 01:31:01 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1716

Note You need to log in before you can comment on or make changes to this bug.