Bug 1538370
| Summary: | broker returning 500 instead of 410 on unbind | ||||||
|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Eric Jones <erjones> | ||||
| Component: | Service Broker | Assignee: | Todd Sanders <tsanders> | ||||
| Status: | CLOSED DUPLICATE | QA Contact: | Jian Zhang <jiazha> | ||||
| Severity: | urgent | Docs Contact: | |||||
| Priority: | urgent | ||||||
| Version: | 3.7.0 | CC: | aos-bugs, chezhang, erjones, jokerman, jpeeler, mmccomas, pmorie, wzheng | ||||
| Target Milestone: | --- | ||||||
| Target Release: | --- | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2018-02-08 17:05:52 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
Custom resource definition garbage support was introduced in Kube 1.8/OCP 3.8 and was actively disabled for CRD in OCP 3.5-3.7 (inclusive). If the service broker was setting ownerReferences on ServiceBinding resources in 3.7, the bug is with the broker using an unsupported feature. CRD garbage collector support will not be backported to OCP 3.7. The options seem to be: 1. Patch to the broker code (not sure whether this is actually viable) 2. Upgrade to OCP 3.8+ Jeff, will you take a look at this? Based on what Dan said above, I'm not sure anything can be done here. This functions correctly with a test using PostgreSQL on origin master and origin 3.7.1. As far as OwnerReferences go, the only reference created is on the secret backing the binding. When a binding is deleted, the secret is too. (Is it even possible for a broker to add ownerReferences on a binding?) I'm wondering what I did incorrectly to not have the binding deletion fail. Eric, can you further outline exactly what you deployed? The issue that was presented by the reporter is a little different from the original issue (I think). Here's the binding deletion failure scenario I observed: - service instance is blocked on deletion due to binding still existing - binding is failing on unbind call due to being forbidden to execute by namespace admission controller because the namespace is in a terminated state Will keep looking. We're missing part of the picture here. The garbage collector is not expected to do any work on behalf of ServiceBindings. Would you attach the logs from the service-catalog controller manager, please? Created attachment 1392775 [details]
controller manager logs
I was incorrect about the binding failing due to the terminated namespace. The first call to unbind returns with an error of 500: W0205 16:44:39.262868 1 controller_binding.go:657] ServiceBinding "test/eap64-amq-persistent-s2i-kqtwm-5lbqg": Error unbinding from ServiceInstance "test/eap64-amq-persistent-s2i-kqtwm" of ClusterServiceClass (K8S: "d9d28705-0a89-11e8-a9b3-fa163effd122" ExternalName: "eap64-amq-persistent-s2i") at ClusterServiceBroker "template-service-broker": Status: 500; ErrorMessage: <nil>; Description: templateinstances.template.openshift.io "b7a58efa-4b43-41c6-b219-48fd21ccd2a8" not found; ResponseError: <nil> According to the OSB spec, a response code of 500 is unexpected and in this case I think catalog is functioning as expected. I think the broker should be returning 410 instead. https://github.com/openservicebrokerapi/servicebroker/blob/master/spec.md#response-5 *** This bug has been marked as a duplicate of bug 1540819 *** |
Description of problem: OCP 3.7 uses the service catalog to deploy applications using templates. With this comes the autocreation of servicebindings. ServiceBindings cannot be deleted. oc delete servicebinding <name> servicebinding "<name>" deleted Version-Release number of selected component (if applicable): # openshift version openshift v3.7.14 kubernetes v1.7.6+a08f5eeb62 etcd 3.2.8 How reproducible: 100% Steps to Reproduce: <deploy app from a template> # oc delete servicebinding <name> servicebinding "<name>" deleted Actual results: # oc get servicebinding NAME AGE <name> <SAME_AGE_AS_BEFORE> Expected results: serviceBinding is actually deleted Additional info: Active Controller notes that atomic-openshift-master-controllers told the garbagecollector to delete it and then the garbagecollector ignores it and moves on: <DATE_TIME> <hostname> atomic-openshift-master-controllers[39698]: I0124 <TIMESTAMP> 39698 garbagecollector.go:291] processing item [servicecatalog.k8s.io/v1beta1/ServiceBinding, namespace: javatest, name: jws31-tomcat8-basic-s2i-sb7dl-tzjbz, uid: d7f10240-f7e2-11e7-8f16-0a580a800002] <DATE_TIME> <hostname> atomic-openshift-master-controllers[39698]: E0124 <TIMESTAMP> 39698 garbagecollector.go:172] Ignore syncing item &garbagecollector.node{identity:garbagecollector.objectReference{OwnerReference:v1.OwnerReference{APIVersion:"servicecatalog.k8s.io/v1beta1", Kind:"ServiceBinding", Name:"jws31-tomcat8-basic-s2i-sb7dl-tzjbz", UID:"d7f10240-f7e2-11e7-8f16-0a580a800002", Controller:(*bool)(0xc420eadd98), BlockOwnerDeletion:(*bool)(0xc420eadd99)}, Namespace:"javatest"}, dependentsLock:sync.RWMutex{w:sync.Mutex{state:0, sema:0x0}, writerSem:0x0, readerSem:0x0, readerCount:0, readerWait:0}, dependents:map[*garbagecollector.node]struct {}{(*garbagecollector.node)(0xc4209eec30):struct {}{}}, deletingDependents:false, deletingDependentsLock:sync.RWMutex{w:sync.Mutex{state:0, sema:0x0}, writerSem:0x0, readerSem:0x0, readerCount:0, readerWait:0}, beingDeleted:false, beingDeletedLock:sync.RWMutex{w:sync.Mutex{state:0, sema:0x0}, writerSem:0x0, readerSem:0x0, readerCount:0, readerWait:0}, owners:[]v1.OwnerReference(nil)}: unable to get REST mapping for servicecatalog.k8s.io/v1beta1/ServiceBinding. If servicecatalog.k8s.io/v1beta1/ServiceBinding is a non-core resource (e.g. thirdparty resource, custom resource from aggregated apiserver), please note that the garbage collector doesn't support non-core resources yet. Once they are supported, object with ownerReferences referring non-existing non-core objects will be deleted by the garbage collector. If servicecatalog.k8s.io/v1beta1/ServiceBinding is an invalid resource, then you should manually remove ownerReferences that refer servicecatalog.k8s.io/v1beta1/ServiceBinding objects.