Bug 1567047

Summary: templateservicebroker Unable to delete servicebinding
Product: OpenShift Container Platform Reporter: Juan Luis de Sousa-Valadas <jdesousa>
Component: Service CatalogAssignee: Paul Morie <pmorie>
Status: CLOSED DUPLICATE QA Contact: Zhang Cheng <chezhang>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.9.0CC: jaboyd, nbhatt
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-04-13 15:50:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Juan Luis de Sousa-Valadas 2018-04-13 10:52:49 UTC
Description of problem:
Cannot delete servicebinding because it has a finalizer:
apiVersion: servicecatalog.k8s.io/v1beta1
kind: ServiceBinding
metadata:
  creationTimestamp: 2018-03-22T07:30:44Z
  deletionGracePeriodSeconds: 0
  deletionTimestamp: 2018-04-11T11:03:10Z
  finalizers:
  - kubernetes-incubator/service-catalog
  generateName: mysql-ephemeral-bv7dk-
  generation: 2
  name: mysql-ephemeral-bv7dk-w8f2c
  namespace: XXXX
  resourceVersion: "15317996"
  selfLink: /apis/servicecatalog.k8s.io/v1beta1/namespaces/capbenchmark/servicebindings/mysql-ephemeral-bv7dk-w8f2c
  uid: ee981f7e-2da2-11e8-992c-0a580a800030
spec:
  externalID: 6df7568a-a9fc-42be-b312-96ad725ec13d
  instanceRef:
    name: mysql-ephemeral-bv7dk
  secretName: mysql-ephemeral-bv7dk-credentials-48uk9
  userInfo:
    groups:
    - system:serviceaccounts
    - system:serviceaccounts:kube-system
    - system:authenticated
    uid: ""
    username: system:serviceaccount:kube-system:namespace-controller
status:
  asyncOpInProgress: false
  conditions:
  - lastTransitionTime: 2018-03-22T09:28:59Z
    message: 'Unbind call failed. Error unbinding from ServiceInstance "capbenchmark/mysql-ephemeral-bv7dk"
      of ClusterServiceClass (K8S: "d29d6b5e-0b4f-11e8-bbd8-005056ba41e9" ExternalName:
      "mysql-ephemeral") at ClusterServiceBroker "template-service-broker": Status:
      403; ErrorMessage: <nil>; Description: templateinstances.template.openshift.io
      "025928df-a863-4139-b92b-c016ca3472ba" is forbidden: User "system:serviceaccount:kube-system:namespace-controller"
      cannot update templateinstances.template.openshift.io in project "capbenchmark";
      ResponseError: <nil>'
    reason: UnbindCallFailed
    status: Unknown
    type: Ready
  - lastTransitionTime: 2018-03-22T09:28:59Z
    message: 'Unbind call failed. Error unbinding from ServiceInstance "XXXX/mysql-ephemeral-bv7dk"
      of ClusterServiceClass (K8S: "d29d6b5e-0b4f-11e8-bbd8-005056ba41e9" ExternalName:
      "mysql-ephemeral") at ClusterServiceBroker "template-service-broker": Status:
      403; ErrorMessage: <nil>; Description: templateinstances.template.openshift.io
      "025928df-a863-4139-b92b-c016ca3472ba" is forbidden: User "system:serviceaccount:kube-system:namespace-controller"
      cannot update templateinstances.template.openshift.io in project "XXXX";
      ResponseError: <nil>'
    reason: UnbindCallFailed
    status: "True"
    type: Failed
  externalProperties:
    userInfo:
      extra:
        scopes.authorization.openshift.io:
        - user:full
      groups:
      - system:authenticated:oauth
      - system:authenticated
      uid: ""
      username: "66115034"
  orphanMitigationInProgress: false
  reconciledGeneration: 2
  unbindStatus: ""


To workaround this I advised the customer to delete the following lines using oc edit:

  finalizers:
  - kubernetes-incubator/service-catalog

However it fails with error:

* status.unbindStatus: Unsupported value: "": supported values: "Required", "Succeeded", "Failed", "NotRequired"

The serviceInstance has already been deleted.

Version-Release number of selected component (if applicable):
v3.9.14 (All the related components, service catalog, tsb, openshift masters and nodes)

How reproducible:
Intermittent.

Steps to Reproduce:
1. Provision a template using the template service broker
2. Wait for it to finish
3. Delete the project

Actual results:
Project deletion is stuck blocked a stuck serviceBinding

Expected results:
Project and all its dependencies are deleted


Additional info:
There are a few related bugs:
https://bugzilla.redhat.com/show_bug.cgi?id=1503829
https://bugzilla.redhat.com/show_bug.cgi?id=1554141
https://bugzilla.redhat.com/show_bug.cgi?id=1503829

But:
* status.unbindStatus: Unsupported value: "": supported values: "Required", "Succeeded", "Failed", "NotRequired"
Seems unique.

Comment 1 Jay Boyd 2018-04-13 15:50:28 UTC
root error 

templateinstances.template.openshift.io
      "025928df-a863-4139-b92b-c016ca3472ba" is forbidden: User "system:serviceaccount:kube-system:namespace-controller"
      cannot update templateinstances.template.openshift.io in project "capbenchmark";

is fixed by https://github.com/openshift/origin/commit/b296e3d62ad6f61dc6e7e97ee6a72739c0856e92

https://bugzilla.redhat.com/show_bug.cgi?id=1554141



Prior to attempting to delete the Finalizer, I believe you can more properly clean up by (as cluster admin) doing

oc delete servicebinding mysql-ephemeral-bv7dk-w8f2c -n XXXX

probably followed by

oc delete serviceinstance .....


we really only want to delete the finalizer as a last resort.   If this still fails, try removing the finalizer again along with deleting the  unbindStatus: ""  line.   I'm sorry to be guessing, I don't have a reproduce environment at the moment. Alternatively the unbindStatus might need to be set to "NotRequired".


If the customer applies patch for the root issue, we probably also want to supply the fix for the Ansible Service Broker which I understand has a similar issue: https://bugzilla.redhat.com/show_bug.cgi?id=1554239

*** This bug has been marked as a duplicate of bug 1554141 ***