Bug 1567047 - templateservicebroker Unable to delete servicebinding
Summary: templateservicebroker Unable to delete servicebinding
Keywords:
Status: CLOSED DUPLICATE of bug 1554141
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Service Catalog
Version: 3.9.0
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: ---
: ---
Assignee: Paul Morie
QA Contact: Zhang Cheng
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-13 10:52 UTC by Juan Luis de Sousa-Valadas
Modified: 2018-05-21 07:45 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-04-13 15:50:28 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Juan Luis de Sousa-Valadas 2018-04-13 10:52:49 UTC
Description of problem:
Cannot delete servicebinding because it has a finalizer:
apiVersion: servicecatalog.k8s.io/v1beta1
kind: ServiceBinding
metadata:
  creationTimestamp: 2018-03-22T07:30:44Z
  deletionGracePeriodSeconds: 0
  deletionTimestamp: 2018-04-11T11:03:10Z
  finalizers:
  - kubernetes-incubator/service-catalog
  generateName: mysql-ephemeral-bv7dk-
  generation: 2
  name: mysql-ephemeral-bv7dk-w8f2c
  namespace: XXXX
  resourceVersion: "15317996"
  selfLink: /apis/servicecatalog.k8s.io/v1beta1/namespaces/capbenchmark/servicebindings/mysql-ephemeral-bv7dk-w8f2c
  uid: ee981f7e-2da2-11e8-992c-0a580a800030
spec:
  externalID: 6df7568a-a9fc-42be-b312-96ad725ec13d
  instanceRef:
    name: mysql-ephemeral-bv7dk
  secretName: mysql-ephemeral-bv7dk-credentials-48uk9
  userInfo:
    groups:
    - system:serviceaccounts
    - system:serviceaccounts:kube-system
    - system:authenticated
    uid: ""
    username: system:serviceaccount:kube-system:namespace-controller
status:
  asyncOpInProgress: false
  conditions:
  - lastTransitionTime: 2018-03-22T09:28:59Z
    message: 'Unbind call failed. Error unbinding from ServiceInstance "capbenchmark/mysql-ephemeral-bv7dk"
      of ClusterServiceClass (K8S: "d29d6b5e-0b4f-11e8-bbd8-005056ba41e9" ExternalName:
      "mysql-ephemeral") at ClusterServiceBroker "template-service-broker": Status:
      403; ErrorMessage: <nil>; Description: templateinstances.template.openshift.io
      "025928df-a863-4139-b92b-c016ca3472ba" is forbidden: User "system:serviceaccount:kube-system:namespace-controller"
      cannot update templateinstances.template.openshift.io in project "capbenchmark";
      ResponseError: <nil>'
    reason: UnbindCallFailed
    status: Unknown
    type: Ready
  - lastTransitionTime: 2018-03-22T09:28:59Z
    message: 'Unbind call failed. Error unbinding from ServiceInstance "XXXX/mysql-ephemeral-bv7dk"
      of ClusterServiceClass (K8S: "d29d6b5e-0b4f-11e8-bbd8-005056ba41e9" ExternalName:
      "mysql-ephemeral") at ClusterServiceBroker "template-service-broker": Status:
      403; ErrorMessage: <nil>; Description: templateinstances.template.openshift.io
      "025928df-a863-4139-b92b-c016ca3472ba" is forbidden: User "system:serviceaccount:kube-system:namespace-controller"
      cannot update templateinstances.template.openshift.io in project "XXXX";
      ResponseError: <nil>'
    reason: UnbindCallFailed
    status: "True"
    type: Failed
  externalProperties:
    userInfo:
      extra:
        scopes.authorization.openshift.io:
        - user:full
      groups:
      - system:authenticated:oauth
      - system:authenticated
      uid: ""
      username: "66115034"
  orphanMitigationInProgress: false
  reconciledGeneration: 2
  unbindStatus: ""


To workaround this I advised the customer to delete the following lines using oc edit:

  finalizers:
  - kubernetes-incubator/service-catalog

However it fails with error:

* status.unbindStatus: Unsupported value: "": supported values: "Required", "Succeeded", "Failed", "NotRequired"

The serviceInstance has already been deleted.

Version-Release number of selected component (if applicable):
v3.9.14 (All the related components, service catalog, tsb, openshift masters and nodes)

How reproducible:
Intermittent.

Steps to Reproduce:
1. Provision a template using the template service broker
2. Wait for it to finish
3. Delete the project

Actual results:
Project deletion is stuck blocked a stuck serviceBinding

Expected results:
Project and all its dependencies are deleted


Additional info:
There are a few related bugs:
https://bugzilla.redhat.com/show_bug.cgi?id=1503829
https://bugzilla.redhat.com/show_bug.cgi?id=1554141
https://bugzilla.redhat.com/show_bug.cgi?id=1503829

But:
* status.unbindStatus: Unsupported value: "": supported values: "Required", "Succeeded", "Failed", "NotRequired"
Seems unique.

Comment 1 Jay Boyd 2018-04-13 15:50:28 UTC
root error 

templateinstances.template.openshift.io
      "025928df-a863-4139-b92b-c016ca3472ba" is forbidden: User "system:serviceaccount:kube-system:namespace-controller"
      cannot update templateinstances.template.openshift.io in project "capbenchmark";

is fixed by https://github.com/openshift/origin/commit/b296e3d62ad6f61dc6e7e97ee6a72739c0856e92

https://bugzilla.redhat.com/show_bug.cgi?id=1554141



Prior to attempting to delete the Finalizer, I believe you can more properly clean up by (as cluster admin) doing

oc delete servicebinding mysql-ephemeral-bv7dk-w8f2c -n XXXX

probably followed by

oc delete serviceinstance .....


we really only want to delete the finalizer as a last resort.   If this still fails, try removing the finalizer again along with deleting the  unbindStatus: ""  line.   I'm sorry to be guessing, I don't have a reproduce environment at the moment. Alternatively the unbindStatus might need to be set to "NotRequired".


If the customer applies patch for the root issue, we probably also want to supply the fix for the Ansible Service Broker which I understand has a similar issue: https://bugzilla.redhat.com/show_bug.cgi?id=1554239

*** This bug has been marked as a duplicate of bug 1554141 ***


Note You need to log in before you can comment on or make changes to this bug.