Bug 1535902

Summary:

Failed to deprovision an unready serviceinstance

Product:

OpenShift Container Platform

Reporter:

XiuJuan Wang <xiuwang>

Component:

Service Broker

Assignee:

Jay Boyd <jaboyd>

Status:

CLOSED ERRATA

QA Contact:

XiuJuan Wang <xiuwang>

Severity:

medium

Docs Contact:

Priority:

medium

Version:

3.9.0

CC:

aos-bugs, cbucur, pmorie, xiuwang

Target Milestone:

---

Keywords:

Regression

Target Release:

3.9.0

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

Doc Type:

No Doc Update

Doc Text:

undefined

Story Points:

---

Clone Of:

Environment:

Last Closed:

2018-03-28 14:20:40 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
Can't delete an unready serviceinstance	none

Description XiuJuan Wang 2018-01-18 08:57:13 UTC

Description of problem:
You can deprovision a failed or ready serviceinstance from webconsole or via cmd.
But failed to deprovision an unready serviceinstance,it just is marked as deletion on web console. Then the deletion serviceinstance will be ready finally.

Version-Release number of selected component (if applicable):
openshift v3.9.0-0.20.0
kubernetes v1.9.1+a0ce1bc657

ose-template-service-broker v3.9.0-0.20.0

How reproducible:
always

Steps to Reproduce:
1.Provision an clusterserviceclass(such as jenkins-ephemeral) from browse console
2.Deprovision the serviceinstance from web console before it is ready or failed
3.Check the serviceinstance if be deleted

Actual results:
After 40 mins, the serviceinstance is still not deleted
$oc get serviceinstance  -n xiu
NAME                      AGE
jenkins-ephemeral-94wks   40m

On the webconsole, the serviceinstance is marked as deletion, but go to ready finally.

Expected results:
Should be deleted no matter what's status

Additional info:

Check the events, no deprovision operation be recorded

$oc describe serviceinstance jenkins-ephemeral-94wks
Status:
  Async Op In Progress:  false
  Conditions:
    Last Transition Time:  2018-01-18T08:04:05Z
    Message:               The instance was provisioned successfully
    Reason:                ProvisionedSuccessfully
    Status:                True
    Type:                  Ready
  Deprovision Status:      Required
  External Properties:
    Cluster Service Plan External ID:    7a8f466e-fc18-11e7-b1d8-fa163eff0701
    Cluster Service Plan External Name:  default
    Parameter Checksum:                  463a0602cfe81b81cd7534488607b181e601c12ce8b87dddc59715eb8911cba5
    Parameters:
      ENABLE _ OAUTH:                  <redacted>
      JENKINS _ IMAGE _ STREAM _ TAG:  <redacted>
      JENKINS _ SERVICE _ NAME:        <redacted>
      JNLP _ SERVICE _ NAME:           <redacted>
      MEMORY _ LIMIT:                  <redacted>
      NAMESPACE:                       <redacted>
    User Info:
      Extra:
        Scopes . Authorization . Openshift . Io:
          user:full
      Groups:
        system:authenticated:oauth
        system:authenticated
      UID:                        
      Username:                   xiuwang3
  Orphan Mitigation In Progress:  false
  Reconciled Generation:          2
Events:
  Type     Reason                   Age   From                                Message
  ----     ------                   ----  ----                                -------
  Normal   Provisioning             41m   service-catalog-controller-manager  The instance is being provisioned asynchronously
  Warning  ProvisionCallFailed      41m   service-catalog-controller-manager  Error provisioning ServiceInstance of ClusterServiceClass (K8S: "7a8f466e-fc18-11e7-b1d8-fa163eff0701" ExternalName: "jenkins-ephemeral") at ClusterServiceBroker "template-service-broker": Status: 409; ErrorMessage: <nil>; Description: <nil>; ResponseError: <nil>
  Normal   ProvisionedSuccessfully  36m   service-catalog-controller-manager  The instance was provisioned successfully

Comment 1 XiuJuan Wang 2018-01-18 08:58:11 UTC

Created attachment 1382781 [details]
Can't delete an unready serviceinstance

Comment 2 Jay Boyd 2018-01-19 19:06:41 UTC

Have you verified the same result from the oc command line?  Could you please attach control manager logs.

Comment 3 Jay Boyd 2018-01-25 16:13:05 UTC

I reproduced.

During provisioning:

0125 15:53:26.752904       1 controller_instance.go:1129] ServiceInstance "my-project/jenkins-ephemeral-qzsgl": Polling last operation
I0125 15:53:26.809956       1 controller_instance.go:1212] ServiceInstance "my-project/jenkins-ephemeral-qzsgl": Poll returned "in progress" : Response description: <nil>
I0125 15:53:26.810003       1 controller_instance.go:1318] ServiceInstance "my-project/jenkins-ephemeral-qzsgl": Last operation not completed (still in progress)

I0125 15:53:26.752904       1 controller_instance.go:1129] ServiceInstance "my-project/jenkins-ephemeral-qzsgl": Polling last operation
I0125 15:53:26.809956       1 controller_instance.go:1212] ServiceInstance "my-project/jenkins-ephemeral-qzsgl": Poll returned "in progress" : Response description: <nil>
I0125 15:53:26.810003       1 controller_instance.go:1318] ServiceInstance "my-project/jenkins-ephemeral-qzsgl": Last operation not completed (still in progress)

I0125 15:55:34.812075       1 controller_instance.go:1129] ServiceInstance "my-project/jenkins-ephemeral-qzsgl": Polling last operation
I0125 15:55:34.826858       1 controller_instance.go:1212] ServiceInstance "my-project/jenkins-ephemeral-qzsgl": Poll returned "succeeded" : Response description: <nil>
I0125 15:55:34.826897       1 controller_instance.go:1778] ServiceInstance "my-project/jenkins-ephemeral-qzsgl": Setting condition "Ready" to True
I0125 15:55:34.827411       1 controller_instance.go:1803] ServiceInstance "my-project/jenkins-ephemeral-qzsgl": Found status change, condition "Ready": "False" -> "True"; setting lastTransitionTime to 2018-01-25 15:55:34.826889779 +0000 UTC m=+494.837548472
I0125 15:55:34.827422       1 controller_instance.go:1841] ServiceInstance "my-project/jenkins-ephemeral-qzsgl": Updating status
I0125 15:55:34.840472       1 controller_instance.go:1384] ServiceInstance "my-project/jenkins-ephemeral-qzsgl": Successfully provisioned
I0125 15:55:34.840558       1 event.go:218] Event(v1.ObjectReference{Kind:"ServiceInstance", Namespace:"my-project", Name:"jenkins-ephemeral-qzsgl", UID:"939c30fe-01e7-11e8-ac5f-0242ac110006", APIVersion:"servicecatalog.k8s.io", ResourceVersion:"72", FieldPath:""}): type: 'Normal' reason: 'ProvisionedSuccessfully' The instance was provisioned successfully


I0125 15:55:34.851737       1 controller_instance.go:237] ServiceInstance "my-project/jenkins-ephemeral-qzsgl": Processing deleting event
I0125 15:55:34.855226       1 controller.go:331] ServiceInstance "my-project/jenkins-ephemeral-qzsgl": Creating client for ClusterServiceBroker template-service-broker, URL: https://apiserver.openshift-template-service-broker.svc:443/brokers/template.openshift.io
I0125 15:55:34.855377       1 controller_instance.go:1778] ServiceInstance "my-project/jenkins-ephemeral-qzsgl": Setting condition "Ready" to False
I0125 15:55:34.855394       1 controller_instance.go:1803] ServiceInstance "my-project/jenkins-ephemeral-qzsgl": Found status change, condition "Ready": "True" -> "False"; setting lastTransitionTime to 2018-01-25 15:55:34.855368082 +0000 UTC m=+494.866026779
I0125 15:55:34.855402       1 controller_instance.go:1841] ServiceInstance "my-project/jenkins-ephemeral-qzsgl": Updating status
E0125 15:55:34.858596       1 controller_instance.go:1844] ServiceInstance "my-project/jenkins-ephemeral-qzsgl": Failed to update status: ServiceInstance.servicecatalog.k8s.io "jenkins-ephemeral-qzsgl" is invalid: status.currentOperation: Forbidden: currentOperation must not be present when reconciledGeneration and generation are equal
I0125 15:55:34.858611       1 controller.go:237] Error syncing ServiceInstance my-project/jenkins-ephemeral-qzsgl: ServiceInstance.servicecatalog.k8s.io "jenkins-ephemeral-qzsgl" is invalid: status.currentOperation: Forbidden: currentOperation must not be present when reconciledGeneration and generation are equal


I0125 15:55:34.863790       1 controller_instance.go:237] ServiceInstance "my-project/jenkins-ephemeral-qzsgl": Processing deleting event
I0125 15:55:34.865409       1 controller.go:331] ServiceInstance "my-project/jenkins-ephemeral-qzsgl": Creating client for ClusterServiceBroker template-service-broker, URL: https://apiserver.openshift-template-service-broker.svc:443/brokers/template.openshift.io
I0125 15:55:34.865719       1 controller_instance.go:1778] ServiceInstance "my-project/jenkins-ephemeral-qzsgl": Setting condition "Ready" to False
I0125 15:55:34.865744       1 controller_instance.go:1803] ServiceInstance "my-project/jenkins-ephemeral-qzsgl": Found status change, condition "Ready": "True" -> "False"; setting lastTransitionTime to 2018-01-25 15:55:34.865700843 +0000 UTC m=+494.876359551
I0125 15:55:34.865756       1 controller_instance.go:1841] ServiceInstance "my-project/jenkins-ephemeral-qzsgl": Updating status
E0125 15:55:34.869429       1 controller_instance.go:1844] ServiceInstance "my-project/jenkins-ephemeral-qzsgl": Failed to update status: ServiceInstance.servicecatalog.k8s.io "jenkins-ephemeral-qzsgl" is invalid: status.currentOperation: Forbidden: currentOperation must not be present when reconciledGeneration and generation are equal
I0125 15:55:34.869451       1 controller.go:237] Error syncing ServiceInstance my-project/jenkins-ephemeral-qzsgl: ServiceInstance.servicecatalog.k8s.io "jenkins-ephemeral-qzsgl" is invalid: status.currentOperation: Forbidden: currentOperation must not be present when reconciledGeneration and generation are equal


I0125 15:55:34.879637       1 controller_instance.go:237] ServiceInstance "my-project/jenkins-ephemeral-qzsgl": Processing deleting event
I0125 15:55:34.881381       1 controller.go:331] ServiceInstance "my-project/jenkins-ephemeral-qzsgl": Creating client for ClusterServiceBroker template-service-broker, URL: https://apiserver.openshift-template-service-broker.svc:443/brokers/template.openshift.io
I0125 15:55:34.881619       1 controller_instance.go:1778] ServiceInstance "my-project/jenkins-ephemeral-qzsgl": Setting condition "Ready" to False
I0125 15:55:34.881640       1 controller_instance.go:1803] ServiceInstance "my-project/jenkins-ephemeral-qzsgl": Found status change, condition "Ready": "True" -> "False"; setting lastTransitionTime to 2018-01-25 15:55:34.881606983 +0000 UTC m=+494.892265678
I0125 15:55:34.881651       1 controller_instance.go:1841] ServiceInstance "my-project/jenkins-ephemeral-qzsgl": Updating status
E0125 15:55:34.885396       1 controller_instance.go:1844] ServiceInstance "my-project/jenkins-ephemeral-qzsgl": Failed to update status: ServiceInstance.servicecatalog.k8s.io "jenkins-ephemeral-qzsgl" is invalid: status.currentOperation: Forbidden: currentOperation must not be present when reconciledGeneration and generation are equal
I0125 15:55:34.885413       1 controller.go:237] Error syncing ServiceInstance my-project/jenkins-ephemeral-qzsgl: ServiceInstance.servicecatalog.k8s.io "jenkins-ephemeral-qzsgl" is invalid: status.currentOperation: Forbidden: currentOperation must not be present when reconciledGeneration and generation are equal


I0125 15:55:34.905573       1 controller_instance.go:237] ServiceInstance "my-project/jenkins-ephemeral-qzsgl": Processing deleting event
I0125 15:55:34.907257       1 controller.go:331] ServiceInstance "my-project/jenkins-ephemeral-qzsgl": Creating client for ClusterServiceBroker template-service-broker, URL: https://apiserver.openshift-template-service-broker.svc:443/brokers/template.openshift.io
I0125 15:55:34.907405       1 controller_instance.go:1778] ServiceInstance "my-project/jenkins-ephemeral-qzsgl": Setting condition "Ready" to False
I0125 15:55:34.907427       1 controller_instance.go:1803] ServiceInstance "my-project/jenkins-ephemeral-qzsgl": Found status change, condition "Ready": "True" -> "False"; setting lastTransitionTime to 2018-01-25 15:55:34.907392443 +0000 UTC m=+494.918051140
I0125 15:55:34.907438       1 controller_instance.go:1841] ServiceInstance "my-project/jenkins-ephemeral-qzsgl": Updating status
E0125 15:55:34.909864       1 controller_instance.go:1844] ServiceInstance "my-project/jenkins-ephemeral-qzsgl": Failed to update status: ServiceInstance.servicecatalog.k8s.io "jenkins-ephemeral-qzsgl" is invalid: status.currentOperation: Forbidden: currentOperation must not be present when reconciledGeneration and generation are equal
I0125 15:55:34.909880       1 controller.go:237] Error syncing ServiceInstance my-project/jenkins-ephemeral-qzsgl: ServiceInstance.servicecatalog.k8s.io "jenkins-ephemeral-qzsgl" is invalid: status.currentOperation: Forbidden: currentOperation must not be present when reconciledGeneration and generation are equal



$ oc describe serviceinstance jenkins-ephemeral-qzsgl -n my-project
Name:         jenkins-ephemeral-qzsgl
Namespace:    my-project
Labels:       <none>
Annotations:  <none>
API Version:  servicecatalog.k8s.io/v1beta1
Kind:         ServiceInstance
Metadata:
  Creation Timestamp:             2018-01-25T15:51:16Z
  Deletion Grace Period Seconds:  0
  Deletion Timestamp:             2018-01-25T15:51:37Z
  Finalizers:
    kubernetes-incubator/service-catalog
  Generate Name:     jenkins-ephemeral-
  Generation:        2
  Resource Version:  74
  Self Link:         /apis/servicecatalog.k8s.io/v1beta1/namespaces/my-project/serviceinstances/jenkins-ephemeral-qzsgl
  UID:               939c30fe-01e7-11e8-ac5f-0242ac110006
Spec:
  Cluster Service Class External Name:  jenkins-ephemeral
  Cluster Service Class Ref:
    Name:                              f53edfca-01e6-11e8-b3fc-c85b76aab8ec
  Cluster Service Plan External Name:  default
  Cluster Service Plan Ref:
    Name:       f53edfca-01e6-11e8-b3fc-c85b76aab8ec
  External ID:  2d970b76-1d46-4804-b425-491cdf1afd12
  Parameters From:
    Secret Key Ref:
      Key:          parameters
      Name:         jenkins-ephemeral-parameters84jei
  Update Requests:  0
  User Info:
    Extra:
      Scopes . Authorization . Openshift . Io:
        user:full
    Groups:
      system:authenticated:oauth
      system:authenticated
    UID:       
    Username:  dev
Status:
  Async Op In Progress:  false
  Conditions:
    Last Transition Time:  2018-01-25T15:55:34Z
    Message:               The instance was provisioned successfully
    Reason:                ProvisionedSuccessfully
    Status:                True
    Type:                  Ready
  Deprovision Status:      Required
  External Properties:
    Cluster Service Plan External ID:    f53edfca-01e6-11e8-b3fc-c85b76aab8ec
    Cluster Service Plan External Name:  default
    Parameter Checksum:                  463a0602cfe81b81cd7534488607b181e601c12ce8b87dddc59715eb8911cba5
    Parameters:
      ENABLE _ OAUTH:                  <redacted>
      JENKINS _ IMAGE _ STREAM _ TAG:  <redacted>
      JENKINS _ SERVICE _ NAME:        <redacted>
      JNLP _ SERVICE _ NAME:           <redacted>
      MEMORY _ LIMIT:                  <redacted>
      NAMESPACE:                       <redacted>
    User Info:
      Extra:
        Scopes . Authorization . Openshift . Io:
          user:full
      Groups:
        system:authenticated:oauth
        system:authenticated
      UID:                        
      Username:                   dev
  Orphan Mitigation In Progress:  false
  Reconciled Generation:          2
Events:
  Type     Reason                   Age   From                                Message
  ----     ------                   ----  ----                                -------
  Normal   Provisioning             7m    service-catalog-controller-manager  The instance is being provisioned asynchronously
  Warning  ProvisionCallFailed      7m    service-catalog-controller-manager  Error provisioning ServiceInstance of ClusterServiceClass (K8S: "f53edfca-01e6-11e8-b3fc-c85b76aab8ec" ExternalName: "jenkins-ephemeral") at ClusterServiceBroker "template-service-broker": Status: 409; ErrorMessage: <nil>; Description: <nil>; ResponseError: <nil>
  Normal   ProvisionedSuccessfully  3m    service-catalog-controller-manager  The instance was provisioned successfully

Comment 5 Jay Boyd 2018-02-08 20:19:27 UTC

still in progress upstream, adding link to PR:  https://github.com/kubernetes-incubator/service-catalog/pull/1708

Comment 6 Jay Boyd 2018-02-19 15:04:43 UTC

PR fixed upstream and re-vendored into OpenShift with https://github.com/openshift/origin/pull/18633

Comment 8 XiuJuan Wang 2018-02-22 07:02:00 UTC

After deprovision an unready serviceinstance, the serviceinstance always waits to provision succeed or failed at first, then will be deprovision successfully.

Mark this bug as verified

openshift v3.9.0-0.47.0
kubernetes v1.9.1+a0ce1bc657

Comment 11 errata-xmlrpc 2018-03-28 14:20:40 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0489