Bug 1725724
| Summary: | [Regression] Delete service instance provisioned by apb, pod and secrets can not be deleted | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Cuiping HUO <chuo> |
| Component: | Service Broker | Assignee: | Jesus M. Rodriguez <jesusr> |
| Status: | CLOSED ERRATA | QA Contact: | Zhang Cheng <chezhang> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 3.9.0 | CC: | aos-bugs, chuo |
| Target Milestone: | --- | Keywords: | Regression |
| Target Release: | 3.9.z | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | No Doc Update | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-07-24 08:00:13 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
The pod and secret can be deleted manually I did the following without any issues: 1) deployed OpenShift Cluster 3.9 $ oc version oc v3.9.0+191fece kubernetes v1.9.1+a0ce1bc657 features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://172.17.0.1:8443 openshift v3.9.0+71543b2-33 kubernetes v1.9.1+a0ce1bc657 2) used Ansible Service Broker version 1.1.20 $ oc logs asb-1-6ksq7 -c asb -n ansible-service-broker | grep Version: [2019-07-02T17:50:18.465Z] [INFO] - Ansible Service Broker Version: 1.1.20 3) Created a namespace called: blogpost 4) Deployed mysql (APB) with a binding so that the secret could be created 5) Provision finished, there was a secret, a pod running mysql, and a serviceinstance. [jesusr@transam linux{master}]$ oc describe secret dh-mysql-apb-t5trv-credentials-abnrq Name: dh-mysql-apb-t5trv-credentials-abnrq Namespace: blogpost Labels: <none> Annotations: <none> Type: Opaque Data ==== DB_PASSWORD: 8 bytes DB_PORT: 4 bytes DB_TYPE: 5 bytes DB_USER: 5 bytes DB_HOST: 42 bytes DB_NAME: 5 bytes [jesusr@transam linux{master}]$ oc get pods NAME READY STATUS RESTARTS AGE mysql-f76a00ae-1a89-4ed5-9dfc-281349c02ab8-1-9mf4l 1/1 Running 0 4m [jesusr@transam linux{master}]$ oc get serviceinstance NAME AGE dh-mysql-apb-t5trv 4m 6) I then went into the OpenShift console to my blogpost project 7) I selected the service and clicked the 3 stacked dots. I selected DELETE to delete the serviceinstance [2019-07-02T17:57:02.056Z] [INFO] - ASYNC deprovision in progress [2019-07-02T17:57:02.056Z] [NOTICE] - ============================================================ [2019-07-02T17:57:02.056Z] [NOTICE] - DEPROVISIONING [2019-07-02T17:57:02.056Z] [NOTICE] - ============================================================ [2019-07-02T17:57:02.056Z] [NOTICE] - ServiceInstance.Id: ddd528762894b277001df310a126d5ad [2019-07-02T17:57:02.056Z] [NOTICE] - ServiceInstance.Name: dh-mysql-apb [2019-07-02T17:57:02.056Z] [NOTICE] - ServiceInstance.Image: docker.io/ansibleplaybookbundle/mysql-apb:latest [2019-07-02T17:57:02.056Z] [NOTICE] - ServiceInstance.Description: Software Collections MySQL APB [2019-07-02T17:57:02.057Z] [NOTICE] - ============================================================ [2019-07-02T17:57:02.057Z] [DEBUG] - ExecutingApb: [2019-07-02T17:57:02.057Z] [DEBUG] - name:[ dh-mysql-apb ] [2019-07-02T17:57:02.057Z] [DEBUG] - image:[ docker.io/ansibleplaybookbundle/mysql-apb:latest ] [2019-07-02T17:57:02.057Z] [DEBUG] - action:[ deprovision ] [2019-07-02T17:57:02.057Z] [DEBUG] - pullPolicy:[ IfNotPresent ] [2019-07-02T17:57:02.057Z] [DEBUG] - role:[ edit ] [2019-07-02T17:57:02.057Z] [DEBUG] - No proxy env vars found to be configured. 172.17.0.3 - - [02/Jul/2019:17:57:02 +0000] "DELETE /ansible-service-broker/v2/service_instances/f76a00ae-1a89-4ed5-9dfc-281349c02ab8?accepts_incomplete=true&plan_id=583f053f9ba165125a16cf9aff768017&service_id=ddd528762894b277001df310a126 d5ad HTTP/1.1" 202 58 [2019-07-02T17:57:02.075Z] [DEBUG] - Trying to create apb sandbox: [ apb-80ba5973-bbae-459a-806a-48443a79647d ], with edit permissions in namespace dh-mysql-apb-depr-ntkvd [2019-07-02T17:57:02.075Z] [NOTICE] - Creating RoleBinding apb-80ba5973-bbae-459a-806a-48443a79647d [2019-07-02T17:57:02.091Z] [DEBUG] - Auto Escalate has been set to true, we are escalating permissions [2019-07-02T17:57:02.092Z] [DEBUG] - Dao::GetSvcInstJobsByState [2019-07-02T17:57:02.092Z] [DEBUG] - Dao::getJobsForSvcInst [2019-07-02T17:57:02.093Z] [DEBUG] - Successfully loaded [ 2 ] jobs objects from [ /state/f76a00ae-1a89-4ed5-9dfc-281349c02ab8/job ] [2019-07-02T17:57:02.093Z] [DEBUG] - Filtered on state: [ in progress ], returning 1 jobs [2019-07-02T17:57:02.093Z] [INFO] - Deprovision requested for instance f76a00ae-1a89-4ed5-9dfc-281349c02ab8, but job is already in progress 172.17.0.3 - - [02/Jul/2019:17:57:02 +0000] "DELETE /ansible-service-broker/v2/service_instances/f76a00ae-1a89-4ed5-9dfc-281349c02ab8?accepts_incomplete=true&plan_id=583f053f9ba165125a16cf9aff768017&service_id=ddd528762894b277001df310a126 d5ad HTTP/1.1" 202 3 [2019-07-02T17:57:02.263Z] [DEBUG] - Auto Escalate has been set to true, we are escalating permissions [2019-07-02T17:57:02.263Z] [DEBUG] - Dao::GetSvcInstJobsByState [2019-07-02T17:57:02.263Z] [DEBUG] - Dao::getJobsForSvcInst [2019-07-02T17:57:02.264Z] [DEBUG] - Successfully loaded [ 2 ] jobs objects from [ /state/f76a00ae-1a89-4ed5-9dfc-281349c02ab8/job ] [2019-07-02T17:57:02.264Z] [DEBUG] - Filtered on state: [ in progress ], returning 1 jobs [2019-07-02T17:57:02.264Z] [INFO] - Deprovision requested for instance f76a00ae-1a89-4ed5-9dfc-281349c02ab8, but job is already in progress 172.17.0.3 - - [02/Jul/2019:17:57:02 +0000] "DELETE /ansible-service-broker/v2/service_instances/f76a00ae-1a89-4ed5-9dfc-281349c02ab8?accepts_incomplete=true&plan_id=583f053f9ba165125a16cf9aff768017&service_id=ddd528762894b277001df310a126d5ad HTTP/1.1" 202 3 [2019-07-02T17:57:02.267Z] [NOTICE] - Creating RoleBinding apb-80ba5973-bbae-459a-806a-48443a79647d [2019-07-02T17:57:02.309Z] [INFO] - No network policies found. Assuming things are open, skip network policy creation [2019-07-02T17:57:02.309Z] [INFO] - Successfully created apb sandbox: [ apb-80ba5973-bbae-459a-806a-48443a79647d ], with edit permissions in namespace dh-mysql-apb-depr-ntkvd [2019-07-02T17:57:02.309Z] [INFO] - Running post create sandbox fuctions if defined. [2019-07-02T17:57:02.309Z] [NOTICE] - Creating pod "apb-80ba5973-bbae-459a-806a-48443a79647d" in the dh-mysql-apb-depr-ntkvd namespace [2019-07-02T17:57:02.32Z] [DEBUG] - Watching pod [ apb-80ba5973-bbae-459a-806a-48443a79647d ] in namespace [ dh-mysql-apb-depr-ntkvd ] for completion [2019-07-02T17:57:02.321Z] [INFO] - Watch pod [ apb-80ba5973-bbae-459a-806a-48443a79647d ] tick 1 [2019-07-02T17:57:02.324Z] [DEBUG] - Pod [ apb-80ba5973-bbae-459a-806a-48443a79647d ] Pending [2019-07-02T17:57:02.466Z] [DEBUG] - service_id: ddd528762894b277001df310a126d5ad [2019-07-02T17:57:02.466Z] [DEBUG] - plan_id: 583f053f9ba165125a16cf9aff768017 [2019-07-02T17:57:02.466Z] [DEBUG] - operation: 325e46ab-537a-4add-b98e-c3993d7adf90 [2019-07-02T17:57:02.467Z] [DEBUG] - state: in progress 172.17.0.3 - - [02/Jul/2019:17:57:02 +0000] "GET /ansible-service-broker/v2/service_instances/f76a00ae-1a89-4ed5-9dfc-281349c02ab8/last_operation?operation=325e46ab-537a-4add-b98e-c3993d7adf90&plan_id=583f053f9ba165125a16cf9aff768017&service_id=ddd528762894b277001df310a126d5ad HTTP/1.1" 200 29 [2019-07-02T17:57:03.079Z] [DEBUG] - service_id: ddd528762894b277001df310a126d5ad [2019-07-02T17:57:03.079Z] [DEBUG] - plan_id: 583f053f9ba165125a16cf9aff768017 [2019-07-02T17:57:03.079Z] [DEBUG] - operation: 325e46ab-537a-4add-b98e-c3993d7adf90 [2019-07-02T17:57:03.08Z] [DEBUG] - state: in progress 172.17.0.3 - - [02/Jul/2019:17:57:03 +0000] "GET /ansible-service-broker/v2/service_instances/f76a00ae-1a89-4ed5-9dfc-281349c02ab8/last_operation?operation=325e46ab-537a-4add-b98e-c3993d7adf90&plan_id=583f053f9ba165125a16cf9aff768017&service_id=ddd528762894b277001df310a126d5ad HTTP/1.1" 200 29 [2019-07-02T17:57:07.097Z] [DEBUG] - service_id: ddd528762894b277001df310a126d5ad [2019-07-02T17:57:07.097Z] [DEBUG] - plan_id: 583f053f9ba165125a16cf9aff768017 [2019-07-02T17:57:07.097Z] [DEBUG] - operation: 325e46ab-537a-4add-b98e-c3993d7adf90 [2019-07-02T17:57:07.098Z] [DEBUG] - state: in progress 172.17.0.3 - - [02/Jul/2019:17:57:07 +0000] "GET /ansible-service-broker/v2/service_instances/f76a00ae-1a89-4ed5-9dfc-281349c02ab8/last_operation?operation=325e46ab-537a-4add-b98e-c3993d7adf90&plan_id=583f053f9ba165125a16cf9aff768017&service_id=ddd528762894b277001df310a126d5ad HTTP/1.1" 200 29 [2019-07-02T17:57:07.324Z] [INFO] - Watch pod [ apb-80ba5973-bbae-459a-806a-48443a79647d ] tick 2 [2019-07-02T17:57:07.329Z] [DEBUG] - Pod [ apb-80ba5973-bbae-459a-806a-48443a79647d ] Running [2019-07-02T17:57:12.329Z] [INFO] - Watch pod [ apb-80ba5973-bbae-459a-806a-48443a79647d ] tick 3 [2019-07-02T17:57:12.34Z] [DEBUG] - Pod [ apb-80ba5973-bbae-459a-806a-48443a79647d ] completed [2019-07-02T17:57:12.34Z] [INFO] - Destroying APB sandbox... [2019-07-02T17:57:12.343Z] [DEBUG] - Deleting namespace dh-mysql-apb-depr-ntkvd [2019-07-02T17:57:12.349Z] [DEBUG] - Deleting rolebinding apb-80ba5973-bbae-459a-806a-48443a79647d, namespace dh-mysql-apb-depr-ntkvd [2019-07-02T17:57:12.361Z] [NOTICE] - Successfully deleted rolebinding apb-80ba5973-bbae-459a-806a-48443a79647d, namespace dh-mysql-apb-depr-ntkvd [2019-07-02T17:57:12.361Z] [DEBUG] - Deleting rolebinding apb-80ba5973-bbae-459a-806a-48443a79647d, namespace blogpost [2019-07-02T17:57:12.367Z] [NOTICE] - Successfully deleted rolebinding apb-80ba5973-bbae-459a-806a-48443a79647d, namespace blogpost [2019-07-02T17:57:12.37Z] [DEBUG] - Running post sandbox destroy hooks [2019-07-02T17:57:12.37Z] [DEBUG] - sending deprovision complete msg to channel [2019-07-02T17:57:12.37Z] [DEBUG] - received deprovision message from buffer [2019-07-02T17:57:12.375Z] [DEBUG] - Dao::DeleteServiceInstance -> [ f76a00ae-1a89-4ed5-9dfc-281349c02ab8 ] [2019-07-02T17:57:15.127Z] [DEBUG] - service_id: ddd528762894b277001df310a126d5ad [2019-07-02T17:57:15.127Z] [DEBUG] - plan_id: 583f053f9ba165125a16cf9aff768017 [2019-07-02T17:57:15.127Z] [DEBUG] - operation: 325e46ab-537a-4add-b98e-c3993d7adf90 [2019-07-02T17:57:15.128Z] [DEBUG] - state: succeeded The broker reported it being deprovisioned correctly. The serviceinstance, pods and secret are all gone as well: [jesusr@transam linux{master}]$ oc get serviceinstance No resources found. [jesusr@transam linux{master}]$ oc get pods No resources found. [jesusr@transam linux{master}]$ oc describe secret dh-mysql-apb-t5trv-credentials-abnrq Error from server (NotFound): secrets "dh-mysql-apb-t5trv-credentials-abnrq" not found Created a new project, deployed another mysql-apb
[jesusr@transam linux{master}]$ oc project mydeployment
Now using project "mydeployment" on server "https://172.17.0.1:8443".
[jesusr@transam linux{master}]$ oc get pods
NAME READY STATUS RESTARTS AGE
mysql-440298ac-af04-4437-859b-a46b56625564-1-446jf 1/1 Running 0 4m
[jesusr@transam linux{master}]$ oc get secrets | grep apb
dh-mysql-apb-mjzd4-credentials-eks90 Opaque 6 4m
dh-mysql-apb-parameters9zj8v Opaque 1 5m
[jesusr@transam linux{master}]$ oc get serviceinstance
NAME AGE
dh-mysql-apb-mjzd4 5m
This time I deleted the serviceinstance using oc
[jesusr@transam linux{master}]$ oc delete serviceinstance dh-mysql-apb-mjzd4
serviceinstance "dh-mysql-apb-mjzd4" deleted
When I do a `oc describe` you can see that the service bindings need to be removed before deleting the serviceinstance.
[jesusr@transam linux{master}]$ oc describe serviceinstance | grep -A10 Events:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Provisioning 10m service-catalog-controller-manager The instance is being provisioned asynchronously
Normal ProvisionedSuccessfully 9m service-catalog-controller-manager The instance was provisioned successfully
Warning DeprovisionBlockedByExistingCredentials 51s (x10 over 2m) service-catalog-controller-manager All associated ServiceBindings must be removed before this ServiceInstance can be deleted
Let's delete the servicebinding first:
[jesusr@transam linux{master}]$ oc get servicebindings
NAME AGE
dh-mysql-apb-mjzd4-v4llb 10m
[jesusr@transam linux{master}]$ oc delete servicebinding dh-mysql-apb-mjzd4-v4llb
servicebinding "dh-mysql-apb-mjzd4-v4llb" deleted
[jesusr@transam linux{master}]$ oc get servicebindings
No resources found.
Try to delete the serviceinstance again
[jesusr@transam linux{master}]$ oc delete serviceinstance dh-mysql-apb-mjzd4
serviceinstance "dh-mysql-apb-mjzd4" deleted
[jesusr@transam linux{master}]$ oc get serviceinstances
No resources found.
Everything looks to be deprovisioned correctly:
[jesusr@transam linux{master}]$ oc project
Using project "mydeployment" on server "https://172.17.0.1:8443".
[jesusr@transam linux{master}]$ oc project
Using project "mydeployment" on server "https://172.17.0.1:8443".
[jesusr@transam linux{master}]$ oc get secrets | grep apb
[jesusr@transam linux{master}]$
Can you give me the broker-config from the broker namespace? oc get configmap broker-config -n INSERTBROKERNAMESPACE -o yaml The cluster version you provided as oc v3.9.0+191fece seems far more behind what is tested as oc v3.9.85. Besides, oc v3.9.78 and former release of 3.9.z work well for this scenario. So I tried the new environment you left for me. It worked just fine from both the webconsole and from the CLI. [root@qe-chuo-master-etcd-1 ~]# oc get pods -n zeus-use-oc-delete-test No resources found. [root@qe-chuo-master-etcd-1 ~]# oc get pods -n zeus-test No resources found. [root@qe-chuo-master-etcd-1 ~]# oc get serviceinstances --all-namespaces No resources found. The one thing I saw that is important is that when you do oc delete serviceinstance rh-mysql-apb-fzb89, you will get a message like this: serviceinstance "rh-mysql-apb-fzb89" deleted But you MUST do an oc describe serviceinstance rh-mysql-apb-fzb89 to see if there are any servicebindings still present. Because if there are the serviceinstance will NOT be deleted despite the message you see above. [root@qe-chuo-master-etcd-1 ~]# oc describe serviceinstance rh-mysql-apb-fzb89 Name: rh-mysql-apb-fzb89 Namespace: zeus-use-oc-delete-test Labels: <none> Annotations: <none> API Version: servicecatalog.k8s.io/v1beta1 Kind: ServiceInstance .... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Provisioning 7m service-catalog-controller-manager The instance is being provisioned asynchronously Normal ProvisionedSuccessfully 7m service-catalog-controller-manager The instance was provisioned successfully Warning DeprovisionBlockedByExistingCredentials 5s (x10 over 16s) service-catalog-controller-manager All associated ServiceBindings must be removed before this ServiceInstance can be deleted As long as that servicebinding exists, NOTHING will go away. [root@qe-chuo-master-etcd-1 ~]# oc get servicebindings NAME AGE rh-mysql-apb-fzb89-nrq8d 8m [root@qe-chuo-master-etcd-1 ~]# oc delete servicebindings rh-mysql-apb-fzb89-nrq8d servicebinding "rh-mysql-apb-fzb89-nrq8d" deleted [root@qe-chuo-master-etcd-1 ~]# oc get servicebindings No resources found. [root@qe-chuo-master-etcd-1 ~]# oc describe serviceinstance rh-mysql-apb-fzb89 Name: rh-mysql-apb-fzb89 Namespace: zeus-use-oc-delete-test Labels: <none> Annotations: <none> API Version: servicecatalog.k8s.io/v1beta1 Kind: ServiceInstance ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Provisioning 8m service-catalog-controller-manager The instance is being provisioned asynchronously Normal ProvisionedSuccessfully 8m service-catalog-controller-manager The instance was provisioned successfully Warning DeprovisionBlockedByExistingCredentials 23s (x11 over 44s) service-catalog-controller-manager All associated ServiceBindings must be removed before this ServiceInstance can be deleted Normal Deprovisioning 2s service-catalog-controller-manager The instance is being deprovisioned asynchronously I then delete it again after deleting the servicebinding, after a bit you see it goes away. Also the pods are gone as well. [root@qe-chuo-master-etcd-1 ~]# oc delete serviceinstance rh-mysql-apb-fzb89 serviceinstance "rh-mysql-apb-fzb89" deleted [root@qe-chuo-master-etcd-1 ~]# oc get pods No resources found. [root@qe-chuo-master-etcd-1 ~]# oc get serviceinstance NAME AGE rh-mysql-apb-fzb89 8m [root@qe-chuo-master-etcd-1 ~]# oc describe serviceinstance rh-mysql-apb-fzb89 Error from server (NotFound): serviceinstances.servicecatalog.k8s.io "rh-mysql-apb-fzb89" not found [root@qe-chuo-master-etcd-1 ~]# oc get serviceinstance No resources found. Verified. oc version:3.9.85 asb:v3.9.85 [root@preserve-wmeng-39-master-etcd-1 ~]# oc get serviceinstance NAME AGE dh-mediawiki-apb-z6gjb 1h [root@preserve-wmeng-39-master-etcd-1 ~]# oc get po NAME READY STATUS RESTARTS AGE mediawiki123-1-26fbz 1/1 Running 0 1h [root@preserve-wmeng-39-master-etcd-1 ~]# oc get servicebinding No resources found. [root@preserve-wmeng-39-master-etcd-1 ~]# oc get secret NAME TYPE DATA AGE builder-dockercfg-p6nlc kubernetes.io/dockercfg 1 1h builder-token-g885r kubernetes.io/service-account-token 4 1h builder-token-p4cnw kubernetes.io/service-account-token 4 1h default-dockercfg-tmkfp kubernetes.io/dockercfg 1 1h default-token-5x97h kubernetes.io/service-account-token 4 1h default-token-rhkms kubernetes.io/service-account-token 4 1h deployer-dockercfg-np7pf kubernetes.io/dockercfg 1 1h deployer-token-8xcp4 kubernetes.io/service-account-token 4 1h deployer-token-njhnc kubernetes.io/service-account-token 4 1h dh-mediawiki-apb-parametersa687x Opaque 1 1h [root@preserve-wmeng-39-master-etcd-1 ~]# oc delete serviceinstance dh-mediawiki-apb-z6gjb serviceinstance "dh-mediawiki-apb-z6gjb" deleted [root@preserve-wmeng-39-master-etcd-1 ~]# oc describe serviceinstance dh-mediawiki-apb-z6gjb Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning ErrorWithParameters 1h (x2 over 1h) service-catalog-controller-manager failed to prepare parameters nil: secrets "dh-mediawiki-apb-parametersa687x" not found Normal Provisioning 1h service-catalog-controller-manager The instance is being provisioned asynchronously Normal ProvisionedSuccessfully 1h service-catalog-controller-manager The instance was provisioned successfully Normal Deprovisioning 1m service-catalog-controller-manager The instance is being deprovisioned asynchronously oc get serviceinstance No resources found. [root@preserve-wmeng-39-master-etcd-1 ~]# oc get po No resources found. [root@preserve-wmeng-39-master-etcd-1 ~]# oc get secret NAME TYPE DATA AGE builder-dockercfg-p6nlc kubernetes.io/dockercfg 1 1h builder-token-g885r kubernetes.io/service-account-token 4 1h builder-token-p4cnw kubernetes.io/service-account-token 4 1h default-dockercfg-tmkfp kubernetes.io/dockercfg 1 1h default-token-5x97h kubernetes.io/service-account-token 4 1h default-token-rhkms kubernetes.io/service-account-token 4 1h deployer-dockercfg-np7pf kubernetes.io/dockercfg 1 1h deployer-token-8xcp4 kubernetes.io/service-account-token 4 1h deployer-token-njhnc kubernetes.io/service-account-token 4 1h Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:1757 |
Description of problem: After delete service instance provisioned by apb, pod and secrets are not deleted Version-Release number of selected component (if applicable): ose-ansible-service-broker:v3.9.85 ASB version: 1.1.20 *-apb::v3.9.85 How reproducible: always Steps to Reproduce: 1. provision mysql-apb 2. delete serviceinstance Actual results: service instance can be deleted, but pod and secrets still exists #oc get serviceinstance -n zitang No resources found. # oc get po -n zitang NAME READY STATUS RESTARTS AGE mysql-5.7-dev-1-4wnn5 1/1 Running 0 2h #oc get secret -n zitang dh-mysql-apb-parametersglhc9 Opaque 1 2h no error logs in deprovision sandbox pod: oc get pod -n dh-mysql-apb-depr-xgkx5 NAME READY STATUS RESTARTS AGE apb-936a5c34-79e9-4e14-bb5c-72f6dd6bb7f4 0/1 Completed 0 2h [root@qe-chuo-master-etcd-1 ~]# oc logs -f apb-936a5c34-79e9-4e14-bb5c-72f6dd6bb7f4 -n dh-mysql-apb-depr-xgkx5 + [[ deprovision --extra-vars {"_apb_plan_id":"dev","_apb_service_class_id":"ddd528762894b277001df310a126d5ad","_apb_service_instance_id":"e6feea51-4f1a-46d2-ad2d-d59bdacac6de","cluster":"openshift","mysql_database":"devel","mysql_password":"dddd","mysql_user":"devel","mysql_version":"5.7","namespace":"zitang","service_name":"mysql"} == *\s\2\i\/\a\s\s\e\m\b\l\e* ]] + ACTION=deprovision + shift + playbooks=/opt/apb/actions + CREDS=/var/tmp/bind-creds + TEST_RESULT=/var/tmp/test-result + whoami + '[' -w /etc/passwd ']' ++ id -u + echo 'apb:x:1000510000:0:apb user:/opt/apb:/sbin/nologin' + set +x + [[ -e /opt/apb/actions/deprovision.yaml ]] + [[ -e /opt/apb/actions/deprovision.yml ]] + ANSIBLE_ROLES_PATH=/etc/ansible/roles:/opt/ansible/roles + ansible-playbook /opt/apb/actions/deprovision.yml --extra-vars '{"_apb_plan_id":"dev","_apb_service_class_id":"ddd528762894b277001df310a126d5ad","_apb_service_instance_id":"e6feea51-4f1a-46d2-ad2d-d59bdacac6de","cluster":"openshift","mysql_database":"devel","mysql_password":"dddd","mysql_user":"devel","mysql_version":"5.7","namespace":"zitang","service_name":"mysql"}' PLAY [mysql-apb playbook to deprovision the application] *********************** TASK [ansible.kubernetes-modules : Install latest openshift client] ************ skipping: [localhost] TASK [ansibleplaybookbundle.asb-modules : debug] ******************************* skipping: [localhost] TASK [rhscl-mysql-apb-openshift : Find pod we need to update] ****************** skipping: [localhost] TASK [rhscl-mysql-apb-openshift : Find dc we will clean up] ******************** skipping: [localhost] TASK [rhscl-mysql-apb-openshift : Prepare for downgrade] *********************** skipping: [localhost] TASK [rhscl-mysql-apb-openshift : Create db backup directory] ****************** skipping: [localhost] TASK [rhscl-mysql-apb-openshift : Backup source database] ********************** skipping: [localhost] TASK [rhscl-mysql-apb-openshift : rsync db backup to apb] ********************** skipping: [localhost] TASK [rhscl-mysql-apb-openshift : Set mysql service state to absent] *********** changed: [localhost] TASK [rhscl-mysql-apb-openshift : include_tasks] ******************************* included: /opt/ansible/roles/rhscl-mysql-apb-openshift/tasks/dev.yml for localhost TASK [rhscl-mysql-apb-openshift : set MySQL deployment with ephemeral storage to absent] *** changed: [localhost] TASK [rhscl-mysql-apb-openshift : include_tasks] ******************************* skipping: [localhost] TASK [rhscl-mysql-apb-openshift : Wait for mysql to come up] ******************* skipping: [localhost] TASK [rhscl-mysql-apb-openshift : Find pod we need to restore] ***************** skipping: [localhost] TASK [rhscl-mysql-apb-openshift : rsync db backup to new pod] ****************** skipping: [localhost] TASK [rhscl-mysql-apb-openshift : Restore database] **************************** skipping: [localhost] TASK [rhscl-mysql-apb-openshift : Run mysql_upgrade] *************************** skipping: [localhost] TASK [rhscl-mysql-apb-openshift : Remove old dc] ******************************* skipping: [localhost] TASK [rhscl-mysql-apb-openshift : ensure production volume is absent] ********** skipping: [localhost] => (item=5.6) skipping: [localhost] => (item=5.7) TASK [rhscl-mysql-apb-openshift : encode bind credentials] ********************* skipping: [localhost] PLAY RECAP ********************************************************************* localhost : ok=3 changed=2 unreachable=0 failed=0 + EXIT_CODE=0 + set +ex + '[' -f /var/tmp/test-result ']' + exit 0 Expected results: after service instance deletes，pod and secrets should also be deleted Additional info: asb:v3.9.78 work well for this scenario, but failed in asb::v3.9.85