Bug 1577810 - deprovision the serviceinstance, the pod is not deleted when the bundle is deleted from registry
Summary: deprovision the serviceinstance, the pod is not deleted when the bundle is de...
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Service Broker
Version: 3.10.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 3.11.0
Assignee: Jesus M. Rodriguez
QA Contact: Zihan Tang
Depends On:
TreeView+ depends on / blocked
Reported: 2018-05-14 07:58 UTC by Zihan Tang
Modified: 2018-10-11 07:20 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Broker's reconciliation feature would delete its image references before getting the updated information from the registry. Consequence: there would be a period of time before the records showed up in the broker's data store while other jobs were still running. Fix: the reconciliation feature was redesigned to do an in place update for items that have changed. For items that were removed from the registry, the broker will now delete only those that have not been already provisioned. It will also mark those items for deletion, which filters them out of the UI preventing future provisions of those items. Result: The broker is more robust in its reconciliation feature thereby making the provision and deprovision more resilient to registry changes.
Clone Of:
Last Closed: 2018-10-11 07:20:00 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:2652 0 None None None 2018-10-11 07:20:21 UTC

Description Zihan Tang 2018-05-14 07:58:41 UTC
Description of problem:
When asb lose connection to registry, some apb are lost , and the clusterserviceclass 'removedFromBrokerCatalog' is set to true by service catalog, 
deprovision the serviceinstance of the lost apb, the instance is deleted immediately, but the pod is not deleted.

Version-Release number of selected component (if applicable):
asb : 1.2.10
service-catalog: v3.10.0-0.38.0;Upstream:v0.1.16
How reproducible:

Steps to Reproduce:
1. provision mariadb-apb
2. delete the postgresql-apb bundle to simulate the scenario of asb losing connection to registry.
[root@qe-zitang-gcemaster-etcd-1 ~]# oc get bundles -o=custom-columns=NAME:.metadata.name,FQ\ NAME:.spec.fq_name   -n openshift-ansible-service-broker 
NAME                               FQ NAME
1dda1477cace09730bd8ed7a6505607e   dh-postgresql-apb
67042296c7c95e84142f21f58da2ebfe   dh-mariadb-apb
ddd528762894b277001df310a126d5ad   dh-mysql-apb
f6c4486b7fb0cdac4b58e193607f7011   dh-mediawiki-apb

[root@qe-zitang-gcemaster-etcd-1 ~]# oc delete bundle 67042296c7c95e84142f21f58da2ebfe -n openshift-ansible-service-broker 
bundle.automationbroker.io "67042296c7c95e84142f21f58da2ebfe" deleted

3.  edit ansible-service-broker  'relistRequests' to sync with asb, the 'removedFromBrokerCatalog ' will be set to true
[root@qe-zitang-gcemaster-etcd-1 ~]# oc get clusterserviceclass -o=custom-columns=NAME:.metadata.name,EXTERNAL\ NAME:.spec.externalName,REMOVED:.status.removedFromBrokerCatalog 
NAME                               EXTERNAL NAME       REMOVED
1dda1477cace09730bd8ed7a6505607e   dh-postgresql-apb   false
67042296c7c95e84142f21f58da2ebfe   dh-mariadb-apb      true
ddd528762894b277001df310a126d5ad   dh-mysql-apb        false
f6c4486b7fb0cdac4b58e193607f7011   dh-mediawiki-apb    false

4. deprovision the serviceinstance

Actual results:
4. the serviceinstance can not be deprovisoin completely 
ASB Log: 
[2018-05-14T07:01:40.873Z] [DEBUG] - get service instance: 8bc2fbc7-5742-11e8-be4e-0a580a800003
[2018-05-14T07:01:40.877Z] [DEBUG] - get spec: 67042296c7c95e84142f21f58da2ebfe
[2018-05-14T07:01:40.879Z] [ERROR] - unable to get bundle from k8s api - bundles.automationbroker.io "67042296c7c95e84142f21f58da2ebfe" not found

serviceinstance is deleted.
[root@qe-zitang-gcemaster-etcd-1 ~]# oc get serviceinstance -n maria
No resources found.

pod not deleted
[root@qe-zitang-gcemaster-etcd-1 ~]# oc get pod -n maria
NAME                             READY     STATUS    RESTARTS   AGE
rhscl-mariadb-10.2-dev-1-cvntt   1/1       Running   0          34m

Expected results:
deprovision succeed.

Additional info:

Comment 1 John Matthews 2018-05-24 19:22:53 UTC
Realigning to 3.11 as this issue requires a larger amount of work than we are comfortable with at end of 3.10.

Likely fix will rework how we handle our reconcile logic.

Comment 2 Zihan Tang 2018-05-28 02:44:22 UTC
I tried this scenario in v3.9 
the serviceinstance and pod can both be deleted succeed.

Comment 3 Zihan Tang 2018-08-09 07:29:56 UTC
this scenario is the same with bug 1583495, update the title.

Comment 4 Zihan Tang 2018-08-09 08:09:12 UTC
The related issue https://github.com/openshift/ansible-service-broker/issues/970 is closed and those changes should be in the latest broker release "openshift-enterprise-asb-container-v3.11.0-"

Checking it's fixed 
asb: 1.3.7

Verify step:
1. provision mariadb-apb 
2. edit broker-config registy tag to an invaild tag to simulate asb lose connection to this registry, restart asb pod.

3. check bundles and clusterserviceclass:
# oc get bundles -o=custom-columns=NAME:.metadata.name,FQ\ NAME:.spec.fq_name,Delete:.spec.delete
NAME                               FQ NAME               Delete
0300d1ae1841c23a9df0a179ad0605fd   brew-mariadb-apb      true
0e5dbb6592fec99057f94fbb095ec558   brew-mediawiki-apb    true
48749329dd289591e11ba737f15fc71b   brew-postgresql-apb   true
bd8dff760b959264f3ab38d42ba5e7a8   brew-mysql-apb        true

# oc get clusterserviceclass -o=custom-columns=NAME:.metadata.name,EXTERNAL\ NAME:.spec.externalName,REMOVED:.status.removedFromBrokerCatalog 
NAME                               EXTERNAL NAME         REMOVED
0300d1ae1841c23a9df0a179ad0605fd   brew-mariadb-apb      false
0e5dbb6592fec99057f94fbb095ec558   brew-mediawiki-apb    false
48749329dd289591e11ba737f15fc71b   brew-postgresql-apb   false
bd8dff760b959264f3ab38d42ba5e7a8   brew-mysql-apb        false

3. delete serviceinstance , check the pod , pod is deleted.

Comment 6 errata-xmlrpc 2018-10-11 07:20:00 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.