Bug 1583495 - bundlebindings and binding credentials can not be deleted when asb lose connection to registry
Summary: bundlebindings and binding credentials can not be deleted when asb lose conne...
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Service Broker
Version: 3.10.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 3.11.0
Assignee: David Zager
QA Contact: Zihan Tang
Depends On:
TreeView+ depends on / blocked
Reported: 2018-05-29 06:46 UTC by Zihan Tang
Modified: 2018-10-11 07:20 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Last Closed: 2018-10-11 07:20:02 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:2652 0 None None None 2018-10-11 07:20:30 UTC

Description Zihan Tang 2018-05-29 06:46:19 UTC
Description of problem:
when asb lose connection to registry , the bundles will be deleted.  when unbind at this time, servicebinding will be deleted, but bundlebinding and bining credentials(in asb ns) can not be deleted.

[2018-05-29T05:58:22.607Z] [ERROR] - unable to get bundle from k8s api - bundles.automationbroker.io "d5915e05b253df421efe6e41fb6a66ba" not found
[2018-05-29T05:58:22.607Z] [INFO] - Could not find a service instance in dao - bundles.automationbroker.io "d5915e05b253df421efe6e41fb6a66ba" not found

Version-Release number of selected component (if applicable):
asb: 1.2.14

How reproducible:

Steps to Reproduce:
1. provision postgresql and create a binding.
2. delete postgresql bundle to simulate the scenario that asb losing connection to registry
3. unbind

Actual results:
servicebinding is deleted ,but bundlebinding and secret in openshift-ansible-service-broker namespace are not deleted.

[root@qe-zitang-r4-master-etcd-1 ~]# oc get servicebinding --all-namespaces
No resources found.
[root@qe-zitang-r4-master-etcd-1 ~]# oc get bundlebindings
NAME                                   AGE
3f9ffcf4-6305-11e8-9173-0a580a800005   14m

[root@qe-zitang-r4-master-etcd-1 ~]# oc get secret  | grep 3f9ffcf4-6305-11e8-9173-0a580a800005
3f9ffcf4-6305-11e8-9173-0a580a800005   Opaque                                1         14m

asb log:
[2018-05-29T05:58:22.607Z] [ERROR] - unable to get bundle from k8s api - bundles.automationbroker.io "d5915e05b253df421efe6e41fb6a66ba" not found
[2018-05-29T05:58:22.607Z] [INFO] - Could not find a service instance in dao - bundles.automationbroker.io "d5915e05b253df421efe6e41fb6a66ba" not found - - [29/May/2018:05:58:22 +0000] "DELETE /ansible-service-broker/v2/service_instances/26932224-6303-11e8-9173-0a580a800005/service_bindings/3f9ffcf4-6305-11e8-9173-0a580a800005?plan_id=9783fc2e859f9179833a7dd003baa841&service_id=d5915e05b253df421efe6e41fb6a66ba HTTP/1.1" 410 5

Expected results:
bundlebinding and secret can be delete.

Additional info:
in this sceanrio, v3.9 , the binding in etcd can be delete when unbind.

Comment 1 David Zager 2018-05-30 14:35:26 UTC
This is similar to an issue already filed upstream https://github.com/openshift/ansible-service-broker/issues/813

The broker's bootstrap procedure (at a high-level):

1. Remove "all" APB specs from our backing store (etcd or crds)
2. Go get all APBs based on our configuration
3. Write APBs specs back to backing  store (etcd or crds)

There is a window here when a request to provision|update|bind|unbind|deprovision from the servicecatalog would fail because the referenced APB (or bundle) would not exist.

Comment 2 David Zager 2018-05-30 14:49:32 UTC
Talked with shurley, the root cause of ansible-service-broker#813 is different. Reference https://github.com/openshift/ansible-service-broker/issues/970

Comment 3 David Zager 2018-08-03 20:33:35 UTC
The related issue https://github.com/openshift/ansible-service-broker/issues/970 is closed and those changes should be in the latest broker release "openshift-enterprise-asb-container-v3.11.0-"

Comment 4 Zihan Tang 2018-08-06 08:10:56 UTC
1. provision mariadb-apb and create a binding.
2. edit broker-config registy tag to an invaild tag to simulate asb lose connection to this registry, restart asb pod.

3. check bundles and clusterserviceclass:
# oc get bundles -o=custom-columns=NAME:.metadata.name,FQ\ NAME:.spec.fq_name,Delete:.spec.delete
NAME                               FQ NAME               Delete
0300d1ae1841c23a9df0a179ad0605fd   brew-mariadb-apb      true
0e5dbb6592fec99057f94fbb095ec558   brew-mediawiki-apb    true
48749329dd289591e11ba737f15fc71b   brew-postgresql-apb   true
bd8dff760b959264f3ab38d42ba5e7a8   brew-mysql-apb        true

# oc get clusterserviceclass -o=custom-columns=NAME:.metadata.name,EXTERNAL\ NAME:.spec.externalName,REMOVED:.status.removedFromBrokerCatalog 
NAME                               EXTERNAL NAME         REMOVED
0300d1ae1841c23a9df0a179ad0605fd   brew-mariadb-apb      false
0e5dbb6592fec99057f94fbb095ec558   brew-mediawiki-apb    false
48749329dd289591e11ba737f15fc71b   brew-postgresql-apb   false
bd8dff760b959264f3ab38d42ba5e7a8   brew-mysql-apb        false

3. delete servicebinding, check bundlebindings and secret in asb ns. they all deleted.

Comment 5 Zihan Tang 2018-08-06 09:19:14 UTC
In v3.11, bundles will not be directly deleted, I want to confirm that:
when ASB lose connection to the registry or find the bundle in the registry is deleted, ASB will:
1. if first time syncing registry, mark the bundle as deleted.
2. if second time syncing (the bundle has already been marked as deleted), delete the bundle directly.

are the above scenarios right as designed ?

Comment 6 David Zager 2018-08-06 16:51:30 UTC
The broker, by default, runs the bootstrap procedure on startup and simply adds all of the APBs it finds into it's datastore. On subsequent calls to bootstrap:

1. If an APB, for which I have a bundle, is not found in the registry we mark it for deletion
2. If an APB, for which I have a bundle marked for deletion, is not found we remove the bundle

I believe this agrees with what you are suggesting. I simply want to be careful with "first time" and "second time". Hope this helps.

Comment 7 Zihan Tang 2018-08-07 02:02:31 UTC
David, thanks for your clarification.

Comment 9 errata-xmlrpc 2018-10-11 07:20:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.