Bug 1534715

Summary:	multiple DELETEs to the same binding result in multiple APB runs when async is turned on
Product:	OpenShift Container Platform	Reporter:	Michael Hrivnak <mhrivnak>
Component:	Service Broker	Assignee:	Michael Hrivnak <mhrivnak>
Status:	CLOSED ERRATA	QA Contact:	Jian Zhang <jiazha>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	3.9.0	CC:	aos-bugs, chezhang, mhrivnak
Target Milestone:	---
Target Release:	3.9.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:	undefined	Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-03-28 14:19:05 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Michael Hrivnak 2018-01-15 19:15:40 UTC

Copied from upstream: https://github.com/openshift/ansible-service-broker/issues/640

Description of problem:
I initiated an unbind with the service catalog. The broker was configured for async bind and unbind. The catalog made multiple DELETE requests to the same endpoint:

172.17.0.2 - - [12/Jan/2018:21:17:47 +0000] "DELETE /ansible-service-broker/v2/service_instances/76593dc4-39d0-4561-a031-e0b4376e5532/service_bindings/7a477324-eb4c-4fb7-9aa6-64a52b2a688d?plan_id=7f4a5e35e4af2beb70076e72fab0b7ff&service_id=1dda1477cace09730bd8ed7a6505607e HTTP/1.1" 200 3
The second one returns an HTTP 500 response code:

172.17.0.2 - - [12/Jan/2018:21:17:57 +0000] "DELETE /ansible-service-broker/v2/service_instances/76593dc4-39d0-4561-a031-e0b4376e5532/service_bindings/7a477324-eb4c-4fb7-9aa6-64a52b2a688d?plan_id=7f4a5e35e4af2beb70076e72fab0b7ff&service_id=1dda1477cace09730bd8ed7a6505607e HTTP/1.1" 500 82
The broker logs make it clear that a second APB container was launched, and I was able to see both pods and both namespaces.

Version-Release number of selected component (if applicable):
current master branch of broker
openshift v3.9.0-alpha.1+96f2f77-79
kubernetes v1.9.0-beta1


How reproducible:
always

Steps to Reproduce:
1. configure the broker for async bind and unbind
2. create a binding
3. delete the binding

Actual results:
multiple APBs run and some of the DELETE requests get a 500 response

Expected results:
only one APB runs, no 500 responses.

Comment 1 Michael Hrivnak 2018-01-15 19:42:26 UTC

https://github.com/openshift/ansible-service-broker/pull/642

Comment 2 Michael Hrivnak 2018-01-16 22:14:45 UTC

There is a bit more work to do on this it turns out. I'll have an additional PR shortly.

Comment 3 Michael Hrivnak 2018-01-17 19:22:16 UTC

https://github.com/openshift/ansible-service-broker/pull/648

Comment 5 Jian Zhang 2018-01-26 07:40:08 UTC

Michael,

I encounter the same issue "launch_apb_on_bind is enabled, but accepts_incomplete is false, unbinding may fail" as I described comment3(step3) in the https://bugzilla.redhat.com/show_bug.cgi?id=1535955.
So, the question is how to enable the "accepts_incomplete"?

The setting below did not take effect. I'm sure I have restarted the controller-manager pod.
[root@host-172-16-120-121 ~]# oc edit daemonset controller-manager -n kube-service-catalog
      containers:
      - args:
        - controller-manager
        - -v
        - "5"
        - --leader-election-namespace
        - kube-service-catalog
        - --broker-relist-interval
        - 5m
        - --feature-gates
        - OriginatingIdentity=true
        - --feature-gates
        - AsyncBindingOperations=true

The warning info:
[2018-01-26T07:24:47.347Z] [DEBUG] - Dao::DeleteBindInstance -> [ b8c86072-7b96-4b08-a3fb-a37abe0b471b ]
10.129.0.8 - - [26/Jan/2018:07:24:42 +0000] "DELETE /ansible-service-broker/v2/service_instances/550c4447-7a8e-4206-8ad2-3cd9f3ef8947/service_bindings/b8c86072-7b96-4b08-a3fb-a37abe0b471b?plan_id=9783fc2e859f9179833a7dd003baa841&service_id=d5915e05b253df421efe6e41fb6a66ba HTTP/1.1" 200 3
[2018-01-26T07:24:47.388Z] [WARNING] - launch_apb_on_bind is enabled, but accepts_incomplete is false, unbinding may fail
[2018-01-26T07:24:47.393Z] [DEBUG] - Binding not found.
10.129.0.8 - - [26/Jan/2018:07:24:47 +0000] "DELETE /ansible-service-broker/v2/service_instances/550c4447-7a8e-4206-8ad2-3cd9f3ef8947/service_bindings/b8c86072-7b96-4b08-a3fb-a37abe0b471b?plan_id=9783fc2e859f9179833a7dd003baa841&service_id=d5915e05b253df421efe6e41fb6a66ba HTTP/1.1" 404 33
[2018-01-26T07:24:47.416Z] [WARNING] - launch_apb_on_bind is enabled, but accepts_incomplete is false, unbinding may fail
[2018-01-26T07:24:47.423Z] [DEBUG] - Binding not found.

The ASB/service-catalog image and version:
[root@host-172-16-120-121 ~]# docker run --rm --entrypoint=asbd registry.access.stage.redhat.com/openshift3/ose-ansible-service-broker:v3.9.0-0.23.0.1 --version
1.1.7
[root@host-172-16-120-121 ~]# docker run --rm --entrypoint=service-catalog registry.reg-aws.openshift.com:443/openshift3/ose-service-catalog:v3.9 --version
v0.1.3

Comment 7 Michael Hrivnak 2018-02-08 15:11:05 UTC

Hi Jian, just to follow up, I've been using curl to directly make requests to the broker that include the query parameter "accepts_incomplete". I know you've seen those commands in little bash scripts this week on other issues, so just let me know if you have any other questions about it.

Comment 10 errata-xmlrpc 2018-03-28 14:19:05 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0489