Bug 1534715 - multiple DELETEs to the same binding result in multiple APB runs when async is turned on
Summary: multiple DELETEs to the same binding result in multiple APB runs when async i...
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Service Broker   
(Show other bugs)
Version: 3.9.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 3.9.0
Assignee: Michael Hrivnak
QA Contact: Jian Zhang
Depends On:
TreeView+ depends on / blocked
Reported: 2018-01-15 19:15 UTC by Michael Hrivnak
Modified: 2018-03-28 14:19 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2018-03-28 14:19:05 UTC
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

External Trackers
Tracker ID Priority Status Summary Last Updated
Github openshift ansible-service-broker issues 640 None None None 2018-01-15 19:16 UTC
Red Hat Product Errata RHBA-2018:0489 None None None 2018-03-28 14:19 UTC

Description Michael Hrivnak 2018-01-15 19:15:40 UTC
Copied from upstream: https://github.com/openshift/ansible-service-broker/issues/640

Description of problem:
I initiated an unbind with the service catalog. The broker was configured for async bind and unbind. The catalog made multiple DELETE requests to the same endpoint: - - [12/Jan/2018:21:17:47 +0000] "DELETE /ansible-service-broker/v2/service_instances/76593dc4-39d0-4561-a031-e0b4376e5532/service_bindings/7a477324-eb4c-4fb7-9aa6-64a52b2a688d?plan_id=7f4a5e35e4af2beb70076e72fab0b7ff&service_id=1dda1477cace09730bd8ed7a6505607e HTTP/1.1" 200 3
The second one returns an HTTP 500 response code: - - [12/Jan/2018:21:17:57 +0000] "DELETE /ansible-service-broker/v2/service_instances/76593dc4-39d0-4561-a031-e0b4376e5532/service_bindings/7a477324-eb4c-4fb7-9aa6-64a52b2a688d?plan_id=7f4a5e35e4af2beb70076e72fab0b7ff&service_id=1dda1477cace09730bd8ed7a6505607e HTTP/1.1" 500 82
The broker logs make it clear that a second APB container was launched, and I was able to see both pods and both namespaces.

Version-Release number of selected component (if applicable):
current master branch of broker
openshift v3.9.0-alpha.1+96f2f77-79
kubernetes v1.9.0-beta1

How reproducible:

Steps to Reproduce:
1. configure the broker for async bind and unbind
2. create a binding
3. delete the binding

Actual results:
multiple APBs run and some of the DELETE requests get a 500 response

Expected results:
only one APB runs, no 500 responses.

Comment 2 Michael Hrivnak 2018-01-16 22:14:45 UTC
There is a bit more work to do on this it turns out. I'll have an additional PR shortly.

Comment 5 Jian Zhang 2018-01-26 07:40:08 UTC

I encounter the same issue "launch_apb_on_bind is enabled, but accepts_incomplete is false, unbinding may fail" as I described comment3(step3) in the https://bugzilla.redhat.com/show_bug.cgi?id=1535955.
So, the question is how to enable the "accepts_incomplete"?

The setting below did not take effect. I'm sure I have restarted the controller-manager pod.
[root@host-172-16-120-121 ~]# oc edit daemonset controller-manager -n kube-service-catalog
      - args:
        - controller-manager
        - -v
        - "5"
        - --leader-election-namespace
        - kube-service-catalog
        - --broker-relist-interval
        - 5m
        - --feature-gates
        - OriginatingIdentity=true
        - --feature-gates
        - AsyncBindingOperations=true

The warning info:
[2018-01-26T07:24:47.347Z] [DEBUG] - Dao::DeleteBindInstance -> [ b8c86072-7b96-4b08-a3fb-a37abe0b471b ] - - [26/Jan/2018:07:24:42 +0000] "DELETE /ansible-service-broker/v2/service_instances/550c4447-7a8e-4206-8ad2-3cd9f3ef8947/service_bindings/b8c86072-7b96-4b08-a3fb-a37abe0b471b?plan_id=9783fc2e859f9179833a7dd003baa841&service_id=d5915e05b253df421efe6e41fb6a66ba HTTP/1.1" 200 3
[2018-01-26T07:24:47.388Z] [WARNING] - launch_apb_on_bind is enabled, but accepts_incomplete is false, unbinding may fail
[2018-01-26T07:24:47.393Z] [DEBUG] - Binding not found. - - [26/Jan/2018:07:24:47 +0000] "DELETE /ansible-service-broker/v2/service_instances/550c4447-7a8e-4206-8ad2-3cd9f3ef8947/service_bindings/b8c86072-7b96-4b08-a3fb-a37abe0b471b?plan_id=9783fc2e859f9179833a7dd003baa841&service_id=d5915e05b253df421efe6e41fb6a66ba HTTP/1.1" 404 33
[2018-01-26T07:24:47.416Z] [WARNING] - launch_apb_on_bind is enabled, but accepts_incomplete is false, unbinding may fail
[2018-01-26T07:24:47.423Z] [DEBUG] - Binding not found.

The ASB/service-catalog image and version:
[root@host-172-16-120-121 ~]# docker run --rm --entrypoint=asbd registry.access.stage.redhat.com/openshift3/ose-ansible-service-broker:v3.9.0- --version
[root@host-172-16-120-121 ~]# docker run --rm --entrypoint=service-catalog registry.reg-aws.openshift.com:443/openshift3/ose-service-catalog:v3.9 --version

Comment 7 Michael Hrivnak 2018-02-08 15:11:05 UTC
Hi Jian, just to follow up, I've been using curl to directly make requests to the broker that include the query parameter "accepts_incomplete". I know you've seen those commands in little bash scripts this week on other issues, so just let me know if you have any other questions about it.

Comment 10 errata-xmlrpc 2018-03-28 14:19:05 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.