Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.
Bug 1534715 - multiple DELETEs to the same binding result in multiple APB runs when async is turned on
multiple DELETEs to the same binding result in multiple APB runs when async i...
Status: CLOSED ERRATA
Product: OpenShift Container Platform
Classification: Red Hat
Component: Service Broker (Show other bugs)
3.9.0
Unspecified Unspecified
unspecified Severity unspecified
: ---
: 3.9.0
Assigned To: Michael Hrivnak
Jian Zhang
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2018-01-15 14:15 EST by Michael Hrivnak
Modified: 2018-03-28 10:19 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: No Doc Update
Doc Text:
undefined
Story Points: ---
Clone Of:
Environment:
Last Closed: 2018-03-28 10:19:05 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Github openshift/ansible-service-broker/issues/640 None None None 2018-01-15 14:16 EST
Red Hat Product Errata RHBA-2018:0489 None None None 2018-03-28 10:19 EDT

  None (edit)
Description Michael Hrivnak 2018-01-15 14:15:40 EST
Copied from upstream: https://github.com/openshift/ansible-service-broker/issues/640

Description of problem:
I initiated an unbind with the service catalog. The broker was configured for async bind and unbind. The catalog made multiple DELETE requests to the same endpoint:

172.17.0.2 - - [12/Jan/2018:21:17:47 +0000] "DELETE /ansible-service-broker/v2/service_instances/76593dc4-39d0-4561-a031-e0b4376e5532/service_bindings/7a477324-eb4c-4fb7-9aa6-64a52b2a688d?plan_id=7f4a5e35e4af2beb70076e72fab0b7ff&service_id=1dda1477cace09730bd8ed7a6505607e HTTP/1.1" 200 3
The second one returns an HTTP 500 response code:

172.17.0.2 - - [12/Jan/2018:21:17:57 +0000] "DELETE /ansible-service-broker/v2/service_instances/76593dc4-39d0-4561-a031-e0b4376e5532/service_bindings/7a477324-eb4c-4fb7-9aa6-64a52b2a688d?plan_id=7f4a5e35e4af2beb70076e72fab0b7ff&service_id=1dda1477cace09730bd8ed7a6505607e HTTP/1.1" 500 82
The broker logs make it clear that a second APB container was launched, and I was able to see both pods and both namespaces.

Version-Release number of selected component (if applicable):
current master branch of broker
openshift v3.9.0-alpha.1+96f2f77-79
kubernetes v1.9.0-beta1


How reproducible:
always

Steps to Reproduce:
1. configure the broker for async bind and unbind
2. create a binding
3. delete the binding

Actual results:
multiple APBs run and some of the DELETE requests get a 500 response

Expected results:
only one APB runs, no 500 responses.
Comment 2 Michael Hrivnak 2018-01-16 17:14:45 EST
There is a bit more work to do on this it turns out. I'll have an additional PR shortly.
Comment 5 Jian Zhang 2018-01-26 02:40:08 EST
Michael,

I encounter the same issue "launch_apb_on_bind is enabled, but accepts_incomplete is false, unbinding may fail" as I described comment3(step3) in the https://bugzilla.redhat.com/show_bug.cgi?id=1535955.
So, the question is how to enable the "accepts_incomplete"?

The setting below did not take effect. I'm sure I have restarted the controller-manager pod.
[root@host-172-16-120-121 ~]# oc edit daemonset controller-manager -n kube-service-catalog
      containers:
      - args:
        - controller-manager
        - -v
        - "5"
        - --leader-election-namespace
        - kube-service-catalog
        - --broker-relist-interval
        - 5m
        - --feature-gates
        - OriginatingIdentity=true
        - --feature-gates
        - AsyncBindingOperations=true

The warning info:
[2018-01-26T07:24:47.347Z] [DEBUG] - Dao::DeleteBindInstance -> [ b8c86072-7b96-4b08-a3fb-a37abe0b471b ]
10.129.0.8 - - [26/Jan/2018:07:24:42 +0000] "DELETE /ansible-service-broker/v2/service_instances/550c4447-7a8e-4206-8ad2-3cd9f3ef8947/service_bindings/b8c86072-7b96-4b08-a3fb-a37abe0b471b?plan_id=9783fc2e859f9179833a7dd003baa841&service_id=d5915e05b253df421efe6e41fb6a66ba HTTP/1.1" 200 3
[2018-01-26T07:24:47.388Z] [WARNING] - launch_apb_on_bind is enabled, but accepts_incomplete is false, unbinding may fail
[2018-01-26T07:24:47.393Z] [DEBUG] - Binding not found.
10.129.0.8 - - [26/Jan/2018:07:24:47 +0000] "DELETE /ansible-service-broker/v2/service_instances/550c4447-7a8e-4206-8ad2-3cd9f3ef8947/service_bindings/b8c86072-7b96-4b08-a3fb-a37abe0b471b?plan_id=9783fc2e859f9179833a7dd003baa841&service_id=d5915e05b253df421efe6e41fb6a66ba HTTP/1.1" 404 33
[2018-01-26T07:24:47.416Z] [WARNING] - launch_apb_on_bind is enabled, but accepts_incomplete is false, unbinding may fail
[2018-01-26T07:24:47.423Z] [DEBUG] - Binding not found.

The ASB/service-catalog image and version:
[root@host-172-16-120-121 ~]# docker run --rm --entrypoint=asbd registry.access.stage.redhat.com/openshift3/ose-ansible-service-broker:v3.9.0-0.23.0.1 --version
1.1.7
[root@host-172-16-120-121 ~]# docker run --rm --entrypoint=service-catalog registry.reg-aws.openshift.com:443/openshift3/ose-service-catalog:v3.9 --version
v0.1.3
Comment 7 Michael Hrivnak 2018-02-08 10:11:05 EST
Hi Jian, just to follow up, I've been using curl to directly make requests to the broker that include the query parameter "accepts_incomplete". I know you've seen those commands in little bash scripts this week on other issues, so just let me know if you have any other questions about it.
Comment 10 errata-xmlrpc 2018-03-28 10:19:05 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0489

Note You need to log in before you can comment on or make changes to this bug.