Bug 1536659 - async bind returns 201 instead of 202
Summary: async bind returns 201 instead of 202
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Service Broker
Version: 3.9.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: 3.9.0
Assignee: Michael Hrivnak
QA Contact: Jian Zhang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-01-19 20:52 UTC by Michael Hrivnak
Modified: 2018-03-28 14:22 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
undefined
Clone Of:
Environment:
Last Closed: 2018-03-28 14:21:18 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
script to create a binding (723 bytes, application/x-shellscript)
2018-01-19 20:52 UTC, Michael Hrivnak
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github openshift ansible-service-broker issues 667 0 None None None 2018-01-19 20:52:42 UTC
Red Hat Product Errata RHBA-2018:0489 0 None None None 2018-03-28 14:22:06 UTC

Description Michael Hrivnak 2018-01-19 20:52:43 UTC
Created attachment 1383609 [details]
script to create a binding

Reported upstream: https://github.com/openshift/ansible-service-broker/issues/667

Description of problem:
I used the broker API to create a binding by making this request:

PUT /ansible-service-broker/v2/service_instances/c4aa708d-8c64-4c37-a131-557a4a155148/service_bindings/745ddeb7-395e-412e-b4ed-8a0b52b11ac0?accepts_incomplete=true

The response code was 201, but should have been 202 according to the OSB API spec proposed changes around async bind/unbind.

HTTP/1.1 201 Created
Content-Type: application/json
Date: Fri, 19 Jan 2018 19:42:53 GMT
Content-Length: 58
Set-Cookie: d05351dd24520dce41f310fcfc112523=1591f023a03f3453b1d2364f14c83799; path=/; HttpOnly; Secure

{
  "operation": "338c289e-8792-4ab4-95fa-4fd9b4752cf0"
}

Version-Release number of selected component (if applicable):
master

How reproducible:
always

Steps to Reproduce:
1. configure the broker to run APBs on bind/unbind and auto-escalate
2. Run the attached script, supplying a valid instance ID as the only argument

Actual results:
201 response code

Expected results:
202 response code

Comment 9 Michael Hrivnak 2018-02-08 15:05:41 UTC
Apologies. I left out the detail that you need to use an APB that supports async bind and unbind. Currently there's just an experimental one in my repo: https://hub.docker.com/r/mhrivnak/postgresql-apb/

Comment 10 Jian Zhang 2018-02-09 03:25:19 UTC
Michael,

I used the example APB you're provided, configure the below:

      - type: dockerhub
        name: dh
        url:  https://registry.hub.docker.com
        org:  mhrivnak
        tag:  latest
        white_list:
          - ".*-apb$"

The ASB version: 1.1.9

Test steps as the above, the response of async binding is 202, looks good to me.
[root@localhost jzhang]# ./bind.sh 
binding ID: 40a35cdd-5f51-4559-ab79-6a685709cd96
HTTP/1.1 202 Accepted
Content-Type: application/json
Date: Fri, 09 Feb 2018 02:58:37 GMT
Content-Length: 58
Set-Cookie: 89a6d633054ded194d4e1360cdc1fbef=b37e683520b874bc37db0badfda2d6dd; path=/; HttpOnly; Secure

{
  "operation": "b4f065ed-2010-47b0-8d61-3bc438f63ac3"
}

Check the state:
[root@localhost jzhang]# ./last_operation.sh b4f065ed-2010-47b0-8d61-3bc438f63ac3
{
  "state": "succeeded"
}

[root@localhost jzhang]# cat last_operation.sh 
OPERATION_ID=$1
INSTANCE_ID="d10a0ca6-c609-4d29-ab9d-b25c3b8c332c"
PLAN_UUID="7f4a5e35e4af2beb70076e72fab0b7ff"
SERVICE_UUID="dh-postgresql-apb-b7gbr"
IDENTITY=$(echo -n '{"groups":["system:authenticated:oauth","system:authenticated"],"scopes.authorization.openshift.io":["user:full"],"uid":"","username":"jiazha"}'|base64 -w 0)

curl \
    -k \
    -X GET \
    -H "Authorization: bearer $(oc whoami -t)" \
    -H "Content-type: application/json" \
    -H "Accept: application/json" \
    -H "X-Broker-API-Originating-Identity: kubernetes $IDENTITY" \
    "https://asb-1338-openshift-ansible-service-broker.apps.0207-nvc.qe.rhcloud.com/ansible-service-broker/v2/service_instances/$INSTANCE_ID/last_operation?operation=$OPERATION_ID&service_id=$SERVICE_UUID&plan_id=$PLAN_UUID"

But, I didn't find the created binding although it returned the "succeed" status. Why? I think the users should get the servicebindings.
[root@host-172-16-120-54 ~]# oc get servicebinding --all-namespaces
No resources found.

And, according to comment 9, I want to know whether our released APBs(such as PostgreSQL, MariaDB, MySQL, Mediawiki) will support the ASYNC in 3.9?

Comment 11 Jian Zhang 2018-02-09 07:00:15 UTC
Michael,

Another thing is your example APB failed when running async unbind. Details as below:
For creating binding steps, please refer to the above.

Run the unbind action by using below script:
[root@localhost jzhang]# cat unbind.sh 
INSTANCE_ID="0a0b861b-3045-4d98-b137-71d1db572d2e"
BINDING_ID=$1
PLAN_UUID="7f4a5e35e4af2beb70076e72fab0b7ff"
IDENTITY=$(echo -n '{"groups":["system:authenticated:oauth","system:authenticated"],"scopes.authorization.openshift.io":["user:full"],"uid":"","username":"jiazha"}'|base64 -w 0)

curl \
    -k -i \
    -X DELETE \
    -H "Authorization: bearer $(oc whoami -t)" \
    -H "Content-type: application/json" \
    -H "Accept: application/json" \
    -H "X-Broker-API-Originating-Identity: kubernetes $IDENTITY" \
    "https://asb-1338-openshift-ansible-service-broker.apps.0207-nvc.qe.rhcloud.com/ansible-service-broker/v2/service_instances/$INSTANCE_ID/service_bindings/$BINDING_ID?plan_id=$PLAN_UUID"
[root@localhost jzhang]# ./unbind.sh ef42a7b3-1d74-4e43-9447-a0af58719c55
HTTP/1.1 500 Internal Server Error
Content-Type: application/json
Date: Fri, 09 Feb 2018 06:46:03 GMT
Content-Length: 98
Set-Cookie: 89a6d633054ded194d4e1360cdc1fbef=45b7fdf950d87e8c3a786f6dc8075bc5; path=/; HttpOnly; Secure

{
  "description": "Pod [ apb-2a8a1fdb-5324-4459-846a-ff3fcf218787 ] failed with exit code [2]"
}

The ASB related logs:
[2018-02-09T06:46:03.652Z] [ERROR] - Unbind action failed - Pod [ apb-2a8a1fdb-5324-4459-846a-ff3fcf218787 ] failed with exit code [2]
[2018-02-09T06:46:03.652Z] [INFO] - Destroying APB sandbox...
[2018-02-09T06:46:03.655Z] [DEBUG] - Keeping namespace alive due to configuration
[2018-02-09T06:46:03.655Z] [DEBUG] - Deleting rolebinding apb-2a8a1fdb-5324-4459-846a-ff3fcf218787, namespace dh-postgresql-apb-unbi-qsrnn
[2018-02-09T06:46:03.658Z] [NOTICE] - Successfully deleted rolebinding apb-2a8a1fdb-5324-4459-846a-ff3fcf218787, namespace dh-postgresql-apb-unbi-qsrnn
[2018-02-09T06:46:03.658Z] [DEBUG] - Deleting rolebinding apb-2a8a1fdb-5324-4459-846a-ff3fcf218787, namespace async3
[2018-02-09T06:46:03.664Z] [NOTICE] - Successfully deleted rolebinding apb-2a8a1fdb-5324-4459-846a-ff3fcf218787, namespace async3
[2018-02-09T06:46:03.664Z] [DEBUG] - Deleting network policy for pod: apb-2a8a1fdb-5324-4459-846a-ff3fcf218787 to grant network access to ns: async3
[2018-02-09T06:46:03.671Z] [DEBUG] - Successfully deleted network policy for pod: apb-2a8a1fdb-5324-4459-846a-ff3fcf218787 to grant network access to ns: async3
[2018-02-09T06:46:03.671Z] [DEBUG] - Running post sandbox destroy hooks
[2018-02-09T06:46:03.671Z] [ERROR] - Unknown error: Pod [ apb-2a8a1fdb-5324-4459-846a-ff3fcf218787 ] failed with exit code [2]
10.128.0.1 - - [09/Feb/2018:06:45:53 +0000] "DELETE /ansible-service-broker/v2/service_instances/0a0b861b-3045-4d98-b137-71d1db572d2e/service_bindings/ef42a7b3-1d74-4e43-9447-a0af58719c55?plan_id=7f4a5e35e4af2beb70076e72fab0b7ff HTTP/1.1" 500 98

I intended to new a bug for tracking this, but as you know, the above example APB is NOT our released version, so, how do you think?

Comment 12 Michael Hrivnak 2018-02-09 15:31:05 UTC
(In reply to Jian Zhang from comment #10)
> 
> But, I didn't find the created binding although it returned the "succeed"
> status. Why? I think the users should get the servicebindings.
> [root@host-172-16-120-54 ~]# oc get servicebinding --all-namespaces
> No resources found.

That's expected. According to the OSB API spec, the "Platform", which in our case is the console, is responsible for managing its records of service instances and bindings. While our broker does happen to manage resources within the cluster for the sake of provisioning services, that should not be confused with managing state on behalf of the "Platform". It took me a while for that to sink in. This Terminology section may help clear things up:

https://github.com/openservicebrokerapi/servicebroker/blob/v2.13/spec.md#terminology

> 
> And, according to comment 9, I want to know whether our released APBs(such
> as PostgreSQL, MariaDB, MySQL, Mediawiki) will support the ASYNC in 3.9?

My understanding is that we are not planning to have async bind and unbind officially supported by an APB in 3.9. But, we are looking for a way to make that feature available for testing and experimentation, perhaps by having an unofficial or upstream APB available elsewhere. Having an image in my personal docker hub org is clearly not the best, so suggestions are welcome!

Comment 13 Michael Hrivnak 2018-02-09 15:37:33 UTC
(In reply to Jian Zhang from comment #11)
> Michael,
> 
> Another thing is your example APB failed when running async unbind.

Ok, feel free to NEEDSINFO me when you file that bug.

I've found the best (and maybe only) way to get more detail about such a failure is to enable the broker's feature "keep_namespace_on_error: True", and then go look at the logs for the APB pod. It should show you the output from ansible.

Comment 14 Jian Zhang 2018-02-11 04:05:29 UTC
Michael,

Thanks for your clear explanation! I got it. As for the storing of the example APB, I think maybe we can put it on the https://hub.docker.com/r/ansibleplaybookbundle. And, we can rename or label it with "async experimentation" or something.

For the unbind failure, I could NOT reproduce it with the latest version.
The ASB version: 1.1.10
reproduce steps:

1, Enable the async of the ASB and enable the develop mode, like below:
    openshift:
      ...
      keep_namespace_on_error: true
    broker:
      ...
      dev_broker: true
      launch_apb_on_bind: true

2, Config the registry which stored the example async APB, like:
    registry:
      - type: dockerhub
        name: dh
        url:  https://registry.hub.docker.com
        org:  mhrivnak
        tag:  latest
        white_list:
          - ".*-apb$"

3, New a porject and provision the example APB(PostgreSQL) in webconsole.
[root@host-172-16-120-48 ~]# oc get pods -n jian
NAME                         READY     STATUS    RESTARTS   AGE
postgresql-9.6-dev-1-vtxm6   1/1       Running   0          9m
[root@host-172-16-120-48 ~]# oc get serviceinstance -n jian
NAME                      AGE
dh-postgresql-apb-5mmtr   10m

4, Create a binding by using the script below and replace the related values with your own.
[jzhang@localhost ~]$ ./bind.sh 
binding ID: 0683134d-5f4d-41ae-b32b-bb7108d3021e
HTTP/1.1 202 Accepted
Content-Type: application/json
Date: Sun, 11 Feb 2018 03:53:43 GMT
Content-Length: 58
Set-Cookie: 89a6d633054ded194d4e1360cdc1fbef=a3d8c4c08e2850819c78cf452f68a11f; path=/; HttpOnly; Secure

{
  "operation": "96232702-20ab-469e-830f-4e1a1fb212a1"
}

5, Check status:
[jzhang@localhost ~]$ ./last_operation.sh 96232702-20ab-469e-830f-4e1a1fb212a1
{
  "state": "succeeded"
}

6, Run the async unbind action, looks worked well!
[jzhang@localhost ~]$ ./unbind.sh 0683134d-5f4d-41ae-b32b-bb7108d3021e
HTTP/1.1 202 Accepted
Content-Type: application/json
Date: Sun, 11 Feb 2018 03:54:11 GMT
Content-Length: 58
Set-Cookie: 89a6d633054ded194d4e1360cdc1fbef=a3d8c4c08e2850819c78cf452f68a11f; path=/; HttpOnly; Secure

{
  "operation": "11c2110b-45a3-472c-9a80-fbf1cad0c22f"
}

[jzhang@localhost ~]$ cat unbind.sh 
INSTANCE_ID="3ac081f5-0605-4462-b354-127a99466ce1"
BINDING_ID=$1
PLAN_UUID="7f4a5e35e4af2beb70076e72fab0b7ff"
IDENTITY=$(echo -n '{"groups":["system:authenticated:oauth","system:authenticated"],"scopes.authorization.openshift.io":["user:full"],"uid":"","username":"jiazha"}'|base64 -w 0)

curl \
    -k -i \
    -X DELETE \
    -H "Authorization: bearer $(oc whoami -t)" \
    -H "Content-type: application/json" \
    -H "Accept: application/json" \
    -H "X-Broker-API-Originating-Identity: kubernetes $IDENTITY" \
    "https://asb-1338-openshift-ansible-service-broker.apps.0211-dxf.qe.rhcloud.com/ansible-service-broker/v2/service_instances/$INSTANCE_ID/service_bindings/$BINDING_ID?accepts_incomplete=true&plan_id=$PLAN_UUID"


The ASB logs:
...
[2018-02-11T03:54:11.787Z] [INFO] - ASYNC unbinding in progress
[2018-02-11T03:54:11.787Z] [DEBUG] - unbindjob: unbinding job (11c2110b-45a3-472c-9a80-fbf1cad0c22f) started, calling apb.Unbind
[2018-02-11T03:54:11.787Z] [NOTICE] - ============================================================
[2018-02-11T03:54:11.787Z] [NOTICE] -                        UNBINDING                            
[2018-02-11T03:54:11.787Z] [NOTICE] - ============================================================
[2018-02-11T03:54:11.787Z] [NOTICE] - ServiceInstance.ID: 1dda1477cace09730bd8ed7a6505607e
[2018-02-11T03:54:11.787Z] [NOTICE] - ServiceInstance.Name: dh-postgresql-apb
[2018-02-11T03:54:11.787Z] [NOTICE] - ServiceInstance.Image: docker.io/mhrivnak/postgresql-apb:latest
[2018-02-11T03:54:11.787Z] [NOTICE] - ServiceInstance.Description: SCL PostgreSQL apb implementation
[2018-02-11T03:54:11.787Z] [NOTICE] - ============================================================
[2018-02-11T03:54:11.787Z] [DEBUG] - ExecutingApb:
[2018-02-11T03:54:11.787Z] [DEBUG] - name:[ dh-postgresql-apb ]
[2018-02-11T03:54:11.787Z] [DEBUG] - image:[ docker.io/mhrivnak/postgresql-apb:latest ]
[2018-02-11T03:54:11.787Z] [DEBUG] - action:[ unbind ]
[2018-02-11T03:54:11.787Z] [DEBUG] - pullPolicy:[ IfNotPresent ]
[2018-02-11T03:54:11.787Z] [DEBUG] - role:[ edit ]
[2018-02-11T03:54:11.787Z] [DEBUG] - No proxy env vars found to be configured.
[2018-02-11T03:54:11.811Z] [DEBUG] - Dao::DeleteBindInstance -> [ 0683134d-5f4d-41ae-b32b-bb7108d3021e ]
10.129.0.1 - - [11/Feb/2018:03:54:11 +0000] "DELETE /ansible-service-broker/v2/service_instances/3ac081f5-0605-4462-b354-127a99466ce1/service_bindings/0683134d-5f4d-41ae-b32b-bb7108d3021e?accepts_incomplete=true&plan_id=7f4a5e35e4af2beb70076e72fab0b7ff HTTP/1.1" 202 58
[2018-02-11T03:54:11.887Z] [DEBUG] - Trying to create apb sandbox: [ apb-333ca488-fb13-4115-b498-6f416dd79d72 ], with edit permissions in namespace dh-postgresql-apb-unbi-swj7s
[2018-02-11T03:54:11.887Z] [NOTICE] - Creating RoleBinding apb-333ca488-fb13-4115-b498-6f416dd79d72
[2018-02-11T03:54:12.033Z] [NOTICE] - Creating RoleBinding apb-333ca488-fb13-4115-b498-6f416dd79d72
[2018-02-11T03:54:12.091Z] [DEBUG] - Creating network policy for pod: apb-333ca488-fb13-4115-b498-6f416dd79d72 to grant network access to ns: jian
[2018-02-11T03:54:12.096Z] [DEBUG] - Successfully created network policy for pod: apb-333ca488-fb13-4115-b498-6f416dd79d72 to grant network access to ns: jian
[2018-02-11T03:54:12.096Z] [INFO] - Successfully created apb sandbox: [ apb-333ca488-fb13-4115-b498-6f416dd79d72 ], with edit permissions in namespace dh-postgresql-apb-unbi-swj7s
[2018-02-11T03:54:12.096Z] [INFO] - Running post create sandbox fuctions if defined.
[2018-02-11T03:54:12.096Z] [NOTICE] - Creating pod "apb-333ca488-fb13-4115-b498-6f416dd79d72" in the dh-postgresql-apb-unbi-swj7s namespace
[2018-02-11T03:54:12.11Z] [DEBUG] - Watching pod [ apb-333ca488-fb13-4115-b498-6f416dd79d72 ] in namespace [ dh-postgresql-apb-unbi-swj7s ] for completion
[2018-02-11T03:54:12.11Z] [INFO] - Watch pod [ apb-333ca488-fb13-4115-b498-6f416dd79d72 ] tick 1
[2018-02-11T03:54:12.127Z] [DEBUG] - Pod [ apb-333ca488-fb13-4115-b498-6f416dd79d72 ] Pending
[2018-02-11T03:54:17.13Z] [INFO] - Watch pod [ apb-333ca488-fb13-4115-b498-6f416dd79d72 ] tick 2
[2018-02-11T03:54:17.156Z] [DEBUG] - Pod [ apb-333ca488-fb13-4115-b498-6f416dd79d72 ] Running
[2018-02-11T03:54:22.156Z] [INFO] - Watch pod [ apb-333ca488-fb13-4115-b498-6f416dd79d72 ] tick 3
[2018-02-11T03:54:22.16Z] [DEBUG] - Pod [ apb-333ca488-fb13-4115-b498-6f416dd79d72 ] completed
[2018-02-11T03:54:22.16Z] [INFO] - Destroying APB sandbox...
[2018-02-11T03:54:22.162Z] [DEBUG] - Deleting namespace dh-postgresql-apb-unbi-swj7s
[2018-02-11T03:54:22.168Z] [DEBUG] - Deleting rolebinding apb-333ca488-fb13-4115-b498-6f416dd79d72, namespace dh-postgresql-apb-unbi-swj7s
[2018-02-11T03:54:22.182Z] [NOTICE] - Successfully deleted rolebinding apb-333ca488-fb13-4115-b498-6f416dd79d72, namespace dh-postgresql-apb-unbi-swj7s
[2018-02-11T03:54:22.182Z] [DEBUG] - Deleting rolebinding apb-333ca488-fb13-4115-b498-6f416dd79d72, namespace jian
[2018-02-11T03:54:22.186Z] [NOTICE] - Successfully deleted rolebinding apb-333ca488-fb13-4115-b498-6f416dd79d72, namespace jian
[2018-02-11T03:54:22.186Z] [DEBUG] - Deleting network policy for pod: apb-333ca488-fb13-4115-b498-6f416dd79d72 to grant network access to ns: jian
[2018-02-11T03:54:22.193Z] [DEBUG] - Successfully deleted network policy for pod: apb-333ca488-fb13-4115-b498-6f416dd79d72 to grant network access to ns: jian
[2018-02-11T03:54:22.193Z] [DEBUG] - Running post sandbox destroy hooks
[2018-02-11T03:54:22.193Z] [DEBUG] - unbindjob: returned from apb.Unbind
[2018-02-11T03:54:22.193Z] [DEBUG] - unbindjob: Looks like we're done
[2018-02-11T03:54:22.193Z] [DEBUG] - Processed binding message from buffer

At present, the async unbind works well, I will keep an eye on it.
And, for this fix, it looks good to me, thanks!

Comment 17 errata-xmlrpc 2018-03-28 14:21:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0489


Note You need to log in before you can comment on or make changes to this bug.