Description of problem: As the https://github.com/kubernetes-incubator/service-catalog/blob/a604bc398bc990285eb47861060a50177c166697/charts/catalog/templates/controller-manager-deployment.yaml#L77 shows, we can enable the ASYNC of the service-catalog by setting "AsyncBindingOperations=true", but, it does NOT work actually! Version-Release number of selected component (if applicable): Service catalog image and version: [root@host-172-16-120-40 ~]# docker run --rm --entrypoint=service-catalog registry.reg-aws.openshift.com:443/openshift3/ose-service-catalog:v3.9 --version v0.1.3 ASB image and version: [root@host-172-16-120-40 ~]# docker run --rm --entrypoint=asbd registry.reg-aws.openshift.com:443/openshift3/ose-ansible-service-broker:v3.9 --version 1.1.7 How reproducible: Always Steps to Reproduce: 1, Config the "AsyncBindingOperations=true" in the controller-manager of the service-catalog as the following: [root@host-172-16-120-40 ~]# oc edit daemonset controller-manager -n kube-service-catalog containers: - args: - controller-manager - -v - "5" - --leader-election-namespace - kube-service-catalog - --broker-relist-interval - 5m - --feature-gates - OriginatingIdentity=true - --feature-gates - AsyncBindingOperations=true 2, Provision the PostgreSQL APB in web console after the controller-manager pod restarted. 3, Enable the ASYNC of the ASB by setting the "launch_apb_on_bind: true" in its ConfigMap. And, restart the ASB pod. 4, Watch the ASB logs, like below: [root@host-172-16-120-40 ~]# oc logs -f asb-1-8lbxq ... 5, Create a binding of the PostgreSQL. Actual results: Did not find the "accepts_incomplete=true" in the PUT info. Details: [2018-01-29T07:00:51.869Z] [DEBUG] - provision bind creds: map[DB_HOST:postgresql DB_NAME:admin DB_PASSWORD:CjNf0mAEYSFyWOaF6gQt DB_PORT:5432 DB_TYPE:postgres DB_USER:admin] 10.128.0.5 - - [29/Jan/2018:07:00:46 +0000] "PUT /ansible-service-broker/v2/service_instances/571578ca-f87e-4141-8ecb-1ba7ef804365/service_bindings/a26fe9cc-edd1-4acc-a574-6ed5c1424570 HTTP/1.1" 201 196 [2018-01-29T07:00:51.911Z] [WARNING] - launch_apb_on_bind is enabled, but accepts_incomplete is false, binding may fail Expected results: Should add the accepts_incomplete=true in the binding PUT. Should NOT get the above WARNING info anymore. Additional info:
I believe that you have to collapse the feature gate options into a single call for this, like: [root@host-172-16-120-40 ~]# oc edit daemonset controller-manager -n kube-service-catalog containers: - args: - controller-manager - -v - "5" - --leader-election-namespace - kube-service-catalog - --broker-relist-interval - 5m - --feature-gates - OriginatingIdentity=true,AsyncBindingOperations=true Jeff, can you corroborate that?
./service-catalog controller-manager -v 5 --feature-gates AsyncBindingOperations=false --feature-gates Initializers=true I0205 15:32:58.714991 2643 feature_gate.go:184] feature gates: map[AsyncBindingOperations:false] I0205 15:32:58.715070 2643 feature_gate.go:184] feature gates: map[Initializers:true AsyncBindingOperations:false] Additional features appear to be respected with a cumulative effect, still investigating.
The problem here is that in addition to passing AsyncBindingOperations=true to the catalog controller and setting the template parameter LAUNCH_APB_ON_BIND=true, the service class also has to have binding_retrievable (soon to be bindingRetrievable) set to true. Since this parameter in the case of ASB is set in the base64 encoded com.redhat.asb.spec label, eventually I think they'll need to update their images and then code to properly extract the value here: https://github.com/openshift/ansible-service-broker/blob/5e58b6f00cafc8d9b6b13e9bf1afde008cdffa38/pkg/broker/util.go#L44 One may try manually editing the service class with binding_retrievable set to true as a workaround: 172.17.0.4 - - [05/Feb/2018:19:40:48 +0000] "GET /ansible-service-broker/v2/service_instances/42e9c882-e7a8-483f-9b2e-7ff244f03f83/last_operation?operation=a0bb9d37-79ee-403a-8af3-b15cba146719&plan_id=7f4a5e35e4af2beb70076e72fab0b7ff&service_id=1dda1477cace09730bd8ed7a6505607e HTTP/1.1" 200 27 [2018-02-05T19:40:55.3Z] [INFO] - Request: "PUT /ansible-service-broker/v2/service_instances/42e9c882-e7a8-483f-9b2e-7ff244f03f83/service_bindings/516a4861-4963-4a42-a0e1-3711a701f8b3?accepts_incomplete=true HTTP/1.1\r\nHost: asb.ansible-service-broker.svc:1338\r\nAccept-Encoding: gzip\r\nContent-Length: 162\r\nContent-Type: application/json\r\nUser-Agent: Go-http-client/1.1\r\nX-Broker-Api-Originating-Identity: kubernetes eyJncm91cHMiOlsic3lzdGVtOmF1dGhlbnRpY2F0ZWQ6b2F1dGgiLCJzeXN0ZW06YXV0aGVudGljYXRlZCJdLCJzY29wZXMuYXV0aG9yaXphdGlvbi5vcGVuc2hpZnQuaW8iOlsidXNlcjpmdWxsIl0sInVpZCI6IiIsInVzZXJuYW1lIjoiZGV2ZWxvcGVyIn0=\r\nX-Broker-Api-Version: 2.13\r\n\r\n{\"service_id\":\"1dda1477cace09730bd8ed7a6505607e\",\"plan_id\":\"7f4a5e35e4af2beb70076e72fab0b7ff\",\"bind_resource\":{\"app_guid\":\"592947b3-0aa5-11e8-9749-54e1ad6bac3b\"}}" [2018-02-05T19:40:55.377Z] [DEBUG] - Injecting PlanID as parameter: { _apb_plan_id: dev } [2018-02-05T19:40:55.377Z] [DEBUG] - Injecting ServiceClassID as parameter: { _apb_service_class_id: 1dda1477cace09730bd8ed7a6505607e } [2018-02-05T19:40:55.377Z] [DEBUG] - Injecting ServiceInstanceID as parameter: { _apb_service_instance_id: 42e9c882-e7a8-483f-9b2e-7ff244f03f83 } [2018-02-05T19:40:55.378Z] [INFO] - ASYNC binding in progress [2018-02-05T19:40:55.378Z] [DEBUG] - bindjob: binding job started, calling apb.Bind [2018-02-05T19:40:55.378Z] [NOTICE] - ============================================================ [2018-02-05T19:40:55.378Z] [NOTICE] - BINDING [2018-02-05T19:40:55.378Z] [NOTICE] - ============================================================ [2018-02-05T19:40:55.378Z] [NOTICE] - ServiceInstance.ID: 1dda1477cace09730bd8ed7a6505607e [2018-02-05T19:40:55.378Z] [NOTICE] - ServiceInstance.Name: dh-postgresql-apb [2018-02-05T19:40:55.378Z] [NOTICE] - ServiceInstance.Image: docker.io/ansibleplaybookbundle/postgresql-apb:latest [2018-02-05T19:40:55.378Z] [NOTICE] - ServiceInstance.Description: SCL PostgreSQL apb implementation [2018-02-05T19:40:55.378Z] [NOTICE] - ============================================================
John- The issue here is that the service in question isn't setting the right field to let the catalog know that it supports async binding. Can someone from your team take a look?
Confirmed we are NOT returning binding_retrievable. Sorry for missing this in the spec.
release-1.1 PR (3.9) https://github.com/openshift/ansible-service-broker/pull/774 master PR https://github.com/openshift/ansible-service-broker/pull/776
http://pkgs.devel.redhat.com/cgit/rpms/ansible-service-broker/commit/?h=rhaos-3.9-asb-rhel-7&id=2a6d5f66b42911ca26c534c97eef3d8e051bdd9b
The ASB version: 1.1.14 [root@host-172-16-120-33 ~]# docker run --rm --entrypoint=asbd registry.reg-aws.openshift.com:443/openshift3/ose-ansible-service-broker:v3.9.0 --version 1.1.14 The service catalog version: 0.1.8 [root@host-172-16-120-33 ~]# docker run --rm --entrypoint=service-catalog registry.reg-aws.openshift.com:443/openshift3/ose-service-catalog:v3.9.0 --version v0.1.8 1) Config the async of the service-catalog and the ASB as the above shows. Like below: The configure of the service-catalog: # oc edit daemonset controller-manager -n kube-service-catalog ... containers: - args: ... - --feature-gates - OriginatingIdentity=true - --feature-gates - AsyncBindingOperations=true The configure of the ASB: #oc edit cm broker-config -n openshift-ansible-service-broker ... broker: ... launch_apb_on_bind: true 2) Provision an APB(for example, PostgreSQL) on web [root@host-172-16-120-33 ~]# oc get pods -n test NAME READY STATUS RESTARTS AGE postgresql-9.6-dev-1-lc6k9 1/1 Running 0 13m [root@host-172-16-120-33 ~]# oc get serviceinstance -n test NAME AGE rh-postgresql-apb-jkwnf 14m 3) Create a binding, watch the ASB logs: [2018-02-26T07:41:49.723Z] [WARNING] - launch_apb_on_bind is enabled, but accepts_incomplete is false, binding may fail [2018-02-26T07:41:49.796Z] [DEBUG] - Injecting PlanID as parameter: { _apb_plan_id: dev } [2018-02-26T07:41:49.797Z] [DEBUG] - Injecting ServiceClassID as parameter: { _apb_service_class_id: d5915e05b253df421efe6e41fb6a66ba } [2018-02-26T07:41:49.797Z] [DEBUG] - Injecting ServiceInstanceID as parameter: { _apb_service_instance_id: 9b58c5f1-1e35-4f7f-b81e-ad982228fdb8 } [2018-02-26T07:41:49.814Z] [INFO] - Broker configured to run APB bind [2018-02-26T07:41:49.814Z] [NOTICE] - ============================================================ [2018-02-26T07:41:49.814Z] [NOTICE] - BINDING [2018-02-26T07:41:49.814Z] [NOTICE] - ============================================================ [2018-02-26T07:41:49.814Z] [NOTICE] - ServiceInstance.ID: d5915e05b253df421efe6e41fb6a66ba [2018-02-26T07:41:49.814Z] [NOTICE] - ServiceInstance.Name: rh-postgresql-apb [2018-02-26T07:41:49.814Z] [NOTICE] - ServiceInstance.Image: registry.access.stage.redhat.com/openshift3/postgresql-apb:v3.9 [2018-02-26T07:41:49.814Z] [NOTICE] - ServiceInstance.Description: SCL PostgreSQL apb implementation [2018-02-26T07:41:49.814Z] [NOTICE] - ============================================================ [2018-02-26T07:41:49.814Z] [DEBUG] - ExecutingApb: [2018-02-26T07:41:49.814Z] [DEBUG] - name:[ rh-postgresql-apb ] [2018-02-26T07:41:49.814Z] [DEBUG] - image:[ registry.access.stage.redhat.com/openshift3/postgresql-apb:v3.9 ] [2018-02-26T07:41:49.814Z] [DEBUG] - action:[ bind ] [2018-02-26T07:41:49.814Z] [DEBUG] - pullPolicy:[ IfNotPresent ] [2018-02-26T07:41:49.814Z] [DEBUG] - role:[ edit ] [2018-02-26T07:41:49.814Z] [DEBUG] - No proxy env vars found to be configured. ... 10.128.0.14 - - [26/Feb/2018:07:41:49 +0000] "PUT /ansible-service-broker/v2/service_instances/9b58c5f1-1e35-4f7f-b81e-ad982228fdb8/service_bindings/72aae0c4-4150-47fb-83d4-d855987542f9 HTTP/1.1" 400 40 [2018-02-26T07:41:55.202Z] [WARNING] - launch_apb_on_bind is enabled, but accepts_incomplete is false, binding may fail [2018-02-26T07:41:55.372Z] [DEBUG] - Injecting PlanID as parameter: { _apb_plan_id: dev } [2018-02-26T07:41:55.372Z] [DEBUG] - Injecting ServiceClassID as parameter: { _apb_service_class_id: d5915e05b253df421efe6e41fb6a66ba } [2018-02-26T07:41:55.372Z] [DEBUG] - Injecting ServiceInstanceID as parameter: { _apb_service_instance_id: 9b58c5f1-1e35-4f7f-b81e-ad982228fdb8 } [2018-02-26T07:41:55.373Z] [DEBUG] - already have this binding instance, returning 200 [2018-02-26T07:41:55.373Z] [DEBUG] - provision bind creds: map[DB_PORT:5432 DB_TYPE:postgres DB_USER:admin DB_HOST:postgresql DB_NAME:admin DB_PASSWORD:test] 10.128.0.14 - - [26/Feb/2018:07:41:55 +0000] "PUT /ansible-service-broker/v2/service_instances/9b58c5f1-1e35-4f7f-b81e-ad982228fdb8/service_bindings/72aae0c4-4150-47fb-83d4-d855987542f9 HTTP/1.1" 200 180 Still got the same warning: "launch_apb_on_bind is enabled, but accepts_incomplete is false, binding may fail", and the PUT request did NOT contain the "accepts_incomplete=true". So, failed to verify.
The ASB is now returning the `bindings_retrievable` in the catalog response: grep bindings_retrievable catalog.out "bindings_retrievable": true, "bindings_retrievable": true, "bindings_retrievable": true, "bindings_retrievable": false, But the serviceclasses in the Service Catalog are still marked as false: bindingRetrievable: false bindingRetrievable: false bindingRetrievable: false bindingRetrievable: false bindingRetrievable: false bindingRetrievable: false bindingRetrievable: false bindingRetrievable: false bindingRetrievable: false bindingRetrievable: false bindingRetrievable: false bindingRetrievable: false bindingRetrievable: false bindingRetrievable: false bindingRetrievable: false
I found that the broker client was looking for binding_retrievable instead of bindings_retrievable. Client PR is here: https://github.com/pmorie/go-open-service-broker-client/pull/106. Will need to bump the version in service-catalog once that PR is merged and will update the bug here with that PR.
Origin PR with fix (not yet merged): https://github.com/openshift/origin/pull/18788/
Since the fix merged in the v0.1.9, so I used it for this test. The service catalog version: v0.1.9 [root@host-172-16-120-35 ~]# docker run --rm --entrypoint=service-catalog brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/ose-service-catalog:v3.9 --version v0.1.9 The ASB version: 1.1.15 [root@host-172-16-120-35 ~]# docker run --rm --entrypoint=asbd brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/ose-ansible-service-broker:v3.9.3 --version 1.1.15 1) Config the async of the service-catalog and the ASB as comment11 shows. 2) Provision an APB(for example, PostgreSQL) on web [root@host-172-16-120-35 ~]# oc get pods -n jian NAME READY STATUS RESTARTS AGE postgresql-9.6-dev-1-dnf7x 1/1 Running 0 53m 3) Create a binding, watch the ASB logs: [root@host-172-16-120-35 ~]# oc logs -f asb-25-jqbw5 ... ... [2018-03-09T03:01:44.707Z] [WARNING] - launch_apb_on_bind is enabled, but accepts_incomplete is false, binding may fail [2018-03-09T03:01:44.744Z] [DEBUG] - Injecting PlanID as parameter: { _apb_plan_id: dev } [2018-03-09T03:01:44.744Z] [DEBUG] - Injecting ServiceClassID as parameter: { _apb_service_class_id: d5915e05b253df421efe6e41fb6a66ba } [2018-03-09T03:01:44.744Z] [DEBUG] - Injecting ServiceInstanceID as parameter: { _apb_service_instance_id: bd4b385f-7a43-473d-8b51-b7388f101b13 } [2018-03-09T03:01:44.747Z] [INFO] - Broker configured to run APB bind [2018-03-09T03:01:44.747Z] [NOTICE] - ============================================================ [2018-03-09T03:01:44.747Z] [NOTICE] - BINDING [2018-03-09T03:01:44.747Z] [NOTICE] - ============================================================ [2018-03-09T03:01:44.747Z] [NOTICE] - ServiceInstance.ID: d5915e05b253df421efe6e41fb6a66ba [2018-03-09T03:01:44.747Z] [NOTICE] - ServiceInstance.Name: rh-postgresql-apb [2018-03-09T03:01:44.747Z] [NOTICE] - ServiceInstance.Image: registry.access.stage.redhat.com/openshift3/postgresql-apb:v3.9 [2018-03-09T03:01:44.747Z] [NOTICE] - ServiceInstance.Description: SCL PostgreSQL apb implementation ... ... [2018-03-09T03:01:50.031Z] [DEBUG] - Running post sandbox destroy hooks 10.128.0.1 - - [09/Mar/2018:03:01:44 +0000] "PUT /ansible-service-broker/v2/service_instances/bd4b385f-7a43-473d-8b51-b7388f101b13/service_bindings/935632d9-c664-4b97-a07c-99845e9dbdf7 HTTP/1.1" 400 40 [2018-03-09T03:01:50.062Z] [WARNING] - launch_apb_on_bind is enabled, but accepts_incomplete is false, binding may fail [2018-03-09T03:01:50.101Z] [DEBUG] - Injecting PlanID as parameter: { _apb_plan_id: dev } [2018-03-09T03:01:50.101Z] [DEBUG] - Injecting ServiceClassID as parameter: { _apb_service_class_id: d5915e05b253df421efe6e41fb6a66ba } [2018-03-09T03:01:50.101Z] [DEBUG] - Injecting ServiceInstanceID as parameter: { _apb_service_instance_id: bd4b385f-7a43-473d-8b51-b7388f101b13 } [2018-03-09T03:01:50.103Z] [DEBUG] - already have this binding instance, returning 200 [2018-03-09T03:01:50.103Z] [DEBUG] - provision bind creds: map[DB_PASSWORD:test DB_PORT:5432 DB_TYPE:postgres DB_USER:admin DB_HOST:postgresql DB_NAME:admin] Still got the same warning: "launch_apb_on_bind is enabled, but accepts_incomplete is false, binding may fail", and the PUT request did NOT contain the "accepts_incomplete=true". So, verify failed. Inadditional info: [root@host-172-16-120-35 ~]# oc version oc v3.9.3 kubernetes v1.9.1+a0ce1bc657 features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://172.16.120.35:8443 openshift v3.9.3 kubernetes v1.9.1+a0ce1bc657 [root@host-172-16-120-35 ~]# oc get pods controller-manager-hdxmn -o yaml | grep Async - AsyncBindingOperations=true
Jeff, Retest it on the v3.9.4 cluster, and it works! I used the same images, the same version, the same configure as used in the v3.9.3 cluster. I'm not very clear about why it did not work on the v3.9.3 cluster, please change the status if LGTY. I will verify it, thanks! [root@host-172-16-120-17 ~]# oc version oc v3.9.4 kubernetes v1.9.1+a0ce1bc657 features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://172.16.120.17:8443 openshift v3.9.4 kubernetes v1.9.1+a0ce1bc657 service catalog info: [root@host-172-16-120-17 ~]# docker run --rm --entrypoint=service-catalog brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/ose-service-catalog:v3.9 --version v0.1.9 ASB info: [root@host-172-16-120-17 ~]# docker run --rm --entrypoint=asbd brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/ose-ansible-service-broker:v3.9 --version 1.1.15 The ASB logs: ... [2018-03-09T09:36:38.362Z] [INFO] - ASYNC binding in progress [2018-03-09T09:36:38.362Z] [DEBUG] - bindjob: binding job started, calling apb.Bind [2018-03-09T09:36:38.362Z] [NOTICE] - ============================================================ [2018-03-09T09:36:38.362Z] [NOTICE] - BINDING [2018-03-09T09:36:38.362Z] [NOTICE] - ============================================================ [2018-03-09T09:36:38.362Z] [NOTICE] - ServiceInstance.ID: 1dda1477cace09730bd8ed7a6505607e [2018-03-09T09:36:38.362Z] [NOTICE] - ServiceInstance.Name: dh-postgresql-apb [2018-03-09T09:36:38.362Z] [NOTICE] - ServiceInstance.Image: docker.io/ansibleplaybookbundle/postgresql-apb:latest [2018-03-09T09:36:38.362Z] [NOTICE] - ServiceInstance.Description: SCL PostgreSQL apb implementation [2018-03-09T09:36:38.362Z] [NOTICE] - ============================================================ [2018-03-09T09:36:38.362Z] [DEBUG] - ExecutingApb: [2018-03-09T09:36:38.362Z] [DEBUG] - name:[ dh-postgresql-apb ] [2018-03-09T09:36:38.362Z] [DEBUG] - image:[ docker.io/ansibleplaybookbundle/postgresql-apb:latest ] [2018-03-09T09:36:38.362Z] [DEBUG] - action:[ bind ] [2018-03-09T09:36:38.362Z] [DEBUG] - pullPolicy:[ IfNotPresent ] [2018-03-09T09:36:38.362Z] [DEBUG] - role:[ edit ] [2018-03-09T09:36:38.373Z] [DEBUG] - Trying to create apb sandbox: [ apb-74507b9a-80d9-4b57-8e98-af78c139ec6c ], with edit permissions in namespace dh-postgresql-apb-bind-kmz9g [2018-03-09T09:36:38.373Z] [NOTICE] - Creating RoleBinding apb-74507b9a-80d9-4b57-8e98-af78c139ec6c 10.129.0.1 - - [09/Mar/2018:09:36:38 +0000] "PUT /ansible-service-broker/v2/service_instances/e597bbea-2add-469a-ab56-05d3a335b408/service_bindings/91a2e89a-aba8-48f6-8ac4-c44114790bd9?accepts_incomplete=true HTTP/1.1" 202 58 And, no warning info, we can see the "accepts_incomplete=true" in the PUT info. Looks good.
The red hat registry image doesn't look right: $ skopeo inspect docker://registry.access.stage.redhat.com/openshift3/postgresql-apb:v3.9 { "Name": "registry.access.stage.redhat.com/openshift3/postgresql-apb", "Tag": "v3.9", "Digest": "sha256:7d7e17e6403a90703075aca33b3a9542bdd006fe9be2101d220bb10e4dbf952b", "RepoTags": [ "0.0.1", "v3.6", "v3.6.173.0.5-4", "v3.6.173.0.5-7", "v3.6.173.0.5", "0.0.1-5", "0.0.1-4", "latest", "0.0.1-3", "0.0.1-2" ], "Created": "2018-03-02T13:32:46.636921Z", "DockerVersion": "1.12.6", "Labels": {}, "Architecture": "amd64", "Os": "linux", "Layers": [ "sha256:9a32f102e6778e4b3c677f1f93fa121808006d3c9e002237677248de9acb9589", "sha256:b8aa42cec17a56aea47fa45bd8029d1e877b21213017e849a839aadba9e1486c", "sha256:23439b9e9c5805ae52802c11007ebc29d22fd0fa1674194cab6110fce8ef7501", "sha256:15dc245e1cdbc149ac82d72a0ca2c1156f061ff940ca49be906ba8a163c2021f" ] } There's no com.redhat.apb.spec label on that image, where as the docker.io image has one. I believe everything is working as it's supposed to be.
Jeff Yes, you're right. I used the dockerhub registry since these APB images stored in our stage registry do not support the async. Anyway, it works as we expected. LGTM.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:0489