Bug 1533425

Additional info: 
The key point for the issue may be the config:
     broker:  
        launch_apb_on_bind: true
If set 'launch_apb_on_bind' to false, binding will be created successfully , and the secret can be found and add to other application (such as mediawiki). 

ASB log: 

[2018-01-15T07:28:00.77Z] [DEBUG] - Injecting PlanID as parameter: { _apb_plan_id: dev }
[2018-01-15T07:28:00.77Z] [DEBUG] - Injecting ServiceClassID as parameter: { _apb_service_class_id: 3bcbc1f42ae47b10e9015b7d7a8a9b97 }
[2018-01-15T07:28:00.77Z] [DEBUG] - Injecting ServiceInstanceID as parameter: { _apb_service_instance_id: e6569f60-b8e7-43a1-8c7e-cad399201a3a }
[2018-01-15T07:28:00.772Z] [WARNING] - Broker configured to *NOT* launch and run APB bind

If I understand this bug, it is actually in reference to our bind credential extraction process and not async bind. The downstream broker likely does not support async bind at this stage and should not be expected to work.

If I am correct and we are speaking of the bind credential extraction process then the important thing to know is that bind credential extraction happens during 'provision' and not 'bind'. I recommend that we close this bug, examine the test procedure for bind credential extraction updates (https://trello.com/c/rT9jfi2P/583-improve-ansible-brokers-bind-credential-extraction-process), and after making any changes to the procedure creating new bugs if there are any.

This bug was found in the test for 'bind credential extraction process'.
But if still set  'launch_apb_on_bind: true', create binding will fail with error : 
[ERROR] - Unable to load secret 'apb-fece1389-17f1-483a-9fc2-fa1e32b09bfe' from namespace 'new-postgresql-apb-bind-cvmsz'

The secret is created in provsion sandbox, but when creating binding , the broker still goes to 'bind' sandbox to load secret.

After looking into this bug, asb 1.1.4 did not have async bind feature. This means that launch_apb_on_bind may have some adverse affects without async bind.

Please retest with asb 1.1.5 or greater since that is where async bind feature was introduced.

I was able to recreate this bug using the latest postgresql apb in ansibleplaybookbundle org. Looks like that apb does not have the bind playbook which is why when launch_apb_on _bind is enabled it fails.

There are 2 problems here. 1) there should be more of an indication that the job failed to the UI from the synchronous call. 2) we need to update the apb to include bind.

PRs created to fix both issues in this bugzilla.

1) add ability to detect an error when action is not found:

apb-base change required:

https://github.com/ansibleplaybookbundle/apb-base/pull/17

broker change required:

https://github.com/openshift/ansible-service-broker/pull/716

2) add bind support to postgresql-apb (depends on apb-base PR):

https://github.com/ansibleplaybookbundle/postgresql-apb/pull/32

This will require 3 new images:

* new apb-base
* new postgresql-apb
* new broker image

Correction to comment #11 there will be no new postgresql-apb with bind support. The async bind feature will be more like tech preview, in order to test this feature you need to use an example apb that has bind support in it. Might I suggest the hello-world-apb from the ansibleplaybookbundle.

Also consider doing regression testing with Launch_apb_on_bind set to false to ensure that bind works as it did in 3.7.0.

http://pkgs.devel.redhat.com/cgit/rpms/ansible-service-broker/commit/?h=rhaos-3.9-asb-rhel-7&id=2a6d5f66b42911ca26c534c97eef3d8e051bdd9b

Image is ready , change to ON_QA.

Verify failed.
ASB: 1.1.14
hello-world-db-apb:latest (in dockerhub, ansibleplaybookbundle)

step:
1. set broker-config:
 registry:
  - type: dockerhub
    name: dh
    url:  https://registry.hub.docker.com
    org:  ansibleplaybookbundle
    tag: "latest"
    white_list:
      - ".*-apb$"  
 broker:
   bootstrap_on_startup: true
2. provision hello-world-db apb,
3. create binding failed with error:
 Asb Log:
   [2018-02-26T09:36:27.696Z] [WARNING] - launch_apb_on_bind is enabled, but accepts_incomplete is false, binding may fail
[2018-02-26T09:36:27.739Z] [INFO] - Broker configured to run APB bind
[2018-02-26T09:36:27.739Z] [NOTICE] - ============================================================
[2018-02-26T09:36:27.739Z] [NOTICE] -                        BINDING                              
[2018-02-26T09:36:27.739Z] [NOTICE] - ============================================================
[2018-02-26T09:36:27.739Z] [NOTICE] - ServiceInstance.ID: b43a4272a6efcaaa3e0b9616324f1099
[2018-02-26T09:36:27.739Z] [NOTICE] - ServiceInstance.Name: dh-hello-world-db-apb
[2018-02-26T09:36:27.739Z] [NOTICE] - ServiceInstance.Image: docker.io/ansibleplaybookbundle/hello-world-db-apb:latest
[2018-02-26T09:36:27.739Z] [NOTICE] - ServiceInstance.Description: A sample APB which deploys Hello World Database
[2018-02-26T09:36:27.739Z] [NOTICE] - ============================================================
[2018-02-26T09:36:27.801Z] [NOTICE] - Creating RoleBinding apb-e30aa480-d7ea-4761-9702-ccf2f011c139
[2018-02-26T09:36:27.933Z] [NOTICE] - Creating RoleBinding apb-e30aa480-d7ea-4761-9702-ccf2f011c139
[2018-02-26T09:36:27.985Z] [INFO] - Successfully created apb sandbox: [ apb-e30aa480-d7ea-4761-9702-ccf2f011c139 ], with edit permissions in namespace dh-hello-world-db-apb-bind-q5qs5
[2018-02-26T09:36:27.986Z] [INFO] - Running post create sandbox fuctions if defined.
[2018-02-26T09:36:27.986Z] [NOTICE] - Creating pod "apb-e30aa480-d7ea-4761-9702-ccf2f011c139" in the dh-hello-world-db-apb-bind-q5qs5 namespace
[2018-02-26T09:36:28.001Z] [INFO] - Watch pod [ apb-e30aa480-d7ea-4761-9702-ccf2f011c139 ] tick 1
[2018-02-26T09:36:33.005Z] [INFO] - Watch pod [ apb-e30aa480-d7ea-4761-9702-ccf2f011c139 ] tick 2
[2018-02-26T09:36:38.009Z] [INFO] - Watch pod [ apb-e30aa480-d7ea-4761-9702-ccf2f011c139 ] tick 3
[2018-02-26T09:36:38.025Z] [ERROR] - Unable to load secret 'apb-e30aa480-d7ea-4761-9702-ccf2f011c139' from namespace 'dh-hello-world-db-apb-bind-q5qs5'
[2018-02-26T09:36:38.025Z] [ERROR] - apb::bind error occurred - Unable to retrieve secret [ apb-e30aa480-d7ea-4761-9702-ccf2f011c139 ] - secrets "apb-e30aa480-d7ea-4761-9702-ccf2f011c139" not found

[root@host-172-16-120-90 ~]# oc describe servicebinding dh-hello-world-db-apb-hllv4-57c85
Name:         dh-hello-world-db-apb-hllv4-57c85
Namespace:    helloworlddb
Labels:       <none>
Annotations:  <none>
API Version:  servicecatalog.k8s.io/v1beta1
Kind:         ServiceBinding
Metadata:
  Creation Timestamp:  2018-02-26T09:36:27Z
  Finalizers:
    kubernetes-incubator/service-catalog
  Generate Name:     dh-hello-world-db-apb-hllv4-
  Generation:        1
  Resource Version:  61803
  Self Link:         /apis/servicecatalog.k8s.io/v1beta1/namespaces/helloworlddb/servicebindings/dh-hello-world-db-apb-hllv4-57c85
  UID:               84b2e62b-1ad8-11e8-9d42-0a580a800004
Spec:
  External ID:  82f74560-f524-4e55-a01a-e330c26d595a
  Instance Ref:
    Name:       dh-hello-world-db-apb-hllv4
  Secret Name:  dh-hello-world-db-apb-hllv4-credentials-u4mel
  User Info:
    Extra:
      Scopes . Authorization . Openshift . Io:
        user:full
    Groups:
      system:authenticated:oauth
      system:authenticated
    UID:       
    Username:  zitang
Status:
  Async Op In Progress:  false
  Conditions:
    Last Transition Time:         2018-02-26T09:36:27Z
    Message:                      ServiceBroker returned failure; bind operation will not be retried: Status: 400; ErrorMessage: <nil>; Description: Unable to retrieve secret [ apb-e30aa480-d7ea-4761-9702-ccf2f011c139 ] - secrets "apb-e30aa480-d7ea-4761-9702-ccf2f011c139" not found; ResponseError: <nil>
    Reason:                       BindCallFailed
    Status:                       False
    Type:                         Ready
    Last Transition Time:         2018-02-26T09:36:38Z
    Message:                      ServiceBroker returned failure; bind operation will not be retried: Status: 400; ErrorMessage: <nil>; Description: Unable to retrieve secret [ apb-e30aa480-d7ea-4761-9702-ccf2f011c139 ] - secrets "apb-e30aa480-d7ea-4761-9702-ccf2f011c139" not found; ResponseError: <nil>
    Reason:                       ServiceBindingReturnedFailure
    Status:                       True
    Type:                         Failed
  Orphan Mitigation In Progress:  false
  Reconciled Generation:          1
  Unbind Status:                  Required
Events:
  Type     Reason                         Age   From                                Message
  ----     ------                         ----  ----                                -------
  Warning  BindCallFailed                 9m    service-catalog-controller-manager  ServiceBroker returned failure; bind operation will not be retried: Status: 400; ErrorMessage: <nil>; Description: Unable to retrieve secret [ apb-e30aa480-d7ea-4761-9702-ccf2f011c139 ] - secrets "apb-e30aa480-d7ea-4761-9702-ccf2f011c139" not found; ResponseError: <nil>
  Warning  ServiceBindingReturnedFailure  9m    service-catalog-controller-manager  ServiceBroker returned failure; bind operation will not be retried: Status: 400; ErrorMessage: <nil>; Description: Unable to retrieve secret [ apb-e30aa480-d7ea-4761-9702-ccf2f011c139 ] - secrets "apb-e30aa480-d7ea-4761-9702-ccf2f011c139" not found; ResponseError: <nil>

About async binding, if I only config async in asb(enable launch_apb_on_bind) , not config service-catalog with async binding, when create binding, is it still performed as synchronous binding ? 
If it is still synchronous binding, so create binding in exist apb(postgresql-apb,mariadb-apb,mysql-apb,hello-world-db-apb) should succeed.
If not support only set async in asb without service-catalog, we'd better update the doc： https://github.com/openshift/ansible-service-broker/blob/master/docs/config.md#broker-configuration.

Moving this to 3.10.0 since we are not releasing a reference APB with async bind for 3.9.

Please use docker.io/jmrodri/postgresql-apb:demo this is an example apb that supports async binds.

$ docker images
REPOSITORY                                             TAG                 IMAGE ID            CREATED             SIZE
docker.io/jmrodri/postgresql-apb                       demo                1bdd30040082        6 weeks ago         1.47 GB

The above APB is purely an example only.

Moving back to assigned because master with CRDs will be broken.

This bug requires the following PR to work with CRDs and release 3.10 (master).

https://github.com/openshift/ansible-service-broker/pull/898

(In reply to Jesus M. Rodriguez from comment #21)
> This bug requires the following PR to work with CRDs and release 3.10
> (master).
> 
> https://github.com/openshift/ansible-service-broker/pull/898

This bug ACTUALLY requires PR 924.
https://github.com/openshift/ansible-service-broker/pull/924

https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=15985275

image is ready , change it to ON_QA.

1. Using apb https://registry.hub.docker.com/mhrivnak/postgresql-apb which support async binding. 
    then only set 'launch_apb_on_bind:true' in asb, the binding will be created succeessfully.
This is verified in asb 1.2.10

2. if using normal apb(postgresql-apb) , and only set 'launch_apb_on_bind:true' in asb, in my point of view, it should works as sync binding and binding can be created successfully.
   but acctually , in asb 1.2.10 . 
the bundlebinding status and servicebinding status are different.
what's the expected result for this scenario ?

# oc describe bundlebinding 98a4aae6-5290-11e8-8906-0a580a80000b
Name:         98a4aae6-5290-11e8-8906-0a580a80000b
Namespace:    openshift-ansible-service-broker
Labels:       <none>
Annotations:  <none>
API Version:  automationbroker.io/v1alpha1
Kind:         BundleBinding
Metadata:
  Cluster Name:        
  Creation Timestamp:  2018-05-08T07:22:42Z
  Generation:          1
  Resource Version:    141125
  Self Link:           /apis/automationbroker.io/v1alpha1/namespaces/openshift-ansible-service-broker/bundlebindings/98a4aae6-5290-11e8-8906-0a580a80000b
  UID:                 98ee42bd-5290-11e8-8260-fa163e868f10
Spec:
  Bundle Instance:
    Name:      464f1b47-5290-11e8-8906-0a580a80000b
  Parameters:  {"_apb_last_requesting_user":"zitang","_apb_plan_id":"dev","_apb_service_binding_id":"98a4aae6-5290-11e8-8906-0a580a80000b","_apb_service_class_id":"d5915e05b253df421efe6e41fb6a66ba","_apb_service_instance_id":"464f1b47-5290-11e8-8906-0a580a80000b"}
Status:
  Jobs:
    07883539 - E 97 B - 46 B 5 - 8972 - C 2 E 11387 F 234:
      Description:         Error occurred during bind. Please contact administrator if the issue persists.
      Error:               action not found
      Last Modified Time:  2018-05-08T07:22:46Z
      Method:              bind
      Podname:             
      State:               failed
  Last Description:        Error occurred during bind. Please contact administrator if the issue persists.
  State:                   failed
Events:                    <none>


# oc describe servicebinding -n post
Name:         rh-postgresql-apb-brtpf-xf8s4
Namespace:    post
Labels:       <none>
Annotations:  <none>
API Version:  servicecatalog.k8s.io/v1beta1
Kind:         ServiceBinding
Metadata:
  Creation Timestamp:  2018-05-08T07:22:42Z
  Finalizers:
    kubernetes-incubator/service-catalog
  Generate Name:     rh-postgresql-apb-brtpf-
  Generation:        1
  Resource Version:  141131
  Self Link:         /apis/servicecatalog.k8s.io/v1beta1/namespaces/post/servicebindings/rh-postgresql-apb-brtpf-xf8s4
  UID:               98a4af7a-5290-11e8-8906-0a580a80000b
Spec:
  External ID:  98a4aae6-5290-11e8-8906-0a580a80000b
  Instance Ref:
    Name:       rh-postgresql-apb-brtpf
  Secret Name:  rh-postgresql-apb-brtpf-credentials-kmc7y
  User Info:
    Extra:
      Scopes . Authorization . Openshift . Io:
        user:full
    Groups:
      system:authenticated:oauth
      system:authenticated
    UID:       
    Username:  zitang
Status:
  Async Op In Progress:  false
  Conditions:
    Last Transition Time:  2018-05-08T07:22:46Z
    Message:               Injected bind result
    Reason:                InjectedBindResult
    Status:                True
    Type:                  Ready
  External Properties:
    User Info:
      Extra:
        Scopes . Authorization . Openshift . Io:
          user:full
      Groups:
        system:authenticated:oauth
        system:authenticated
      UID:                        
      Username:                   zitang
  Orphan Mitigation In Progress:  false
  Reconciled Generation:          1
  Unbind Status:                  Required
Events:
  Type    Reason              Age   From                                Message
  ----    ------              ----  ----                                -------
  Normal  InjectedBindResult  1h    service-catalog-controller-manager  Injected bind result

As comment 25 /scenario 2 described, set it to ASSIGNED. 
if scenario 2 is expected , please set back to ON_QA

Scenario 2 is actually invalid. When you enable launch_apb_on_bind, that will instruct the broker to actually RUN the APB image and attempt to call the bind playbook on it. The "normal apb(postgresql-apb)" does not have a bind playbook to run and will FAIL. This failure is expected.

If you want to use the "normal apb(postgresql-apb)" you MUST disable launch_apb_on_bind, that is, set launch_apb_on_bind=false. Setting this to false instructs the broker to NOT run the APB image, but to return any credentials that were created during the provision if there were any.

Thanks for your clarification,
Based on comment 25 & comment 27, marked as verified.