Bug 1554145

Summary: [apb] Newer version of APB tool fails with `apb remove` on a 3.7 version of broker
Product: OpenShift Container Platform Reporter: daniel <dmoessne>
Component: Service BrokerAssignee: Dylan Murray <dymurray>
Status: CLOSED ERRATA QA Contact: Jian Zhang <jiazha>
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.7.0CC: aos-bugs, chezhang, dmoessne, dymurray, dzhukous, hongkliu, jiazha, jmatthew, rfoyle, smunilla, zitang
Target Milestone: ---   
Target Release: 3.9.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Added a small compatibility check to eliminate a pain point with API endpoints changing from 3.7->3.9.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2018-06-27 18:01:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description daniel 2018-03-11 21:28:17 UTC 
Description of problem:
When trying to run apb push from remote host (not part of ocp cluster) the following error meaasage is shown, no matter if cluster admin or not:
~~~
Exception deleting old images: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Date': 'Sun, 11 Mar 2018 16:39:28 GMT', 'Content-Length': '110', 'Content-Type': 'text/plain', 'X-Content-Type-Options': 'nosniff', 'Cache-Control': 'no-store'})
HTTP response body: User "quicklab" cannot list images at the cluster scope: User "quicklab" cannot list all images in the cluster
~~~
Not erroring out, this may cause duplicate images in the registry. Try: `oc get images`.

Version-Release number of selected component (if applicable):
openshift v3.7.23
kubernetes v1.7.6+a08f5eeb62
apb-1.1.15-1.20180305205029.el7.centos.noarch 

How reproducible:


Steps to Reproduce:
1. Install openshift with asb
2. install apb from EPEL (see https://bugzilla.redhat.com/show_bug.cgi?id=1554138) on remote machine (fedora 27)
3. create none admin user : quicklab
4. create admin user 
5. add permissions as follows:
- follow https://github.com/ansibleplaybookbundle/ansible-playbook-bundle/blob/master/docs/apb_cli.md#test-apb-tooling
- wget https://raw.githubusercontent.com/ansibleplaybookbundle/ansible-playbook-bundle/master/templates/openshift-permissions.template.yaml
- cp openshift-permissions.template.yaml openshift-permissions.template2.yaml
- change template as follows
- # diff -u openshift-permissions.template.yaml openshift-permissions.template2.yaml
--- openshift-permissions.template.yaml 2018-03-11 06:52:30.334924794 -0400
+++ openshift-permissions.template2.yaml        2018-03-11 06:40:42.738924794 -0400
@@ -1,6 +1,6 @@
 # Template used for permissioning a user to allow full apb tooling authorization
-# Primarily used by the ansible-service-broker minishift addon:
-# https://github.com/minishift/minishift-addons/tree/master/add-ons/ansible-service-broker
+# Primarily used by the openshift-ansible-service-broker minishift addon:
+# https://github.com/minishift/minishift-addons/tree/master/add-ons/openshift-ansible-service-broker
 apiVersion: v1
 kind: Template
 metadata:
@@ -18,10 +18,10 @@
     name: ${USER}
   roleRef:
     kind: ClusterRole
-    name: access-asb-role
+    name: asb-access
     apiGroup: rbac.authorization.k8s.io
 ################################################################################
-# apb must be able to view the ansible-service-broker project
+# apb must be able to view the openshift-ansible-service-broker project
 ################################################################################
 - apiVersion: authorization.openshift.io/v1
   kind: RoleBinding
@@ -122,7 +122,7 @@
 - displayname: Broker Namespace
   description: The Ansible Broker's namespace
   name: BROKER_NAMESPACE
-  value: ansible-service-broker
+  value: openshift-ansible-service-broker
 - displayname: Global Image Project
   description: Global Image Project
   name: GLOBAL_IMAGE_PROJECT

- # oc process -f openshift-permissions.template2.yaml -p USER=quicklab |oc create -f -

6 on remote host
- # oc login -u quicklab -p <password>
- # oc whoami -t (check if we got a token)
- # apb list --broker  https://asb-1338-openshift-ansible-service-broker.apps.example.com 
  (check if we get the list of available apbs - not working w/o step 5)
- # oc project test04-apb
- following https://blog.openshift.com/apb-development-tutorial-mediawiki-1-23-postgresql-9-5/
- create template as outlined in above docu
- take https://bugzilla.redhat.com/show_bug.cgi?id=1526147#c5 into consideration 
- # docker login docker-registry-default.apps.example.com -u quicklab -p $(oc whoami -t)
- # apb build --tag docker-registry-default.apps.example.com/openshift/test04-apb
- # apb push --registry-route docker-registry-default.apps.example.com --namespace test04-apb --broker https://asb-1338-openshift-ansible-service-broker.apps.example.com
[...]

Actual results:
Finished writing dockerfile.
Building APB using tag: [docker-registry-default.apps.example.com/test04-apb/mediawiki-apb]
Successfully built APB image: docker-registry-default.apps.example.com/test04-apb/mediawiki-apb
Exception deleting old images: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Date': 'Sun, 11 Mar 2018 16:39:28 GMT', 'Content-Length': '110', 'Content-Type': 'text/plain', 'X-Content-Type-Options': 'nosniff', 'Cache-Control': 'no-store'})
HTTP response body: User "quicklab" cannot list images at the cluster scope: User "quicklab" cannot list all images in the cluster

Not erroring out, this may cause duplicate images in the registry. Try: `oc get images`.
Pushing the image, this could take a minute...
Successfully pushed image: docker-registry-default.apps.example.com/test04-apb/mediawiki-apb
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.example.com/ansible-service-broker/v2/bootstrap
Successfully bootstrapped Ansible Service Broker
Successfully relisted the Service Catalog

Expected results:
- works without error messages 


Additional info:
 config map:
[...]
registry:
  - type: rhcc
    name: rh
    url:  https://registry.access.redhat.com
    org:  
    tag:  v3.7
    white_list: [.*-apb$]

  - type: local_openshift
    name: localregistry
    namespaces: ['openshift']
    white_list: [.*-apb$]
[...]
broker:
  dev_broker: true
  bootstrap_on_startup: true
  refresh_interval: 600s
  launch_apb_on_bind: false
  output_request: false
  recovery: true
  ssl_cert_key: /etc/tls/private/tls.key
  ssl_cert: /etc/tls/private/tls.crt
  auto_escalate: False
  auth:
    - type: basic
      enabled: false
[...]

full protocol:
[root@frodo mediawiki-apb]# apb list --broker  https://asb-1338-openshift-ansible-service-broker.apps.example.com
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.example.com/ansible-service-broker/v2/catalog
ID                                NAME               DESCRIPTION                        
2c259ddd8059b9bc65081e07bf20058f  rh-mariadb-apb     Mariadb apb implementation         
03b69500305d9859bb9440d9f9023784  rh-mediawiki-apb   Mediawiki123 apb implementation    
73ead67495322cc462794387fa9884f5  rh-mysql-apb       Software Collections MySQL APB     
d5915e05b253df421efe6e41fb6a66ba  rh-postgresql-apb  SCL PostgreSQL apb implementation  
[root@frodo mediawiki-apb]# 
[root@frodo mediawiki-apb]# oc whoami
quicklab
[root@frodo mediawiki-apb]# oc project 
Using project "test04-apb" on server "https://openshift.example.com:443".
[root@frodo mediawiki-apb]# 
[root@frodo mediawiki-apb]# docker login docker-registry-default.apps.example.com -u quicklab -p $(oc whoami -t)
Login Succeeded
[root@frodo mediawiki-apb]# apb build --tag docker-registry-default.apps.example.com/openshift/test04-apb
Finished writing dockerfile.
Building APB using tag: [docker-registry-default.apps.example.com/openshift/test04-apb]
Successfully built APB image: docker-registry-default.apps.example.com/openshift/test04-apb
[root@frodo mediawiki-apb]# 
[root@frodo mediawiki-apb]# apb push --registry-route docker-registry-default.apps.example.com --namespace test04-apb --broker https://asb-1338-openshift-ansible-service-broker.apps.example.com
# apb.yml
version: 1.0
name: mediawiki-apb
description: This APB deploys Mediawiki123.
bindable: False
async: optional
metadata:
  displayName: Mediawiki (APB)
plans:
  - name: default
    description: This plan deploys a Mediawiki instance
    free: True
    metadata: {}
    parameters:
      - name: mediawiki_db_schema
        default: mediawiki
        type: string
        title: Mediawiki DB Schema
        required: True
      - name: mediawiki_site_name
        default: MediaWiki
        type: string
        title: Mediawiki Site Name
        required: True
      - name: mediawiki_site_lang
        default: en
        type: string
        title: Mediawiki Site Language
        required: True
      - name: mediawiki_admin_user
        default: admin
        type: string
        title: Mediawiki Admin User
        required: True
      - name: mediawiki_admin_pass
        type: string
        title: Mediawiki Admin User Password
        required: True

Finished writing dockerfile.
Building APB using tag: [docker-registry-default.apps.example.com/test04-apb/mediawiki-apb]
Successfully built APB image: docker-registry-default.apps.example.com/test04-apb/mediawiki-apb
Exception deleting old images: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Date': 'Sun, 11 Mar 2018 16:39:28 GMT', 'Content-Length': '110', 'Content-Type': 'text/plain', 'X-Content-Type-Options': 'nosniff', 'Cache-Control': 'no-store'})
HTTP response body: User "quicklab" cannot list images at the cluster scope: User "quicklab" cannot list all images in the cluster

Not erroring out, this may cause duplicate images in the registry. Try: `oc get images`.
Pushing the image, this could take a minute...
Successfully pushed image: docker-registry-default.apps.example.com/test04-apb/mediawiki-apb
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.example.com/ansible-service-broker/v2/bootstrap
Successfully bootstrapped Ansible Service Broker
Successfully relisted the Service Catalog
[root@frodo mediawiki-apb]#


- tried this also with a cluster admin user to check if this could be a permission issue, but same result, even complaining about user quicklab:

----------------------------
[root@frodo mediawiki-apb]# oc whoami
admin
[root@frodo mediawiki-apb]# oc whoami -t 
_mz-mYB2qfJPpAc1AvxiQH5dYpL_NYuG1BFnxwQHRnc
[root@frodo mediawiki-apb]# apb list --broker  https://asb-1338-openshift-ansible-service-broker.apps.example.com
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.example.com/ansible-service-broker/v2/catalog
ID                                NAME               DESCRIPTION                        
2c259ddd8059b9bc65081e07bf20058f  rh-mariadb-apb     Mariadb apb implementation         
03b69500305d9859bb9440d9f9023784  rh-mediawiki-apb   Mediawiki123 apb implementation    
73ead67495322cc462794387fa9884f5  rh-mysql-apb       Software Collections MySQL APB     
d5915e05b253df421efe6e41fb6a66ba  rh-postgresql-apb  SCL PostgreSQL apb implementation  
[root@frodo mediawiki-apb]# docker login docker-registry-default.apps.example.com -u admin -p $(oc whoami -t)
Login Succeeded
[root@frodo mediawiki-apb]# apb build --tag docker-registry-default.apps.example.com/openshift/test04-apb
Finished writing dockerfile.
Building APB using tag: [docker-registry-default.apps.example.com/openshift/test04-apb]
Successfully built APB image: docker-registry-default.apps.example.com/openshift/test04-apb
[root@frodo mediawiki-apb]# apb push --registry-route docker-registry-default.apps.example.com --namespace test04-apb --broker https://asb-1338-openshift-ansible-service-broker.apps.example.com
# apb.yml
version: 1.0
name: mediawiki-apb
description: This APB deploys Mediawiki123.
bindable: False
async: optional
metadata:
  displayName: Mediawiki (APB)
plans:
  - name: default
    description: This plan deploys a Mediawiki instance
    free: True
    metadata: {}
    parameters:
      - name: mediawiki_db_schema
        default: mediawiki
        type: string
        title: Mediawiki DB Schema
        required: True
      - name: mediawiki_site_name
        default: MediaWiki
        type: string
        title: Mediawiki Site Name
        required: True
      - name: mediawiki_site_lang
        default: en
        type: string
        title: Mediawiki Site Language
        required: True
      - name: mediawiki_admin_user
        default: admin
        type: string
        title: Mediawiki Admin User
        required: True
      - name: mediawiki_admin_pass
        type: string
        title: Mediawiki Admin User Password
        required: True

Finished writing dockerfile.
Building APB using tag: [docker-registry-default.apps.example.com/test04-apb/mediawiki-apb]
Successfully built APB image: docker-registry-default.apps.example.com/test04-apb/mediawiki-apb
Exception deleting old images: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Date': 'Sun, 11 Mar 2018 21:21:08 GMT', 'Content-Length': '110', 'Content-Type': 'text/plain', 'X-Content-Type-Options': 'nosniff', 'Cache-Control': 'no-store'})
HTTP response body: User "quicklab" cannot list images at the cluster scope: User "quicklab" cannot list all images in the cluster

Not erroring out, this may cause duplicate images in the registry. Try: `oc get images`.
Pushing the image, this could take a minute...
Successfully pushed image: docker-registry-default.apps.example.com/test04-apb/mediawiki-apb
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.example.com/ansible-service-broker/v2/bootstrap
Successfully bootstrapped Ansible Service Broker
Successfully relisted the Service Catalog
[root@frodo mediawiki-apb]# apb push --registry-route docker-registry-default.apps.example.com --namespace test04-apb --broker https://asb-1338-openshift-ansible-service-broker.apps.example.com
# apb.yml
version: 1.0
name: mediawiki-apb
description: This APB deploys Mediawiki123.
bindable: False
async: optional
metadata:
  displayName: Mediawiki (APB)
plans:
  - name: default
    description: This plan deploys a Mediawiki instance
    free: True
    metadata: {}
    parameters:
      - name: mediawiki_db_schema
        default: mediawiki
        type: string
        title: Mediawiki DB Schema
        required: True
      - name: mediawiki_site_name
        default: MediaWiki
        type: string
        title: Mediawiki Site Name
        required: True
      - name: mediawiki_site_lang
        default: en
        type: string
        title: Mediawiki Site Language
        required: True
      - name: mediawiki_admin_user
        default: admin
        type: string
        title: Mediawiki Admin User
        required: True
      - name: mediawiki_admin_pass
        type: string
        title: Mediawiki Admin User Password
        required: True

Finished writing dockerfile.
Building APB using tag: [docker-registry-default.apps.example.com/test04-apb/mediawiki-apb]
Successfully built APB image: docker-registry-default.apps.example.com/test04-apb/mediawiki-apb
Exception deleting old images: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Date': 'Sun, 11 Mar 2018 21:21:38 GMT', 'Content-Length': '110', 'Content-Type': 'text/plain', 'X-Content-Type-Options': 'nosniff', 'Cache-Control': 'no-store'})
HTTP response body: User "quicklab" cannot list images at the cluster scope: User "quicklab" cannot list all images in the cluster

Not erroring out, this may cause duplicate images in the registry. Try: `oc get images`.
Pushing the image, this could take a minute...
Successfully pushed image: docker-registry-default.apps.example.com/test04-apb/mediawiki-apb
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.example.com/ansible-service-broker/v2/bootstrap
Error: Attempt to bootstrap Broker returned status: 403
Unable to bootstrap Ansible Service Broker.
[root@frodo mediawiki-apb]# 

----------------------------
Comment 1 daniel 2018-03-11 21:30:09 UTC 
also even if it seems push was successful, apb list dos not show pushed image
Comment 2 Dylan Murray 2018-03-14 13:30:03 UTC 
Daniel,

I see a few problems with the commands you are typing. You did not change GLOBAL_IMAGE_PROJECT yet you are passing in test04-apb as the --namespace argument. By default GLOBAL_IMAGE_PROJECT is set to `openshift`. The openshift namespace is special because any authenticated user can view images in this namespace by default. (You can check this by doing `oc get images`). I *think* the issue is that namespace test04-apb is NOT configured to expose images at the cluster level like the openshfit namespace is. When the tooling attempts to delete an image that would theoretically live in namespace test04-apb (docker-registry-default.apps.example.com/test04-apb/mediawiki-apb) it fails because the user cannot get images from that namespace.


Additionally, the openshift-permissions template allows you to never have to use the --broker flag since you can access the openshift-ansible-service-broker namespace to get the route.

Can you please output the return of: `apb push --registry-route docker-registry-default.apps.example.com`?
Comment 3 Jian Zhang 2018-03-15 05:47:24 UTC 
Daniel,

The usage of `apb push` for your reference.
The apb tool version: 1.1.15

1) Login the remote cluster as a usual user. And then, give the cluster-admin role to this user.
#oc adm policy add-cluster-role-to-user cluster-admin xxx

2) Get the route of the cluster default registry. And add it to your docker INSECURE_REGISTRY. Restart docker.

[root@localhost hello-world-apb]# oc get route -n default
NAME               HOST/PORT                                               PATH      SERVICES           PORT      TERMINATION   WILDCARD
docker-registry    docker-registry-default.apps.0314-p5u.qe.rhcloud.com              docker-registry    <all>     passthrough   None

[root@localhost hello-world-apb]# vim /etc/sysconfig/docker
INSECURE_REGISTRY='--insecure-registry docker-registry-default.apps.0314-p5u.qe.rhcloud.com ...
[root@localhost hello-world-apb]# systemctl restart docker


3) Config the `white_list` of local_openshift as below so that can list the pushed images.
registry:
  - type: rhcc
    name: rh
    url:  https://registry.access.redhat.com
    org:  
    tag:  v3.7
    white_list: [.*-apb$]

  - type: local_openshift
    name: localregistry
    namespaces: ['openshift']
    white_list: [.*]

4) Run the `apb push --registry-route xxx`, the default namespace is the "openshift", no need to specify the `--broker`.

[root@localhost hello-world-apb]# apb push --registry-route docker-registry-default.apps.0314-p5u.qe.rhcloud.com
Didn't find OpenShift Ansible Broker route in namespace: ansible-service-broker.                    Trying openshift-ansible-service-broker
version: 1.0
name: hello-world-apb
description: deploys hello-world web application
bindable: False
async: optional
metadata:
  displayName: Hello World Jian (APB)
  longDescription: A sample APB which deploys a containerized Hello World web application
  dependencies: ['docker.io/ansibleplaybookbundle/hello-world:latest']
  providerDisplayName: "Red Hat, Inc."
plans:
  - name: default
    description: A sample APB which deploys Hello World
    free: True
    metadata:
      displayName: Default
      longDescription: This plan deploys a Python web application displaying Hello World
      cost: $0.00
    parameters: []

Finished writing dockerfile.
Building APB using tag: [docker-registry-default.apps.0314-p5u.qe.rhcloud.com/openshift/hello-world-apb]
Successfully built APB image: docker-registry-default.apps.0314-p5u.qe.rhcloud.com/openshift/hello-world-apb
Pushing the image, this could take a minute...
Successfully pushed image: docker-registry-default.apps.0314-p5u.qe.rhcloud.com/openshift/hello-world-apb
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.0314-p5u.qe.rhcloud.com/ansible-service-broker/v2/bootstrap
Successfully bootstrapped Ansible Service Broker
Successfully relisted the Service Catalog

5) Run the `apb list`, then we can see the pushed images.

[root@localhost hello-world-apb]# apb list
Didn't find OpenShift Ansible Broker route in namespace: ansible-service-broker.                    Trying openshift-ansible-service-broker
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.0314-p5u.qe.rhcloud.com/ansible-service-broker/v2/catalog
ID                                NAME                           DESCRIPTION                          
30ba0018270996447917c32b0d6c4e6b  localregistry-hello-world-apb  deploys hello-world web application  
2c259ddd8059b9bc65081e07bf20058f  rh-mariadb-apb                 Mariadb apb implementation           
03b69500305d9859bb9440d9f9023784  rh-mediawiki-apb               Mediawiki123 apb implementation      
73ead67495322cc462794387fa9884f5  rh-mysql-apb                   Software Collections MySQL APB       
d5915e05b253df421efe6e41fb6a66ba  rh-postgresql-apb              SCL PostgreSQL apb implementation 

We can also see this image in the cluster.

[root@host-172-16-120-32 ~]# oc get images | grep hello
sha256:7418e5b141ff78166c6c0878aff78be7c175b6153e9b89c7d96f4a67b55c8e84   docker-registry.default.svc:5000/openshift/hello-world-apb@sha256:7418e5b141ff78166c6c0878aff78be7c175b6153e9b89c7d96f4a67b55c8e84

[root@host-172-16-120-32 ~]# oc get imagestream -n openshift | grep hello
hello-world-apb                       docker-registry.default.svc:5000/openshift/hello-world-apb                       latest                       21 minutes ago


Additional info:
Will get the 403 errors(as the same as yours) when run the `apb push` if login the cluster as a usual user, details:

[root@host-172-16-120-32 ~]# oc adm policy remove-cluster-role-from-user cluster-admin jiazha
cluster role "cluster-admin" removed: "jiazha"

[root@localhost hello-world-apb]# apb push --registry-route docker-registry-default.apps.0314-p5u.qe.rhcloud.com
version: 1.0
name: hello-world-apb
description: deploys hello-world web application
bindable: False
async: optional
metadata:
  displayName: Hello World Jian (APB)
  longDescription: A sample APB which deploys a containerized Hello World web application
  dependencies: ['docker.io/ansibleplaybookbundle/hello-world:latest']
  providerDisplayName: "Red Hat, Inc."
plans:
  - name: default
    description: A sample APB which deploys Hello World
    free: True
    metadata:
      displayName: Default
      longDescription: This plan deploys a Python web application displaying Hello World
      cost: $0.00
    parameters: []

Finished writing dockerfile.
Building APB using tag: [docker-registry-default.apps.0314-p5u.qe.rhcloud.com/openshift/hello-world-apb]
Successfully built APB image: docker-registry-default.apps.0314-p5u.qe.rhcloud.com/openshift/hello-world-apb
Exception deleting old images: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Date': 'Thu, 15 Mar 2018 05:43:17 GMT', 'Content-Length': '106', 'Content-Type': 'text/plain', 'X-Content-Type-Options': 'nosniff', 'Cache-Control': 'no-store'})
HTTP response body: User "jiazha" cannot list images at the cluster scope: User "jiazha" cannot list all images in the cluster

Not erroring out, this may cause duplicate images in the registry. Try: `oc get images`.
Pushing the image, this could take a minute...
Successfully pushed image: docker-registry-default.apps.0314-p5u.qe.rhcloud.com/openshift/hello-world-apb
Exception occurred! Could not find route to ansible-service-broker. Use --broker or log into the cluster using "oc login"
Comment 4 Zhang Cheng 2018-03-15 08:16:29 UTC 
QE tested both in ocp3.7 + apb 1.0.4 and ocp3.7 + apb 1.1.15, apb push function work well inside cluster, will hit a known issue while running outside cluster, https://bugzilla.redhat.com/show_bug.cgi?id=1526147
Comment 5 Jian Zhang 2018-03-15 08:28:36 UTC 
Daniel,

For the 1.0.4 version of the apb tool, the `apb push` does not work when running outside of the cluster, there is a bug for this: https://bugzilla.redhat.com/show_bug.cgi?id=1526147. But, the `apb push` works well when running inside of the cluster, below steps for your reference.

[root@host-172-16-120-128 ~]# rpm -qa | grep apb
apb-1.0.4-1.el7.noarch

1) In the cluster, log in as a cluster-admin user.
[root@host-172-16-120-128 ~]# oc adm policy add-cluster-role-to-user cluster-admin jiazha(replace it with yours)
cluster role "cluster-admin" added: "jiazha"
[root@host-172-16-120-128 ~]# oc login https://host-8-241-83.host.centralci.eng.rdu2.redhat.com:8443
Authentication required for https://host-8-241-83.host.centralci.eng.rdu2.redhat.com:8443 (openshift)
Username: jiazha
Password: 
Login successful.

2) Config the "white_list" of the local_openshift so that you can list the images successfully. And, rollout the ASB pod.
registry:
  - type: rhcc
    name: rh
    url:  https://registry.access.redhat.com
    org:  
    tag:  v3.7
    white_list: [.*-apb$]

  - type: local_openshift
    name: localregistry
    namespaces: ['openshift']
    white_list: [.*]

3) Push the image as below:
[root@host-172-16-120-128 hello-world-apb]# apb push --openshift --broker https://asb-1338-openshift-ansible-service-broker.apps.0315-9tr.qe.rhcloud.com/ansible-service-broker
version: 1.0
name: hello-world-apb
description: deploys hello-world web application
bindable: False
async: optional
metadata:
  displayName: Hello World (APB)
  longDescription: A sample APB which deploys a containerized Hello World web application
  dependencies: ['docker.io/ansibleplaybookbundle/hello-world:latest']
  providerDisplayName: "Red Hat, Inc."
plans:
  - name: default
    description: A sample APB which deploys Hello World
    free: True
    metadata:
      displayName: Default
      longDescription: This plan deploys a Python web application displaying Hello World
      cost: $0.00
    parameters: []

Found registry IP at: 172.30.230.160:5000
Building image with the tag: 172.30.230.160:5000/openshift/hello-world-apb
Successfully pushed image: 172.30.230.160:5000/openshift/hello-world-apb
https://asb-1338-openshift-ansible-service-broker.apps.0315-9tr.qe.rhcloud.com/ansible-service-broker
Successfully bootstrapped Ansible Service Broker
Successfully relisted the Service Catalog

4) Check the pushed image and list them.

[root@host-172-16-120-128 hello-world-apb]# oc get images | grep hello
sha256:d738149574aef142bc8d19087f8b43b606aab677d13260af010db71c6c932c10   docker-registry.default.svc:5000/openshift/hello-world-apb@sha256:d738149574aef142bc8d19087f8b43b606aab677d13260af010db71c6c932c10

[root@host-172-16-120-128 hello-world-apb]# apb list --broker https://asb-1338-openshift-ansible-service-broker.apps.0315-9tr.qe.rhcloud.com/ansible-service-broker
ID                                NAME                           DESCRIPTION                          
30ba0018270996447917c32b0d6c4e6b  localregistry-hello-world-apb  deploys hello-world web application  
2c259ddd8059b9bc65081e07bf20058f  rh-mariadb-apb                 Mariadb apb implementation           
03b69500305d9859bb9440d9f9023784  rh-mediawiki-apb               Mediawiki123 apb implementation      
73ead67495322cc462794387fa9884f5  rh-mysql-apb                   Software Collections MySQL APB       
d5915e05b253df421efe6e41fb6a66ba  rh-postgresql-apb              SCL PostgreSQL apb implementation   

So, for 1.0.4 and 1.1.15, both of them work well.
Comment 6 daniel 2018-03-18 08:56:42 UTC 
(In reply to Dylan Murray from comment #2)

> 
> Can you please output the return of: `apb push --registry-route
> docker-registry-default.apps.example.com`?

Dylan,

please find that below:

used cluster admin role user for all the tests

apb from openshift repo:
~~~~
[root@test01 mediawiki-apb]# apb version
usage: apb [-h] [--debug] [--project BASE_PATH]
           {relist,init,test,help,prepare,push,bootstrap,list,remove,build}
           ...
apb: error: argument subcommand: invalid choice: 'version' (choose from 'relist', 'init', 'test', 'help', 'prepare', 'push', 'bootstrap', 'list', 'remove', 'build')
[root@test01 mediawiki-apb]# 

[root@test01 mediawiki-apb]# apb push --broker https://asb-1338-openshift-ansible-service-broker.apps.dmoessneapb2.example.com/ansible-service-broker 
version: 1.0
name: mediawiki-apb
description: This is a sample application generated by apb init
bindable: False
async: optional
metadata:
  displayName: mediawiki
plans:
  - name: default
    description: This default plan deploys mediawiki-apb
    free: True
    metadata: {}
    parameters: []
Error: Attempt to add APB to the Broker returned status: 404
Unable to add APB to Ansible Service Broker.
[root@test01 mediawiki-apb]# 

[root@test01 mediawiki-apb]# apb push --broker https://asb-1338-openshift-ansible-service-broker.apps.dmoessneapb2.example.com/ansible-service-broker  --openshift
version: 1.0
name: mediawiki-apb
description: This is a sample application generated by apb init
bindable: False
async: optional
metadata:
  displayName: mediawiki
plans:
  - name: default
    description: This default plan deploys mediawiki-apb
    free: True
    metadata: {}
    parameters: []
Found registry IP at: 172.30.170.177:5000
Building image with the tag: 172.30.170.177:5000/openshift/mediawiki-apb
Error accessing the docker API. Is the daemon running?
Exception occurred! 500 Server Error: Internal Server Error ("Get https://172.30.170.177:5000/v1/users/: dial tcp 172.30.170.177:5000: getsockopt: no route to host")
[root@test01 mediawiki-apb]# 
~~~~
I found nothing how to add the route to registry in the help 


apb from epel repo:
~~~~
[root@test02 mediawiki-apb]# apb version
Version: apb-1.1.15
[root@test02 mediawiki-apb]# 
[root@test02 mediawiki-apb]# apb push --registry-route docker-registry-default.apps.dmoessneapb2.example.com
Didn't find OpenShift Ansible Broker route in namespace: ansible-service-broker.                    Trying openshift-ansible-service-broker
version: 1.0
name: mediawiki-apb
description: This is a sample application generated by apb init
bindable: False
async: optional
metadata:
  displayName: mediawiki
plans:
  - name: default
    description: This default plan deploys mediawiki-apb
    free: True
    metadata: {}
    parameters: []
Finished writing dockerfile.
Building APB using tag: [docker-registry-default.apps.dmoessneapb2.example.com/openshift/mediawiki-apb]
Successfully built APB image: docker-registry-default.apps.dmoessneapb2.example.com/openshift/mediawiki-apb
Pushing the image, this could take a minute...
Successfully pushed image: docker-registry-default.apps.dmoessneapb2.example.com/openshift/mediawiki-apb
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.dmoessneapb2.example.com/ansible-service-broker/v2/bootstrap
Successfully bootstrapped Ansible Service Broker
Successfully relisted the Service Catalog
[root@test02 mediawiki-apb]# 
[root@test02 mediawiki-apb]# apb push --registry-route docker-registry-default.apps.dmoessneapb2.example.com --namespace openshift 
Didn't find OpenShift Ansible Broker route in namespace: ansible-service-broker.                    Trying openshift-ansible-service-broker
version: 1.0
name: mediawiki-apb
description: This is a sample application generated by apb init
bindable: False
async: optional
metadata:
  displayName: mediawiki
plans:
  - name: default
    description: This default plan deploys mediawiki-apb
    free: True
    metadata: {}
    parameters: []
Finished writing dockerfile.
Building APB using tag: [docker-registry-default.apps.dmoessneapb2.example.com/openshift/mediawiki-apb]
Successfully built APB image: docker-registry-default.apps.dmoessneapb2.example.com/openshift/mediawiki-apb
Pushing the image, this could take a minute...
Successfully pushed image: docker-registry-default.apps.dmoessneapb2.example.com/openshift/mediawiki-apb
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.dmoessneapb2.example.com/ansible-service-broker/v2/bootstrap
Successfully bootstrapped Ansible Service Broker
Successfully relisted the Service Catalog
[root@test02 mediawiki-apb]#
~~~~

however, even if pushed this way I cannot list it afterwards:
~~~~
[root@test02 mediawiki-apb]# apb list 
Didn't find OpenShift Ansible Broker route in namespace: ansible-service-broker.                    Trying openshift-ansible-service-broker
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.dmoessneapb2.example.com/ansible-service-broker/v2/catalog
ID                                NAME               DESCRIPTION                        
2c259ddd8059b9bc65081e07bf20058f  rh-mariadb-apb     Mariadb apb implementation         
03b69500305d9859bb9440d9f9023784  rh-mediawiki-apb   Mediawiki123 apb implementation    
73ead67495322cc462794387fa9884f5  rh-mysql-apb       Software Collections MySQL APB     
d5915e05b253df421efe6e41fb6a66ba  rh-postgresql-apb  SCL PostgreSQL apb implementation  
[root@test02 mediawiki-apb]# 
[root@test02 mediawiki-apb]# apb bootstrap
Didn't find OpenShift Ansible Broker route in namespace: ansible-service-broker.                    Trying openshift-ansible-service-broker
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.dmoessneapb2.example.com/ansible-service-broker/v2/bootstrap
Successfully bootstrapped Ansible Service Broker
Successfully relisted the Service Catalog
[root@test02 mediawiki-apb]# apb list 
Didn't find OpenShift Ansible Broker route in namespace: ansible-service-broker.                    Trying openshift-ansible-service-broker
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.dmoessneapb2.example.com/ansible-service-broker/v2/catalog
ID                                NAME               DESCRIPTION                        
2c259ddd8059b9bc65081e07bf20058f  rh-mariadb-apb     Mariadb apb implementation         
03b69500305d9859bb9440d9f9023784  rh-mediawiki-apb   Mediawiki123 apb implementation    
73ead67495322cc462794387fa9884f5  rh-mysql-apb       Software Collections MySQL APB     
d5915e05b253df421efe6e41fb6a66ba  rh-postgresql-apb  SCL PostgreSQL apb implementation  
[root@test02 mediawiki-apb]# 
~~~~
Comment 7 daniel 2018-03-18 10:28:07 UTC 
[root@test02 tiki-apb]# apb version 
Version: apb-1.1.15
[root@test02 tiki-apb]# oc whoami 
user1  # user with cluster admin role
[root@test02 tiki-apb]# 

[root@test02 hello-world-apb]# apb build --tag docker-registry-default.apps.dmoessneapb2.example.com/openshift
Finished writing dockerfile.
Building APB using tag: [docker-registry-default.apps.dmoessneapb2.example.com/openshift]
Successfully built APB image: docker-registry-default.apps.dmoessneapb2.example.com/openshift
[root@test02 hello-world-apb]# 
[root@test02 hello-world-apb]# apb push --registry-route docker-registry-default.apps.dmoessneapb2.example.com 
Didn't find OpenShift Ansible Broker route in namespace: ansible-service-broker.                    Trying openshift-ansible-service-broker
version: 1.0
name: hello-world-apb
description: deploys hello-world web application
bindable: False
async: optional
metadata:
  displayName: Hello World (APB)
  longDescription: A sample APB which deploys a containerized Hello World web application
  dependencies: ['docker.io/ansibleplaybookbundle/hello-world:latest']
  providerDisplayName: "Red Hat, Inc."
plans:
  - name: default
    description: A sample APB which deploys Hello World
    free: True
    metadata:
      displayName: Default
      longDescription: This plan deploys a Python web application displaying Hello World
      cost: $0.00
    parameters: []

Finished writing dockerfile.
Building APB using tag: [docker-registry-default.apps.dmoessneapb2.example.com/openshift/hello-world-apb]
Successfully built APB image: docker-registry-default.apps.dmoessneapb2.example.com/openshift/hello-world-apb
Pushing the image, this could take a minute...
Successfully pushed image: docker-registry-default.apps.dmoessneapb2.example.com/openshift/hello-world-apb
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.dmoessneapb2.example.com/ansible-service-broker/v2/bootstrap
Successfully bootstrapped Ansible Service Broker
Successfully relisted the Service Catalog
[root@test02 hello-world-apb]# apb list 
Didn't find OpenShift Ansible Broker route in namespace: ansible-service-broker.                    Trying openshift-ansible-service-broker
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.dmoessneapb2.example.com/ansible-service-broker/v2/catalog
ID                                NAME                           DESCRIPTION                          
30ba0018270996447917c32b0d6c4e6b  localregistry-hello-world-apb  deploys hello-world web application  
2c259ddd8059b9bc65081e07bf20058f  rh-mariadb-apb                 Mariadb apb implementation           
03b69500305d9859bb9440d9f9023784  rh-mediawiki-apb               Mediawiki123 apb implementation      
73ead67495322cc462794387fa9884f5  rh-mysql-apb                   Software Collections MySQL APB       
d5915e05b253df421efe6e41fb6a66ba  rh-postgresql-apb              SCL PostgreSQL apb implementation    
[root@test02 hello-world-apb]# 

doing the same for mediawiki:
[root@test02 mediawiki-apb]# apb build --tag docker-registry-default.apps.dmoessneapb2.example.com/openshift
Finished writing dockerfile.
Building APB using tag: [docker-registry-default.apps.dmoessneapb2.example.com/openshift]
Successfully built APB image: docker-registry-default.apps.dmoessneapb2.example.com/openshift
[root@test02 mediawiki-apb]# 
[root@test02 mediawiki-apb]# apb push --registry-route docker-registry-default.apps.dmoessneapb2.example.com
Didn't find OpenShift Ansible Broker route in namespace: ansible-service-broker.                    Trying openshift-ansible-service-broker
version: 1.0
name: mediawiki-apb
description: This is a sample application generated by apb init
bindable: False
async: optional
metadata:
  displayName: mediawiki
plans:
  - name: default
    description: This default plan deploys mediawiki-apb
    free: True
    metadata: {}
    parameters: []
Finished writing dockerfile.
Building APB using tag: [docker-registry-default.apps.dmoessneapb2.example.com/openshift/mediawiki-apb]
Successfully built APB image: docker-registry-default.apps.dmoessneapb2.example.com/openshift/mediawiki-apb
Pushing the image, this could take a minute...
Successfully pushed image: docker-registry-default.apps.dmoessneapb2.example.com/openshift/mediawiki-apb
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.dmoessneapb2.example.com/ansible-service-broker/v2/bootstrap
Successfully bootstrapped Ansible Service Broker
Successfully relisted the Service Catalog
[root@test02 mediawiki-apb]# 
[root@test02 mediawiki-apb]# 
[root@test02 mediawiki-apb]# apb list 
Didn't find OpenShift Ansible Broker route in namespace: ansible-service-broker.                    Trying openshift-ansible-service-broker
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.dmoessneapb2.example.com/ansible-service-broker/v2/catalog
ID                                NAME                           DESCRIPTION                          
30ba0018270996447917c32b0d6c4e6b  localregistry-hello-world-apb  deploys hello-world web application  
2c259ddd8059b9bc65081e07bf20058f  rh-mariadb-apb                 Mariadb apb implementation           
03b69500305d9859bb9440d9f9023784  rh-mediawiki-apb               Mediawiki123 apb implementation      
73ead67495322cc462794387fa9884f5  rh-mysql-apb                   Software Collections MySQL APB       
d5915e05b253df421efe6e41fb6a66ba  rh-postgresql-apb              SCL PostgreSQL apb implementation    
[root@test02 mediawiki-apb]#


but after editing the mediawiki app i.e. replacing mediawiki by just wiki or mediatiki it works, and after editing broker cm like so

~~~~
registry:
  - type: rhcc
    name: rh
    url:  https://registry.access.redhat.com
    org:  
    tag:  v3.7
    white_list: [.*-apb$]

  - type: local_openshift
    name: localregistry
    namespaces: ['openshift','test01-apb']
    white_list: [.*]
~~~~

I am even able to see this in my test01-apb namespace

[root@test02 tiki-apb]# apb push --registry-route docker-registry-default.apps.dmoessneapb2.example.com --registry-namespace test01-apb
Didn't find OpenShift Ansible Broker route in namespace: ansible-service-broker.                    Trying openshift-ansible-service-broker
version: 1.0
name: mediatiki-apb
description: This is a sample application generated by apb init
bindable: False
async: optional
metadata:
  displayName: mediatiki
plans:
  - name: default
    description: This default plan deploys mediatiki-apb
    free: True
    metadata: {}
    parameters: []
Finished writing dockerfile.
Building APB using tag: [docker-registry-default.apps.dmoessneapb2.example.com/openshift/mediatiki-apb]
Successfully built APB image: docker-registry-default.apps.dmoessneapb2.example.com/openshift/mediatiki-apb
Pushing the image, this could take a minute...
Successfully pushed image: docker-registry-default.apps.dmoessneapb2.example.com/openshift/mediatiki-apb
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.dmoessneapb2.example.com/ansible-service-broker/v2/bootstrap
Successfully bootstrapped Ansible Service Broker
Successfully relisted the Service Catalog
[root@test02 tiki-apb]#

pushed that as well to openshift namespace:
[root@test02 tiki-apb]# apb list 
Didn't find OpenShift Ansible Broker route in namespace: ansible-service-broker.                    Trying openshift-ansible-service-broker
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.dmoessneapb2.example.com/ansible-service-broker/v2/catalog
ID                                NAME                           DESCRIPTION                                         
30ba0018270996447917c32b0d6c4e6b  localregistry-hello-world-apb  deploys hello-world web application                 
b82394a6f097d4c5358fc18c91e01869  localregistry-mediatiki-apb    This is a sample application generated by apb init  
38557f67b73882767fcc0f2d12b91baf  localregistry-prometheus-apb   Deploy Prometheus on your Project                   
c7b73951ab058437898a8dcfc03c344f  localregistry-wiki-apb         This is a sample application generated by apb init  
2c259ddd8059b9bc65081e07bf20058f  rh-mariadb-apb                 Mariadb apb implementation                          
03b69500305d9859bb9440d9f9023784  rh-mediawiki-apb               Mediawiki123 apb implementation                     
73ead67495322cc462794387fa9884f5  rh-mysql-apb                   Software Collections MySQL APB                      
d5915e05b253df421efe6e41fb6a66ba  rh-postgresql-apb              SCL PostgreSQL apb implementation                   
[root@test02 tiki-apb]# 

after editing cm:
[root@test02 tiki-apb]# apb list 
Didn't find OpenShift Ansible Broker route in namespace: ansible-service-broker.                    Trying openshift-ansible-service-broker
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.dmoessneapb2.example.com/ansible-service-broker/v2/catalog
ID                                NAME                           DESCRIPTION                                         
30ba0018270996447917c32b0d6c4e6b  localregistry-hello-world-apb  deploys hello-world web application                 
b82394a6f097d4c5358fc18c91e01869  localregistry-mediatiki-apb    This is a sample application generated by apb init  
c23ec213bb8dea1577230c5ce005b9c2  localregistry-mediawiki-apb    This is a sample application generated by apb init  
38557f67b73882767fcc0f2d12b91baf  localregistry-prometheus-apb   Deploy Prometheus on your Project                   
c7b73951ab058437898a8dcfc03c344f  localregistry-wiki-apb         This is a sample application generated by apb init  
2c259ddd8059b9bc65081e07bf20058f  rh-mariadb-apb                 Mariadb apb implementation                          
03b69500305d9859bb9440d9f9023784  rh-mediawiki-apb               Mediawiki123 apb implementation                     
73ead67495322cc462794387fa9884f5  rh-mysql-apb                   Software Collections MySQL APB                      
d5915e05b253df421efe6e41fb6a66ba  rh-postgresql-apb              SCL PostgreSQL apb implementation                   
[root@test02 tiki-apb]# 

--> it's there twice then
Comment 8 daniel 2018-03-18 10:29:02 UTC 
getting adventurous and trying with apb from ocp repo and pushing to different namespace:

 [root@test01 ~]# oc whoami 
user1
[root@test01 ~]# apb version
usage: apb [-h] [--debug] [--project BASE_PATH]
           {relist,init,test,help,prepare,push,bootstrap,list,remove,build}
           ...
apb: error: argument subcommand: invalid choice: 'version' (choose from 'relist', 'init', 'test', 'help', 'prepare', 'push', 'bootstrap', 'list', 'remove', 'build')
[root@test01 ~]# rpm -qa apb
apb-1.0.4-1.el7.noarch
[root@test01 ~]# 

[root@test01 ~]# git clone https://github.com/ansibleplaybookbundle/photo-album-demo.git
[root@test01 ~]# cd photo-album-demo/
[root@test01 photo-album-demo]# find . -name apb*
./api-apb/apb.yml
./app-apb/apb.yml
./ext-api-apb/apb.yml
[root@test01 photo-album-demo]# 
[root@test01 photo-album-demo]# cd api-apb/
[root@test01 api-apb]# apb build --tag docker-registry-default.apps.dmoessneapb2.example.com
[root@test01 api-apb]# apb push --broker https://asb-1338-openshift-ansible-service-broker.apps.dmoessneapb2.example.com/ansible-service-broker --namespace test01-apb 
[root@test01 api-apb]# apb list --broker https://asb-1338-openshift-ansible-service-broker.apps.dmoessneapb2.example.com/ansible-service-broker
ID                                NAME                           DESCRIPTION                                         
cb8f56dc6074c4949ad3bb7f164c224c  apb-push-demo-api-apb          Internal Dog API                                    
30ba0018270996447917c32b0d6c4e6b  localregistry-hello-world-apb  deploys hello-world web application                 
b82394a6f097d4c5358fc18c91e01869  localregistry-mediatiki-apb    This is a sample application generated by apb init  
c23ec213bb8dea1577230c5ce005b9c2  localregistry-mediawiki-apb    This is a sample application generated by apb init  
38557f67b73882767fcc0f2d12b91baf  localregistry-prometheus-apb   Deploy Prometheus on your Project                   
c7b73951ab058437898a8dcfc03c344f  localregistry-wiki-apb         This is a sample application generated by apb init  
2c259ddd8059b9bc65081e07bf20058f  rh-mariadb-apb                 Mariadb apb implementation                          
03b69500305d9859bb9440d9f9023784  rh-mediawiki-apb               Mediawiki123 apb implementation                     
73ead67495322cc462794387fa9884f5  rh-mysql-apb                   Software Collections MySQL APB                      
d5915e05b253df421efe6e41fb6a66ba  rh-postgresql-apb              SCL PostgreSQL apb implementation                   
[root@test01 api-apb]# 
[root@test01 api-apb]# cd ../app-apb
[root@test01 app-apb]# apb build --tag docker-registry-default.apps.dmoessneapb2.example.com
[root@test01 app-apb]# apb push --broker https://asb-1338-openshift-ansible-service-broker.apps.dmoessneapb2.example.com/ansible-service-broker --namespace test01-apb 
[root@test01 app-apb]# 
[root@test01 app-apb]# apb list --broker https://asb-1338-openshift-ansible-service-broker.apps.dmoessneapb2.example.com/ansible-service-broker
ID                                NAME                           DESCRIPTION                                          
cb8f56dc6074c4949ad3bb7f164c224c  apb-push-demo-api-apb          Internal Dog API                                     
9573e996d70d193ae3eeee589acb75ea  apb-push-demo-app-apb          Deploys a demo application showing random pictures.  
30ba0018270996447917c32b0d6c4e6b  localregistry-hello-world-apb  deploys hello-world web application                  
b82394a6f097d4c5358fc18c91e01869  localregistry-mediatiki-apb    This is a sample application generated by apb init   
c23ec213bb8dea1577230c5ce005b9c2  localregistry-mediawiki-apb    This is a sample application generated by apb init   
38557f67b73882767fcc0f2d12b91baf  localregistry-prometheus-apb   Deploy Prometheus on your Project                    
c7b73951ab058437898a8dcfc03c344f  localregistry-wiki-apb         This is a sample application generated by apb init   
2c259ddd8059b9bc65081e07bf20058f  rh-mariadb-apb                 Mariadb apb implementation                           
03b69500305d9859bb9440d9f9023784  rh-mediawiki-apb               Mediawiki123 apb implementation                      
73ead67495322cc462794387fa9884f5  rh-mysql-apb                   Software Collections MySQL APB                       
d5915e05b253df421efe6e41fb6a66ba  rh-postgresql-apb              SCL PostgreSQL apb implementation                    
[root@test01 app-apb]# 
[root@test01 app-apb]# cd ../ext-api-apb
[root@test01 ext-api-apb]# apb build --tag docker-registry-default.apps.dmoessneapb2.example.com
Finished writing dockerfile.
Building APB using tag: [docker-registry-default.apps.dmoessneapb2.example.com]
Successfully built APB image: docker-registry-default.apps.dmoessneapb2.example.com
[root@test01 ext-api-apb]# apb push --broker https://asb-1338-openshift-ansible-service-broker.apps.dmoessneapb2.example.com/ansible-service-broker --namespace test01-apb 
[root@test01 ext-api-apb]# apb list --broker https://asb-1338-openshift-ansible-service-broker.apps.dmoessneapb2.example.com/ansible-service-broker
ID                                NAME                           DESCRIPTION                                          
cb8f56dc6074c4949ad3bb7f164c224c  apb-push-demo-api-apb          Internal Dog API                                     
9573e996d70d193ae3eeee589acb75ea  apb-push-demo-app-apb          Deploys a demo application showing random pictures.  
efa321d3bcc4bad5af56b40a4c7b34de  apb-push-demo-ext-api-apb      Connection to external API of cats (thecatapi.com)   
30ba0018270996447917c32b0d6c4e6b  localregistry-hello-world-apb  deploys hello-world web application                  
b82394a6f097d4c5358fc18c91e01869  localregistry-mediatiki-apb    This is a sample application generated by apb init   
c23ec213bb8dea1577230c5ce005b9c2  localregistry-mediawiki-apb    This is a sample application generated by apb init   
38557f67b73882767fcc0f2d12b91baf  localregistry-prometheus-apb   Deploy Prometheus on your Project                    
c7b73951ab058437898a8dcfc03c344f  localregistry-wiki-apb         This is a sample application generated by apb init   
2c259ddd8059b9bc65081e07bf20058f  rh-mariadb-apb                 Mariadb apb implementation                           
03b69500305d9859bb9440d9f9023784  rh-mediawiki-apb               Mediawiki123 apb implementation                      
73ead67495322cc462794387fa9884f5  rh-mysql-apb                   Software Collections MySQL APB                       
d5915e05b253df421efe6e41fb6a66ba  rh-postgresql-apb              SCL PostgreSQL apb implementation                    
[root@test01 ext-api-apb]#
Comment 10 daniel 2018-03-18 10:48:56 UTC 
btw, anything special needed for deleting an apb again?

[root@test02 ~]# apb list 
Didn't find OpenShift Ansible Broker route in namespace: ansible-service-broker.                    Trying openshift-ansible-service-broker
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.dmoessneapb2.example.com/ansible-service-broker/v2/catalog
ID                                NAME                           DESCRIPTION                                          
cb8f56dc6074c4949ad3bb7f164c224c  apb-push-demo-api-apb          Internal Dog API                                     
9573e996d70d193ae3eeee589acb75ea  apb-push-demo-app-apb          Deploys a demo application showing random pictures.  
efa321d3bcc4bad5af56b40a4c7b34de  apb-push-demo-ext-api-apb      Connection to external API of cats (thecatapi.com)   
30ba0018270996447917c32b0d6c4e6b  localregistry-hello-world-apb  deploys hello-world web application                  
b82394a6f097d4c5358fc18c91e01869  localregistry-mediatiki-apb    This is a sample application generated by apb init   
c23ec213bb8dea1577230c5ce005b9c2  localregistry-mediawiki-apb    This is a sample application generated by apb init   
38557f67b73882767fcc0f2d12b91baf  localregistry-prometheus-apb   Deploy Prometheus on your Project                    
c7b73951ab058437898a8dcfc03c344f  localregistry-wiki-apb         This is a sample application generated by apb init   
2c259ddd8059b9bc65081e07bf20058f  rh-mariadb-apb                 Mariadb apb implementation                           
03b69500305d9859bb9440d9f9023784  rh-mediawiki-apb               Mediawiki123 apb implementation                      
73ead67495322cc462794387fa9884f5  rh-mysql-apb                   Software Collections MySQL APB                       
d5915e05b253df421efe6e41fb6a66ba  rh-postgresql-apb              SCL PostgreSQL apb implementation                    
[root@test02 ~]# 
[root@test02 ~]# apb remove --id 2c259ddd8059b9bc65081e07bf20058f
Didn't find OpenShift Ansible Broker route in namespace: ansible-service-broker.                    Trying openshift-ansible-service-broker
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.dmoessneapb2.example.com/ansible-service-broker/v2/apb/2c259ddd8059b9bc65081e07bf20058f
Error: Attempt to remove an APB from Broker returned status: 404
Unable to remove APB from Ansible Service Broker.
[root@test02 ~]# 
[root@test02 ~]# apb remove --id b82394a6f097d4c5358fc18c91e01869
Didn't find OpenShift Ansible Broker route in namespace: ansible-service-broker.                    Trying openshift-ansible-service-broker
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.dmoessneapb2.example.com/ansible-service-broker/v2/apb/b82394a6f097d4c5358fc18c91e01869
Error: Attempt to remove an APB from Broker returned status: 404
Unable to remove APB from Ansible Service Broker.
[root@test02 ~]#
Comment 11 Jian Zhang 2018-03-19 09:15:46 UTC 
Daniel,

For the `apb remove` command, you need to enable the develop mode first. As below:

broker:
  dev_broker: true

[root@localhost apb]# apb version
Version: apb-1.1.15
[root@localhost apb]# apb remove --id 30ba0018270996447917c32b0d6c4e6b
Didn't find OpenShift Ansible Broker route in namespace: ansible-service-broker.                    Trying openshift-ansible-service-broker
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.0319-rcj.qe.rhcloud.com/ansible-service-broker/v2/apb/30ba0018270996447917c32b0d6c4e6b
Successfully relisted the Service Catalog
Successfully deleted APB


And, if you want to remove the image forever, please use the "apb remove --local" command, but, you need to run it under the target APB folder. Details are here: https://bugzilla.redhat.com/show_bug.cgi?id=1550017
Comment 12 daniel 2018-03-19 11:15:58 UTC 
Jian,

I have that set:

~~~
broker-config:
----
registry:
  - type: rhcc
    name: rh
    url:  https://registry.access.redhat.com
    org:  
    tag:  v3.7
    white_list: [.*-apb$]

  - type: local_openshift
    name: localregistry
    namespaces: ['openshift','test02-apb']
    white_list: [.*]

dao:
  etcd_host: asb-etcd.openshift-ansible-service-broker.svc
  etcd_port: 2379
  etcd_ca_file: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
  etcd_client_cert: /var/run/asb-etcd-auth/client.crt
  etcd_client_key: /var/run/asb-etcd-auth/client.key
log:
  stdout: true
  level: info
  color: true
openshift:
  host: ""
  ca_file: ""
  bearer_token_file: ""
  sandbox_role: edit
  image_pull_policy: IfNotPresent
  keep_namespace: false
  keep_namespace_on_error: true
broker:
  dev_broker: true
  bootstrap_on_startup: true
  refresh_interval: 600s
  launch_apb_on_bind: false
  output_request: false
  recovery: true
  ssl_cert_key: /etc/tls/private/tls.key
  ssl_cert: /etc/tls/private/tls.crt
  auto_escalate: False
  auth:
    - type: basic
      enabled: false

Events:	<none>
[root@master-2 ~]# 

~~~

I even restarted the pod as I am unsure if the change is picked up automatically.
user1 has cluster admin role, but still it is failing:
 

[root@test02 ~]# oc whoami 
user1
[root@test02 ~]# apb version
Version: apb-1.1.15
[root@test02 ~]# 
[root@test02 ~]# apb list 
Didn't find OpenShift Ansible Broker route in namespace: ansible-service-broker.                    Trying openshift-ansible-service-broker
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.dmoessneabptest1.quicklab.rdu2.cee.redhat.com/ansible-service-broker/v2/catalog
ID                                NAME                    DESCRIPTION                                         
c7b73951ab058437898a8dcfc03c344f  localregistry-wiki-apb  This is a sample application generated by apb init  
2c259ddd8059b9bc65081e07bf20058f  rh-mariadb-apb          Mariadb apb implementation                          
03b69500305d9859bb9440d9f9023784  rh-mediawiki-apb        Mediawiki123 apb implementation                     
73ead67495322cc462794387fa9884f5  rh-mysql-apb            Software Collections MySQL APB                      
d5915e05b253df421efe6e41fb6a66ba  rh-postgresql-apb       SCL PostgreSQL apb implementation                   
[root@test02 ~]# 
[root@test02 ~]# 
[root@test02 ~]# apb remove --id c7b73951ab058437898a8dcfc03c344f -l
Didn't find OpenShift Ansible Broker route in namespace: ansible-service-broker.                    Trying openshift-ansible-service-broker
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.dmoessneabptest1.quicklab.rdu2.cee.redhat.com/ansible-service-broker/v2/apb/c7b73951ab058437898a8dcfc03c344f
Error: Attempt to remove an APB from Broker returned status: 404
Unable to remove APB from Ansible Service Broker.
[root@test02 ~]# apb remove --id c7b73951ab058437898a8dcfc03c344f 
Didn't find OpenShift Ansible Broker route in namespace: ansible-service-broker.                    Trying openshift-ansible-service-broker
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.dmoessneabptest1.quicklab.rdu2.cee.redhat.com/ansible-service-broker/v2/apb/c7b73951ab058437898a8dcfc03c344f
Error: Attempt to remove an APB from Broker returned status: 404
Unable to remove APB from Ansible Service Broker.
[root@test02 ~]# 

I wonder if I am missing sth
Comment 13 Jian Zhang 2018-03-20 10:33:17 UTC 
Daniel,

Yes, you need to restart the ASB pods manually.

>  apb remove --id c7b73951ab058437898a8dcfc03c344f -l

please remove the "-l" parameter and have a try.
If you want to use the "-l" parameter, please run this command under the target APB folder. And, you do not need to specify the "--id".
Comment 14 daniel 2018-03-20 14:12:12 UTC 
(In reply to Jian Zhang from comment #13)
> Daniel,
> 
> Yes, you need to restart the ASB pods manually.
> 
> >  apb remove --id c7b73951ab058437898a8dcfc03c344f -l
> 
> please remove the "-l" parameter and have a try.
> If you want to use the "-l" parameter, please run this command under the
> target APB folder. And, you do not need to specify the "--id".

Jian, I did, but probably hard to read and I should have probably put some more space in between the lines :

[root@test02 ~]# 
Didn't find OpenShift Ansible Broker route in namespace: ansible-service-broker.                    Trying openshift-ansible-service-broker
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.dmoessneabptest1.quicklab.rdu2.cee.redhat.com/ansible-service-broker/v2/apb/c7b73951ab058437898a8dcfc03c344f
Error: Attempt to remove an APB from Broker returned status: 404
Unable to remove APB from Ansible Service Broker.
[root@test02 ~]# apb remove --id c7b73951ab058437898a8dcfc03c344f 
Didn't find OpenShift Ansible Broker route in namespace: ansible-service-broker.                    Trying openshift-ansible-service-broker
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.dmoessneabptest1.quicklab.rdu2.cee.redhat.com/ansible-service-broker/v2/apb/c7b73951ab058437898a8dcfc03c344f
Error: Attempt to remove an APB from Broker returned status: 404
Unable to remove APB from Ansible Service Broker.
[root@test02 ~]# 

but anyway, let's give it a try again:

[root@test02 ~]# oc login -u user1 -p pass1     #### cluster admin role user
Login successful.
[...]
[root@test02 ~]# docker login docker-registry-default.apps.dmoessneabptest1.quicklab.rdu2.cee.redhat.com -u user1 -p $(oc whoami -t)
Login Succeeded
[root@test02 ~]# 
[root@test02 ~]# 
[root@test02 ~]# apb version
Version: apb-1.1.15
[root@test02 ~]# 
[root@test02 ~]# 
[root@test02 ~]# apb list 
Didn't find OpenShift Ansible Broker route in namespace: ansible-service-broker.                    Trying openshift-ansible-service-broker
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.dmoessneabptest1.quicklab.rdu2.cee.redhat.com/ansible-service-broker/v2/catalog
ID                                NAME                    DESCRIPTION                                         
c7b73951ab058437898a8dcfc03c344f  localregistry-wiki-apb  This is a sample application generated by apb init  
2c259ddd8059b9bc65081e07bf20058f  rh-mariadb-apb          Mariadb apb implementation                          
03b69500305d9859bb9440d9f9023784  rh-mediawiki-apb        Mediawiki123 apb implementation                     
73ead67495322cc462794387fa9884f5  rh-mysql-apb            Software Collections MySQL APB                      
d5915e05b253df421efe6e41fb6a66ba  rh-postgresql-apb       SCL PostgreSQL apb implementation                   
[root@test02 ~]# 
[root@test02 ~]# 
[root@test02 ~]# 
[root@test02 ~]# apb remove --id c7b73951ab058437898a8dcfc03c344f
Didn't find OpenShift Ansible Broker route in namespace: ansible-service-broker.                    Trying openshift-ansible-service-broker
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.dmoessneabptest1.quicklab.rdu2.cee.redhat.com/ansible-service-broker/v2/apb/c7b73951ab058437898a8dcfc03c344f
Error: Attempt to remove an APB from Broker returned status: 404
Unable to remove APB from Ansible Service Broker.
[root@test02 ~]# 

--> so I really wonder what I am missing and doing wrong here ?

on the other hand if I - as advice by you (thanks!) - go to the folder and do a -l it is removed afterwards:

[root@test02 ~]# cd wiki-apb/
[root@test02 wiki-apb]# apb remove -l
Attempting to delete associated registry image.
Found registry IP at: 172.30.5.52:5000
Image: [172.30.5.52:5000/openshift/wiki-apb]
Found image: docker-registry.default.svc:5000/openshift/wiki-apb
Warning: Tagged image registry prefix doesn't match. Deleting anyway. Given: 172.30.5.52:5000; Found: docker-registry.default.svc:5000
Successfully deleted sha256:5741a5a9fa39f63a3f31f52e94cdaae6eab675a07f75aee1e005635a7a7c8c22
Didn't find OpenShift Ansible Broker route in namespace: ansible-service-broker.                    Trying openshift-ansible-service-broker
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.dmoessneabptest1.quicklab.rdu2.cee.redhat.com/ansible-service-broker/v2/bootstrap
Successfully bootstrapped Ansible Service Broker
[root@test02 wiki-apb]# 
[root@test02 wiki-apb]# 
[root@test02 wiki-apb]# apb list 
Didn't find OpenShift Ansible Broker route in namespace: ansible-service-broker.                    Trying openshift-ansible-service-broker
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.dmoessneabptest1.quicklab.rdu2.cee.redhat.com/ansible-service-broker/v2/catalog
ID                                NAME               DESCRIPTION                        
2c259ddd8059b9bc65081e07bf20058f  rh-mariadb-apb     Mariadb apb implementation         
03b69500305d9859bb9440d9f9023784  rh-mediawiki-apb   Mediawiki123 apb implementation    
73ead67495322cc462794387fa9884f5  rh-mysql-apb       Software Collections MySQL APB     
d5915e05b253df421efe6e41fb6a66ba  rh-postgresql-apb  SCL PostgreSQL apb implementation  
[root@test02 wiki-apb]#
Comment 15 Dylan Murray 2018-03-20 19:57:36 UTC 
I have actually found the issue that is causing this for Daniel. He is using the latest APB tooling with a 3.7 broker. The development endpoint for `apb remove` has changed in 3.9 so I have went ahead and added an additional check to attempt to remove from the old endpoint if we receive a 404.

https://github.com/ansibleplaybookbundle/ansible-playbook-bundle/pull/246
Comment 16 Jian Zhang 2018-03-23 08:44:39 UTC 
Dylan,

One more question, if we should change the 'target release' to `3.9.0` for this fix.
Comment 17 Jian Zhang 2018-03-23 08:46:37 UTC 
Dylan,

oh, sorry, my mistake, this fix only work for 3.10, right?
Comment 19 Richard Foyle 2018-04-09 14:25:20 UTC 
Hi,

Is there a plan to back port to 3.7.x? Or is there a workaround/story that can be shared with the customer?

Thanks,

Rich
Comment 20 Dylan Murray 2018-04-09 14:48:03 UTC 
Marking my previous comment as private as I was confused... the PR above had the wrong bug # linked. It has actually been merged and built into apb 1.1.16.

Targeting 3.7.x does not make sense since this is a PR which affects someone using the 3.9 tooling on a 3.7 broker image. I am changing the target release to 3.9.z and attaching the build that this was merged in.

PR: https://github.com/ansibleplaybookbundle/ansible-playbook-bundle/pull/246
Brew: https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=668623
Comment 21 Jian Zhang 2018-04-10 01:49:34 UTC 
Dylan,

As we synced before, we do not support the backward compatible officially. So, I just want to confirm with you that this fix is a special support, right? We won't fully support the backward compatible.
Comment 22 Dylan Murray 2018-04-13 12:50:42 UTC 
Jian,

Yes we do not support backwards compatibility. This is simply a small code change to alleviate pain on the developer in case they are using a newer version of the tooling.

You are correct.
Comment 24 Jian Zhang 2018-04-17 08:00:34 UTC 
The APB version: 1.1.16
[root@localhost hello-world-apb]# apb version
Version: apb-1.1.16

The ASB version: 1.0.21
[root@qe-jiazha-37master-etcd-1 ~]# docker run --rm --entrypoint=asbd registry.reg-aws.openshift.com:443/openshift3/ose-ansible-service-broker:v3.7 --version
1.0.21

1) Push an image as the comment 3 described.

2) List the remote host APB images, like below:
[root@localhost hello-world-apb]# apb list
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.0417-4s7.qe.rhcloud.com/ansible-service-broker/v2/catalog
ID                                NAME                           DESCRIPTION                          
30ba0018270996447917c32b0d6c4e6b  localregistry-hello-world-apb  deploys hello-world web application  
2c259ddd8059b9bc65081e07bf20058f  rh-mariadb-apb                 Mariadb apb implementation           
03b69500305d9859bb9440d9f9023784  rh-mediawiki-apb               Mediawiki123 apb implementation      
73ead67495322cc462794387fa9884f5  rh-mysql-apb                   Software Collections MySQL APB       
d5915e05b253df421efe6e41fb6a66ba  rh-postgresql-apb              SCL PostgreSQL apb implementation    

3)Remove the pushed image, it worked well! LGTM, verify succeed.

[root@localhost hello-world-apb]# apb remove --id 30ba0018270996447917c32b0d6c4e6b
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.0417-4s7.qe.rhcloud.com/ansible-service-broker/v2/apb/30ba0018270996447917c32b0d6c4e6b
Received a 404 trying to remove APB with id: 30ba0018270996447917c32b0d6c4e6b
Attempting to contact 3.7 endpoint before erroring out.
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.0417-4s7.qe.rhcloud.com/ansible-service-broker/apb/spec/30ba0018270996447917c32b0d6c4e6b
Successfully relisted the Service Catalog
Successfully deleted APB


The 3.7 cluster info:
[root@qe-jiazha-37master-etcd-1 ~]# oc version
oc v3.7.44
kubernetes v1.7.6+a08f5eeb62
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://qe-jiazha-37master-etcd-1:8443
openshift v3.7.44
kubernetes v1.7.6+a08f5eeb62

Note that: the `apb remove --id xxx` is a temporary delete action, please use the `apb remove -l` if you want to delete an image forever. More info please refer to bug 1550017
Comment 30 errata-xmlrpc 2018-06-27 18:01:34 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2013