Bug 1550017 - [APB] The APB which deleted via the `apb remove` command still exist after restarting the ASB
Summary: [APB] The APB which deleted via the `apb remove` command still exist after re...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Service Broker
Version: 3.9.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 3.9.0
Assignee: Dylan Murray
QA Contact: Jian Zhang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-02-28 10:05 UTC by Jian Zhang
Modified: 2018-06-18 18:29 UTC (History)
6 users (show)

Fixed In Version: v3.9.11
Doc Type: No Doc Update
Doc Text:
undefined
Clone Of:
Environment:
Last Closed: 2018-06-18 18:02:53 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Jian Zhang 2018-02-28 10:05:09 UTC
Description of problem:
The APB which deleted via the `apb remove` command still exist after restarting the ASB
Version-Release number of selected component (if applicable):
The apb tool version: 1.1.12
[root@localhost postgresql-apb]# apb version
Version: apb-1.1.12
The ASB version: 1.1.15
[root@host-172-16-120-67 ~]# docker run --rm --entrypoint=asbd registry.reg-aws.openshift.com:443/openshift3/ose-ansible-service-broker:v3.9.1 --version
1.1.15

How reproducible:
Always

Steps to Reproduce:
1, Enable the develop mode of the ASB, and log in the cluster as an admin role.

2, Push an APB image to the cluster docker registry.

[root@localhost hello-world-db-apb]# apb push --registry-route docker-registry-default.apps.0228-gtj.qe.rhcloud.com 
Didn't find OpenShift Ansible Broker route in namespace: ansible-service-broker.                    Trying openshift-ansible-service-broker
version: 1.0
name: hello-world-db-apb
description: A sample APB which deploys Hello World Database
bindable: True
async: optional
metadata:
  displayName: Hello World Database (APB)
  dependencies: ['docker.io/centos/postgresql-94-centos7']
  providerDisplayName: "Red Hat, Inc."
plans:
  - name: default
    description: A sample APB which deploys Hello World Database
    free: True
    metadata:
      displayName: Default
      longDescription: This plan deploys a Postgres Database the Hello World application can connect to
      cost: $0.00
    parameters:
      - name: postgresql_database
        title: PostgreSQL Database Name
        type: string
        default: admin
      - name: postgresql_user
        title: PostgreSQL User
        type: string
        default: admin
      - name: postgresql_password
        title: PostgreSQL Password
        type: string
        default: admin

Finished writing dockerfile.
Building APB using tag: [docker-registry-default.apps.0228-gtj.qe.rhcloud.com/openshift/hello-world-db-apb]
Successfully built APB image: docker-registry-default.apps.0228-gtj.qe.rhcloud.com/openshift/hello-world-db-apb
Pushing the image, this could take a minute...
Successfully pushed image: docker-registry-default.apps.0228-gtj.qe.rhcloud.com/openshift/hello-world-db-apb
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.0228-gtj.qe.rhcloud.com/ansible-service-broker/v2/bootstrap
Successfully bootstrapped Ansible Service Broker
Successfully relisted the Service Catalog

We can get the imagestream from the docker registry, like below:
[root@host-172-16-120-67 ~]# oc get is -n openshift | grep hello-world
hello-world-db-apb                    docker-registry.default.svc:5000/openshift/hello-world-db-apb                    latest                         28 seconds ago

3, Run `apb list`, we can get the pushed image.
[root@localhost hello-world-db-apb]# apb list
Didn't find OpenShift Ansible Broker route in namespace: ansible-service-broker.                    Trying openshift-ansible-service-broker
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.0228-gtj.qe.rhcloud.com/ansible-service-broker/v2/catalog
ID                                NAME                              DESCRIPTION                                      
6398639b6039c3fa98fcb7cb0ee7777e  localregistry-hello-world-db-apb  A sample APB which deploys Hello World Database                 
2c259ddd8059b9bc65081e07bf20058f  rh-mariadb-apb                    Mariadb apb implementation                       
03b69500305d9859bb9440d9f9023784  rh-mediawiki-apb                  Mediawiki apb implementation                     
73ead67495322cc462794387fa9884f5  rh-mysql-apb                      Software Collections MySQL APB                   
d5915e05b253df421efe6e41fb6a66ba  rh-postgresql-apb                 SCL PostgreSQL apb implementation   

4, Run the `apb remove` command. And run `apb list`, we can see the pushed image has been deleted. But, the corresponding imagestream object wasn't deleted. Like below:

[root@localhost hello-world-db-apb]# apb remove --id 6398639b6039c3fa98fcb7cb0ee7777e
Didn't find OpenShift Ansible Broker route in namespace: ansible-service-broker.                    Trying openshift-ansible-service-broker
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.0228-gtj.qe.rhcloud.com/ansible-service-broker/v2/apb/6398639b6039c3fa98fcb7cb0ee7777e
Successfully relisted the Service Catalog
Successfully deleted APB

[root@localhost hello-world-db-apb]# apb list
Didn't find OpenShift Ansible Broker route in namespace: ansible-service-broker.                    Trying openshift-ansible-service-broker
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.0228-gtj.qe.rhcloud.com/ansible-service-broker/v2/catalog
ID                                NAME                          DESCRIPTION                        
2c259ddd8059b9bc65081e07bf20058f  rh-mariadb-apb                Mariadb apb implementation         
03b69500305d9859bb9440d9f9023784  rh-mediawiki-apb              Mediawiki apb implementation       
73ead67495322cc462794387fa9884f5  rh-mysql-apb                  Software Collections MySQL APB     
d5915e05b253df421efe6e41fb6a66ba  rh-postgresql-apb             SCL PostgreSQL apb implementation  

[root@host-172-16-120-67 ~]# oc get is -n openshift | grep hello-world
hello-world-db-apb                    docker-registry.default.svc:5000/openshift/hello-world-db-apb                    latest                       4 minutes ago

5, Restart the ABS and run the `apb list` command.

[root@localhost hello-world-db-apb]# apb bootstrap
Didn't find OpenShift Ansible Broker route in namespace: ansible-service-broker.                    Trying openshift-ansible-service-broker
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.0228-gtj.qe.rhcloud.com/ansible-service-broker/v2/bootstrap
Successfully bootstrapped Ansible Service Broker
Successfully relisted the Service Catalog

Actual results:
The pushed apb image(hello-world-db-apb) still exist.

[root@localhost hello-world-db-apb]# apb list
Didn't find OpenShift Ansible Broker route in namespace: ansible-service-broker.                    Trying openshift-ansible-service-broker
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.0228-gtj.qe.rhcloud.com/ansible-service-broker/v2/catalog
ID                                NAME                              DESCRIPTION                                      
6398639b6039c3fa98fcb7cb0ee7777e  localregistry-hello-world-db-apb  A sample APB which deploys Hello World Database                
2c259ddd8059b9bc65081e07bf20058f  rh-mariadb-apb                    Mariadb apb implementation                       
03b69500305d9859bb9440d9f9023784  rh-mediawiki-apb                  Mediawiki apb implementation                     
73ead67495322cc462794387fa9884f5  rh-mysql-apb                      Software Collections MySQL APB                   
d5915e05b253df421efe6e41fb6a66ba  rh-postgresql-apb                 SCL PostgreSQL apb implementation 

Expected results:
The apb image which deleted by `apb remove` should be deleted forever.

Additional info:

Comment 1 Dylan Murray 2018-03-02 19:09:52 UTC
The `apb remove` command is meant to remove APBs from the list of bootstrapped specs from the Openshift Ansible Broker. To delete the image and to prevent the broker from bootstrapping the image again you can do `oc delete image <tagged_apb>`. I can look into adding a flag to apb remove to delete the associated image for development.

Comment 2 Jian Zhang 2018-03-05 09:43:13 UTC
Dylan,

Did you mean the `oc delete imagestreams <tagged_apb> -n openshift` command?
I test this command but still cannot delete the pushed APB image. Details:

[root@localhost hello-world-apb]# apb remove --id 30ba0018270996447917c32b0d6c4e6b
Didn't find OpenShift Ansible Broker route in namespace: ansible-service-broker.                    Trying openshift-ansible-service-broker
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.0305-f77.qe.rhcloud.com/ansible-service-broker/v2/apb/30ba0018270996447917c32b0d6c4e6b
Successfully relisted the Service Catalog
Successfully deleted APB

[root@localhost hello-world-apb]# apb list
Didn't find OpenShift Ansible Broker route in namespace: ansible-service-broker.                    Trying openshift-ansible-service-broker
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.0305-f77.qe.rhcloud.com/ansible-service-broker/v2/catalog
ID                                NAME               DESCRIPTION                        
2c259ddd8059b9bc65081e07bf20058f  rh-mariadb-apb     Mariadb apb implementation         
03b69500305d9859bb9440d9f9023784  rh-mediawiki-apb   Mediawiki apb implementation       
73ead67495322cc462794387fa9884f5  rh-mysql-apb       Software Collections MySQL APB     
d5915e05b253df421efe6e41fb6a66ba  rh-postgresql-apb  SCL PostgreSQL apb implementation  

Delete the imagestreams:
[root@host-172-16-120-58 ~]# oc delete is hello-world-apb -n openshift
imagestream "hello-world-apb" deleted

[root@host-172-16-120-58 ~]# oc get is -n openshift | grep hello

apb bootstrap:
[root@localhost hello-world-apb]# apb bootstrap 
Didn't find OpenShift Ansible Broker route in namespace: ansible-service-broker.                    Trying openshift-ansible-service-broker
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.0305-f77.qe.rhcloud.com/ansible-service-broker/v2/bootstrap
Successfully bootstrapped Ansible Service Broker
Successfully relisted the Service Catalog

The pushed image(hello-world-apb) still exist:

[root@localhost hello-world-apb]# apb list
Didn't find OpenShift Ansible Broker route in namespace: ansible-service-broker.                    Trying openshift-ansible-service-broker
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.0305-f77.qe.rhcloud.com/ansible-service-broker/v2/catalog
ID                                NAME                           DESCRIPTION                          
30ba0018270996447917c32b0d6c4e6b  localregistry-hello-world-apb  deploys hello-world web application  
2c259ddd8059b9bc65081e07bf20058f  rh-mariadb-apb                 Mariadb apb implementation           
03b69500305d9859bb9440d9f9023784  rh-mediawiki-apb               Mediawiki apb implementation         
73ead67495322cc462794387fa9884f5  rh-mysql-apb                   Software Collections MySQL APB       
d5915e05b253df421efe6e41fb6a66ba  rh-postgresql-apb              SCL PostgreSQL apb implementation

Comment 3 Dylan Murray 2018-03-05 14:11:59 UTC
Jian,

No actually the imagestream is not relevant in this discussion. The broker checks all of the images present in the registry and attempts to bootstrap them, bypassing imagestreams. So when the broker is bootstrapping after the remove, it is still finding the image in the internal registry.

To delete it so that the broker won't bootstrap it again you must do this:

`oc get images | grep apb` - Take note of the sha256 hash of the image you want to delete. Then:

`oc delete image <sha256_hash>`.

For example:

[dymurray@dymurray mediawiki-apb]$ oc get images | grep apb
sha256:7f6da2dffe0da82a018a00e2e37698e6dcfc3864a1deae4421f305762f42e657   172.30.1.1:5000/openshift/mediawiki-apb@sha256:7f6da2dffe0da82a018a00e2e37698e6dcfc3864a1deae4421f305762f42e657
[dymurray@dymurray mediawiki-apb]$ oc delete image sha256:7f6da2dffe0da82a018a00e2e37698e6dcfc3864a1deae4421f305762f42e657
image "sha256:7f6da2dffe0da82a018a00e2e37698e6dcfc3864a1deae4421f305762f42e657" deleted

If you then do `apb bootstrap` and `apb list` you will no longer see the APB.

Comment 4 Dylan Murray 2018-03-05 20:00:21 UTC
https://github.com/ansibleplaybookbundle/ansible-playbook-bundle/pull/240


This now adds the flag `apb remove --local` which will delete the associated registry image and bootstrap the broker.

Comment 5 Jian Zhang 2018-03-07 02:50:49 UTC
Thanks Dylan and David.

The apb tool version: 1.1.15
[root@localhost hello-world-apb]# apb version
Version: apb-1.1.15

Test Steps:
1, Push an APB image called "hello-world-apb"

[root@localhost hello-world-apb]# apb push --registry-route docker-registry-default.apps.0305-f77.qe.rhcloud.com
Didn't find OpenShift Ansible Broker route in namespace: ansible-service-broker.                    Trying openshift-ansible-service-broker
version: 1.0
name: hello-world-apb
description: deploys hello-world web application
bindable: False
async: optional
metadata:
  displayName: Hello World Jian (APB)
  longDescription: A sample APB which deploys a containerized Hello World web application
  dependencies: ['docker.io/ansibleplaybookbundle/hello-world:latest']
  providerDisplayName: "Red Hat, Inc."
plans:
  - name: default
    description: A sample APB which deploys Hello World
    free: True
    metadata:
      displayName: Default
      longDescription: This plan deploys a Python web application displaying Hello World
      cost: $0.00
    parameters: []

Finished writing dockerfile.
Building APB using tag: [docker-registry-default.apps.0305-f77.qe.rhcloud.com/openshift/hello-world-apb]
Successfully built APB image: docker-registry-default.apps.0305-f77.qe.rhcloud.com/openshift/hello-world-apb
Pushing the image, this could take a minute...
Successfully pushed image: docker-registry-default.apps.0305-f77.qe.rhcloud.com/openshift/hello-world-apb
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.0305-f77.qe.rhcloud.com/ansible-service-broker/v2/bootstrap
Successfully bootstrapped Ansible Service Broker
Successfully relisted the Service Catalog

2, apb list

[root@localhost hello-world-apb]# apb list
Didn't find OpenShift Ansible Broker route in namespace: ansible-service-broker.                    Trying openshift-ansible-service-broker
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.0305-f77.qe.rhcloud.com/ansible-service-broker/v2/catalog
ID                                NAME                           DESCRIPTION                          
30ba0018270996447917c32b0d6c4e6b  localregistry-hello-world-apb  deploys hello-world web application  
5b17e93bf711e6d0d7949d977b1da091  localregistry-mariadb-apb      Mariadb apb implementation           
c23ec213bb8dea1577230c5ce005b9c2  localregistry-mediawiki-apb    Mediawiki apb implementation         
2c259ddd8059b9bc65081e07bf20058f  rh-mariadb-apb                 Mariadb apb implementation           
03b69500305d9859bb9440d9f9023784  rh-mediawiki-apb               Mediawiki apb implementation         
73ead67495322cc462794387fa9884f5  rh-mysql-apb                   Software Collections MySQL APB       
d5915e05b253df421efe6e41fb6a66ba  rh-postgresql-apb              SCL PostgreSQL apb implementation    

3, Delete the "hello-world-apb" by using the "apb remove --local" command.
[root@localhost hello-world-apb]# apb remove --local
Attempting to delete associated registry image.
Found registry IP at: 172.30.38.249:5000
Image: [172.30.38.249:5000/openshift/hello-world-apb]
Found image: docker-registry.default.svc:5000/openshift/hello-world-apb
Warning: Tagged image registry prefix doesn't match. Deleting anyway. Given: 172.30.38.249:5000; Found: docker-registry.default.svc:5000
Successfully deleted sha256:7418e5b141ff78166c6c0878aff78be7c175b6153e9b89c7d96f4a67b55c8e84
Didn't find OpenShift Ansible Broker route in namespace: ansible-service-broker.                    Trying openshift-ansible-service-broker
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.0305-f77.qe.rhcloud.com/ansible-service-broker/v2/bootstrap
Successfully bootstrapped Ansible Service Broker

4, list them again.
[root@localhost hello-world-apb]# apb list
Didn't find OpenShift Ansible Broker route in namespace: ansible-service-broker.                    Trying openshift-ansible-service-broker
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.0305-f77.qe.rhcloud.com/ansible-service-broker/v2/catalog
ID                                NAME                         DESCRIPTION                        
5b17e93bf711e6d0d7949d977b1da091  localregistry-mariadb-apb    Mariadb apb implementation         
c23ec213bb8dea1577230c5ce005b9c2  localregistry-mediawiki-apb  Mediawiki apb implementation       
2c259ddd8059b9bc65081e07bf20058f  rh-mariadb-apb               Mariadb apb implementation         
03b69500305d9859bb9440d9f9023784  rh-mediawiki-apb             Mediawiki apb implementation       
73ead67495322cc462794387fa9884f5  rh-mysql-apb                 Software Collections MySQL APB     
d5915e05b253df421efe6e41fb6a66ba  rh-postgresql-apb            SCL PostgreSQL apb implementation  

5, apb bootstrap and list apb again.
[root@localhost hello-world-apb]# apb bootstrap
Didn't find OpenShift Ansible Broker route in namespace: ansible-service-broker.                    Trying openshift-ansible-service-broker
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.0305-f77.qe.rhcloud.com/ansible-service-broker/v2/bootstrap
Successfully bootstrapped Ansible Service Broker
Successfully relisted the Service Catalog
[root@localhost hello-world-apb]# apb list
Didn't find OpenShift Ansible Broker route in namespace: ansible-service-broker.                    Trying openshift-ansible-service-broker
Contacting the ansible-service-broker at: https://asb-1338-openshift-ansible-service-broker.apps.0305-f77.qe.rhcloud.com/ansible-service-broker/v2/catalog
ID                                NAME                         DESCRIPTION                        
5b17e93bf711e6d0d7949d977b1da091  localregistry-mariadb-apb    Mariadb apb implementation         
c23ec213bb8dea1577230c5ce005b9c2  localregistry-mediawiki-apb  Mediawiki apb implementation       
2c259ddd8059b9bc65081e07bf20058f  rh-mariadb-apb               Mariadb apb implementation         
03b69500305d9859bb9440d9f9023784  rh-mediawiki-apb             Mediawiki apb implementation       
73ead67495322cc462794387fa9884f5  rh-mysql-apb                 Software Collections MySQL APB     
d5915e05b253df421efe6e41fb6a66ba  rh-postgresql-apb            SCL PostgreSQL apb implementation  

6, Check the internal images.
[root@host-172-16-120-58 ~]# oc get images | grep hello
[root@host-172-16-120-58 ~]# 

The APB image was deleted forever by the `apb remove --local` command. LGTM.


Note You need to log in before you can comment on or make changes to this bug.