1538777 – [OSP13] docker containers and images remains on the overcloud nodes after running "#openstack overcloud delete overcloud" .

Bug 1538777 - [OSP13] docker containers and images remains on the overcloud nodes after running "#openstack overcloud delete overcloud" .

Summary: [OSP13] docker containers and images remains on the overcloud nodes after run...

Keywords:
Status:	CLOSED NOTABUG
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	rhosp-director
Sub Component:
Version:	13.0 (Queens)
Hardware:	x86_64
OS:	Linux
Priority:	medium
Severity:	medium
Target Milestone:	beta
Target Release:	---
Assignee:	Angus Thomas
QA Contact:	Amit Ugol
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2018-01-25 19:46 UTC by Omri Hochman
Modified:	2018-02-22 22:38 UTC (History)
CC List:	8 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2018-02-22 22:38:25 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Description Omri Hochman 2018-01-25 19:46:55 UTC

[OSP13] docker containers and images remains on the overcloud nodes after running "#openstack overcloud delete overcloud" . 


Environment ( containers image tag 2018-01-22.1):
---------------------------------------------------
openstack-tripleo-heat-templates-8.0.0-0.20180103192341.el7ost.noarch
openstack-tripleo-ui-7.4.3-4.el7ost.noarch
openstack-tripleo-common-containers-8.3.1-0.20180103233643.el7ost.noarch
openstack-tripleo-puppet-elements-8.0.0-0.20171228195253.002c4ca.el7ost.noarch
openstack-tripleo-common-8.3.1-0.20180103233643.el7ost.noarch
python-tripleoclient-8.1.1-0.20171231084755.el7ost.noarch
openstack-tripleo-validations-8.1.1-0.20171221173840.ac39a91.el7ost.noarch
puppet-tripleo-8.1.1-0.20180102165828.el7ost.noarch
openstack-tripleo-image-elements-8.0.0-0.20180103224254.aad6322.el7ost.noarch


Steps to reproduce:
-------------------
- post overcloud deployment run the following steps
- openstack overcloud delete overcloud 
- start the nodes using ironic power on 
- ssh the nodes 
- run sudo docker ps  / run sudo docker images 

results:
--------

- there are leftovers of containers and container-images on the overcloud nodes. 


(undercloud) [stack@undercloud74 ~]$ openstack stack list

(undercloud) [stack@undercloud74 ~]$
(reverse-i-search)`': n^Ca list
(reverse-i-search)`for': sudo  docker rmi `docker images --format "{{.ID}}"` --^Crce
(undercloud) [stack@undercloud74 ~]$ ^C
(undercloud) [stack@undercloud74 ~]$ ssh heat-admin.0.18
Last login: Thu Jan 25 19:20:42 2018 from 192.168.0.1
[heat-admin@overcloud-controller-0 ~]$ rsu
-bash: rsu: command not found
[heat-admin@overcloud-controller-0 ~]$ rsu
-bash: rsu: command not found
[heat-admin@overcloud-controller-0 ~]$ sudo su -
Last login: Thu Jan 25 18:50:56 UTC 2018 on pts/0
[root@overcloud-controller-0 ~]#
[root@overcloud-controller-0 ~]#
[root@overcloud-controller-0 ~]#
[root@overcloud-controller-0 ~]# docker ps
CONTAINER ID        IMAGE                                                                       COMMAND                  CREATED             STATUS                          PORTS               NAMES
865cef16f9dc        192.168.0.1:8787/rhosp13/openstack-haproxy:pcmklatest                       "/bin/bash /usr/local"   50 minutes ago      Up 50 minutes                                       haproxy-bundle-docker-0
ab8b69c6818b        192.168.0.1:8787/rhosp13/openstack-rabbitmq:pcmklatest                      "/bin/bash /usr/local"   50 minutes ago      Up 50 minutes                                       rabbitmq-bundle-docker-0
1ad84bb0c3bd        192.168.0.1:8787/rhosp13/openstack-redis:pcmklatest                         "/bin/bash /usr/local"   50 minutes ago      Up 50 minutes                                       redis-bundle-docker-0
dac16da41ca8        192.168.0.1:8787/rhosp13/openstack-mariadb:pcmklatest                       "/bin/bash /usr/local"   50 minutes ago      Up 50 minutes                                       galera-bundle-docker-0
d44daed98bf8        192.168.0.1:8787/rhceph/rhceph-3-rhel7:latest                               "/entrypoint.sh"         50 minutes ago      Up 50 minutes                                       ceph-mgr-overcloud-controller-0
165f12538668        192.168.0.1:8787/rhceph/rhceph-3-rhel7:latest                               "/entrypoint.sh"         50 minutes ago      Up 50 minutes                                       ceph-mon-overcloud-controller-0
536ad7829c3d        192.168.0.1:8787/rhosp13/openstack-neutron-openvswitch-agent:2018-01-22.1   "kolla_start"            20 hours ago        Up 50 minutes (healthy)                             neutron_ovs_agent
4a26e788b266        192.168.0.1:8787/rhosp13/openstack-neutron-l3-agent:2018-01-22.1            "kolla_start"            20 hours ago        Up 51 minutes (healthy)                             neutron_l3_agent
67f96fe8ec57        192.168.0.1:8787/rhosp13/openstack-neutron-metadata-agent:2018-01-22.1      "kolla_start"            20 hours ago        Up 51 minutes (healthy)                             neutron_metadata_agent
091148be7292        192.168.0.1:8787/rhosp13/openstack-neutron-dhcp-agent:2018-01-22.1          "kolla_start"            20 hours ago        Up 50 minutes (healthy)                             neutron_dhcp
ec7c8623b0c9        192.168.0.1:8787/rhosp13/openstack-glance-api:2018-01-22.1                  "kolla_start"            20 hours ago        Up 51 minutes (healthy)                             glance_api
468b852e83e0        192.168.0.1:8787/rhosp13/openstack-panko-api:2018-01-22.1                   "kolla_start"            20 hours ago        Up 50 minutes                                       panko_api
a6d38e876210        192.168.0.1:8787/rhosp13/openstack-nova-api:2018-01-22.1                    "kolla_start"            20 hours ago        Up 51 minutes                                       nova_metadata
052c43b8361b        192.168.0.1:8787/rhosp13/openstack-nova-api:2018-01-22.1                    "kolla_start"            20 hours ago        Up 51 minutes (healthy)                             nova_api
9b10972ee85d        192.168.0.1:8787/rhosp13/openstack-cron:2018-01-22.1                        "kolla_start"            20 hours ago        Up 50 minutes                                       logrotate_crond
882311c1331f        192.168.0.1:8787/rhosp13/openstack-swift-account:2018-01-22.1               "kolla_start"            20 hours ago        Up 51 minutes (healthy)                             swift_account_server
51b923964cc7        192.168.0.1:8787/rhosp13/openstack-neutron-server:2018-01-22.1              "kolla_start"            20 hours ago        Up 51 minutes                                       neutron_api
23884fe997ee        192.168.0.1:8787/rhosp13/openstack-aodh-listener:2018-01-22.1               "kolla_start"            20 hours ago        Up 51 minutes (healthy)                             aodh_listener
a9b4c936a267        192.168.0.1:8787/rhosp13/openstack-swift-container:2018-01-22.1             "kolla_start"            20 hours ago        Up 51 minutes                                       swift_container_auditor
c91149e57c89        192.168.0.1:8787/rhosp13/openstack-heat-api:2018-01-22.1                    "kolla_start"            20 hours ago        Up 51 minutes                                       heat_api_cron
d87ce68ecaec        192.168.0.1:8787/rhosp13/openstack-swift-proxy-server:2018-01-22.1          "kolla_start"            20 hours ago        Up 51 minutes                                       swift_object_expirer
0cb8f73d748e        192.168.0.1:8787/rhosp13/openstack-swift-object:2018-01-22.1                "kolla_start"            20 hours ago        Up 51 minutes                                       swift_object_updater
43341e8bef6b        192.168.0.1:8787/rhosp13/openstack-swift-container:2018-01-22.1             "kolla_start"            20 hours ago        Up 51 minutes                                       swift_container_replicator
0c14f97d8a3f        192.168.0.1:8787/rhosp13/openstack-swift-account:2018-01-22.1               "kolla_start"            20 hours ago        Up 51 minutes                                       swift_account_auditor
d18502b2fbab        192.168.0.1:8787/rhosp13/openstack-cinder-api:2018-01-22.1                  "kolla_start"            20 hours ago        Up 51 minutes                                       cinder_api_cron
3a1090982f26        192.168.0.1:8787/rhosp13/openstack-heat-api-cfn:2018-01-22.1                "kolla_start"            20 hours ago        Up 50 minutes (healthy)                             heat_api_cfn
3165f1f7cbc5        192.168.0.1:8787/rhosp13/openstack-nova-conductor:2018-01-22.1              "kolla_start"            20 hours ago        Up 50 minutes (healthy)                             nova_conductor
6f968fea9c95        192.168.0.1:8787/rhosp13/openstack-cinder-scheduler:2018-01-22.1            "kolla_start"            20 hours ago        Up 51 minutes (healthy)                             cinder_scheduler
69d7e1dd8677        192.168.0.1:8787/rhosp13/openstack-swift-object:2018-01-22.1                "kolla_start"            20 hours ago        Up 51 minutes                                       swift_object_replicator
2eea664d75d7        192.168.0.1:8787/rhosp13/openstack-swift-container:2018-01-22.1             "kolla_start"            20 hours ago        Up 50 minutes (healthy)                             swift_container_server
a703a69a3b54        192.168.0.1:8787/rhosp13/openstack-heat-engine:2018-01-22.1                 "kolla_start"            20 hours ago        Up 50 minutes (healthy)                             heat_engine
1454f16e1ce1        192.168.0.1:8787/rhosp13/openstack-aodh-api:2018-01-22.1                    "kolla_start"            20 hours ago        Up 51 minutes                                       aodh_api
22142129ce7a        192.168.0.1:8787/rhosp13/openstack-swift-object:2018-01-22.1                "kolla_start"            20 hours ago        Up 50 minutes                                       swift_rsync
2d2019a2c8d0        192.168.0.1:8787/rhosp13/openstack-nova-novncproxy:2018-01-22.1             "kolla_start"            20 hours ago        Up 51 minutes (healthy)                             nova_vnc_proxy
f40e15007fe5        192.168.0.1:8787/rhosp13/openstack-ceilometer-notification:2018-01-22.1     "kolla_start"            20 hours ago        Up 50 minutes (healthy)                             ceilometer_agent_notification
18d4f3a163e1        192.168.0.1:8787/rhosp13/openstack-swift-account:2018-01-22.1               "kolla_start"            20 hours ago        Up 51 minutes                                       swift_account_reaper
f9353a44af93        192.168.0.1:8787/rhosp13/openstack-nova-consoleauth:2018-01-22.1            "kolla_start"            20 hours ago        Up 51 minutes (healthy)                             nova_consoleauth
15b74e00a365        192.168.0.1:8787/rhosp13/openstack-nova-api:2018-01-22.1                    "kolla_start"            20 hours ago        Up 51 minutes                                       nova_api_cron
3147fe296860        192.168.0.1:8787/rhosp13/openstack-aodh-notifier:2018-01-22.1               "kolla_start"            20 hours ago        Up 51 minutes (healthy)                             aodh_notifier
6164f7510141        192.168.0.1:8787/rhosp13/openstack-ceilometer-central:2018-01-22.1          "kolla_start"            20 hours ago        Up 50 minutes (healthy)                             ceilometer_agent_central
235fa4c873f9        192.168.0.1:8787/rhosp13/openstack-swift-account:2018-01-22.1               "kolla_start"            20 hours ago        Up 50 minutes                                       swift_account_replicator
1e0df81c8f66        192.168.0.1:8787/rhosp13/openstack-swift-object:2018-01-22.1                "kolla_start"            20 hours ago        Up 51 minutes                                       swift_object_auditor
2af98abfbcb6        192.168.0.1:8787/rhosp13/openstack-heat-api:2018-01-22.1                    "kolla_start"            20 hours ago        Up 50 minutes (healthy)                             heat_api
d7aa64074312        192.168.0.1:8787/rhosp13/openstack-swift-proxy-server:2018-01-22.1          "kolla_start"            20 hours ago        Up 50 minutes (healthy)                             swift_proxy
a7dcead1187b        192.168.0.1:8787/rhosp13/openstack-cinder-api:2018-01-22.1                  "kolla_start"            20 hours ago        Up 51 minutes                                       cinder_api
6f288e58ebeb        192.168.0.1:8787/rhosp13/openstack-swift-object:2018-01-22.1                "kolla_start"            20 hours ago        Up 50 minutes (healthy)                             swift_object_server
fa067734a0a1        192.168.0.1:8787/rhosp13/openstack-nova-scheduler:2018-01-22.1              "kolla_start"            20 hours ago        Up 48 minutes (healthy)                             nova_scheduler
c62e95976adc        192.168.0.1:8787/rhosp13/openstack-aodh-evaluator:2018-01-22.1              "kolla_start"            20 hours ago        Up 51 minutes (healthy)                             aodh_evaluator
0e2a55ebf5c6        192.168.0.1:8787/rhosp13/openstack-swift-container:2018-01-22.1             "kolla_start"            20 hours ago        Up 51 minutes                                       swift_container_updater
bc3e7ce6f17e        192.168.0.1:8787/rhosp13/openstack-keystone:2018-01-22.1                    "/bin/bash -c '/usr/l"   20 hours ago        Up 51 minutes                                       keystone_cron
3e2637e5ff1e        192.168.0.1:8787/rhosp13/openstack-keystone:2018-01-22.1                    "kolla_start"            20 hours ago        Up 51 minutes (healthy)                             keystone
a876153d6bb3        192.168.0.1:8787/rhosp13/openstack-iscsid:2018-01-22.1                      "kolla_start"            20 hours ago        Up 51 minutes                                       iscsid
93f7397d41bd        192.168.0.1:8787/rhosp13/openstack-nova-placement-api:2018-01-22.1          "kolla_start"            20 hours ago        Up 51 minutes                                       nova_placement
37c9d6891420        192.168.0.1:8787/rhosp13/openstack-horizon:2018-01-22.1                     "kolla_start"            20 hours ago        Restarting (1) 24 minutes ago                       horizon
8a3b962ddd46        192.168.0.1:8787/rhosp13/openstack-mariadb:2018-01-22.1                     "kolla_start"            21 hours ago        Up 50 minutes                                       clustercheck
54b05aa6b991        192.168.0.1:8787/rhosp13/openstack-memcached:2018-01-22.1                   "/bin/bash -c 'source"   21 hours ago        Up 51 minutes                                       memcached
[root@overcloud-controller-0 ~]# docker images
REPOSITORY                                                     TAG                 IMAGE ID            CREATED             SIZE
192.168.0.1:8787/rhosp13/openstack-swift-proxy-server          2018-01-22.1        2404f4e5a16a        9 days ago          903.1 MB
192.168.0.1:8787/rhosp13/openstack-rabbitmq                    2018-01-22.1        e96b694baf66        9 days ago          1.099 GB
192.168.0.1:8787/rhosp13/openstack-rabbitmq                    pcmklatest          e96b694baf66        9 days ago          1.099 GB
192.168.0.1:8787/rhosp13/openstack-swift-account               2018-01-22.1        306066cb3813        9 days ago          849.8 MB
192.168.0.1:8787/rhosp13/openstack-swift-object                2018-01-22.1        ba6d8b1cd2d5        9 days ago          849.8 MB
192.168.0.1:8787/rhosp13/openstack-panko-api                   2018-01-22.1        46f4eb5824e5        9 days ago          904.9 MB
192.168.0.1:8787/rhosp13/openstack-swift-container             2018-01-22.1        33f3d9f0bc7a        9 days ago          849.8 MB
192.168.0.1:8787/rhosp13/openstack-redis                       2018-01-22.1        15d9fd272204        9 days ago          1.064 GB
192.168.0.1:8787/rhosp13/openstack-redis                       pcmklatest          15d9fd272204        9 days ago          1.064 GB
192.168.0.1:8787/rhosp13/openstack-nova-api                    2018-01-22.1        58e48705c40a        9 days ago          967.4 MB
192.168.0.1:8787/rhosp13/openstack-nova-placement-api          2018-01-22.1        ef21284059ed        9 days ago          967.4 MB
192.168.0.1:8787/rhosp13/openstack-nova-novncproxy             2018-01-22.1        0e67da0a34be        9 days ago          930.2 MB
192.168.0.1:8787/rhosp13/openstack-nova-conductor              2018-01-22.1        27349bc0bf5c        9 days ago          929.3 MB
192.168.0.1:8787/rhosp13/openstack-nova-consoleauth            2018-01-22.1        e4bd56d25527        9 days ago          929.6 MB
192.168.0.1:8787/rhosp13/openstack-neutron-openvswitch-agent   2018-01-22.1        445a1392862c        9 days ago          958.7 MB
192.168.0.1:8787/rhosp13/openstack-neutron-l3-agent            2018-01-22.1        0a3c3e8d5841        9 days ago          929.6 MB
192.168.0.1:8787/rhosp13/openstack-neutron-server              2018-01-22.1        844ed2b63839        9 days ago          996.5 MB
192.168.0.1:8787/rhosp13/openstack-nova-scheduler              2018-01-22.1        ec351454137f        9 days ago          948 MB
192.168.0.1:8787/rhosp13/openstack-neutron-dhcp-agent          2018-01-22.1        9b73e5d8ff40        9 days ago          929.6 MB
192.168.0.1:8787/rhosp13/openstack-neutron-metadata-agent      2018-01-22.1        76b042c395a3        9 days ago          929.6 MB
192.168.0.1:8787/rhosp13/openstack-mariadb                     2018-01-22.1        453e56b55550        9 days ago          1.195 GB
192.168.0.1:8787/rhosp13/openstack-mariadb                     pcmklatest          453e56b55550        9 days ago          1.195 GB
192.168.0.1:8787/rhosp13/openstack-heat-engine                 2018-01-22.1        0e323b476abf        9 days ago          894.5 MB
192.168.0.1:8787/rhosp13/openstack-heat-api                    2018-01-22.1        c244157fbf43        9 days ago          894.5 MB
192.168.0.1:8787/rhosp13/openstack-horizon                     2018-01-22.1        ef5b04e7941d        9 days ago          995.4 MB
192.168.0.1:8787/rhosp13/openstack-heat-api-cfn                2018-01-22.1        dad2dbc712e8        9 days ago          894.5 MB
192.168.0.1:8787/rhosp13/openstack-memcached                   2018-01-22.1        6991221d910a        9 days ago          668.6 MB
192.168.0.1:8787/rhosp13/openstack-keystone                    2018-01-22.1        bca4d763b24a        9 days ago          866.6 MB
192.168.0.1:8787/rhosp13/openstack-iscsid                      2018-01-22.1        61ab04cc2691        9 days ago          629.5 MB
192.168.0.1:8787/rhosp13/openstack-gnocchi-api                 2018-01-22.1        25157a4a951f        9 days ago          1.179 GB
192.168.0.1:8787/rhosp13/openstack-cinder-volume               2018-01-22.1        5bcc23c706bb        9 days ago          1.488 GB
192.168.0.1:8787/rhosp13/openstack-cinder-volume               pcmklatest          5bcc23c706bb        9 days ago          1.488 GB
192.168.0.1:8787/rhosp13/openstack-cinder-api                  2018-01-22.1        b8f212288938        9 days ago          1.151 GB
192.168.0.1:8787/rhosp13/openstack-haproxy                     2018-01-22.1        ab13191de079        9 days ago          1.065 GB
192.168.0.1:8787/rhosp13/openstack-haproxy                     pcmklatest          ab13191de079        9 days ago          1.065 GB
192.168.0.1:8787/rhosp13/openstack-glance-api                  2018-01-22.1        f2633b001576        9 days ago          1.049 GB
192.168.0.1:8787/rhosp13/openstack-cron                        2018-01-22.1        7892d9a718cb        9 days ago          625.1 MB
192.168.0.1:8787/rhosp13/openstack-ceilometer-notification     2018-01-22.1        14674c660c8c        9 days ago          864.1 MB
192.168.0.1:8787/rhosp13/openstack-ceilometer-central          2018-01-22.1        88541a94a6dd        9 days ago          913.1 MB
192.168.0.1:8787/rhosp13/openstack-cinder-scheduler            2018-01-22.1        bd127d4a61d1        9 days ago          1.071 GB
192.168.0.1:8787/rhosp13/openstack-aodh-listener               2018-01-22.1        04929594651d        9 days ago          892.5 MB
192.168.0.1:8787/rhosp13/openstack-aodh-notifier               2018-01-22.1        71233564145a        9 days ago          892.5 MB
192.168.0.1:8787/rhosp13/openstack-aodh-evaluator              2018-01-22.1        a31d9b12d506        9 days ago          892.5 MB
192.168.0.1:8787/rhosp13/openstack-aodh-api                    2018-01-22.1        80fe424f8bd7        9 days ago          892.7 MB
192.168.0.1:8787/rhceph/rhceph-3-rhel7                         latest              363666c92064        8 weeks ago         567.5 MB
[root@overcloud-controller-0 ~]#

Comment 1 Omri Hochman 2018-01-25 19:47:48 UTC

unclean environment when attempted to redeploy overcloud might cause to the issue on ticker -> https://bugzilla.redhat.com/show_bug.cgi?id=1538336

Comment 3 Pradeep Kilambi 2018-01-26 20:15:57 UTC

*** Bug 1538336 has been marked as a duplicate of this bug. ***

Comment 4 Omri Hochman 2018-01-29 18:10:33 UTC

*** Bug 1538336 has been marked as a duplicate of this bug. ***

Comment 5 Alex Schultz 2018-02-22 22:38:25 UTC

So "openstack overcloud delete overcloud" should remove the nodes just like a stack delete. In order to ensure that they were cleaned you would have to ensure that ironic is set to wipe all the disks.  I don't think this is an actual feature issue as the same would be true about image based deployment where configuration files may still be in place if you delete an overcloud and then turn the nodes back on.

Note You need to log in before you can comment on or make changes to this bug.