Bug 1477457 - [OSP12][documentation] A lot of containers with "exited" status
[OSP12][documentation] A lot of containers with "exited" status
Status: NEW
Product: Red Hat OpenStack
Classification: Red Hat
Component: documentation (Show other bugs)
12.0 (Pike)
x86_64 All
high Severity high
: rc
: ---
Assigned To: RHOS Documentation Team
RHOS Documentation Team
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-08-02 03:05 EDT by Artem Hrechanychenko
Modified: 2017-08-16 09:24 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Artem Hrechanychenko 2017-08-02 03:05:05 EDT
Description of problem:

1) For  right validation of non-up containers - need to know - which containers must be deployed and have "--filter status=running" at the end of deployment, because we have several containers which have "-init" in container name and status=exited 

2) we checking services state inside of docker containers by next way:

sudo docker exec it service_container_namme "ps -aux |grep service_container_name in short format."

Exceptions:

if name == "horizon":
                cmd = 'sudo docker exec %s ps -aux |grep %s|grep -v ps' % (name, "httpd")
            elif name == "swift_xinetd_rsync":
                cmd = 'sudo docker exec %s ps -aux |grep %s|grep -v ps' % (name, "xinet")
            elif name == "nova_api_cron":
                cmd = 'sudo docker exec %s ps -aux |grep %s|grep -v ps' % (name, "crond")
            elif name == "swift_rsync":
                cmd = 'sudo docker exec %s ps -aux |grep %s|grep -v ps' % (name, "rsync")
            elif name == "clustercheck":
                cmd = 'sudo docker exec clustercheck clustercheck'





So for correct validation need to have some file with description of which containers must be "Up" on specific nodes and which services must be "Up" inside of docker container

For example:
yaml structure:

node_role:
- container 1:
    name of processes that must running
- container 2
...
- container m
Comment 2 Artem Hrechanychenko 2017-08-02 03:08:06 EDT
[heat-admin@overcloud-controller-0 ~]$ sudo docker ps -a |grep "Exited"
2c67b49f560e        docker-registry.engineering.redhat.com/rhosp12/openstack-ceilometer-central-docker:2017-07-26.10        "/usr/bin/bootstrap_h"   38 hours ago        Exited (1) 38 hours ago                          ceilometer_gnocchi_upgrade
5b1f8fc5dcfd        docker-registry.engineering.redhat.com/rhosp12/openstack-nova-api-docker:2017-07-26.10                  "/usr/bin/bootstrap_h"   38 hours ago        Exited (0) 38 hours ago                          nova_api_discover_hosts
6748aa5bbe8e        docker-registry.engineering.redhat.com/rhosp12/openstack-nova-api-docker:2017-07-26.10                  "/usr/bin/bootstrap_h"   38 hours ago        Exited (0) 38 hours ago                          nova_db_sync
0fee61bec424        docker-registry.engineering.redhat.com/rhosp12/openstack-nova-api-docker:2017-07-26.10                  "/usr/bin/bootstrap_h"   38 hours ago        Exited (0) 38 hours ago                          nova_api_create_default_cell
d5c6e1bd24e3        docker-registry.engineering.redhat.com/rhosp12/openstack-nova-api-docker:2017-07-26.10                  "/usr/bin/bootstrap_h"   38 hours ago        Exited (0) 38 hours ago                          nova_api_map_cell0
77b7cbef770c        docker-registry.engineering.redhat.com/rhosp12/openstack-swift-account-docker:2017-07-26.10             "chown -R swift: /srv"   38 hours ago        Exited (0) 38 hours ago                          swift_setup_srv
02734047a714        docker-registry.engineering.redhat.com/rhosp12/openstack-gnocchi-api-docker:2017-07-26.10               "/usr/bin/bootstrap_h"   38 hours ago        Exited (0) 38 hours ago                          gnocchi_db_sync
3c9b3e948ae1        docker-registry.engineering.redhat.com/rhosp12/openstack-ceilometer-notification-docker:2017-07-26.10   "/bin/bash -c 'chown "   38 hours ago        Exited (0) 38 hours ago                          ceilometer_init_log
a8f686214379        docker-registry.engineering.redhat.com/rhosp12/openstack-keystone-docker:2017-07-26.10                  "/usr/bin/bootstrap_h"   38 hours ago        Exited (0) 38 hours ago                          keystone_db_sync
23f5233bc830        docker-registry.engineering.redhat.com/rhosp12/openstack-nova-api-docker:2017-07-26.10                  "/usr/bin/bootstrap_h"   38 hours ago        Exited (0) 38 hours ago                          nova_api_db_sync
6713a20d7fa9        docker-registry.engineering.redhat.com/rhosp12/openstack-panko-api-docker:2017-07-26.10                 "/usr/bin/bootstrap_h"   38 hours ago        Exited (0) 38 hours ago                          panko_db_sync
472672d0e788        docker-registry.engineering.redhat.com/rhosp12/openstack-glance-api-docker:2017-07-26.10                "/usr/bin/bootstrap_h"   38 hours ago        Exited (0) 38 hours ago                          glance_api_db_sync
dcf299d4a6bb        docker-registry.engineering.redhat.com/rhosp12/openstack-heat-engine-docker:2017-07-26.10               "/usr/bin/bootstrap_h"   38 hours ago        Exited (0) 38 hours ago                          heat_engine_db_sync
2bcadb56e0fa        docker-registry.engineering.redhat.com/rhosp12/openstack-aodh-api-docker:2017-07-26.10                  "/usr/bin/bootstrap_h"   38 hours ago        Exited (0) 38 hours ago                          aodh_db_sync
03f2a36bb232        docker-registry.engineering.redhat.com/rhosp12/openstack-haproxy-docker:2017-07-26.10                   "/bin/bash -c 'cp -a "   38 hours ago        Exited (0) 38 hours ago                          haproxy_init_bundle
8ec3685a5ac0        docker-registry.engineering.redhat.com/rhosp12/openstack-redis-docker:2017-07-26.10                     "/bin/bash -c 'cp -a "   38 hours ago        Exited (0) 38 hours ago                          redis_init_bundle
8690e09e3216        docker-registry.engineering.redhat.com/rhosp12/openstack-mariadb-docker:2017-07-26.10                   "/bin/bash -c 'cp -a "   38 hours ago        Exited (0) 38 hours ago                          mysql_init_bundle
545c1b091b84        docker-registry.engineering.redhat.com/rhosp12/openstack-aodh-api-docker:2017-07-26.10                  "/bin/bash -c 'mkdir "   38 hours ago        Exited (0) 38 hours ago                          aodh_init_log
73d50dab8a55        docker-registry.engineering.redhat.com/rhosp12/openstack-panko-api-docker:2017-07-26.10                 "/bin/bash -c 'mkdir "   38 hours ago        Exited (0) 38 hours ago                          panko_init_log
6a9ab5b4b955        docker-registry.engineering.redhat.com/rhosp12/openstack-horizon-docker:2017-07-26.10                   "/bin/bash -c 'touch "   38 hours ago        Exited (0) 38 hours ago                          horizon_fix_perms
a67551a6800c        docker-registry.engineering.redhat.com/rhosp12/openstack-keystone-docker:2017-07-26.10                  "/bin/bash -c 'mkdir "   38 hours ago        Exited (0) 38 hours ago                          keystone_init_log
8a4e1fc6b9c8        docker-registry.engineering.redhat.com/rhosp12/openstack-glance-api-docker:2017-07-26.10                "/bin/bash -c 'chown "   38 hours ago        Exited (0) 38 hours ago                          glance_init_logs
f343a0c9f488        docker-registry.engineering.redhat.com/rhosp12/openstack-heat-engine-docker:2017-07-26.10               "/bin/bash -c 'chown "   38 hours ago        Exited (0) 38 hours ago                          heat_init_log
0c54f89af96f        docker-registry.engineering.redhat.com/rhosp12/openstack-nova-api-docker:2017-07-26.10                  "/bin/bash -c 'chown "   38 hours ago        Exited (0) 38 hours ago                          nova_init_logs
1e848e20dcaf        docker-registry.engineering.redhat.com/rhosp12/openstack-rabbitmq-docker:2017-07-26.10                  "/bin/bash -c 'cp -a "   38 hours ago        Exited (0) 38 hours ago                          rabbitmq_init_bundle
110ef79864d1        docker-registry.engineering.redhat.com/rhosp12/openstack-gnocchi-api-docker:2017-07-26.10               "/bin/bash -c 'mkdir "   38 hours ago        Exited (0) 38 hours ago                          gnocchi_init_log
d019ba7efe15        docker-registry.engineering.redhat.com/rhosp12/openstack-mariadb-docker:2017-07-26.10                   "bash -c 'test -e /va"   38 hours ago        Exited (0) 38 hours ago                          mysql_bootstrap
912bc0285aa2        docker-registry.engineering.redhat.com/rhosp12/openstack-mariadb-docker:2017-07-26.10                   "chown -R mysql: /var"   38 hours ago        Exited (0) 38 hours ago                          mysql_data_ownership
2fbb0b82b5f2        docker-registry.engineering.redhat.com/rhosp12/openstack-memcached-docker:2017-07-26.10                 "/bin/bash -c 'source"   38 hours ago        Exited (0) 38 hours ago                          memcached_init_logs
a4f1ded36e77        docker-registry.engineering.redhat.com/rhosp12/openstack-rabbitmq-docker:2017-07-26.10                  "kolla_start"            38 hours ago        Exited (0) 38 hours ago                          rabbitmq_bootstrap
Comment 3 Dan Prince 2017-08-02 16:08:44 EDT
I think this is expected behavior. All of the above containers are exiting with a status of 0 ATM.

If you remove the -a option from your 'docker ps' command these would no longer show up.

While we could entertain the option to remove these containers it would make debugging and troubleshooting much harded in some cases I think.
Comment 4 Artem Hrechanychenko 2017-08-03 02:45:02 EDT
(In reply to Dan Prince from comment #3)
> I think this is expected behavior. All of the above containers are exiting
> with a status of 0 ATM.
> 
> If you remove the -a option from your 'docker ps' command these would no
> longer show up.
> 
> While we could entertain the option to remove these containers it would make
> debugging and troubleshooting much harded in some cases I think.

But I don't know which containers must exist on controller, compute, ceph node..
I don't see any documentations about it?
How to validate that our deployment with containerized oc really succeeded?
Comment 5 Ian Main 2017-08-03 13:47:17 EDT
Yes I think this is perfectly normal.  The ones you have to watch out for when doing a 'docker ps' is ones in 'restart' status.  These are the ones which are trying to run but are failing over and over.  Exited containers means something was ran and it exited cleanly and we're not asking it to restart.

You can make a list with 'docker ps' or we can come up with one based on the templates or even the json generated by docker-puppet.py.
Comment 6 Omri Hochman 2017-08-04 10:23:37 EDT
Agreed that whatever have Exited (0) is good , otherwise (any other exit code) needs to be investigated. 

The other thing to check is container status == UP ( any other status need to be investigated.) 

- > we can add usage of health-check inside the sanity script.
Comment 7 Omri Hochman 2017-08-04 10:27:21 EDT
Adding documentation ?  

should we document the information from this BZ,  that said in normal env  
"we're going to have containers with Exit 0"

Note You need to log in before you can comment on or make changes to this bug.