Bug 1477457
| Summary: | [OSP12][documentation] A lot of containers with "exited" status | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Artem Hrechanychenko <ahrechan> |
| Component: | documentation | Assignee: | RHOS Documentation Team <rhos-docs> |
| Status: | CLOSED NOTABUG | QA Contact: | RHOS Documentation Team <rhos-docs> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 12.0 (Pike) | CC: | ahrechan, imain, lbopf, m.andre, ohochman, rhallise, sasha, srevivo |
| Target Milestone: | rc | Keywords: | Triaged |
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | All | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-11-15 07:11:42 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
[heat-admin@overcloud-controller-0 ~]$ sudo docker ps -a |grep "Exited" 2c67b49f560e docker-registry.engineering.redhat.com/rhosp12/openstack-ceilometer-central-docker:2017-07-26.10 "/usr/bin/bootstrap_h" 38 hours ago Exited (1) 38 hours ago ceilometer_gnocchi_upgrade 5b1f8fc5dcfd docker-registry.engineering.redhat.com/rhosp12/openstack-nova-api-docker:2017-07-26.10 "/usr/bin/bootstrap_h" 38 hours ago Exited (0) 38 hours ago nova_api_discover_hosts 6748aa5bbe8e docker-registry.engineering.redhat.com/rhosp12/openstack-nova-api-docker:2017-07-26.10 "/usr/bin/bootstrap_h" 38 hours ago Exited (0) 38 hours ago nova_db_sync 0fee61bec424 docker-registry.engineering.redhat.com/rhosp12/openstack-nova-api-docker:2017-07-26.10 "/usr/bin/bootstrap_h" 38 hours ago Exited (0) 38 hours ago nova_api_create_default_cell d5c6e1bd24e3 docker-registry.engineering.redhat.com/rhosp12/openstack-nova-api-docker:2017-07-26.10 "/usr/bin/bootstrap_h" 38 hours ago Exited (0) 38 hours ago nova_api_map_cell0 77b7cbef770c docker-registry.engineering.redhat.com/rhosp12/openstack-swift-account-docker:2017-07-26.10 "chown -R swift: /srv" 38 hours ago Exited (0) 38 hours ago swift_setup_srv 02734047a714 docker-registry.engineering.redhat.com/rhosp12/openstack-gnocchi-api-docker:2017-07-26.10 "/usr/bin/bootstrap_h" 38 hours ago Exited (0) 38 hours ago gnocchi_db_sync 3c9b3e948ae1 docker-registry.engineering.redhat.com/rhosp12/openstack-ceilometer-notification-docker:2017-07-26.10 "/bin/bash -c 'chown " 38 hours ago Exited (0) 38 hours ago ceilometer_init_log a8f686214379 docker-registry.engineering.redhat.com/rhosp12/openstack-keystone-docker:2017-07-26.10 "/usr/bin/bootstrap_h" 38 hours ago Exited (0) 38 hours ago keystone_db_sync 23f5233bc830 docker-registry.engineering.redhat.com/rhosp12/openstack-nova-api-docker:2017-07-26.10 "/usr/bin/bootstrap_h" 38 hours ago Exited (0) 38 hours ago nova_api_db_sync 6713a20d7fa9 docker-registry.engineering.redhat.com/rhosp12/openstack-panko-api-docker:2017-07-26.10 "/usr/bin/bootstrap_h" 38 hours ago Exited (0) 38 hours ago panko_db_sync 472672d0e788 docker-registry.engineering.redhat.com/rhosp12/openstack-glance-api-docker:2017-07-26.10 "/usr/bin/bootstrap_h" 38 hours ago Exited (0) 38 hours ago glance_api_db_sync dcf299d4a6bb docker-registry.engineering.redhat.com/rhosp12/openstack-heat-engine-docker:2017-07-26.10 "/usr/bin/bootstrap_h" 38 hours ago Exited (0) 38 hours ago heat_engine_db_sync 2bcadb56e0fa docker-registry.engineering.redhat.com/rhosp12/openstack-aodh-api-docker:2017-07-26.10 "/usr/bin/bootstrap_h" 38 hours ago Exited (0) 38 hours ago aodh_db_sync 03f2a36bb232 docker-registry.engineering.redhat.com/rhosp12/openstack-haproxy-docker:2017-07-26.10 "/bin/bash -c 'cp -a " 38 hours ago Exited (0) 38 hours ago haproxy_init_bundle 8ec3685a5ac0 docker-registry.engineering.redhat.com/rhosp12/openstack-redis-docker:2017-07-26.10 "/bin/bash -c 'cp -a " 38 hours ago Exited (0) 38 hours ago redis_init_bundle 8690e09e3216 docker-registry.engineering.redhat.com/rhosp12/openstack-mariadb-docker:2017-07-26.10 "/bin/bash -c 'cp -a " 38 hours ago Exited (0) 38 hours ago mysql_init_bundle 545c1b091b84 docker-registry.engineering.redhat.com/rhosp12/openstack-aodh-api-docker:2017-07-26.10 "/bin/bash -c 'mkdir " 38 hours ago Exited (0) 38 hours ago aodh_init_log 73d50dab8a55 docker-registry.engineering.redhat.com/rhosp12/openstack-panko-api-docker:2017-07-26.10 "/bin/bash -c 'mkdir " 38 hours ago Exited (0) 38 hours ago panko_init_log 6a9ab5b4b955 docker-registry.engineering.redhat.com/rhosp12/openstack-horizon-docker:2017-07-26.10 "/bin/bash -c 'touch " 38 hours ago Exited (0) 38 hours ago horizon_fix_perms a67551a6800c docker-registry.engineering.redhat.com/rhosp12/openstack-keystone-docker:2017-07-26.10 "/bin/bash -c 'mkdir " 38 hours ago Exited (0) 38 hours ago keystone_init_log 8a4e1fc6b9c8 docker-registry.engineering.redhat.com/rhosp12/openstack-glance-api-docker:2017-07-26.10 "/bin/bash -c 'chown " 38 hours ago Exited (0) 38 hours ago glance_init_logs f343a0c9f488 docker-registry.engineering.redhat.com/rhosp12/openstack-heat-engine-docker:2017-07-26.10 "/bin/bash -c 'chown " 38 hours ago Exited (0) 38 hours ago heat_init_log 0c54f89af96f docker-registry.engineering.redhat.com/rhosp12/openstack-nova-api-docker:2017-07-26.10 "/bin/bash -c 'chown " 38 hours ago Exited (0) 38 hours ago nova_init_logs 1e848e20dcaf docker-registry.engineering.redhat.com/rhosp12/openstack-rabbitmq-docker:2017-07-26.10 "/bin/bash -c 'cp -a " 38 hours ago Exited (0) 38 hours ago rabbitmq_init_bundle 110ef79864d1 docker-registry.engineering.redhat.com/rhosp12/openstack-gnocchi-api-docker:2017-07-26.10 "/bin/bash -c 'mkdir " 38 hours ago Exited (0) 38 hours ago gnocchi_init_log d019ba7efe15 docker-registry.engineering.redhat.com/rhosp12/openstack-mariadb-docker:2017-07-26.10 "bash -c 'test -e /va" 38 hours ago Exited (0) 38 hours ago mysql_bootstrap 912bc0285aa2 docker-registry.engineering.redhat.com/rhosp12/openstack-mariadb-docker:2017-07-26.10 "chown -R mysql: /var" 38 hours ago Exited (0) 38 hours ago mysql_data_ownership 2fbb0b82b5f2 docker-registry.engineering.redhat.com/rhosp12/openstack-memcached-docker:2017-07-26.10 "/bin/bash -c 'source" 38 hours ago Exited (0) 38 hours ago memcached_init_logs a4f1ded36e77 docker-registry.engineering.redhat.com/rhosp12/openstack-rabbitmq-docker:2017-07-26.10 "kolla_start" 38 hours ago Exited (0) 38 hours ago rabbitmq_bootstrap I think this is expected behavior. All of the above containers are exiting with a status of 0 ATM. If you remove the -a option from your 'docker ps' command these would no longer show up. While we could entertain the option to remove these containers it would make debugging and troubleshooting much harded in some cases I think. (In reply to Dan Prince from comment #3) > I think this is expected behavior. All of the above containers are exiting > with a status of 0 ATM. > > If you remove the -a option from your 'docker ps' command these would no > longer show up. > > While we could entertain the option to remove these containers it would make > debugging and troubleshooting much harded in some cases I think. But I don't know which containers must exist on controller, compute, ceph node.. I don't see any documentations about it? How to validate that our deployment with containerized oc really succeeded? Yes I think this is perfectly normal. The ones you have to watch out for when doing a 'docker ps' is ones in 'restart' status. These are the ones which are trying to run but are failing over and over. Exited containers means something was ran and it exited cleanly and we're not asking it to restart. You can make a list with 'docker ps' or we can come up with one based on the templates or even the json generated by docker-puppet.py. Agreed that whatever have Exited (0) is good , otherwise (any other exit code) needs to be investigated. The other thing to check is container status == UP ( any other status need to be investigated.) - > we can add usage of health-check inside the sanity script. Adding documentation ? should we document the information from this BZ, that said in normal env "we're going to have containers with Exit 0" |
Description of problem: 1) For right validation of non-up containers - need to know - which containers must be deployed and have "--filter status=running" at the end of deployment, because we have several containers which have "-init" in container name and status=exited 2) we checking services state inside of docker containers by next way: sudo docker exec it service_container_namme "ps -aux |grep service_container_name in short format." Exceptions: if name == "horizon": cmd = 'sudo docker exec %s ps -aux |grep %s|grep -v ps' % (name, "httpd") elif name == "swift_xinetd_rsync": cmd = 'sudo docker exec %s ps -aux |grep %s|grep -v ps' % (name, "xinet") elif name == "nova_api_cron": cmd = 'sudo docker exec %s ps -aux |grep %s|grep -v ps' % (name, "crond") elif name == "swift_rsync": cmd = 'sudo docker exec %s ps -aux |grep %s|grep -v ps' % (name, "rsync") elif name == "clustercheck": cmd = 'sudo docker exec clustercheck clustercheck' So for correct validation need to have some file with description of which containers must be "Up" on specific nodes and which services must be "Up" inside of docker container For example: yaml structure: node_role: - container 1: name of processes that must running - container 2 ... - container m