Bug 1745950
| Summary: | [OSP15] paunch restarts almost all containers at every redeploy (with no changes) | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Michele Baldessari <michele> |
| Component: | python-paunch | Assignee: | Emilien Macchi <emacchi> |
| Status: | CLOSED ERRATA | QA Contact: | nlevinki <nlevinki> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 15.0 (Stein) | CC: | emacchi, jcoufal, lshort, mcornea |
| Target Milestone: | rc | Keywords: | Regression, Triaged |
| Target Release: | 15.0 (Stein) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | python-paunch-4.5.1-0.20190829080435.f9349e0.el8ost | Doc Type: | No Doc Update |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-09-21 11:24:31 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
[root@controller-0 heat-admin]# rpm -q python3-paunch python3-paunch-4.5.1-0.20190829080435.f9349e0.el8ost.noarch After a compute scale out operation we can see that only the swift containers were restarted: [root@controller-0 heat-admin]# podman ps -a | grep mysql dbbc6adf7dd1 192.168.24.1:8787/rhosp15/openstack-mariadb:20190829.2 dumb-init -- /con... 22 minutes ago Exited (0) 21 minutes ago mysql_init_bundle fe2547751d8c 192.168.24.1:8787/rhosp15/openstack-mariadb:20190829.2 dumb-init -- /usr... 3 hours ago Exited (0) 3 hours ago mysql_restart_bundle 738ebe634e7d 192.168.24.1:8787/rhosp15/openstack-mariadb:20190829.2 dumb-init -- bash... 3 hours ago Exited (0) 3 hours ago mysql_bootstrap b08b590b59e9 192.168.24.1:8787/rhosp15/openstack-mariadb:20190829.2 dumb-init -- chow... 3 hours ago Exited (0) 3 hours ago mysql_data_ownership [root@controller-0 heat-admin]# podman ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 0336c0d7f7fa 192.168.24.1:8787/rhosp15/openstack-swift-proxy-server:20190829.2 dumb-init --singl... 6 minutes ago Up 6 minutes ago swift_proxy c5a64f30e3fb 192.168.24.1:8787/rhosp15/openstack-swift-object:20190829.2 dumb-init --singl... 6 minutes ago Up 6 minutes ago swift_rsync f899d63cc46c 192.168.24.1:8787/rhosp15/openstack-swift-object:20190829.2 dumb-init --singl... 6 minutes ago Up 6 minutes ago swift_object_updater c34624b40354 192.168.24.1:8787/rhosp15/openstack-swift-object:20190829.2 dumb-init --singl... 6 minutes ago Up 6 minutes ago swift_object_server e637743d665c 192.168.24.1:8787/rhosp15/openstack-swift-object:20190829.2 dumb-init --singl... 6 minutes ago Up 6 minutes ago swift_object_replicator 91419a279809 192.168.24.1:8787/rhosp15/openstack-swift-proxy-server:20190829.2 dumb-init --singl... 6 minutes ago Up 6 minutes ago swift_object_expirer 2e52bb4acf7d 192.168.24.1:8787/rhosp15/openstack-swift-object:20190829.2 dumb-init --singl... 6 minutes ago Up 6 minutes ago swift_object_auditor 916814386c77 192.168.24.1:8787/rhosp15/openstack-swift-container:20190829.2 dumb-init --singl... 6 minutes ago Up 6 minutes ago swift_container_updater d985c91bb25d 192.168.24.1:8787/rhosp15/openstack-swift-container:20190829.2 dumb-init --singl... 6 minutes ago Up 6 minutes ago swift_container_server 3ba378dd7db0 192.168.24.1:8787/rhosp15/openstack-swift-container:20190829.2 dumb-init --singl... 6 minutes ago Up 6 minutes ago swift_container_replicator 3eb8629cb192 192.168.24.1:8787/rhosp15/openstack-swift-container:20190829.2 dumb-init --singl... 6 minutes ago Up 6 minutes ago swift_container_auditor 35114106eda7 192.168.24.1:8787/rhosp15/openstack-swift-account:20190829.2 dumb-init --singl... 6 minutes ago Up 6 minutes ago swift_account_server 4a861c624a57 192.168.24.1:8787/rhosp15/openstack-swift-account:20190829.2 dumb-init --singl... 6 minutes ago Up 6 minutes ago swift_account_replicator 01e7ffd1f653 192.168.24.1:8787/rhosp15/openstack-swift-account:20190829.2 dumb-init --singl... 6 minutes ago Up 6 minutes ago swift_account_reaper 433d84e606bf 192.168.24.1:8787/rhosp15/openstack-swift-account:20190829.2 dumb-init --singl... 6 minutes ago Up 6 minutes ago swift_account_auditor 6b7a3cba0df9 192.168.24.1:8787/ceph/rhceph-4.0-rhel8:stable /opt/ceph-contain... 31 minutes ago Up 31 minutes ago ceph-mgr-controller-0 1b6e38c410aa 192.168.24.1:8787/ceph/rhceph-4.0-rhel8:stable /opt/ceph-contain... 32 minutes ago Up 32 minutes ago ceph-mon-controller-0 20c193472ffe 192.168.24.1:8787/rhosp15/openstack-gnocchi-statsd:20190829.2 dumb-init --singl... 2 hours ago Up 2 hours ago gnocchi_statsd c38f5d2ff1d4 192.168.24.1:8787/rhosp15/openstack-gnocchi-metricd:20190829.2 dumb-init --singl... 2 hours ago Up 2 hours ago gnocchi_metricd 3640720ceb51 192.168.24.1:8787/rhosp15/openstack-gnocchi-api:20190829.2 dumb-init --singl... 2 hours ago Up 2 hours ago gnocchi_api 1badf617703b 192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190829.2 dumb-init --singl... 2 hours ago Up 2 hours ago openstack-cinder-volume-podman-0 87e337749a21 192.168.24.1:8787/rhosp15/openstack-panko-api:20190829.2 dumb-init --singl... 2 hours ago Up 2 hours ago panko_api 31378c9734e5 192.168.24.1:8787/rhosp15/openstack-nova-api:20190829.2 dumb-init --singl... 2 hours ago Up 2 hours ago nova_metadata 048788b00e29 192.168.24.1:8787/rhosp15/openstack-nova-api:20190829.2 dumb-init --singl... 2 hours ago Up 2 hours ago nova_api 78b13c3efc33 192.168.24.1:8787/rhosp15/openstack-glance-api:20190829.2 dumb-init --singl... 2 hours ago Up 2 hours ago glance_api f8864ae725ef 192.168.24.1:8787/rhosp15/openstack-ovn-controller:20190829.2 dumb-init --singl... 2 hours ago Up 2 hours ago ovn_controller 69c4bda75f48 192.168.24.1:8787/rhosp15/openstack-nova-placement-api:20190829.2 dumb-init --singl... 2 hours ago Up 2 hours ago nova_placement 74d8ed93487d 192.168.24.1:8787/rhosp15/openstack-panko-api:20190829.2 dumb-init --singl... 2 hours ago Up 2 hours ago panko_api_cron ef4d6a10a09f 192.168.24.1:8787/rhosp15/openstack-nova-novncproxy:20190829.2 dumb-init --singl... 2 hours ago Up 2 hours ago nova_vnc_proxy 1da917c4eefb 192.168.24.1:8787/rhosp15/openstack-nova-scheduler:20190829.2 dumb-init --singl... 2 hours ago Up 2 hours ago nova_scheduler 41e0650f506b 192.168.24.1:8787/rhosp15/openstack-nova-consoleauth:20190829.2 dumb-init --singl... 2 hours ago Up 2 hours ago nova_consoleauth 4ae1e1fbd6ac 192.168.24.1:8787/rhosp15/openstack-nova-conductor:20190829.2 dumb-init --singl... 2 hours ago Up 2 hours ago nova_conductor 05f8586dfc40 192.168.24.1:8787/rhosp15/openstack-nova-api:20190829.2 dumb-init --singl... 2 hours ago Up 2 hours ago nova_api_cron 89e8f278cfb3 192.168.24.1:8787/rhosp15/openstack-neutron-server-ovn:20190829.2 dumb-init --singl... 2 hours ago Up 2 hours ago neutron_api 1f0bb63600c3 192.168.24.1:8787/rhosp15/openstack-cron:20190829.2 dumb-init --singl... 2 hours ago Up 2 hours ago logrotate_crond 7e0a6beb5d74 192.168.24.1:8787/rhosp15/openstack-heat-engine:20190829.2 dumb-init --singl... 2 hours ago Up 2 hours ago heat_engine 4acc7f00fb69 192.168.24.1:8787/rhosp15/openstack-heat-api:20190829.2 dumb-init --singl... 2 hours ago Up 2 hours ago heat_api_cron 97a69074f875 192.168.24.1:8787/rhosp15/openstack-heat-api-cfn:20190829.2 dumb-init --singl... 2 hours ago Up 2 hours ago heat_api_cfn 5f65b7fd21b3 192.168.24.1:8787/rhosp15/openstack-heat-api:20190829.2 dumb-init --singl... 2 hours ago Up 2 hours ago heat_api 4fda9d892a62 192.168.24.1:8787/rhosp15/openstack-cinder-scheduler:20190829.2 dumb-init --singl... 2 hours ago Up 2 hours ago cinder_scheduler 2731a15fb325 192.168.24.1:8787/rhosp15/openstack-cinder-api:20190829.2 dumb-init --singl... 2 hours ago Up 2 hours ago cinder_api_cron 15c53cfafb03 192.168.24.1:8787/rhosp15/openstack-cinder-api:20190829.2 dumb-init --singl... 2 hours ago Up 2 hours ago cinder_api 139af0c98706 192.168.24.1:8787/rhosp15/openstack-ceilometer-notification:20190829.2 dumb-init --singl... 2 hours ago Up 2 hours ago ceilometer_agent_notification 47faca0bcf4d 192.168.24.1:8787/rhosp15/openstack-ceilometer-central:20190829.2 dumb-init --singl... 2 hours ago Up 2 hours ago ceilometer_agent_central 029446aaecdc 192.168.24.1:8787/rhosp15/openstack-aodh-notifier:20190829.2 dumb-init --singl... 2 hours ago Up 2 hours ago aodh_notifier 6d50233720fe 192.168.24.1:8787/rhosp15/openstack-aodh-listener:20190829.2 dumb-init --singl... 2 hours ago Up 2 hours ago aodh_listener bf7793253f6e 192.168.24.1:8787/rhosp15/openstack-aodh-evaluator:20190829.2 dumb-init --singl... 2 hours ago Up 2 hours ago aodh_evaluator 2e7577e0e9f3 192.168.24.1:8787/rhosp15/openstack-aodh-api:20190829.2 dumb-init --singl... 2 hours ago Up 2 hours ago aodh_api ac969a88c09c 192.168.24.1:8787/rhosp15/openstack-keystone:20190829.2 dumb-init --singl... 2 hours ago Up 2 hours ago keystone_cron 3863765bf67e 192.168.24.1:8787/rhosp15/openstack-keystone:20190829.2 dumb-init --singl... 2 hours ago Up 2 hours ago keystone 36b2f0260292 192.168.24.1:8787/rhosp15/openstack-iscsid:20190829.2 dumb-init --singl... 2 hours ago Up 2 hours ago iscsid 94dbd3558a55 192.168.24.1:8787/rhosp15/openstack-ovn-northd:20190829.2 dumb-init --singl... 2 hours ago Up 2 hours ago ovn-dbs-bundle-podman-0 62cc839401bf 192.168.24.1:8787/rhosp15/openstack-horizon:20190829.2 dumb-init --singl... 3 hours ago Up 3 hours ago horizon 1468a70e3908 192.168.24.1:8787/rhosp15/openstack-haproxy:20190829.2 dumb-init --singl... 3 hours ago Up 3 hours ago haproxy-bundle-podman-0 6613d2a6a4dc 192.168.24.1:8787/rhosp15/openstack-redis:20190829.2 dumb-init --singl... 3 hours ago Up 3 hours ago redis-bundle-podman-0 36796c594261 192.168.24.1:8787/rhosp15/openstack-rabbitmq:20190829.2 dumb-init --singl... 3 hours ago Up 3 hours ago rabbitmq-bundle-podman-0 eb8459f93ef9 192.168.24.1:8787/rhosp15/openstack-mariadb:20190829.2 dumb-init -- /bin... 3 hours ago Up 3 hours ago galera-bundle-podman-0 4e2d798dc79c 192.168.24.1:8787/rhosp15/openstack-mariadb:20190829.2 dumb-init -- koll... 3 hours ago Up 3 hours ago clustercheck 4e9a12b7a460 192.168.24.1:8787/rhosp15/openstack-memcached:20190829.2 dumb-init --singl... 3 hours ago Up 3 hours ago memcached Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:2811 |
Problem statement: Damien and I observed that during an OSP15 redeploy without any changes to THT a number of containers are re-run by paunch and that should not happen. It causes long delays and creates a lot of extra unneeded downtime. Only containers that have the following snippet in THT should be rerun on a redeploy (i.e. containers where the config or some environment changes). For example mysql_init_bundle: environment: # NOTE: this should force this container to re-run on each # update (scale-out, etc.) - list_join: - '' - - 'TRIPLEO_DEPLOY_IDENTIFIER=' - {get_param: DeployIdentifier} The above is the typical way in tripleo to say "we want this container to run at every redeploy" But we see a truckload of containers being rerun, even ones that do not have the above code. For example (short list, the real list is a lot longer): mysql_restart_bundle mysql_bootstrap mysql_data_ownership We are reasonably certain that the restart is not being triggered due to changes in /var/lib/config-data/puppet-generated/<service> as we observed no md5sum changes (except swift) during a redeploy. The patch that broke things is the following one: https://review.opendev.org/#/c/642910/ commit 85fb2ed4d3f7e42840842f2bdec617e33d6ba4ef Author: Luke Short <ekultails> Date: Fri Jul 5 11:31:13 2019 -0400 Handle defined containers that are stopped. Delete the stopped container and recreate it to ensure there are no local modifications to the container. Change-Id: Ia3fe06dd221d0894a5de2c18beaa48ba29440003 Resolves: rhbz#1674517 Signed-off-by: Luke Short <ekultails> Reverting this patch fixes things for us. This patch, while it likely did fix Lukas' concerns in https://bugzilla.redhat.com/show_bug.cgi?id=1674517, needs some changes to only delete and start stopped container *if* they have some sort of "restart:" policy in their container definition. For example: docker_config: step_1: memcached: start_order: 0 image: {get_param: ContainerMemcachedImage} net: host privileged: false restart: always healthcheck: test: /openstack/healthcheck volumes: list_concat: - {get_attr: [ContainersCommon, volumes]} - - /var/lib/config-data/memcached/etc/sysconfig/memcached:/etc/sysconfig/memcached:ro command: ['/bin/bash', '-c', 'source /etc/sysconfig/memcached; /usr/bin/memcached -p ${PORT} -u ${USER} -m ${CACHESIZE} -c ${MAXCONN} $OPTIONS'] It should not go around restarting containers what do not have a restart: policy set unless something in the config changed as that is the implied contract paunch exposed since its inception and which we rely on for a number of things (in particular we were debugging the excessive restart cause by the HA restart_bundle containers)