Bug 1745950 - [OSP15] paunch restarts almost all containers at every redeploy (with no changes)
Summary: [OSP15] paunch restarts almost all containers at every redeploy (with no chan...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-paunch
Version: 15.0 (Stein)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: 15.0 (Stein)
Assignee: Emilien Macchi
QA Contact: nlevinki
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-08-27 10:13 UTC by Michele Baldessari
Modified: 2019-09-27 10:44 UTC (History)
4 users (show)

Fixed In Version: python-paunch-4.5.1-0.20190829080435.f9349e0.el8ost
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-09-21 11:24:31 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 679081 0 'None' MERGED Fix idempotency on RHEL8 2021-01-26 06:12:32 UTC
Red Hat Product Errata RHEA-2019:2811 0 None None None 2019-09-21 11:25:34 UTC

Description Michele Baldessari 2019-08-27 10:13:36 UTC
Problem statement:
Damien and I observed that during an OSP15 redeploy without any changes to THT a number of containers are re-run by paunch and that should not happen. It causes long delays and creates a lot of extra unneeded downtime.

Only containers that have the following snippet in THT should be rerun on a redeploy (i.e. containers where the config or some environment changes). For example mysql_init_bundle:
environment:
  # NOTE: this should force this container to re-run on each 
  # update (scale-out, etc.)
  - list_join:
      - ''
      - - 'TRIPLEO_DEPLOY_IDENTIFIER='
        - {get_param: DeployIdentifier}

The above is the typical way in tripleo to say "we want this container to run at every redeploy"

But we see a truckload of containers being rerun, even ones that do not have the above code. For example (short list, the real list is a lot longer):
mysql_restart_bundle
mysql_bootstrap
mysql_data_ownership

We are reasonably certain that the restart is not being triggered due to changes in /var/lib/config-data/puppet-generated/<service> as we observed no md5sum changes (except swift) during a redeploy.

The patch that broke things is the following one:
https://review.opendev.org/#/c/642910/

commit 85fb2ed4d3f7e42840842f2bdec617e33d6ba4ef
Author: Luke Short <ekultails@gmail.com>
Date:   Fri Jul 5 11:31:13 2019 -0400

    Handle defined containers that are stopped.

    Delete the stopped container and recreate it to ensure
    there are no local modifications to the container.

    Change-Id: Ia3fe06dd221d0894a5de2c18beaa48ba29440003
    Resolves: rhbz#1674517
    Signed-off-by: Luke Short <ekultails@gmail.com>


Reverting this patch fixes things for us. This patch, while it likely did fix Lukas' concerns in https://bugzilla.redhat.com/show_bug.cgi?id=1674517, needs some changes to only delete and start stopped container *if* they have some sort of "restart:" policy in their container definition.
For example:
docker_config:
  step_1:
    memcached:
      start_order: 0
      image: {get_param: ContainerMemcachedImage}
      net: host
      privileged: false
      restart: always
      healthcheck:
        test: /openstack/healthcheck
      volumes:
        list_concat:
          - {get_attr: [ContainersCommon, volumes]}
          -
            - /var/lib/config-data/memcached/etc/sysconfig/memcached:/etc/sysconfig/memcached:ro
      command: ['/bin/bash', '-c', 'source /etc/sysconfig/memcached; /usr/bin/memcached -p ${PORT} -u ${USER} -m ${CACHESIZE} -c ${MAXCONN} $OPTIONS']

It should not go around restarting containers what do not have a restart: policy set unless something in the config changed as that is the implied contract paunch exposed since its inception and which we rely on for a number of things (in particular we were debugging the excessive restart cause by the HA restart_bundle containers)

Comment 4 Marius Cornea 2019-08-30 23:16:29 UTC
[root@controller-0 heat-admin]# rpm -q python3-paunch
python3-paunch-4.5.1-0.20190829080435.f9349e0.el8ost.noarch


After a compute scale out operation we can see that only the swift containers were restarted:


[root@controller-0 heat-admin]# podman ps -a | grep mysql
dbbc6adf7dd1  192.168.24.1:8787/rhosp15/openstack-mariadb:20190829.2                  dumb-init -- /con...  22 minutes ago  Exited (0) 21 minutes ago         mysql_init_bundle
fe2547751d8c  192.168.24.1:8787/rhosp15/openstack-mariadb:20190829.2                  dumb-init -- /usr...  3 hours ago     Exited (0) 3 hours ago            mysql_restart_bundle
738ebe634e7d  192.168.24.1:8787/rhosp15/openstack-mariadb:20190829.2                  dumb-init -- bash...  3 hours ago     Exited (0) 3 hours ago            mysql_bootstrap
b08b590b59e9  192.168.24.1:8787/rhosp15/openstack-mariadb:20190829.2                  dumb-init -- chow...  3 hours ago     Exited (0) 3 hours ago            mysql_data_ownership


[root@controller-0 heat-admin]# podman ps
CONTAINER ID  IMAGE                                                                   COMMAND               CREATED         STATUS             PORTS  NAMES
0336c0d7f7fa  192.168.24.1:8787/rhosp15/openstack-swift-proxy-server:20190829.2       dumb-init --singl...  6 minutes ago   Up 6 minutes ago          swift_proxy
c5a64f30e3fb  192.168.24.1:8787/rhosp15/openstack-swift-object:20190829.2             dumb-init --singl...  6 minutes ago   Up 6 minutes ago          swift_rsync
f899d63cc46c  192.168.24.1:8787/rhosp15/openstack-swift-object:20190829.2             dumb-init --singl...  6 minutes ago   Up 6 minutes ago          swift_object_updater
c34624b40354  192.168.24.1:8787/rhosp15/openstack-swift-object:20190829.2             dumb-init --singl...  6 minutes ago   Up 6 minutes ago          swift_object_server
e637743d665c  192.168.24.1:8787/rhosp15/openstack-swift-object:20190829.2             dumb-init --singl...  6 minutes ago   Up 6 minutes ago          swift_object_replicator
91419a279809  192.168.24.1:8787/rhosp15/openstack-swift-proxy-server:20190829.2       dumb-init --singl...  6 minutes ago   Up 6 minutes ago          swift_object_expirer
2e52bb4acf7d  192.168.24.1:8787/rhosp15/openstack-swift-object:20190829.2             dumb-init --singl...  6 minutes ago   Up 6 minutes ago          swift_object_auditor
916814386c77  192.168.24.1:8787/rhosp15/openstack-swift-container:20190829.2          dumb-init --singl...  6 minutes ago   Up 6 minutes ago          swift_container_updater
d985c91bb25d  192.168.24.1:8787/rhosp15/openstack-swift-container:20190829.2          dumb-init --singl...  6 minutes ago   Up 6 minutes ago          swift_container_server
3ba378dd7db0  192.168.24.1:8787/rhosp15/openstack-swift-container:20190829.2          dumb-init --singl...  6 minutes ago   Up 6 minutes ago          swift_container_replicator
3eb8629cb192  192.168.24.1:8787/rhosp15/openstack-swift-container:20190829.2          dumb-init --singl...  6 minutes ago   Up 6 minutes ago          swift_container_auditor
35114106eda7  192.168.24.1:8787/rhosp15/openstack-swift-account:20190829.2            dumb-init --singl...  6 minutes ago   Up 6 minutes ago          swift_account_server
4a861c624a57  192.168.24.1:8787/rhosp15/openstack-swift-account:20190829.2            dumb-init --singl...  6 minutes ago   Up 6 minutes ago          swift_account_replicator
01e7ffd1f653  192.168.24.1:8787/rhosp15/openstack-swift-account:20190829.2            dumb-init --singl...  6 minutes ago   Up 6 minutes ago          swift_account_reaper
433d84e606bf  192.168.24.1:8787/rhosp15/openstack-swift-account:20190829.2            dumb-init --singl...  6 minutes ago   Up 6 minutes ago          swift_account_auditor
6b7a3cba0df9  192.168.24.1:8787/ceph/rhceph-4.0-rhel8:stable                          /opt/ceph-contain...  31 minutes ago  Up 31 minutes ago         ceph-mgr-controller-0
1b6e38c410aa  192.168.24.1:8787/ceph/rhceph-4.0-rhel8:stable                          /opt/ceph-contain...  32 minutes ago  Up 32 minutes ago         ceph-mon-controller-0
20c193472ffe  192.168.24.1:8787/rhosp15/openstack-gnocchi-statsd:20190829.2           dumb-init --singl...  2 hours ago     Up 2 hours ago            gnocchi_statsd
c38f5d2ff1d4  192.168.24.1:8787/rhosp15/openstack-gnocchi-metricd:20190829.2          dumb-init --singl...  2 hours ago     Up 2 hours ago            gnocchi_metricd
3640720ceb51  192.168.24.1:8787/rhosp15/openstack-gnocchi-api:20190829.2              dumb-init --singl...  2 hours ago     Up 2 hours ago            gnocchi_api
1badf617703b  192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190829.2            dumb-init --singl...  2 hours ago     Up 2 hours ago            openstack-cinder-volume-podman-0
87e337749a21  192.168.24.1:8787/rhosp15/openstack-panko-api:20190829.2                dumb-init --singl...  2 hours ago     Up 2 hours ago            panko_api
31378c9734e5  192.168.24.1:8787/rhosp15/openstack-nova-api:20190829.2                 dumb-init --singl...  2 hours ago     Up 2 hours ago            nova_metadata
048788b00e29  192.168.24.1:8787/rhosp15/openstack-nova-api:20190829.2                 dumb-init --singl...  2 hours ago     Up 2 hours ago            nova_api
78b13c3efc33  192.168.24.1:8787/rhosp15/openstack-glance-api:20190829.2               dumb-init --singl...  2 hours ago     Up 2 hours ago            glance_api
f8864ae725ef  192.168.24.1:8787/rhosp15/openstack-ovn-controller:20190829.2           dumb-init --singl...  2 hours ago     Up 2 hours ago            ovn_controller
69c4bda75f48  192.168.24.1:8787/rhosp15/openstack-nova-placement-api:20190829.2       dumb-init --singl...  2 hours ago     Up 2 hours ago            nova_placement
74d8ed93487d  192.168.24.1:8787/rhosp15/openstack-panko-api:20190829.2                dumb-init --singl...  2 hours ago     Up 2 hours ago            panko_api_cron
ef4d6a10a09f  192.168.24.1:8787/rhosp15/openstack-nova-novncproxy:20190829.2          dumb-init --singl...  2 hours ago     Up 2 hours ago            nova_vnc_proxy
1da917c4eefb  192.168.24.1:8787/rhosp15/openstack-nova-scheduler:20190829.2           dumb-init --singl...  2 hours ago     Up 2 hours ago            nova_scheduler
41e0650f506b  192.168.24.1:8787/rhosp15/openstack-nova-consoleauth:20190829.2         dumb-init --singl...  2 hours ago     Up 2 hours ago            nova_consoleauth
4ae1e1fbd6ac  192.168.24.1:8787/rhosp15/openstack-nova-conductor:20190829.2           dumb-init --singl...  2 hours ago     Up 2 hours ago            nova_conductor
05f8586dfc40  192.168.24.1:8787/rhosp15/openstack-nova-api:20190829.2                 dumb-init --singl...  2 hours ago     Up 2 hours ago            nova_api_cron
89e8f278cfb3  192.168.24.1:8787/rhosp15/openstack-neutron-server-ovn:20190829.2       dumb-init --singl...  2 hours ago     Up 2 hours ago            neutron_api
1f0bb63600c3  192.168.24.1:8787/rhosp15/openstack-cron:20190829.2                     dumb-init --singl...  2 hours ago     Up 2 hours ago            logrotate_crond
7e0a6beb5d74  192.168.24.1:8787/rhosp15/openstack-heat-engine:20190829.2              dumb-init --singl...  2 hours ago     Up 2 hours ago            heat_engine
4acc7f00fb69  192.168.24.1:8787/rhosp15/openstack-heat-api:20190829.2                 dumb-init --singl...  2 hours ago     Up 2 hours ago            heat_api_cron
97a69074f875  192.168.24.1:8787/rhosp15/openstack-heat-api-cfn:20190829.2             dumb-init --singl...  2 hours ago     Up 2 hours ago            heat_api_cfn
5f65b7fd21b3  192.168.24.1:8787/rhosp15/openstack-heat-api:20190829.2                 dumb-init --singl...  2 hours ago     Up 2 hours ago            heat_api
4fda9d892a62  192.168.24.1:8787/rhosp15/openstack-cinder-scheduler:20190829.2         dumb-init --singl...  2 hours ago     Up 2 hours ago            cinder_scheduler
2731a15fb325  192.168.24.1:8787/rhosp15/openstack-cinder-api:20190829.2               dumb-init --singl...  2 hours ago     Up 2 hours ago            cinder_api_cron
15c53cfafb03  192.168.24.1:8787/rhosp15/openstack-cinder-api:20190829.2               dumb-init --singl...  2 hours ago     Up 2 hours ago            cinder_api
139af0c98706  192.168.24.1:8787/rhosp15/openstack-ceilometer-notification:20190829.2  dumb-init --singl...  2 hours ago     Up 2 hours ago            ceilometer_agent_notification
47faca0bcf4d  192.168.24.1:8787/rhosp15/openstack-ceilometer-central:20190829.2       dumb-init --singl...  2 hours ago     Up 2 hours ago            ceilometer_agent_central
029446aaecdc  192.168.24.1:8787/rhosp15/openstack-aodh-notifier:20190829.2            dumb-init --singl...  2 hours ago     Up 2 hours ago            aodh_notifier
6d50233720fe  192.168.24.1:8787/rhosp15/openstack-aodh-listener:20190829.2            dumb-init --singl...  2 hours ago     Up 2 hours ago            aodh_listener
bf7793253f6e  192.168.24.1:8787/rhosp15/openstack-aodh-evaluator:20190829.2           dumb-init --singl...  2 hours ago     Up 2 hours ago            aodh_evaluator
2e7577e0e9f3  192.168.24.1:8787/rhosp15/openstack-aodh-api:20190829.2                 dumb-init --singl...  2 hours ago     Up 2 hours ago            aodh_api
ac969a88c09c  192.168.24.1:8787/rhosp15/openstack-keystone:20190829.2                 dumb-init --singl...  2 hours ago     Up 2 hours ago            keystone_cron
3863765bf67e  192.168.24.1:8787/rhosp15/openstack-keystone:20190829.2                 dumb-init --singl...  2 hours ago     Up 2 hours ago            keystone
36b2f0260292  192.168.24.1:8787/rhosp15/openstack-iscsid:20190829.2                   dumb-init --singl...  2 hours ago     Up 2 hours ago            iscsid
94dbd3558a55  192.168.24.1:8787/rhosp15/openstack-ovn-northd:20190829.2               dumb-init --singl...  2 hours ago     Up 2 hours ago            ovn-dbs-bundle-podman-0
62cc839401bf  192.168.24.1:8787/rhosp15/openstack-horizon:20190829.2                  dumb-init --singl...  3 hours ago     Up 3 hours ago            horizon
1468a70e3908  192.168.24.1:8787/rhosp15/openstack-haproxy:20190829.2                  dumb-init --singl...  3 hours ago     Up 3 hours ago            haproxy-bundle-podman-0
6613d2a6a4dc  192.168.24.1:8787/rhosp15/openstack-redis:20190829.2                    dumb-init --singl...  3 hours ago     Up 3 hours ago            redis-bundle-podman-0
36796c594261  192.168.24.1:8787/rhosp15/openstack-rabbitmq:20190829.2                 dumb-init --singl...  3 hours ago     Up 3 hours ago            rabbitmq-bundle-podman-0
eb8459f93ef9  192.168.24.1:8787/rhosp15/openstack-mariadb:20190829.2                  dumb-init -- /bin...  3 hours ago     Up 3 hours ago            galera-bundle-podman-0
4e2d798dc79c  192.168.24.1:8787/rhosp15/openstack-mariadb:20190829.2                  dumb-init -- koll...  3 hours ago     Up 3 hours ago            clustercheck
4e9a12b7a460  192.168.24.1:8787/rhosp15/openstack-memcached:20190829.2                dumb-init --singl...  3 hours ago     Up 3 hours ago            memcached

Comment 9 errata-xmlrpc 2019-09-21 11:24:31 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:2811


Note You need to log in before you can comment on or make changes to this bug.