Description of problem: After a HA overcloud has been successfully deployed, subsequent stack update operation fail in docker-puppet-rabbitmq container. This is because the tripleo puppet module [1] that configures rabbitmq create a rabbitmq_user resource by calling an explicit puppet provider: if $stack_action == 'UPDATE' { # Required for changing password on update scenario. Password will be changed only when # called explicity, if the rabbitmq service is already running. rabbitmq_user { $rabbitmq_user : password => $rabbitmq_pass, provider => 'rabbitmqctl', admin => true, } } By doing so, the usual noop_resource override cannot be used, and puppet triggers a call to /usr/bin/rabbitmqctl. This unwanted call then fails because the container is not configured to allow calls to rabbitmq or access to the Erlang VM. [1] tripleo/manifests/profile/base/rabbitmq.pp How reproducible: Always Steps to Reproduce: 1. deploy a stack 2. re-deploy on the same stack (UPDATE action) Actual results: The redeploy should finish Expected results: The redeploy errors out during the config generation (docker-puppet-rabbitmq) Additional info:
Fixed in Pike upstream https://review.openstack.org/#/c/529856/
Verified , OC stack update finished succesfully , all cluster resources are on good health Details: (undercloud) [stack@undercloud-0 ~]$ rhos-release -L Installed repositories (rhel-7.5): 12 ceph-2 ceph-osd-2 rhel-7.5 [stack@undercloud-0 ~]$ cat core_puddle_version 2018-07-13.1[stack@undercloud-0 ~]$ rpm -qa|grep puppet-tripleo puppet-tripleo-7.4.12-4.el7ost.noarch (undercloud) [stack@undercloud-0 ~]$ openstack stack list +--------------------------------------+------------+----------------------------------+-----------------+----------------------+--------------+ | ID | Stack Name | Project | Stack Status | Creation Time | Updated Time | +--------------------------------------+------------+----------------------------------+-----------------+----------------------+--------------+ | 21941980-681d-44d2-972d-44ad1aeb5563 | overcloud | b53bc16fd86a483ba31f252093bb7ea6 | CREATE_COMPLETE | 2018-07-17T04:48:36Z | None | +--------------------------------------+------------+----------------------------------+-----------------+----------------------+--------------+ Minor update procedure: cat > custom_params.yaml <<EOF parameter_defaults: ExtraConfig: tripleo::haproxy::haproxy_globals_override: 'maxconn': 1111 EOF echo "-e /home/stack/custom_params.yaml >> overcloud_deploy.sh" ./overcloud_deploy.sh ... 2018-07-17 07:54:23Z [overcloud-AllNodesDeploySteps-jcmuxrksrhe5.ObjectStorageExtraConfigPost]: UPDATE_COMPLETE state changed 2018-07-17 07:54:23Z [overcloud-AllNodesDeploySteps-jcmuxrksrhe5.CephStorageExtraConfigPost]: UPDATE_COMPLETE state changed 2018-07-17 07:54:23Z [overcloud-AllNodesDeploySteps-jcmuxrksrhe5.ComputeExtraConfigPost]: UPDATE_COMPLETE state changed 2018-07-17 07:54:34Z [overcloud-AllNodesDeploySteps-jcmuxrksrhe5]: UPDATE_COMPLETE Stack UPDATE completed successfully 2018-07-17 07:54:35Z [AllNodesDeploySteps]: UPDATE_COMPLETE state changed 2018-07-17 07:54:51Z [overcloud]: UPDATE_COMPLETE Stack UPDATE completed successfully Stack overcloud UPDATE_COMPLETE Overcloud Endpoint: http://10.0.0.104:5000/v2.0 Overcloud Deployed ... (undercloud) [stack@undercloud-0 ~]$ openstack stack list +--------------------------------------+------------+----------------------------------+--------------------+----------------------+----------------------+ | ID | Stack Name | Project | Stack Status | Creation Time | Updated Time | +--------------------------------------+------------+----------------------------------+--------------------+----------------------+----------------------+ | 21941980-681d-44d2-972d-44ad1aeb5563 | overcloud | b53bc16fd86a483ba31f252093bb7ea6 | UPDATE_IN_PROGRESS | 2018-07-17T04:48:36Z | 2018-07-17T07:17:53Z | +--------------------------------------+------------+----------------------------------+--------------------+----------------------+----------------------+ (undercloud) [stack@undercloud-0 ~]$ openstack stack list +--------------------------------------+------------+----------------------------------+-----------------+----------------------+----------------------+ | ID | Stack Name | Project | Stack Status | Creation Time | Updated Time | +--------------------------------------+------------+----------------------------------+-----------------+----------------------+----------------------+ | 21941980-681d-44d2-972d-44ad1aeb5563 | overcloud | b53bc16fd86a483ba31f252093bb7ea6 | UPDATE_COMPLETE | 2018-07-17T04:48:36Z | 2018-07-17T07:17:53Z | +--------------------------------------+------------+----------------------------------+-----------------+----------------------+----------------------+ [root@controller-0 ~]# pcs status Cluster name: tripleo_cluster Stack: corosync Current DC: controller-2 (version 1.1.18-11.el7_5.3-2b07d5c5a9) - partition with quorum Last updated: Tue Jul 17 08:00:58 2018 Last change: Tue Jul 17 05:47:48 2018 by root via cibadmin on controller-0 12 nodes configured 37 resources configured Online: [ controller-0 controller-1 controller-2 ] GuestOnline: [ galera-bundle-0@controller-0 galera-bundle-1@controller-1 galera-bundle-2@controller-2 rabbitmq-bundle-0@controller-0 rabbitmq-bundle-1@controller-1 rabbitmq-bundle-2@controller-2 redis-bundle-0@controller-0 redis-bundle-1@controller-1 redis-bundle-2@controller-2 ] Full list of resources: Docker container set: rabbitmq-bundle [192.168.24.1:8787/rhosp12/openstack-rabbitmq:pcmklatest] rabbitmq-bundle-0 (ocf::heartbeat:rabbitmq-cluster): Started controller-0 rabbitmq-bundle-1 (ocf::heartbeat:rabbitmq-cluster): Started controller-1 rabbitmq-bundle-2 (ocf::heartbeat:rabbitmq-cluster): Started controller-2 Docker container set: galera-bundle [192.168.24.1:8787/rhosp12/openstack-mariadb:pcmklatest] galera-bundle-0 (ocf::heartbeat:galera): Master controller-0 galera-bundle-1 (ocf::heartbeat:galera): Master controller-1 galera-bundle-2 (ocf::heartbeat:galera): Master controller-2 Docker container set: redis-bundle [192.168.24.1:8787/rhosp12/openstack-redis:pcmklatest] redis-bundle-0 (ocf::heartbeat:redis): Master controller-0 redis-bundle-1 (ocf::heartbeat:redis): Slave controller-1 redis-bundle-2 (ocf::heartbeat:redis): Slave controller-2 ip-192.168.24.12 (ocf::heartbeat:IPaddr2): Started controller-0 ip-10.0.0.104 (ocf::heartbeat:IPaddr2): Started controller-1 ip-172.17.1.10 (ocf::heartbeat:IPaddr2): Started controller-2 ip-172.17.1.16 (ocf::heartbeat:IPaddr2): Started controller-0 ip-172.17.3.16 (ocf::heartbeat:IPaddr2): Started controller-1 ip-172.17.4.14 (ocf::heartbeat:IPaddr2): Started controller-2 Docker container set: haproxy-bundle [192.168.24.1:8787/rhosp12/openstack-haproxy:pcmklatest] haproxy-bundle-docker-0 (ocf::heartbeat:docker): Started controller-0 haproxy-bundle-docker-1 (ocf::heartbeat:docker): Started controller-1 haproxy-bundle-docker-2 (ocf::heartbeat:docker): Started controller-2 openstack-cinder-volume (systemd:openstack-cinder-volume): Started controller-0 Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:2331