Description of problem: After migration to AODH before OSP 8->9 upgrade, AODH services run on only one controller, even after pcs resource cleanup pacemaker doesn't try to start them on the remaining nodes. The interesting thing is that the node where AODH is running seemed to coincide with it being the redis-master node. There's a redis-master ordering constraint with AODH, but not a colocation constraint, so the behavior seems strange. `crm_resource --wait` returns immediately, so there are no pending operations. Version-Release number of selected component (if applicable): openstack-tripleo-heat-templates-2.0.0-11.el7ost.noarch pacemaker-cluster-libs-1.1.13-10.el7_2.2.x86_64 pacemaker-1.1.13-10.el7_2.2.x86_64 pacemaker-remote-1.1.13-10.el7_2.2.x86_64 pacemaker-cli-1.1.13-10.el7_2.2.x86_64 pacemaker-libs-1.1.13-10.el7_2.2.x86_64 resource-agents-3.9.5-54.el7_2.10.x86_64
Created attachment 1170623 [details] pcs status the httpd monitor timeouts are due to next migration step (keystone wsgi)
Created attachment 1170624 [details] pcs constraint show
I was able to verify bandini's fix today, it no longer occurs.
o/ just to verify that today, using latest poodle, with (upgraded) undercloud packages like [0] and overcloud (still on osp8 repos) like [1] I got through the aodh upgrade successfully with no stopped services. I didn't apply any workarounds. For reference, deployed like: openstack overcloud deploy --templates /usr/share/openstack-tripleo-heat-templates -e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml --control-scale 3 --compute-scale 1 --libvirt-type qemu -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e network_env.yaml --ntp-server '0.fedora.pool.ntp.org' And then apply the aodh migration like: openstack overcloud deploy --templates /usr/share/openstack-tripleo-heat-templates -e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml --control-scale 3 --compute-scale 1 --libvirt-type qemu -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e network_env.yaml --ntp-server '0.fedora.pool.ntp.org' -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-aodh.yaml thanks, marios [0] [stack@instack ~]$ rpm -qa | grep "openstack\|instack\|puppet" openstack-ceilometer-notification-6.1.0-1.el7ost.noarch instack-undercloud-4.0.0-5.el7ost.noarch python-openstacksdk-0.8.3-1.el7ost.noarch openstack-heat-api-cloudwatch-6.0.0-6.el7ost.noarch openstack-nova-cert-13.1.0-2.el7ost.noarch openstack-ceilometer-api-6.1.0-1.el7ost.noarch openstack-swift-account-2.6.0-1.el7ost.noarch openstack-utils-2015.2-1.el7ost.noarch openstack-heat-templates-0-0.3.96a0b0bgit.el7ost.noarch openstack-swift-container-2.6.0-1.el7ost.noarch openstack-tripleo-heat-templates-liberty-2.0.0-13.el7ost.noarch openstack-aodh-evaluator-2.0.1-3.el7ost.noarch openstack-ironic-common-5.1.2-2.el7ost.noarch openstack-heat-common-6.0.0-6.el7ost.noarch openstack-nova-api-13.1.0-2.el7ost.noarch openstack-ceilometer-collector-6.1.0-1.el7ost.noarch openstack-tempest-10.0.0-1.b4a056dgit.el7ost.noarch instack-0.0.8-3.el7ost.noarch openstack-tripleo-heat-templates-kilo-2.0.0-13.el7ost.noarch openstack-ceilometer-common-6.1.0-1.el7ost.noarch openstack-aodh-api-2.0.1-3.el7ost.noarch openstack-neutron-ml2-8.1.0-11.el7ost.noarch openstack-ceilometer-central-6.1.0-1.el7ost.noarch openstack-swift-proxy-2.6.0-1.el7ost.noarch openstack-selinux-0.7.3-3.el7ost.noarch puppet-3.6.2-2.el7.noarch openstack-swift-2.6.0-1.el7ost.noarch openstack-ceilometer-polling-6.1.0-1.el7ost.noarch openstack-aodh-notifier-2.0.1-3.el7ost.noarch openstack-nova-common-13.1.0-2.el7ost.noarch python-openstackclient-2.2.0-1.el7ost.noarch openstack-heat-engine-6.0.0-6.el7ost.noarch openstack-neutron-openvswitch-8.1.0-11.el7ost.noarch openstack-nova-conductor-13.1.0-2.el7ost.noarch openstack-ironic-conductor-5.1.2-2.el7ost.noarch openstack-keystone-9.0.0-1.el7ost.noarch openstack-puppet-modules-8.1.2-1.el7ost.noarch openstack-tripleo-heat-templates-2.0.0-13.el7ost.noarch openstack-tripleo-puppet-elements-2.0.0-3.el7ost.noarch openstack-aodh-listener-2.0.1-3.el7ost.noarch openstack-tripleo-common-2.0.0-6.el7ost.noarch openstack-neutron-8.1.0-11.el7ost.noarch openstack-ironic-api-5.1.2-2.el7ost.noarch openstack-ironic-inspector-3.2.2-4.el7ost.noarch openstack-tripleo-image-elements-0.9.9-6.el7ost.noarch openstack-tripleo-0.0.8-0.2.d81bd6dgit.el7ost.noarch openstack-aodh-expirer-2.0.1-3.el7ost.noarch openstack-neutron-common-8.1.0-11.el7ost.noarch openstack-heat-api-6.0.0-6.el7ost.noarch openstack-nova-compute-13.1.0-2.el7ost.noarch openstack-aodh-compat-2.0.1-3.el7ost.noarch openstack-swift-object-2.6.0-1.el7ost.noarch openstack-swift-plugin-swift3-1.10-1.el7ost.noarch openstack-aodh-common-2.0.1-3.el7ost.noarch openstack-heat-api-cfn-6.0.0-6.el7ost.noarch openstack-nova-scheduler-13.1.0-2.el7ost.noarch openstack-glance-12.0.0-1.el7ost.noarch [1] [root@overcloud-controller-0 ~]# rpm -qa | grep "openstack\|instack\|puppet" openstack-nova-cert-12.0.4-3.el7ost.noarch openstack-puppet-modules-7.1.2-1.el7ost.noarch openstack-aodh-api-1.1.2-1.el7ost.noarch openstack-swift-2.5.0-2.el7ost.noarch openstack-ceilometer-common-5.0.2-2.el7ost.noarch openstack-neutron-7.0.4-11.el7ost.noarch openstack-heat-api-cfn-5.0.1-6.el7ost.noarch openstack-ceilometer-api-5.0.2-2.el7ost.noarch openstack-swift-object-2.5.0-2.el7ost.noarch openstack-ceilometer-compute-5.0.2-2.el7ost.noarch openstack-nova-console-12.0.4-3.el7ost.noarch openstack-aodh-notifier-1.1.2-1.el7ost.noarch python-openstackclient-1.7.2-1.el7ost.noarch openstack-swift-container-2.5.0-2.el7ost.noarch openstack-dashboard-theme-8.0.1-4.el7ost.noarch openstack-cinder-7.0.2-2.el7ost.noarch openstack-heat-engine-5.0.1-6.el7ost.noarch openstack-utils-2014.2-1.el7ost.noarch openstack-nova-common-12.0.4-3.el7ost.noarch openstack-neutron-common-7.0.4-11.el7ost.noarch openstack-swift-proxy-2.5.0-2.el7ost.noarch openstack-nova-api-12.0.4-3.el7ost.noarch openstack-nova-novncproxy-12.0.4-3.el7ost.noarch openstack-aodh-evaluator-1.1.2-1.el7ost.noarch python-django-openstack-auth-2.0.1-3.el7ost.noarch openstack-ceilometer-polling-5.0.2-2.el7ost.noarch openstack-dashboard-8.0.1-4.el7ost.noarch openstack-glance-11.0.1-4.el7ost.noarch openstack-keystone-8.0.1-2.el7ost.noarch openstack-heat-api-5.0.1-6.el7ost.noarch openstack-ceilometer-collector-5.0.2-2.el7ost.noarch openstack-nova-conductor-12.0.4-3.el7ost.noarch openstack-neutron-ml2-7.0.4-11.el7ost.noarch openstack-aodh-common-1.1.2-1.el7ost.noarch openstack-nova-compute-12.0.4-3.el7ost.noarch openstack-ceilometer-central-5.0.2-2.el7ost.noarch openstack-nova-scheduler-12.0.4-3.el7ost.noarch openstack-neutron-metering-agent-7.0.4-11.el7ost.noarch openstack-selinux-0.6.58-1.el7ost.noarch puppet-3.6.2-2.el7.noarch openstack-aodh-listener-1.1.2-1.el7ost.noarch openstack-swift-plugin-swift3-1.9-1.el7ost.noarch openstack-ceilometer-notification-5.0.2-2.el7ost.noarch openstack-neutron-bigswitch-lldp-2015.3.8-1.el7ost.noarch openstack-heat-common-5.0.1-6.el7ost.noarch openstack-neutron-lbaas-7.0.0-2.el7ost.noarch openstack-heat-api-cloudwatch-5.0.1-6.el7ost.noarch openstack-neutron-openvswitch-7.0.4-11.el7ost.noarch openstack-swift-account-2.5.0-2.el7ost.noarch openstack-manila-1.0.1-3.el7ost.noarch openstack-neutron-bigswitch-agent-2015.3.8-1.el7ost.noarch openstack-manila-share-1.0.1-3.el7ost.noarch
Deploy command: openstack overcloud deploy --libvirt-type qemu \ --ntp-server clock.redhat.com --templates \ --control-scale 3 --compute-scale 1 \ --ceph-storage-scale 1 -e storage-environment.yaml storage-environment.yaml: parameter_defaults: CinderEnableIscsiBackend: false CinderEnableRbdBackend: true NovaEnableRbdBackend: false GlanceBackend: rbd GnocchiBackend: rbd Upgrade command: openstack overcloud deploy --libvirt-type qemu \ --ntp-server clock.redhat.com --templates \ --control-scale 3 --compute-scale 1 \ --ceph-storage-scale 1 -e storage-environment.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-aodh.yaml \ --force-postconfig ... 2016-07-26 07:39:32 [NovaComputeDeployment]: SIGNAL_COMPLETE Unknown Stack overcloud UPDATE_COMPLETE PKI initialization in init-keystone is deprecated and will be removed. Overcloud Endpoint: http://192.0.2.6:5000/v2.0 Overcloud Deployed On the controller: pcs status | grep aodh -A1 Clone Set: openstack-aodh-listener-clone [openstack-aodh-listener] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-aodh-notifier-clone [openstack-aodh-notifier] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-aodh-evaluator-clone [openstack-aodh-evaluator] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Packages: --------- openstack-tripleo-heat-templates-2.0.0-18.el7ost.noarch openstack-tripleo-puppet-elements-2.0.0-3.el7ost.noarch puppet-3.6.2-4.el7sat.noarch openstack-puppet-modules-8.1.5-1.el7ost.noarch
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHEA-2016-1599.html