Bug 1348842 - After migration to AODH, its services run only on one controller
Summary: After migration to AODH, its services run only on one controller
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 9.0 (Mitaka)
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ga
: 9.0 (Mitaka)
Assignee: Michele Baldessari
QA Contact: Yurii Prokulevych
URL:
Whiteboard:
Depends On:
Blocks: 1333977
TreeView+ depends on / blocked
 
Reported: 2016-06-22 08:10 UTC by Jiri Stransky
Modified: 2016-08-11 11:33 UTC (History)
7 users (show)

Fixed In Version: openstack-tripleo-heat-templates-2.0.0-12
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-08-11 11:33:10 UTC
Target Upstream Version:


Attachments (Terms of Use)
pcs status (6.63 KB, text/plain)
2016-06-22 08:12 UTC, Jiri Stransky
no flags Details
pcs constraint show (5.85 KB, text/plain)
2016-06-22 08:13 UTC, Jiri Stransky
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2016:1599 normal SHIPPED_LIVE Red Hat OpenStack Platform 9 director Release Candidate Advisory 2016-08-11 15:25:37 UTC
OpenStack gerrit 332804 None None None 2016-06-29 08:35:49 UTC

Description Jiri Stransky 2016-06-22 08:10:11 UTC
Description of problem:

After migration to AODH before OSP 8->9 upgrade, AODH services run on only one controller, even after pcs resource cleanup pacemaker doesn't try to start them on the remaining nodes.

The interesting thing is that the node where AODH is running seemed to coincide with it being the redis-master node. There's a redis-master ordering constraint with AODH, but not a colocation constraint, so the behavior seems strange. `crm_resource --wait` returns immediately, so there are no pending operations.
 

Version-Release number of selected component (if applicable):

openstack-tripleo-heat-templates-2.0.0-11.el7ost.noarch

pacemaker-cluster-libs-1.1.13-10.el7_2.2.x86_64
pacemaker-1.1.13-10.el7_2.2.x86_64
pacemaker-remote-1.1.13-10.el7_2.2.x86_64
pacemaker-cli-1.1.13-10.el7_2.2.x86_64
pacemaker-libs-1.1.13-10.el7_2.2.x86_64

resource-agents-3.9.5-54.el7_2.10.x86_64

Comment 1 Jiri Stransky 2016-06-22 08:12:53 UTC
Created attachment 1170623 [details]
pcs status

the httpd monitor timeouts are due to next migration step (keystone wsgi)

Comment 2 Jiri Stransky 2016-06-22 08:13:17 UTC
Created attachment 1170624 [details]
pcs constraint show

Comment 3 Jiri Stransky 2016-06-29 08:35:49 UTC
I was able to verify bandini's fix today, it no longer occurs.

Comment 4 Marios Andreou 2016-07-06 16:23:38 UTC
o/ just to verify that today, using latest poodle, with (upgraded) undercloud packages like [0] and overcloud (still on osp8 repos) like [1] I got through the aodh upgrade successfully with no stopped services. I didn't apply any workarounds. For reference, deployed like:

    openstack overcloud deploy --templates /usr/share/openstack-tripleo-heat-templates -e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml  --control-scale 3 --compute-scale 1 --libvirt-type qemu -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e network_env.yaml --ntp-server '0.fedora.pool.ntp.org'

And then apply the aodh migration like:

    openstack overcloud deploy --templates /usr/share/openstack-tripleo-heat-templates -e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml  --control-scale 3 --compute-scale 1 --libvirt-type qemu -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e network_env.yaml --ntp-server '0.fedora.pool.ntp.org' -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-aodh.yaml
    
thanks, marios

[0]
[stack@instack ~]$ rpm -qa | grep "openstack\|instack\|puppet"
openstack-ceilometer-notification-6.1.0-1.el7ost.noarch
instack-undercloud-4.0.0-5.el7ost.noarch
python-openstacksdk-0.8.3-1.el7ost.noarch
openstack-heat-api-cloudwatch-6.0.0-6.el7ost.noarch
openstack-nova-cert-13.1.0-2.el7ost.noarch
openstack-ceilometer-api-6.1.0-1.el7ost.noarch
openstack-swift-account-2.6.0-1.el7ost.noarch
openstack-utils-2015.2-1.el7ost.noarch
openstack-heat-templates-0-0.3.96a0b0bgit.el7ost.noarch
openstack-swift-container-2.6.0-1.el7ost.noarch
openstack-tripleo-heat-templates-liberty-2.0.0-13.el7ost.noarch
openstack-aodh-evaluator-2.0.1-3.el7ost.noarch
openstack-ironic-common-5.1.2-2.el7ost.noarch
openstack-heat-common-6.0.0-6.el7ost.noarch
openstack-nova-api-13.1.0-2.el7ost.noarch
openstack-ceilometer-collector-6.1.0-1.el7ost.noarch
openstack-tempest-10.0.0-1.b4a056dgit.el7ost.noarch
instack-0.0.8-3.el7ost.noarch
openstack-tripleo-heat-templates-kilo-2.0.0-13.el7ost.noarch
openstack-ceilometer-common-6.1.0-1.el7ost.noarch
openstack-aodh-api-2.0.1-3.el7ost.noarch
openstack-neutron-ml2-8.1.0-11.el7ost.noarch
openstack-ceilometer-central-6.1.0-1.el7ost.noarch
openstack-swift-proxy-2.6.0-1.el7ost.noarch
openstack-selinux-0.7.3-3.el7ost.noarch
puppet-3.6.2-2.el7.noarch
openstack-swift-2.6.0-1.el7ost.noarch
openstack-ceilometer-polling-6.1.0-1.el7ost.noarch
openstack-aodh-notifier-2.0.1-3.el7ost.noarch
openstack-nova-common-13.1.0-2.el7ost.noarch
python-openstackclient-2.2.0-1.el7ost.noarch
openstack-heat-engine-6.0.0-6.el7ost.noarch
openstack-neutron-openvswitch-8.1.0-11.el7ost.noarch
openstack-nova-conductor-13.1.0-2.el7ost.noarch
openstack-ironic-conductor-5.1.2-2.el7ost.noarch
openstack-keystone-9.0.0-1.el7ost.noarch
openstack-puppet-modules-8.1.2-1.el7ost.noarch
openstack-tripleo-heat-templates-2.0.0-13.el7ost.noarch
openstack-tripleo-puppet-elements-2.0.0-3.el7ost.noarch
openstack-aodh-listener-2.0.1-3.el7ost.noarch
openstack-tripleo-common-2.0.0-6.el7ost.noarch
openstack-neutron-8.1.0-11.el7ost.noarch
openstack-ironic-api-5.1.2-2.el7ost.noarch
openstack-ironic-inspector-3.2.2-4.el7ost.noarch
openstack-tripleo-image-elements-0.9.9-6.el7ost.noarch
openstack-tripleo-0.0.8-0.2.d81bd6dgit.el7ost.noarch
openstack-aodh-expirer-2.0.1-3.el7ost.noarch
openstack-neutron-common-8.1.0-11.el7ost.noarch
openstack-heat-api-6.0.0-6.el7ost.noarch
openstack-nova-compute-13.1.0-2.el7ost.noarch
openstack-aodh-compat-2.0.1-3.el7ost.noarch
openstack-swift-object-2.6.0-1.el7ost.noarch
openstack-swift-plugin-swift3-1.10-1.el7ost.noarch
openstack-aodh-common-2.0.1-3.el7ost.noarch
openstack-heat-api-cfn-6.0.0-6.el7ost.noarch
openstack-nova-scheduler-13.1.0-2.el7ost.noarch
openstack-glance-12.0.0-1.el7ost.noarch


[1] 
[root@overcloud-controller-0 ~]# rpm -qa | grep "openstack\|instack\|puppet"
openstack-nova-cert-12.0.4-3.el7ost.noarch
openstack-puppet-modules-7.1.2-1.el7ost.noarch
openstack-aodh-api-1.1.2-1.el7ost.noarch
openstack-swift-2.5.0-2.el7ost.noarch
openstack-ceilometer-common-5.0.2-2.el7ost.noarch
openstack-neutron-7.0.4-11.el7ost.noarch
openstack-heat-api-cfn-5.0.1-6.el7ost.noarch
openstack-ceilometer-api-5.0.2-2.el7ost.noarch
openstack-swift-object-2.5.0-2.el7ost.noarch
openstack-ceilometer-compute-5.0.2-2.el7ost.noarch
openstack-nova-console-12.0.4-3.el7ost.noarch
openstack-aodh-notifier-1.1.2-1.el7ost.noarch
python-openstackclient-1.7.2-1.el7ost.noarch
openstack-swift-container-2.5.0-2.el7ost.noarch
openstack-dashboard-theme-8.0.1-4.el7ost.noarch
openstack-cinder-7.0.2-2.el7ost.noarch
openstack-heat-engine-5.0.1-6.el7ost.noarch
openstack-utils-2014.2-1.el7ost.noarch
openstack-nova-common-12.0.4-3.el7ost.noarch
openstack-neutron-common-7.0.4-11.el7ost.noarch
openstack-swift-proxy-2.5.0-2.el7ost.noarch
openstack-nova-api-12.0.4-3.el7ost.noarch
openstack-nova-novncproxy-12.0.4-3.el7ost.noarch
openstack-aodh-evaluator-1.1.2-1.el7ost.noarch
python-django-openstack-auth-2.0.1-3.el7ost.noarch
openstack-ceilometer-polling-5.0.2-2.el7ost.noarch
openstack-dashboard-8.0.1-4.el7ost.noarch
openstack-glance-11.0.1-4.el7ost.noarch
openstack-keystone-8.0.1-2.el7ost.noarch
openstack-heat-api-5.0.1-6.el7ost.noarch
openstack-ceilometer-collector-5.0.2-2.el7ost.noarch
openstack-nova-conductor-12.0.4-3.el7ost.noarch
openstack-neutron-ml2-7.0.4-11.el7ost.noarch
openstack-aodh-common-1.1.2-1.el7ost.noarch
openstack-nova-compute-12.0.4-3.el7ost.noarch
openstack-ceilometer-central-5.0.2-2.el7ost.noarch
openstack-nova-scheduler-12.0.4-3.el7ost.noarch
openstack-neutron-metering-agent-7.0.4-11.el7ost.noarch
openstack-selinux-0.6.58-1.el7ost.noarch
puppet-3.6.2-2.el7.noarch
openstack-aodh-listener-1.1.2-1.el7ost.noarch
openstack-swift-plugin-swift3-1.9-1.el7ost.noarch
openstack-ceilometer-notification-5.0.2-2.el7ost.noarch
openstack-neutron-bigswitch-lldp-2015.3.8-1.el7ost.noarch
openstack-heat-common-5.0.1-6.el7ost.noarch
openstack-neutron-lbaas-7.0.0-2.el7ost.noarch
openstack-heat-api-cloudwatch-5.0.1-6.el7ost.noarch
openstack-neutron-openvswitch-7.0.4-11.el7ost.noarch
openstack-swift-account-2.5.0-2.el7ost.noarch
openstack-manila-1.0.1-3.el7ost.noarch
openstack-neutron-bigswitch-agent-2015.3.8-1.el7ost.noarch
openstack-manila-share-1.0.1-3.el7ost.noarch

Comment 6 Yurii Prokulevych 2016-07-26 15:30:19 UTC
Deploy command:
    openstack  overcloud deploy --libvirt-type qemu \
        --ntp-server clock.redhat.com --templates \
        --control-scale 3 --compute-scale 1 \
        --ceph-storage-scale 1 -e storage-environment.yaml

storage-environment.yaml:
  parameter_defaults:
    CinderEnableIscsiBackend: false
    CinderEnableRbdBackend: true
    NovaEnableRbdBackend: false
    GlanceBackend: rbd
    GnocchiBackend: rbd


Upgrade command:
    openstack  overcloud deploy --libvirt-type qemu \
        --ntp-server clock.redhat.com --templates \
        --control-scale 3 --compute-scale 1 \
        --ceph-storage-scale 1 -e storage-environment.yaml \
        -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-aodh.yaml \
        --force-postconfig

...
2016-07-26 07:39:32 [NovaComputeDeployment]: SIGNAL_COMPLETE Unknown
Stack overcloud UPDATE_COMPLETE
PKI initialization in init-keystone is deprecated and will be removed.
Overcloud Endpoint: http://192.0.2.6:5000/v2.0
Overcloud Deployed

On the controller:
pcs status | grep aodh -A1
 Clone Set: openstack-aodh-listener-clone [openstack-aodh-listener]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-aodh-notifier-clone [openstack-aodh-notifier]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-aodh-evaluator-clone [openstack-aodh-evaluator]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]

Packages:
---------
openstack-tripleo-heat-templates-2.0.0-18.el7ost.noarch
openstack-tripleo-puppet-elements-2.0.0-3.el7ost.noarch
puppet-3.6.2-4.el7sat.noarch
openstack-puppet-modules-8.1.5-1.el7ost.noarch

Comment 9 errata-xmlrpc 2016-08-11 11:33:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-1599.html


Note You need to log in before you can comment on or make changes to this bug.