Bug 1348842

Summary: After migration to AODH, its services run only on one controller
Product: Red Hat OpenStack Reporter: Jiri Stransky <jstransk>
Component: openstack-tripleo-heat-templatesAssignee: Michele Baldessari <michele>
Status: CLOSED ERRATA QA Contact: Yurii Prokulevych <yprokule>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 9.0 (Mitaka)CC: jason.dobies, mandreou, mburns, morazi, rhel-osp-director-maint, royoung, tvignaud
Target Milestone: gaKeywords: Triaged
Target Release: 9.0 (Mitaka)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-2.0.0-12 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-08-11 11:33:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1333977    
Attachments:
Description Flags
pcs status
none
pcs constraint show none

Description Jiri Stransky 2016-06-22 08:10:11 UTC
Description of problem:

After migration to AODH before OSP 8->9 upgrade, AODH services run on only one controller, even after pcs resource cleanup pacemaker doesn't try to start them on the remaining nodes.

The interesting thing is that the node where AODH is running seemed to coincide with it being the redis-master node. There's a redis-master ordering constraint with AODH, but not a colocation constraint, so the behavior seems strange. `crm_resource --wait` returns immediately, so there are no pending operations.
 

Version-Release number of selected component (if applicable):

openstack-tripleo-heat-templates-2.0.0-11.el7ost.noarch

pacemaker-cluster-libs-1.1.13-10.el7_2.2.x86_64
pacemaker-1.1.13-10.el7_2.2.x86_64
pacemaker-remote-1.1.13-10.el7_2.2.x86_64
pacemaker-cli-1.1.13-10.el7_2.2.x86_64
pacemaker-libs-1.1.13-10.el7_2.2.x86_64

resource-agents-3.9.5-54.el7_2.10.x86_64

Comment 1 Jiri Stransky 2016-06-22 08:12:53 UTC
Created attachment 1170623 [details]
pcs status

the httpd monitor timeouts are due to next migration step (keystone wsgi)

Comment 2 Jiri Stransky 2016-06-22 08:13:17 UTC
Created attachment 1170624 [details]
pcs constraint show

Comment 3 Jiri Stransky 2016-06-29 08:35:49 UTC
I was able to verify bandini's fix today, it no longer occurs.

Comment 4 Marios Andreou 2016-07-06 16:23:38 UTC
o/ just to verify that today, using latest poodle, with (upgraded) undercloud packages like [0] and overcloud (still on osp8 repos) like [1] I got through the aodh upgrade successfully with no stopped services. I didn't apply any workarounds. For reference, deployed like:

    openstack overcloud deploy --templates /usr/share/openstack-tripleo-heat-templates -e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml  --control-scale 3 --compute-scale 1 --libvirt-type qemu -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e network_env.yaml --ntp-server '0.fedora.pool.ntp.org'

And then apply the aodh migration like:

    openstack overcloud deploy --templates /usr/share/openstack-tripleo-heat-templates -e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml  --control-scale 3 --compute-scale 1 --libvirt-type qemu -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e network_env.yaml --ntp-server '0.fedora.pool.ntp.org' -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-aodh.yaml
    
thanks, marios

[0]
[stack@instack ~]$ rpm -qa | grep "openstack\|instack\|puppet"
openstack-ceilometer-notification-6.1.0-1.el7ost.noarch
instack-undercloud-4.0.0-5.el7ost.noarch
python-openstacksdk-0.8.3-1.el7ost.noarch
openstack-heat-api-cloudwatch-6.0.0-6.el7ost.noarch
openstack-nova-cert-13.1.0-2.el7ost.noarch
openstack-ceilometer-api-6.1.0-1.el7ost.noarch
openstack-swift-account-2.6.0-1.el7ost.noarch
openstack-utils-2015.2-1.el7ost.noarch
openstack-heat-templates-0-0.3.96a0b0bgit.el7ost.noarch
openstack-swift-container-2.6.0-1.el7ost.noarch
openstack-tripleo-heat-templates-liberty-2.0.0-13.el7ost.noarch
openstack-aodh-evaluator-2.0.1-3.el7ost.noarch
openstack-ironic-common-5.1.2-2.el7ost.noarch
openstack-heat-common-6.0.0-6.el7ost.noarch
openstack-nova-api-13.1.0-2.el7ost.noarch
openstack-ceilometer-collector-6.1.0-1.el7ost.noarch
openstack-tempest-10.0.0-1.b4a056dgit.el7ost.noarch
instack-0.0.8-3.el7ost.noarch
openstack-tripleo-heat-templates-kilo-2.0.0-13.el7ost.noarch
openstack-ceilometer-common-6.1.0-1.el7ost.noarch
openstack-aodh-api-2.0.1-3.el7ost.noarch
openstack-neutron-ml2-8.1.0-11.el7ost.noarch
openstack-ceilometer-central-6.1.0-1.el7ost.noarch
openstack-swift-proxy-2.6.0-1.el7ost.noarch
openstack-selinux-0.7.3-3.el7ost.noarch
puppet-3.6.2-2.el7.noarch
openstack-swift-2.6.0-1.el7ost.noarch
openstack-ceilometer-polling-6.1.0-1.el7ost.noarch
openstack-aodh-notifier-2.0.1-3.el7ost.noarch
openstack-nova-common-13.1.0-2.el7ost.noarch
python-openstackclient-2.2.0-1.el7ost.noarch
openstack-heat-engine-6.0.0-6.el7ost.noarch
openstack-neutron-openvswitch-8.1.0-11.el7ost.noarch
openstack-nova-conductor-13.1.0-2.el7ost.noarch
openstack-ironic-conductor-5.1.2-2.el7ost.noarch
openstack-keystone-9.0.0-1.el7ost.noarch
openstack-puppet-modules-8.1.2-1.el7ost.noarch
openstack-tripleo-heat-templates-2.0.0-13.el7ost.noarch
openstack-tripleo-puppet-elements-2.0.0-3.el7ost.noarch
openstack-aodh-listener-2.0.1-3.el7ost.noarch
openstack-tripleo-common-2.0.0-6.el7ost.noarch
openstack-neutron-8.1.0-11.el7ost.noarch
openstack-ironic-api-5.1.2-2.el7ost.noarch
openstack-ironic-inspector-3.2.2-4.el7ost.noarch
openstack-tripleo-image-elements-0.9.9-6.el7ost.noarch
openstack-tripleo-0.0.8-0.2.d81bd6dgit.el7ost.noarch
openstack-aodh-expirer-2.0.1-3.el7ost.noarch
openstack-neutron-common-8.1.0-11.el7ost.noarch
openstack-heat-api-6.0.0-6.el7ost.noarch
openstack-nova-compute-13.1.0-2.el7ost.noarch
openstack-aodh-compat-2.0.1-3.el7ost.noarch
openstack-swift-object-2.6.0-1.el7ost.noarch
openstack-swift-plugin-swift3-1.10-1.el7ost.noarch
openstack-aodh-common-2.0.1-3.el7ost.noarch
openstack-heat-api-cfn-6.0.0-6.el7ost.noarch
openstack-nova-scheduler-13.1.0-2.el7ost.noarch
openstack-glance-12.0.0-1.el7ost.noarch


[1] 
[root@overcloud-controller-0 ~]# rpm -qa | grep "openstack\|instack\|puppet"
openstack-nova-cert-12.0.4-3.el7ost.noarch
openstack-puppet-modules-7.1.2-1.el7ost.noarch
openstack-aodh-api-1.1.2-1.el7ost.noarch
openstack-swift-2.5.0-2.el7ost.noarch
openstack-ceilometer-common-5.0.2-2.el7ost.noarch
openstack-neutron-7.0.4-11.el7ost.noarch
openstack-heat-api-cfn-5.0.1-6.el7ost.noarch
openstack-ceilometer-api-5.0.2-2.el7ost.noarch
openstack-swift-object-2.5.0-2.el7ost.noarch
openstack-ceilometer-compute-5.0.2-2.el7ost.noarch
openstack-nova-console-12.0.4-3.el7ost.noarch
openstack-aodh-notifier-1.1.2-1.el7ost.noarch
python-openstackclient-1.7.2-1.el7ost.noarch
openstack-swift-container-2.5.0-2.el7ost.noarch
openstack-dashboard-theme-8.0.1-4.el7ost.noarch
openstack-cinder-7.0.2-2.el7ost.noarch
openstack-heat-engine-5.0.1-6.el7ost.noarch
openstack-utils-2014.2-1.el7ost.noarch
openstack-nova-common-12.0.4-3.el7ost.noarch
openstack-neutron-common-7.0.4-11.el7ost.noarch
openstack-swift-proxy-2.5.0-2.el7ost.noarch
openstack-nova-api-12.0.4-3.el7ost.noarch
openstack-nova-novncproxy-12.0.4-3.el7ost.noarch
openstack-aodh-evaluator-1.1.2-1.el7ost.noarch
python-django-openstack-auth-2.0.1-3.el7ost.noarch
openstack-ceilometer-polling-5.0.2-2.el7ost.noarch
openstack-dashboard-8.0.1-4.el7ost.noarch
openstack-glance-11.0.1-4.el7ost.noarch
openstack-keystone-8.0.1-2.el7ost.noarch
openstack-heat-api-5.0.1-6.el7ost.noarch
openstack-ceilometer-collector-5.0.2-2.el7ost.noarch
openstack-nova-conductor-12.0.4-3.el7ost.noarch
openstack-neutron-ml2-7.0.4-11.el7ost.noarch
openstack-aodh-common-1.1.2-1.el7ost.noarch
openstack-nova-compute-12.0.4-3.el7ost.noarch
openstack-ceilometer-central-5.0.2-2.el7ost.noarch
openstack-nova-scheduler-12.0.4-3.el7ost.noarch
openstack-neutron-metering-agent-7.0.4-11.el7ost.noarch
openstack-selinux-0.6.58-1.el7ost.noarch
puppet-3.6.2-2.el7.noarch
openstack-aodh-listener-1.1.2-1.el7ost.noarch
openstack-swift-plugin-swift3-1.9-1.el7ost.noarch
openstack-ceilometer-notification-5.0.2-2.el7ost.noarch
openstack-neutron-bigswitch-lldp-2015.3.8-1.el7ost.noarch
openstack-heat-common-5.0.1-6.el7ost.noarch
openstack-neutron-lbaas-7.0.0-2.el7ost.noarch
openstack-heat-api-cloudwatch-5.0.1-6.el7ost.noarch
openstack-neutron-openvswitch-7.0.4-11.el7ost.noarch
openstack-swift-account-2.5.0-2.el7ost.noarch
openstack-manila-1.0.1-3.el7ost.noarch
openstack-neutron-bigswitch-agent-2015.3.8-1.el7ost.noarch
openstack-manila-share-1.0.1-3.el7ost.noarch

Comment 6 Yurii Prokulevych 2016-07-26 15:30:19 UTC
Deploy command:
    openstack  overcloud deploy --libvirt-type qemu \
        --ntp-server clock.redhat.com --templates \
        --control-scale 3 --compute-scale 1 \
        --ceph-storage-scale 1 -e storage-environment.yaml

storage-environment.yaml:
  parameter_defaults:
    CinderEnableIscsiBackend: false
    CinderEnableRbdBackend: true
    NovaEnableRbdBackend: false
    GlanceBackend: rbd
    GnocchiBackend: rbd


Upgrade command:
    openstack  overcloud deploy --libvirt-type qemu \
        --ntp-server clock.redhat.com --templates \
        --control-scale 3 --compute-scale 1 \
        --ceph-storage-scale 1 -e storage-environment.yaml \
        -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-aodh.yaml \
        --force-postconfig

...
2016-07-26 07:39:32 [NovaComputeDeployment]: SIGNAL_COMPLETE Unknown
Stack overcloud UPDATE_COMPLETE
PKI initialization in init-keystone is deprecated and will be removed.
Overcloud Endpoint: http://192.0.2.6:5000/v2.0
Overcloud Deployed

On the controller:
pcs status | grep aodh -A1
 Clone Set: openstack-aodh-listener-clone [openstack-aodh-listener]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-aodh-notifier-clone [openstack-aodh-notifier]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-aodh-evaluator-clone [openstack-aodh-evaluator]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]

Packages:
---------
openstack-tripleo-heat-templates-2.0.0-18.el7ost.noarch
openstack-tripleo-puppet-elements-2.0.0-3.el7ost.noarch
puppet-3.6.2-4.el7sat.noarch
openstack-puppet-modules-8.1.5-1.el7ost.noarch

Comment 9 errata-xmlrpc 2016-08-11 11:33:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-1599.html