Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1348842

Summary:

After migration to AODH, its services run only on one controller

Product:

Red Hat OpenStack

Reporter:

Jiri Stransky <jstransk>

Component:

openstack-tripleo-heat-templates

Assignee:

Michele Baldessari <michele>

Status:

CLOSED ERRATA

QA Contact:

Yurii Prokulevych <yprokule>

Severity:

unspecified

Docs Contact:

Priority:

unspecified

Version:

9.0 (Mitaka)

CC:

jason.dobies, mandreou, mburns, morazi, rhel-osp-director-maint, royoung, tvignaud

Target Milestone:

Keywords:

Triaged

Target Release:

9.0 (Mitaka)

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

openstack-tripleo-heat-templates-2.0.0-12

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2016-08-11 11:33:10 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Bug Depends On:

Bug Blocks:

1333977

Attachments:

Description	Flags
pcs status	none
pcs constraint show	none

Description Jiri Stransky 2016-06-22 08:10:11 UTC

Description of problem:

After migration to AODH before OSP 8->9 upgrade, AODH services run on only one controller, even after pcs resource cleanup pacemaker doesn't try to start them on the remaining nodes.

The interesting thing is that the node where AODH is running seemed to coincide with it being the redis-master node. There's a redis-master ordering constraint with AODH, but not a colocation constraint, so the behavior seems strange. `crm_resource --wait` returns immediately, so there are no pending operations.
 

Version-Release number of selected component (if applicable):

openstack-tripleo-heat-templates-2.0.0-11.el7ost.noarch

pacemaker-cluster-libs-1.1.13-10.el7_2.2.x86_64
pacemaker-1.1.13-10.el7_2.2.x86_64
pacemaker-remote-1.1.13-10.el7_2.2.x86_64
pacemaker-cli-1.1.13-10.el7_2.2.x86_64
pacemaker-libs-1.1.13-10.el7_2.2.x86_64

resource-agents-3.9.5-54.el7_2.10.x86_64

Comment 1 Jiri Stransky 2016-06-22 08:12:53 UTC

Created attachment 1170623 [details]
pcs status

the httpd monitor timeouts are due to next migration step (keystone wsgi)

Comment 2 Jiri Stransky 2016-06-22 08:13:17 UTC

Created attachment 1170624 [details]
pcs constraint show

Comment 3 Jiri Stransky 2016-06-29 08:35:49 UTC

I was able to verify bandini's fix today, it no longer occurs.

Comment 4 Marios Andreou 2016-07-06 16:23:38 UTC

o/ just to verify that today, using latest poodle, with (upgraded) undercloud packages like [0] and overcloud (still on osp8 repos) like [1] I got through the aodh upgrade successfully with no stopped services. I didn't apply any workarounds. For reference, deployed like:

    openstack overcloud deploy --templates /usr/share/openstack-tripleo-heat-templates -e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml  --control-scale 3 --compute-scale 1 --libvirt-type qemu -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e network_env.yaml --ntp-server '0.fedora.pool.ntp.org'

And then apply the aodh migration like:

    openstack overcloud deploy --templates /usr/share/openstack-tripleo-heat-templates -e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml  --control-scale 3 --compute-scale 1 --libvirt-type qemu -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e network_env.yaml --ntp-server '0.fedora.pool.ntp.org' -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-aodh.yaml
    
thanks, marios

[0]
[stack@instack ~]$ rpm -qa | grep "openstack\|instack\|puppet"
openstack-ceilometer-notification-6.1.0-1.el7ost.noarch
instack-undercloud-4.0.0-5.el7ost.noarch
python-openstacksdk-0.8.3-1.el7ost.noarch
openstack-heat-api-cloudwatch-6.0.0-6.el7ost.noarch
openstack-nova-cert-13.1.0-2.el7ost.noarch
openstack-ceilometer-api-6.1.0-1.el7ost.noarch
openstack-swift-account-2.6.0-1.el7ost.noarch
openstack-utils-2015.2-1.el7ost.noarch
openstack-heat-templates-0-0.3.96a0b0bgit.el7ost.noarch
openstack-swift-container-2.6.0-1.el7ost.noarch
openstack-tripleo-heat-templates-liberty-2.0.0-13.el7ost.noarch
openstack-aodh-evaluator-2.0.1-3.el7ost.noarch
openstack-ironic-common-5.1.2-2.el7ost.noarch
openstack-heat-common-6.0.0-6.el7ost.noarch
openstack-nova-api-13.1.0-2.el7ost.noarch
openstack-ceilometer-collector-6.1.0-1.el7ost.noarch
openstack-tempest-10.0.0-1.b4a056dgit.el7ost.noarch
instack-0.0.8-3.el7ost.noarch
openstack-tripleo-heat-templates-kilo-2.0.0-13.el7ost.noarch
openstack-ceilometer-common-6.1.0-1.el7ost.noarch
openstack-aodh-api-2.0.1-3.el7ost.noarch
openstack-neutron-ml2-8.1.0-11.el7ost.noarch
openstack-ceilometer-central-6.1.0-1.el7ost.noarch
openstack-swift-proxy-2.6.0-1.el7ost.noarch
openstack-selinux-0.7.3-3.el7ost.noarch
puppet-3.6.2-2.el7.noarch
openstack-swift-2.6.0-1.el7ost.noarch
openstack-ceilometer-polling-6.1.0-1.el7ost.noarch
openstack-aodh-notifier-2.0.1-3.el7ost.noarch
openstack-nova-common-13.1.0-2.el7ost.noarch
python-openstackclient-2.2.0-1.el7ost.noarch
openstack-heat-engine-6.0.0-6.el7ost.noarch
openstack-neutron-openvswitch-8.1.0-11.el7ost.noarch
openstack-nova-conductor-13.1.0-2.el7ost.noarch
openstack-ironic-conductor-5.1.2-2.el7ost.noarch
openstack-keystone-9.0.0-1.el7ost.noarch
openstack-puppet-modules-8.1.2-1.el7ost.noarch
openstack-tripleo-heat-templates-2.0.0-13.el7ost.noarch
openstack-tripleo-puppet-elements-2.0.0-3.el7ost.noarch
openstack-aodh-listener-2.0.1-3.el7ost.noarch
openstack-tripleo-common-2.0.0-6.el7ost.noarch
openstack-neutron-8.1.0-11.el7ost.noarch
openstack-ironic-api-5.1.2-2.el7ost.noarch
openstack-ironic-inspector-3.2.2-4.el7ost.noarch
openstack-tripleo-image-elements-0.9.9-6.el7ost.noarch
openstack-tripleo-0.0.8-0.2.d81bd6dgit.el7ost.noarch
openstack-aodh-expirer-2.0.1-3.el7ost.noarch
openstack-neutron-common-8.1.0-11.el7ost.noarch
openstack-heat-api-6.0.0-6.el7ost.noarch
openstack-nova-compute-13.1.0-2.el7ost.noarch
openstack-aodh-compat-2.0.1-3.el7ost.noarch
openstack-swift-object-2.6.0-1.el7ost.noarch
openstack-swift-plugin-swift3-1.10-1.el7ost.noarch
openstack-aodh-common-2.0.1-3.el7ost.noarch
openstack-heat-api-cfn-6.0.0-6.el7ost.noarch
openstack-nova-scheduler-13.1.0-2.el7ost.noarch
openstack-glance-12.0.0-1.el7ost.noarch


[1] 
[root@overcloud-controller-0 ~]# rpm -qa | grep "openstack\|instack\|puppet"
openstack-nova-cert-12.0.4-3.el7ost.noarch
openstack-puppet-modules-7.1.2-1.el7ost.noarch
openstack-aodh-api-1.1.2-1.el7ost.noarch
openstack-swift-2.5.0-2.el7ost.noarch
openstack-ceilometer-common-5.0.2-2.el7ost.noarch
openstack-neutron-7.0.4-11.el7ost.noarch
openstack-heat-api-cfn-5.0.1-6.el7ost.noarch
openstack-ceilometer-api-5.0.2-2.el7ost.noarch
openstack-swift-object-2.5.0-2.el7ost.noarch
openstack-ceilometer-compute-5.0.2-2.el7ost.noarch
openstack-nova-console-12.0.4-3.el7ost.noarch
openstack-aodh-notifier-1.1.2-1.el7ost.noarch
python-openstackclient-1.7.2-1.el7ost.noarch
openstack-swift-container-2.5.0-2.el7ost.noarch
openstack-dashboard-theme-8.0.1-4.el7ost.noarch
openstack-cinder-7.0.2-2.el7ost.noarch
openstack-heat-engine-5.0.1-6.el7ost.noarch
openstack-utils-2014.2-1.el7ost.noarch
openstack-nova-common-12.0.4-3.el7ost.noarch
openstack-neutron-common-7.0.4-11.el7ost.noarch
openstack-swift-proxy-2.5.0-2.el7ost.noarch
openstack-nova-api-12.0.4-3.el7ost.noarch
openstack-nova-novncproxy-12.0.4-3.el7ost.noarch
openstack-aodh-evaluator-1.1.2-1.el7ost.noarch
python-django-openstack-auth-2.0.1-3.el7ost.noarch
openstack-ceilometer-polling-5.0.2-2.el7ost.noarch
openstack-dashboard-8.0.1-4.el7ost.noarch
openstack-glance-11.0.1-4.el7ost.noarch
openstack-keystone-8.0.1-2.el7ost.noarch
openstack-heat-api-5.0.1-6.el7ost.noarch
openstack-ceilometer-collector-5.0.2-2.el7ost.noarch
openstack-nova-conductor-12.0.4-3.el7ost.noarch
openstack-neutron-ml2-7.0.4-11.el7ost.noarch
openstack-aodh-common-1.1.2-1.el7ost.noarch
openstack-nova-compute-12.0.4-3.el7ost.noarch
openstack-ceilometer-central-5.0.2-2.el7ost.noarch
openstack-nova-scheduler-12.0.4-3.el7ost.noarch
openstack-neutron-metering-agent-7.0.4-11.el7ost.noarch
openstack-selinux-0.6.58-1.el7ost.noarch
puppet-3.6.2-2.el7.noarch
openstack-aodh-listener-1.1.2-1.el7ost.noarch
openstack-swift-plugin-swift3-1.9-1.el7ost.noarch
openstack-ceilometer-notification-5.0.2-2.el7ost.noarch
openstack-neutron-bigswitch-lldp-2015.3.8-1.el7ost.noarch
openstack-heat-common-5.0.1-6.el7ost.noarch
openstack-neutron-lbaas-7.0.0-2.el7ost.noarch
openstack-heat-api-cloudwatch-5.0.1-6.el7ost.noarch
openstack-neutron-openvswitch-7.0.4-11.el7ost.noarch
openstack-swift-account-2.5.0-2.el7ost.noarch
openstack-manila-1.0.1-3.el7ost.noarch
openstack-neutron-bigswitch-agent-2015.3.8-1.el7ost.noarch
openstack-manila-share-1.0.1-3.el7ost.noarch

Comment 6 Yurii Prokulevych 2016-07-26 15:30:19 UTC

Deploy command:
    openstack  overcloud deploy --libvirt-type qemu \
        --ntp-server clock.redhat.com --templates \
        --control-scale 3 --compute-scale 1 \
        --ceph-storage-scale 1 -e storage-environment.yaml

storage-environment.yaml:
  parameter_defaults:
    CinderEnableIscsiBackend: false
    CinderEnableRbdBackend: true
    NovaEnableRbdBackend: false
    GlanceBackend: rbd
    GnocchiBackend: rbd


Upgrade command:
    openstack  overcloud deploy --libvirt-type qemu \
        --ntp-server clock.redhat.com --templates \
        --control-scale 3 --compute-scale 1 \
        --ceph-storage-scale 1 -e storage-environment.yaml \
        -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-aodh.yaml \
        --force-postconfig

...
2016-07-26 07:39:32 [NovaComputeDeployment]: SIGNAL_COMPLETE Unknown
Stack overcloud UPDATE_COMPLETE
PKI initialization in init-keystone is deprecated and will be removed.
Overcloud Endpoint: http://192.0.2.6:5000/v2.0
Overcloud Deployed

On the controller:
pcs status | grep aodh -A1
 Clone Set: openstack-aodh-listener-clone [openstack-aodh-listener]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-aodh-notifier-clone [openstack-aodh-notifier]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-aodh-evaluator-clone [openstack-aodh-evaluator]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]

Packages:
---------
openstack-tripleo-heat-templates-2.0.0-18.el7ost.noarch
openstack-tripleo-puppet-elements-2.0.0-3.el7ost.noarch
puppet-3.6.2-4.el7sat.noarch
openstack-puppet-modules-8.1.5-1.el7ost.noarch

Comment 9 errata-xmlrpc 2016-08-11 11:33:10 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-1599.html