Bug 1235703 - keystone started by systemd initially, causing pacemaker resource for keystone to fail to start
Summary: keystone started by systemd initially, causing pacemaker resource for keyston...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: Director
Hardware: Unspecified
OS: Unspecified
high
unspecified
Target Milestone: beta
: Director
Assignee: Giulio Fidente
QA Contact: Mike Abrams
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-06-25 14:27 UTC by James Slagle
Modified: 2015-08-05 13:56 UTC (History)
10 users (show)

Fixed In Version: openstack-tripleo-heat-templates-0.8.6-19.el7ost
Doc Type: Bug Fix
Doc Text:
The Keystone service started through Pacemaker and as a systemd dependency of the Ceilometer resource in Pacemaker. This caused conflicts between the two versions of the Keystone service starting that produced failures for the Pacemaker Keystone resource to start. This fix adds a Pacemaker constraint to halt the Ceilometer resource until the Keystone resource starts. Keystone starts before Ceilometer and the Keystone service does not start through systemd.
Clone Of:
Environment:
Last Closed: 2015-08-05 13:56:13 UTC
Target Upstream Version:


Attachments (Terms of Use)
pacemaker.log (2.25 MB, text/plain)
2015-06-25 14:30 UTC, James Slagle
no flags Details
pcs status output (4.98 KB, text/plain)
2015-06-25 14:31 UTC, James Slagle
no flags Details


Links
System ID Priority Status Summary Last Updated
OpenStack gerrit 195622 None None None Never
Red Hat Product Errata RHEA-2015:1549 normal SHIPPED_LIVE Red Hat Enterprise Linux OpenStack Platform director Release 2015-08-05 17:49:10 UTC

Description James Slagle 2015-06-25 14:27:25 UTC
On my overcloud, I'm seeing where keystone is started via systemd initially instead of pacemaker. This causes the pacemaker resource to fail to start keystone, and other resources to subsequently fail after that, typically nova-api, nova-scheduler, nova-conductor.

I think this might be caused by the openstack-heat-engine service having an After= on openstack-keystone, yet there is no pacemaker constraint in overcloud_controller_pacemaker.pp defining this constraint, so keystone is not yet started when heat-engine is started, so systemd tries to start it.

Comment 3 James Slagle 2015-06-25 14:30:12 UTC
was working with giulio to debug this

Comment 4 James Slagle 2015-06-25 14:30:46 UTC
Created attachment 1043137 [details]
pacemaker.log

Comment 5 James Slagle 2015-06-25 14:31:06 UTC
Created attachment 1043138 [details]
pcs status output

Comment 6 Giulio Fidente 2015-06-25 15:25:17 UTC
Ceilometer and Heat constraints groups are missing initial dependency on Keystone

Comment 7 James Slagle 2015-06-25 15:51:28 UTC
testing with the patch I get a failed puppet deployment:

[stack@instack ~]$ heat deployment-show c4337ea9-c192-4d5c-9f80-0ff5fe7d1ff7 | jq .output_values.deploy_stderr | xargs echo -e
\u001b[1;31mWarning: Scope(Class[Keystone]): Execution of db_sync does not depend on $enabled anymore. Please use sync_db instead.\u001b[0m
\u001b[1;31mWarning: Scope(Class[Glance::Registry]): Execution of db_sync does not depend on $manage_service or $enabled anymore. Please use sync_db instead.\u001b[0m
\u001b[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_host'; class ::nova::compute has not been evaluated\u001b[0m
\u001b[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_protocol'; class ::nova::compute has not been evaluated\u001b[0m
\u001b[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_port'; class ::nova::compute has not been evaluated\u001b[0m
\u001b[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_path'; class ::nova::compute has not been evaluated\u001b[0m
\u001b[1;31mWarning: Scope(Class[Concat::Setup]): concat::setup is deprecated as a public API of the concat module and should no longer be directly included in the manifest.\u001b[0m
\u001b[1;31mError: Duplicate declaration: Pacemaker::Constraint::Base[keystone-then-ceilometer-central-constraint] is already declared in file /var/lib/heat-config/heat-config-puppet/eb303a22-ec41-4acd-89d3-bf98fd694fc5.pp:1121; cannot redeclare at /var/lib/heat-config/heat-config-puppet/eb303a22-ec41-4acd-89d3-bf98fd694fc5.pp:1239 on node overcloud-controller-0.localdomain\u001b[0m
\u001b[1;31mError: Duplicate declaration: Pacemaker::Constraint::Base[keystone-then-ceilometer-central-constraint] is already declared in file /var/lib/heat-config/heat-config-puppet/eb303a22-ec41-4acd-89d3-bf98fd694fc5.pp:1121; cannot redeclare at /var/lib/heat-config/heat-config-puppet/eb303a22-ec41-4acd-89d3-bf98fd694fc5.pp:1239 on node overcloud-controller-0.localdomain\u001b[0m

Comment 8 chris alfonso 2015-06-26 12:22:53 UTC
Ben, had a successful deployment with this patch. James, do you want to have this patch pulled in?

Comment 9 James Slagle 2015-06-26 12:30:04 UTC
(In reply to chris alfonso from comment #8)
> Ben, had a successful deployment with this patch. James, do you want to have
> this patch pulled in?

No. If I'm the only person that encountered this problem, then I think we can assume it's environment related for now. And we've had more successful testing without this patch than with it.

Comment 10 Giulio Fidente 2015-06-26 12:42:24 UTC
Meantime patch was merged upstream; James, I volunteer to test it on the environment where this initially appeared. I think it is safe change to merge and can save us from users/customers hitting it.

Comment 16 errata-xmlrpc 2015-08-05 13:56:13 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2015:1549


Note You need to log in before you can comment on or make changes to this bug.