Bug 1235703

Summary: keystone started by systemd initially, causing pacemaker resource for keystone to fail to start
Product: Red Hat OpenStack Reporter: James Slagle <jslagle>
Component: openstack-tripleo-heat-templatesAssignee: Giulio Fidente <gfidente>
Status: CLOSED ERRATA QA Contact: Mike Abrams <mabrams>
Severity: unspecified Docs Contact:
Priority: high    
Version: DirectorCC: calfonso, dmacpher, jason.dobies, jguiditt, jslagle, mburns, ohochman, rhel-osp-director-maint, rrosa
Target Milestone: betaKeywords: Triaged
Target Release: Director   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-0.8.6-19.el7ost Doc Type: Bug Fix
Doc Text:
The Keystone service started through Pacemaker and as a systemd dependency of the Ceilometer resource in Pacemaker. This caused conflicts between the two versions of the Keystone service starting that produced failures for the Pacemaker Keystone resource to start. This fix adds a Pacemaker constraint to halt the Ceilometer resource until the Keystone resource starts. Keystone starts before Ceilometer and the Keystone service does not start through systemd.
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-08-05 13:56:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
pacemaker.log
none
pcs status output none

Description James Slagle 2015-06-25 14:27:25 UTC
On my overcloud, I'm seeing where keystone is started via systemd initially instead of pacemaker. This causes the pacemaker resource to fail to start keystone, and other resources to subsequently fail after that, typically nova-api, nova-scheduler, nova-conductor.

I think this might be caused by the openstack-heat-engine service having an After= on openstack-keystone, yet there is no pacemaker constraint in overcloud_controller_pacemaker.pp defining this constraint, so keystone is not yet started when heat-engine is started, so systemd tries to start it.

Comment 3 James Slagle 2015-06-25 14:30:12 UTC
was working with giulio to debug this

Comment 4 James Slagle 2015-06-25 14:30:46 UTC
Created attachment 1043137 [details]
pacemaker.log

Comment 5 James Slagle 2015-06-25 14:31:06 UTC
Created attachment 1043138 [details]
pcs status output

Comment 6 Giulio Fidente 2015-06-25 15:25:17 UTC
Ceilometer and Heat constraints groups are missing initial dependency on Keystone

Comment 7 James Slagle 2015-06-25 15:51:28 UTC
testing with the patch I get a failed puppet deployment:

[stack@instack ~]$ heat deployment-show c4337ea9-c192-4d5c-9f80-0ff5fe7d1ff7 | jq .output_values.deploy_stderr | xargs echo -e
\u001b[1;31mWarning: Scope(Class[Keystone]): Execution of db_sync does not depend on $enabled anymore. Please use sync_db instead.\u001b[0m
\u001b[1;31mWarning: Scope(Class[Glance::Registry]): Execution of db_sync does not depend on $manage_service or $enabled anymore. Please use sync_db instead.\u001b[0m
\u001b[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_host'; class ::nova::compute has not been evaluated\u001b[0m
\u001b[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_protocol'; class ::nova::compute has not been evaluated\u001b[0m
\u001b[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_port'; class ::nova::compute has not been evaluated\u001b[0m
\u001b[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_path'; class ::nova::compute has not been evaluated\u001b[0m
\u001b[1;31mWarning: Scope(Class[Concat::Setup]): concat::setup is deprecated as a public API of the concat module and should no longer be directly included in the manifest.\u001b[0m
\u001b[1;31mError: Duplicate declaration: Pacemaker::Constraint::Base[keystone-then-ceilometer-central-constraint] is already declared in file /var/lib/heat-config/heat-config-puppet/eb303a22-ec41-4acd-89d3-bf98fd694fc5.pp:1121; cannot redeclare at /var/lib/heat-config/heat-config-puppet/eb303a22-ec41-4acd-89d3-bf98fd694fc5.pp:1239 on node overcloud-controller-0.localdomain\u001b[0m
\u001b[1;31mError: Duplicate declaration: Pacemaker::Constraint::Base[keystone-then-ceilometer-central-constraint] is already declared in file /var/lib/heat-config/heat-config-puppet/eb303a22-ec41-4acd-89d3-bf98fd694fc5.pp:1121; cannot redeclare at /var/lib/heat-config/heat-config-puppet/eb303a22-ec41-4acd-89d3-bf98fd694fc5.pp:1239 on node overcloud-controller-0.localdomain\u001b[0m

Comment 8 chris alfonso 2015-06-26 12:22:53 UTC
Ben, had a successful deployment with this patch. James, do you want to have this patch pulled in?

Comment 9 James Slagle 2015-06-26 12:30:04 UTC
(In reply to chris alfonso from comment #8)
> Ben, had a successful deployment with this patch. James, do you want to have
> this patch pulled in?

No. If I'm the only person that encountered this problem, then I think we can assume it's environment related for now. And we've had more successful testing without this patch than with it.

Comment 10 Giulio Fidente 2015-06-26 12:42:24 UTC
Meantime patch was merged upstream; James, I volunteer to test it on the environment where this initially appeared. I think it is safe change to merge and can save us from users/customers hitting it.

Comment 16 errata-xmlrpc 2015-08-05 13:56:13 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2015:1549