Red Hat Bugzilla – Bug 1235703
keystone started by systemd initially, causing pacemaker resource for keystone to fail to start
Last modified: 2015-08-05 09:56:13 EDT
On my overcloud, I'm seeing where keystone is started via systemd initially instead of pacemaker. This causes the pacemaker resource to fail to start keystone, and other resources to subsequently fail after that, typically nova-api, nova-scheduler, nova-conductor. I think this might be caused by the openstack-heat-engine service having an After= on openstack-keystone, yet there is no pacemaker constraint in overcloud_controller_pacemaker.pp defining this constraint, so keystone is not yet started when heat-engine is started, so systemd tries to start it.
was working with giulio to debug this
Created attachment 1043137 [details] pacemaker.log
Created attachment 1043138 [details] pcs status output
Ceilometer and Heat constraints groups are missing initial dependency on Keystone
testing with the patch I get a failed puppet deployment: [stack@instack ~]$ heat deployment-show c4337ea9-c192-4d5c-9f80-0ff5fe7d1ff7 | jq .output_values.deploy_stderr | xargs echo -e \u001b[1;31mWarning: Scope(Class[Keystone]): Execution of db_sync does not depend on $enabled anymore. Please use sync_db instead.\u001b[0m \u001b[1;31mWarning: Scope(Class[Glance::Registry]): Execution of db_sync does not depend on $manage_service or $enabled anymore. Please use sync_db instead.\u001b[0m \u001b[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_host'; class ::nova::compute has not been evaluated\u001b[0m \u001b[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_protocol'; class ::nova::compute has not been evaluated\u001b[0m \u001b[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_port'; class ::nova::compute has not been evaluated\u001b[0m \u001b[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_path'; class ::nova::compute has not been evaluated\u001b[0m \u001b[1;31mWarning: Scope(Class[Concat::Setup]): concat::setup is deprecated as a public API of the concat module and should no longer be directly included in the manifest.\u001b[0m \u001b[1;31mError: Duplicate declaration: Pacemaker::Constraint::Base[keystone-then-ceilometer-central-constraint] is already declared in file /var/lib/heat-config/heat-config-puppet/eb303a22-ec41-4acd-89d3-bf98fd694fc5.pp:1121; cannot redeclare at /var/lib/heat-config/heat-config-puppet/eb303a22-ec41-4acd-89d3-bf98fd694fc5.pp:1239 on node overcloud-controller-0.localdomain\u001b[0m \u001b[1;31mError: Duplicate declaration: Pacemaker::Constraint::Base[keystone-then-ceilometer-central-constraint] is already declared in file /var/lib/heat-config/heat-config-puppet/eb303a22-ec41-4acd-89d3-bf98fd694fc5.pp:1121; cannot redeclare at /var/lib/heat-config/heat-config-puppet/eb303a22-ec41-4acd-89d3-bf98fd694fc5.pp:1239 on node overcloud-controller-0.localdomain\u001b[0m
Ben, had a successful deployment with this patch. James, do you want to have this patch pulled in?
(In reply to chris alfonso from comment #8) > Ben, had a successful deployment with this patch. James, do you want to have > this patch pulled in? No. If I'm the only person that encountered this problem, then I think we can assume it's environment related for now. And we've had more successful testing without this patch than with it.
Meantime patch was merged upstream; James, I volunteer to test it on the environment where this initially appeared. I think it is safe change to merge and can save us from users/customers hitting it.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2015:1549