Bug 1353637

Summary: External load balancer deployment fails with: Error: Could not find dependency Pacemaker::Resource::Ocf[openstack-core] for Pacemaker::Constraint::Base[openstack-core-then-httpd-constraint]
Product: Red Hat OpenStack Reporter: Marius Cornea <mcornea>
Component: openstack-tripleo-heat-templatesAssignee: Giulio Fidente <gfidente>
Status: CLOSED ERRATA QA Contact: Marius Cornea <mcornea>
Severity: urgent Docs Contact:
Priority: high    
Version: 9.0 (Mitaka)CC: dbecker, dmacpher, gfidente, goneri, jason.dobies, jcoufal, jjoyce, mburns, mkrcmari, morazi, ohochman, rhel-osp-director-maint, tvignaud
Target Milestone: gaKeywords: Triaged
Target Release: 9.0 (Mitaka)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-2.0.0-18.el7ost Doc Type: Bug Fix
Doc Text:
The Overcloud uses a fake openstack-core resource to replace keystone (now accessed through httpd). However, the Overcloud did not create openstack-core when using an external load balancer. This caused puppet to fail when creating Pacemaker constraints that required openstack-core. This fix ensures creation of the openstack-core resource for any deployment configuration. The deployment succeeds and creates the needed constraints.
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-08-11 11:35:47 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Marius Cornea 2016-07-07 15:43:15 UTC
Description of problem:

External load balancer deployment fails with: Error: Could not find dependency Pacemaker::Resource::Ocf[openstack-core] for Pacemaker::Constraint::Base[openstack-core-then-httpd-constraint]

Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-2.0.0-12.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy overcloud:

openstack overcloud deploy --debug \
--templates \
--libvirt-type qemu \
--ntp-server clock.redhat.com \
--neutron-network-type vxlan \
--neutron-tunnel-types vxlan \
--control-scale 3 \
--control-flavor controller-d75f3dec-c770-5f88-9d4c-3fea1bf9c484 \--compute-scale 1 \
--compute-flavor compute-b634c10a-570f-59ba-bdbf-0c313d745a10 \
-e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml \
--ceph-storage-scale 1 \
--ceph-storage-flavor ceph-cf1f074b-dadb-5eb8-9eb0-55828273fab7 \
-e /home/stack/virt/ceph.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/external-loadbalancer-vip.yaml \
-e ~/external-lb.yaml \
--log-file overcloud_deployment_76.log &> overcloud_install.log


Actual results:
Deployment fails:
Error: Could not find dependency Pacemaker::Resource::Ocf[openstack-core] for Pacemaker::Constraint::Base[openstack-core-then-httpd-constraint] at /var/lib/heat-config/heat-config-puppet/e7aecd14-71e2-4d7b-a93c-fd9b45016c87.pp:1317


Expected results:
Deployment is successful.

Additional info:

In /usr/share/openstack-tripleo-heat-templates/puppet/manifests/overcloud_controller_pacemaker.pp the openstack-core resource creation appears to be conditioned by the enable_load_balancer variable which is set to false in /usr/share/openstack-tripleo-heat-templates/environments/external-loadbalancer-vip.yaml

  if $pacemaker_master {

    if $enable_load_balancer {

      include ::pacemaker::resource_defaults

      # Create an openstack-core dummy resource. See RHBZ 1290121
      pacemaker::resource::ocf { 'openstack-core':
        ocf_agent_name => 'heartbeat:Dummy',
        clone_params   => true,
      }

These are the resources that exist on the nodes at the failure time:

[root@overcloud-controller-0 ~]# pcs status
Cluster name: tripleo_cluster
Last updated: Thu Jul  7 15:42:47 2016		Last change: Thu Jul  7 14:36:08 2016 by root via crm_attribute on overcloud-controller-2
Stack: corosync
Current DC: overcloud-controller-0 (version 1.1.13-10.el7_2.2-44eb2dd) - partition with quorum
3 nodes and 15 resources configured

Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]

Full list of resources:

 Master/Slave Set: galera-master [galera]
     Masters: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: memcached-clone [memcached]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: rabbitmq-clone [rabbitmq]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Master/Slave Set: redis-master [redis]
     Masters: [ overcloud-controller-2 ]
     Slaves: [ overcloud-controller-0 overcloud-controller-1 ]
 Clone Set: mongod-clone [mongod]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]

PCSD Status:
  overcloud-controller-0: Online
  overcloud-controller-1: Online
  overcloud-controller-2: Online

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

Comment 2 Jay Dobies 2016-07-12 13:18:36 UTC
Giulio - I'm assigning this back to you (I'm guessing it got moved when the component was changed). It looks very close to landing, so can you just make sure it passes CI and gets the +A necessary?

Comment 5 Marius Cornea 2016-07-20 21:09:31 UTC
I tested the patch[1] with downstream ospd9  and I couldn't reproduce this issue anymore.

[1] https://review.openstack.org/#/c/339100/2/puppet/manifests/overcloud_controller_pacemaker.pp

Comment 8 errata-xmlrpc 2016-08-11 11:35:47 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-1599.html