Bug 1127816

Description of problem:
The HA neutron manifest ends up setting the pcs properties before the neutron-server service starts up.  Here's the excerpt from the puppet log:

Aug 07 15:10:47 mac5254009e4c3c.example.org puppet-agent[3331]: (/Stage[main]/Quickstack::Neutron::All/Exec[neutron-db-manage upgrade]/returns) executed successfully
Aug 07 15:10:47 mac5254009e4c3c.example.org puppet-agent[3331]: (/Stage[main]/Quickstack::Pacemaker::Neutron/Exec[pcs-neutron-server-set-up]/returns) executed successfully
Aug 07 15:10:48 mac5254009e4c3c.example.org puppet-agent[3331]: (/Stage[main]/Quickstack::Pacemaker::Neutron/Exec[pcs-neutron-server-set-up-on-this-node]/returns) executed successfully
Aug 07 15:11:10 mac5254009e4c3c.example.org puppet-agent[3331]: (/Stage[main]/Quickstack::Pacemaker::Neutron/Exec[all-neutron-nodes-are-up]/returns) executed successfully
Aug 07 15:11:10 mac5254009e4c3c.example.org puppet-agent[3331]: (/Stage[main]/Neutron::Plugins::Ml2/Neutron::Plugins::Ml2::Driver[vxlan]/Neutron_plugin_ml2[ml2_type_vxlan/vxlan_group]/ensure) created
Aug 07 15:11:10 mac5254009e4c3c.example.org puppet-agent[3331]: (/Stage[main]/Neutron::Plugins::Ml2/Neutron::Plugins::Ml2::Driver[vlan]/Neutron_plugin_ml2[ml2_type_vlan/network_vlan_ranges]/ensure) created
Aug 07 15:11:10 mac5254009e4c3c.example.org puppet-agent[3331]: (/Stage[main]/Neutron::Plugins::Ml2/Neutron::Plugins::Ml2::Driver[flat]/Neutron_plugin_ml2[ml2_type_flat/flat_networks]/ensure) created
Aug 07 15:11:12 mac5254009e4c3c.example.org puppet-agent[3331]: (/Stage[main]/Neutron::Server/Service[neutron-server]/ensure) ensure changed 'stopped' to 'running'

This means that the other (non-bootstrap) nodes start at the same time and behavior is unpredictable.  In this particular case they conflicted over loading the tunnel ids into the database.

Error from one of the non-bootstrap nodes:

2014-08-07 15:11:11.964 19644 TRACE neutron.service DBDuplicateEntry: (IntegrityError) (1062, "Duplicate entry '10' for key 'PRIMARY'") 'INSERT INTO ml2_gre_allocations (gre_id, allocated) VALUES (%s, %s)' ((10, 0), (11, 0), (12, 0), (13, 0), (14, 0), (15, 0), (16, 0), (17, 0)  ... displaying 10 of 991 total bound parameter sets ...  (999, 0), (1000, 0))


Version-Release number of selected component (if applicable):
openstack-foreman-installer-2.0.17-1.el6ost.noarch

How reproducible:
Not very, since it's dependent on the order that puppet decided to evaluate resources, and even then it's a bit of a race condition between the different nodes to trigger an error condition.