Bug 1127816 - HA neutron sets pcs properties before starting service
Summary: HA neutron sets pcs properties before starting service
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-foreman-installer
Version: Foreman (RHEL 6)
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ga
: Installer
Assignee: John Eckersberg
QA Contact: Leonid Natapov
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-08-07 15:31 UTC by John Eckersberg
Modified: 2014-08-21 18:08 UTC (History)
6 users (show)

Fixed In Version: openstack-foreman-installer-2.0.18-1.el6ost
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-08-21 18:08:28 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2014:1090 normal SHIPPED_LIVE Red Hat Enterprise Linux OpenStack Platform Enhancement Advisory 2014-08-22 15:28:08 UTC

Description John Eckersberg 2014-08-07 15:31:29 UTC
Description of problem:
The HA neutron manifest ends up setting the pcs properties before the neutron-server service starts up.  Here's the excerpt from the puppet log:

Aug 07 15:10:47 mac5254009e4c3c.example.org puppet-agent[3331]: (/Stage[main]/Quickstack::Neutron::All/Exec[neutron-db-manage upgrade]/returns) executed successfully
Aug 07 15:10:47 mac5254009e4c3c.example.org puppet-agent[3331]: (/Stage[main]/Quickstack::Pacemaker::Neutron/Exec[pcs-neutron-server-set-up]/returns) executed successfully
Aug 07 15:10:48 mac5254009e4c3c.example.org puppet-agent[3331]: (/Stage[main]/Quickstack::Pacemaker::Neutron/Exec[pcs-neutron-server-set-up-on-this-node]/returns) executed successfully
Aug 07 15:11:10 mac5254009e4c3c.example.org puppet-agent[3331]: (/Stage[main]/Quickstack::Pacemaker::Neutron/Exec[all-neutron-nodes-are-up]/returns) executed successfully
Aug 07 15:11:10 mac5254009e4c3c.example.org puppet-agent[3331]: (/Stage[main]/Neutron::Plugins::Ml2/Neutron::Plugins::Ml2::Driver[vxlan]/Neutron_plugin_ml2[ml2_type_vxlan/vxlan_group]/ensure) created
Aug 07 15:11:10 mac5254009e4c3c.example.org puppet-agent[3331]: (/Stage[main]/Neutron::Plugins::Ml2/Neutron::Plugins::Ml2::Driver[vlan]/Neutron_plugin_ml2[ml2_type_vlan/network_vlan_ranges]/ensure) created
Aug 07 15:11:10 mac5254009e4c3c.example.org puppet-agent[3331]: (/Stage[main]/Neutron::Plugins::Ml2/Neutron::Plugins::Ml2::Driver[flat]/Neutron_plugin_ml2[ml2_type_flat/flat_networks]/ensure) created
Aug 07 15:11:12 mac5254009e4c3c.example.org puppet-agent[3331]: (/Stage[main]/Neutron::Server/Service[neutron-server]/ensure) ensure changed 'stopped' to 'running'

This means that the other (non-bootstrap) nodes start at the same time and behavior is unpredictable.  In this particular case they conflicted over loading the tunnel ids into the database.

Error from one of the non-bootstrap nodes:

2014-08-07 15:11:11.964 19644 TRACE neutron.service DBDuplicateEntry: (IntegrityError) (1062, "Duplicate entry '10' for key 'PRIMARY'") 'INSERT INTO ml2_gre_allocations (gre_id, allocated) VALUES (%s, %s)' ((10, 0), (11, 0), (12, 0), (13, 0), (14, 0), (15, 0), (16, 0), (17, 0)  ... displaying 10 of 991 total bound parameter sets ...  (999, 0), (1000, 0))


Version-Release number of selected component (if applicable):
openstack-foreman-installer-2.0.17-1.el6ost.noarch

How reproducible:
Not very, since it's dependent on the order that puppet decided to evaluate resources, and even then it's a bit of a race condition between the different nodes to trigger an error condition.

Comment 2 John Eckersberg 2014-08-07 20:54:09 UTC
https://github.com/redhat-openstack/astapor/pull/344

Comment 6 Leonid Natapov 2014-08-18 10:59:32 UTC
Was unable to reproduce on  openstack-foreman-installer-2.0.20-1.el6ost

Comment 7 errata-xmlrpc 2014-08-21 18:08:28 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1090.html


Note You need to log in before you can comment on or make changes to this bug.