Bug 1017281

It looks like foreman gets this for free once puppet modules are updated, but claiming this so it is clear someone looked at it from the foreman side

https://github.com/redhat-openstack/astapor/archive/openstack-foreman-installer-0.0.23.tar.gz

This looks to still be waiting on the packstack-modules-puppet rpm to be updated, as the ml2 stuff was _just_ merged to packstack upstream today.

Bruce, this text makes sense to me until the issue is addressed.  Let me know if anything else is needed here.

(In reply to Jason Guiditta from comment #4)
> Bruce, this text makes sense to me until the issue is addressed.  Let me
> know if anything else is needed here.

Thanks Jason, should be fine as-is.

The ML2 plugin is present and works in RHOS 4. What is missing is support for deploying with ML2 in puppet, packstack and foreman. It is possible to deploy with these tools using the openvswitch plugin and then manually convert to ML2, as is documented for RDO at http://openstack.redhat.com/Modular_Layer_2_%28ML2%29_Plugin.

Instead of stating "OpenStack Networking does not have ML2 plugin support", why not state that packstack and foreman don't yet support deploying with ML2, and provide a reference to a RHOS KB article with instructions for manual conversion? Of course, we also need to decide whether we officially support customers using ML2.

The puppet module exists, but packstack and foreman are experiencing issues pulling this functionality in presently.  Pushing to A2 so they can remain aligned.

Pull request is up:

https://github.com/redhat-openstack/astapor/pull/107

Tested with: openstack-foreman-installer-1.0.4-1.el6ost.noarch

Configured a Controller (Neutron) to work with OVS+GRE ML2 as follows:

ml2_mechanism_drivers: ["openvswitch"]
ml2_network_vlan_range: ["10:50"]
ml2_tenant_network_type: ["gre"]
ml2_tunnel_id_ranges: ["20:100"]
neutron_core_plugin: neutron.plugins.ml2.plugin.Ml2Plugin

changed all relevant IP addresses to match my server address as well.

Result:
# puppet agent -t -v
Info: Retrieving plugin
Info: Loading facts in /var/lib/puppet/lib/facter/netns_support.rb
Info: Loading facts in /var/lib/puppet/lib/facter/network.rb
Info: Loading facts in /var/lib/puppet/lib/facter/iptables_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/hamysql_active_node.rb
Info: Loading facts in /var/lib/puppet/lib/facter/iptables_persistent_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/facter_dot_d.rb
Info: Loading facts in /var/lib/puppet/lib/facter/ip6tables_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rb
Info: Loading facts in /var/lib/puppet/lib/facter/root_home.rb
Info: Loading facts in /var/lib/puppet/lib/facter/pe_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/ipa_client_configured.rb
Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Puppet::Parser::AST::Resource failed with error ArgumentError: Cannot alias Package[neutron-plugin-ml2] to ["openstack-neutron-ml2"] at /usr/share/packstack/modules/neutron/manifests/plugins/ml2.pp:118; resource ["Package", "openstack-neutron-ml2"] already declared at /usr/share/openstack-foreman-installer/puppet/modules/quickstack/manifests/neutron/controller.pp:178 at /usr/share/packstack/modules/neutron/manifests/plugins/ml2.pp:118 on node <FQDN>
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

Looks like neutron puppet module was updated under us with the upstream fix to install those packages.  I'll make a patch to remove the duplication.  Can you tell me which version of packstack-modules-puppet this was using, so I can verify my fix?

Nir, actually, I was just starting to make the change, and remembered that I already accounted for this change getting merged in.  Would you please retest setting the parameter $ml2_install_deps to false instead of true?  We can still remove the call now that we have the needed dep, but if that param does what it was designed for, this should not be a blocker for the A2 release, just a doc note, at least in my opinion. Going to flip this back to you, let me know how you make out.

(In reply to Jason Guiditta from comment #16)
> Nir, actually, I was just starting to make the change, and remembered that I
> already accounted for this change getting merged in.  Would you please
> retest setting the parameter $ml2_install_deps to false instead of true?  We
> can still remove the call now that we have the needed dep, but if that param
> does what it was designed for, this should not be a blocker for the A2
> release, just a doc note, at least in my opinion. Going to flip this back to
> you, let me know how you make out.

Re-tested (with the latest version) but it still doesn't work as expected.

Reopening.
Tested NVR: openstack-foreman-installer-1.0.4-1.el6ost.noarch

Tried to install OpenStack with Neutron ML2 OVS and GRE and Vlans as follows:

Controller (Neutron):
=====================
ml2_install_deps=false
ml2_network_vlan_ranges=["185:185"]
ml2_tenant_network_types=["vlan", "gre"]
ml2_tunnel_id_ranges=["1:1000"]
ml2_type_drivers=["gre", "vlan"]
ovs_vlan_ranges=ext_net:185:185
neutron_core_plugin=neutron.plugins.ml2.plugin.Ml2Plugin

Neutron Networker:
==================
ovs_bridge_mappings=["ext_net:br-ex"]
ovs_bridge_uplinks=["br-ex:eth3.185"]
ovs_tunnel_iface=eth3
ovs_tunnel_types=["gre"]
ovs_vlan_ranges=ext_net:185:185
tenant_network_type=gre,vlan

Compute (Neutron):
==================
neutron_core_plugin=neutron.plugins.openvswitch.ovs_neutron_plugin.OVSNeutronPluginV2
ovs_tunnel_iface=eth3
ovs_tunnel_types=["gre"]


All services are up and running. but this is what I get when I try to use Neutron.

# neutron router-list
404 Not Found

The resource could not be found.
   
snipped from neutron server.log:

[root@puma08 ]# 2014-03-03 16:37:50.016 9700 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 192.168.160.25
2014-03-03 16:37:54.139 9700 ERROR neutron.openstack.common.rpc.amqp [-] Exception during message handling
2014-03-03 16:37:54.139 9700 TRACE neutron.openstack.common.rpc.amqp Traceback (most recent call last):
2014-03-03 16:37:54.139 9700 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/amqp.py", line 438, in _process_data
2014-03-03 16:37:54.139 9700 TRACE neutron.openstack.common.rpc.amqp     **args)
2014-03-03 16:37:54.139 9700 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/neutron/common/rpc.py", line 45, in dispatch
2014-03-03 16:37:54.139 9700 TRACE neutron.openstack.common.rpc.amqp     neutron_ctxt, version, method, namespace, **kwargs)
2014-03-03 16:37:54.139 9700 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2014-03-03 16:37:54.139 9700 TRACE neutron.openstack.common.rpc.amqp     result = getattr(proxyobj, method)(ctxt, **kwargs)
2014-03-03 16:37:54.139 9700 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/neutron/db/dhcp_rpc_base.py", line 63, in get_active_networks_info
2014-03-03 16:37:54.139 9700 TRACE neutron.openstack.common.rpc.amqp     networks = self._get_active_networks(context, **kwargs)
2014-03-03 16:37:54.139 9700 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/neutron/db/dhcp_rpc_base.py", line 43, in _get_active_networks
2014-03-03 16:37:54.139 9700 TRACE neutron.openstack.common.rpc.amqp     context, host)
2014-03-03 16:37:54.139 9700 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/neutron/db/agentschedulers_db.py", line 185, in list_active_networks_on_active_dhcp_agent
2014-03-03 16:37:54.139 9700 TRACE neutron.openstack.common.rpc.amqp     net_ids = [item[0] for item in query]
2014-03-03 16:37:54.139 9700 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2227, in __iter__
2014-03-03 16:37:54.139 9700 TRACE neutron.openstack.common.rpc.amqp     return self._execute_and_instances(context)
2014-03-03 16:37:54.139 9700 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2242, in _execute_and_instances
2014-03-03 16:37:54.139 9700 TRACE neutron.openstack.common.rpc.amqp     result = conn.execute(querycontext.statement, self._params)
2014-03-03 16:37:54.139 9700 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1449, in execute
2014-03-03 16:37:54.139 9700 TRACE neutron.openstack.common.rpc.amqp     params)
2014-03-03 16:37:54.139 9700 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement
2014-03-03 16:37:54.139 9700 TRACE neutron.openstack.common.rpc.amqp     compiled_sql, distilled_params
2014-03-03 16:37:54.139 9700 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1698, in _execute_context
2014-03-03 16:37:54.139 9700 TRACE neutron.openstack.common.rpc.amqp     context)
2014-03-03 16:37:54.139 9700 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1691, in _execute_context
2014-03-03 16:37:54.139 9700 TRACE neutron.openstack.common.rpc.amqp     context)
2014-03-03 16:37:54.139 9700 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/default.py", line 331, in do_execute
2014-03-03 16:37:54.139 9700 TRACE neutron.openstack.common.rpc.amqp     cursor.execute(statement, parameters)
2014-03-03 16:37:54.139 9700 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/site-packages/MySQLdb/cursors.py", line 173, in execute
2014-03-03 16:37:54.139 9700 TRACE neutron.openstack.common.rpc.amqp     self.errorhandler(self, exc, value)
2014-03-03 16:37:54.139 9700 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
2014-03-03 16:37:54.139 9700 TRACE neutron.openstack.common.rpc.amqp     raise errorclass, errorvalue
2014-03-03 16:37:54.139 9700 TRACE neutron.openstack.common.rpc.amqp ProgrammingError: (ProgrammingError) (1146, "Table 'neutron.networkdhcpagentbindings' doesn't exist") 'SELECT networkdhcpagentbindings.network_id AS networkdhcpagentbindings_network_id \nFROM networkdhcpagentbindings \nWHERE networkdhcpagentbindings.dhcp_agent_id = %s' ('9dab8d15-e6b3-4f1a-8084-de3cd07a563d',)
2014-03-03 16:37:54.139 9700 TRACE neutron.openstack.common.rpc.amqp 
2014-03-03 16:37:54.142 9700 ERROR neutron.openstack.common.rpc.common [-] Returning exception (ProgrammingError) (1146, "Table 'neutron.networkdhcpagentbindings' doesn't exist") 'SELECT networkdhcpagentbindings.network_id AS networkdhcpagentbindings_network_id \nFROM networkdhcpagentbindings \nWHERE networkdhcpagentbindings.dhcp_agent_id = %s' ('9dab8d15-e6b3-4f1a-8084-de3cd07a563d',) to caller
2014-03-03 16:37:54.142 9700 ERROR neutron.openstack.common.rpc.common [-] ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/amqp.py", line 438, in _process_data\n    **args)\n', '  File "/usr/lib/python2.6/site-packages/neutron/common/rpc.py", line 45, in dispatch\n    neutron_ctxt, version, method, namespace, **kwargs)\n', '  File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/dispatcher.py", line 172, in dispatch\n    result = getattr(proxyobj, method)(ctxt, **kwargs)\n', '  File "/usr/lib/python2.6/site-packages/neutron/db/dhcp_rpc_base.py", line 63, in get_active_networks_info\n    networks = self._get_active_networks(context, **kwargs)\n', '  File "/usr/lib/python2.6/site-packages/neutron/db/dhcp_rpc_base.py", line 43, in _get_active_networks\n    context, host)\n', '  File "/usr/lib/python2.6/site-packages/neutron/db/agentschedulers_db.py", line 185, in list_active_networks_on_active_dhcp_agent\n    net_ids = [item[0] for item in query]\n', '  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2227, in __iter__\n    return self._execute_and_instances(context)\n', '  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2242, in _execute_and_instances\n    result = conn.execute(querycontext.statement, self._params)\n', '  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1449, in execute\n    params)\n', '  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement\n    compiled_sql, distilled_params\n', '  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1698, in _execute_context\n    context)\n', '  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1691, in _execute_context\n    context)\n', '  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/default.py", line 331, in do_execute\n    cursor.execute(statement, parameters)\n', '  File "/usr/lib64/python2.6/site-packages/MySQLdb/cursors.py", line 173, in execute\n    self.errorhandler(self, exc, value)\n', '  File "/usr/lib64/python2.6/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler\n    raise errorclass, errorvalue\n', 'ProgrammingError: (ProgrammingError) (1146, "Table \'neutron.networkdhcpagentbindings\' doesn\'t exist") \'SELECT networkdhcpagentbindings.network_id AS networkdhcpagentbindings_network_id \\nFROM networkdhcpagentbindings \\nWHERE networkdhcpagentbindings.dhcp_agent_id = %s\' (\'9dab8d15-e6b3-4f1a-8084-de3cd07a563d\',)\n']


Seems like some tables are absent from neutron database (in our case: the networkdhcpagentbindings table)

The tables list in Neutron db post foreman ML2 installation:

+------------------------------+
| Tables_in_neutron            |
+------------------------------+
| agents                       |
| alembic_version              |
| allowedaddresspairs          |
| arista_provisioned_nets      |
| arista_provisioned_tenants   |
| arista_provisioned_vms       |
| cisco_ml2_credentials        |
| cisco_ml2_nexusport_bindings |
| dnsnameservers               |
| externalnetworks             |
| floatingips                  |
| ipallocationpools            |
| ipallocations                |
| ipavailabilityranges         |
| ml2_flat_allocations         |
| ml2_gre_allocations          |
| ml2_gre_endpoints            |
| ml2_network_segments         |
| ml2_port_bindings            |
| ml2_vlan_allocations         |
| ml2_vxlan_allocations        |
| ml2_vxlan_endpoints          |
| networks                     |
| ports                        |
| quotas                       |
| routers                      |
| routes                       |
| securitygroups               |
| servicedefinitions           |
| servicetypes                 |
| subnets                      |
+------------------------------+

Nir, I have actually seen this same behavior on someone else's deployment, and I believe it is something wrong with neutron setup itself (vs our code, at least for this part), but I have been unable to track it down yet.  I have 2 questions that woudl help me determine where the issue is coming from:

1. Have you had any better luck having packstack install neutron?  
2. Can you tell me what versions you have of packstack-modules-puppet, neutron, and neutron-ml2?

(In reply to Jason Guiditta from comment #18)
> Nir, I have actually seen this same behavior on someone else's deployment,
> and I believe it is something wrong with neutron setup itself (vs our code,
> at least for this part), but I have been unable to track it down yet.  I
> have 2 questions that woudl help me determine where the issue is coming from:
> 
> 1. Have you had any better luck having packstack install neutron?

No, But for different reasons. See:
https://bugzilla.redhat.com/show_bug.cgi?id=1068962#c7
https://bugzilla.redhat.com/show_bug.cgi?id=1066519#c5

Nevertheless, When I configured my setup with ML2 manually[1], I did not encounter such issues.

> 2. Can you tell me what versions you have of packstack-modules-puppet,
> neutron, and neutron-ml2?

Unfortunately, the setup is not longer live. Yet since I used the 014-02-28.3 Havana puddle, i would presume it was packstack-modules-puppet-2013.2.1-0.25.dev987.el6ost.noarch



[1] http://openstack.redhat.com/ML2_plugin

This will not make A3, pushing to A4.

Here's why the networkdhcpagentbindings table is missing:

https://git.openstack.org/cgit/openstack/neutron/tree/neutron/db/migration/alembic_migrations/versions/4692d074d587_agent_scheduler.py?id=00281819f3707372e14005b0be8363fca17bd4e1#n32

The ml2 plugin isn't in the array, so it gets skipped in the upgrade method.  Thus table doesn't exist.  If you run the migrations once with one of the supported plugins enabled (e.g. ovs_neutron), the table will get created and then changing plugins later will leave it intact.  Since the manual setup instructions state:

"Start with a working packstack installation with neutron and openvswitch." 

I'm pretty sure that's why it works with a manual install but not with foreman.

How to fix the problem, I don't exactly know.  Add the ml2 plugin to the migration plugin list?  I'm not familiar enough with neutron and its migrations to know for sure.

I wonder then why it also seems to work for packstack?  I do remember Bob Kukura mentioning this bug, and we tried to use this fix (and the one other_ at the time, however I think we were hung up because of other issues.  After that, I must have forgotten about it, as it was at least several weeks ago.  I searched through my irc logs, and the proposed fix (at least for the same kind of problem) was out a while ago, but is marked abandoned:

https://review.openstack.org/#/c/61677/

Not sure if that is any help, but at least some context.

(In reply to Jason Guiditta from comment #24)
> I wonder then why it also seems to work for packstack?  I do remember Bob
> Kukura mentioning this bug, and we tried to use this fix (and the one other_
> at the time, however I think we were hung up because of other issues.  After
> that, I must have forgotten about it, as it was at least several weeks ago. 
> I searched through my irc logs, and the proposed fix (at least for the same
> kind of problem) was out a while ago, but is marked abandoned:
> 
> https://review.openstack.org/#/c/61677/
> 
> Not sure if that is any help, but at least some context.

Aye, I saw that abandoned one.  The one I was most recently looking at
is here:

https://bugs.launchpad.net/neutron/+bug/1260224
https://review.openstack.org/#/c/61663/

I got to that bug via this bug which was marked as a dupe:

https://bugs.launchpad.net/neutron/+bug/1264464

It all sounds related.

I've followed the instructions from comment #17, on:

openstack-foreman-installer-1.0.5-1.el6ost.noarch

And on the controller:

openstack-neutron-2013.2.2-5.el6ost.noarch

The only deviation from the config in comment #17 was changing
ml2_network_vlan_ranges=["185:185"]
to
ml2_network_vlan_ranges=["ext_net:185:185"]

The former gives:

2014-03-15 01:10:59.841 1300 TRACE neutron.plugins.ml2.drivers.type_vlan NetworkVlanRangeError: Invalid network VLAN range: '185:185' - 'need more than 2 values to unpack'

In this setup, I get the following errors.

Instead of the 404 on router-list, instead i get:

# neutron router-list
Request Failed: internal server error while processing your request.

And the traceback from the neutron server.log:

2014-03-15 01:34:20.413 3403 TRACE neutron.api.v2.resource Traceback (most recent call last):
2014-03-15 01:34:20.413 3403 TRACE neutron.api.v2.resource   File "/usr/lib/python2.6/site-packages/neutron/api/v2/resource.py", line 84, in resource
2014-03-15 01:34:20.413 3403 TRACE neutron.api.v2.resource     result = method(request=request, **args)
2014-03-15 01:34:20.413 3403 TRACE neutron.api.v2.resource   File "/usr/lib/python2.6/site-packages/neutron/api/v2/base.py", line 273, in index
2014-03-15 01:34:20.413 3403 TRACE neutron.api.v2.resource     return self._items(request, True, parent_id)
2014-03-15 01:34:20.413 3403 TRACE neutron.api.v2.resource   File "/usr/lib/python2.6/site-packages/neutron/api/v2/base.py", line 227, in _items
2014-03-15 01:34:20.413 3403 TRACE neutron.api.v2.resource     obj_list = obj_getter(request.context, **kwargs)
2014-03-15 01:34:20.413 3403 TRACE neutron.api.v2.resource   File "/usr/lib/python2.6/site-packages/neutron/db/extraroute_db.py", line 165, in get_routers
2014-03-15 01:34:20.413 3403 TRACE neutron.api.v2.resource     marker=marker, page_reverse=page_reverse)
2014-03-15 01:34:20.413 3403 TRACE neutron.api.v2.resource   File "/usr/lib/python2.6/site-packages/neutron/db/l3_db.py", line 266, in get_routers
2014-03-15 01:34:20.413 3403 TRACE neutron.api.v2.resource     page_reverse=page_reverse)
2014-03-15 01:34:20.413 3403 TRACE neutron.api.v2.resource   File "/usr/lib/python2.6/site-packages/neutron/db/db_base_plugin_v2.py", line 196, in _get_collection
2014-03-15 01:34:20.413 3403 TRACE neutron.api.v2.resource     items = [dict_func(c, fields) for c in query]
2014-03-15 01:34:20.413 3403 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2227, in __iter__
2014-03-15 01:34:20.413 3403 TRACE neutron.api.v2.resource     return self._execute_and_instances(context)
2014-03-15 01:34:20.413 3403 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2242, in _execute_and_instances
2014-03-15 01:34:20.413 3403 TRACE neutron.api.v2.resource     result = conn.execute(querycontext.statement, self._params)
2014-03-15 01:34:20.413 3403 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1449, in execute
2014-03-15 01:34:20.413 3403 TRACE neutron.api.v2.resource     params)
2014-03-15 01:34:20.413 3403 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement
2014-03-15 01:34:20.413 3403 TRACE neutron.api.v2.resource     compiled_sql, distilled_params
2014-03-15 01:34:20.413 3403 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1698, in _execute_context
2014-03-15 01:34:20.413 3403 TRACE neutron.api.v2.resource     context)
2014-03-15 01:34:20.413 3403 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1691, in _execute_context
2014-03-15 01:34:20.413 3403 TRACE neutron.api.v2.resource     context)
2014-03-15 01:34:20.413 3403 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/default.py", line 331, in do_execute
2014-03-15 01:34:20.413 3403 TRACE neutron.api.v2.resource     cursor.execute(statement, parameters)
2014-03-15 01:34:20.413 3403 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.6/site-packages/MySQLdb/cursors.py", line 173, in execute
2014-03-15 01:34:20.413 3403 TRACE neutron.api.v2.resource     self.errorhandler(self, exc, value)
2014-03-15 01:34:20.413 3403 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.6/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
2014-03-15 01:34:20.413 3403 TRACE neutron.api.v2.resource     raise errorclass, errorvalue
2014-03-15 01:34:20.413 3403 TRACE neutron.api.v2.resource ProgrammingError: (ProgrammingError) (1146, "Table 'neutron.routerroutes' doesn't exist") 'SELECT routers.tenant_id AS routers_tenant_id, routers.id AS routers_id, routers.name AS routers_name, routers.status AS routers_status, routers.admin_state_up AS routers_admin_state_up, routers.gw_port_id AS routers_gw_port_id, routers.enable_snat AS routers_enable_snat, routerroutes_1.destination AS routerroutes_1_destination, routerroutes_1.nexthop AS routerroutes_1_nexthop, routerroutes_1.router_id AS routerroutes_1_router_id \nFROM routers LEFT OUTER JOIN routerroutes AS routerroutes_1 ON routers.id = routerroutes_1.router_id' ()

Confirmed there is no neutron.routerroutes table:

mysql> select table_name from information_schema.tables where table_schema = 'neutron' and table_name like '%route%';
+------------+
| table_name |
+------------+
| routers    |
| routes     |
+------------+
2 rows in set (0.00 sec)

What version of packstack-modules-puppet do you have?  The latest version was supposed to include a fix for the clan range thing.  They were previously a bit over aggressive with validation, meaning you could not use provider networks.

(In reply to Jason Guiditta from comment #27)
> What version of packstack-modules-puppet do you have?  The latest version
> was supposed to include a fix for the clan range thing.  They were
> previously a bit over aggressive with validation, meaning you could not use
> provider networks.

packstack-modules-puppet-2013.2.1-0.28.dev989.el6ost.noarch

OK, I think I've dug to the root cause here:

https://bugs.launchpad.net/neutron/+bug/1288358/comments/3

The end result is neutron tables created with myisam storage engine, and then table creation failing due to lack of support of foreign keys in myisam and/or foreign keys between tables on different storage engines.

I posted a workaround[1] to force InnoDB.  Repeated here:

cat <<EOF > /etc/mysql/conf.d/innodb.cnf 
[mysqld]
default-storage-engine = innodb
EOF

service mysqld restart

mysql -e 'drop database neutron; create database neutron;'

Then either run puppet agent -tv or neutron-db-manage to rerun the migrations and default all tables to innodb.

After that, I get an empty router list:

[root@control ~]# neutron router-list

[root@control ~]# 


[1] https://www.redhat.com/archives/rdo-list/2014-March/msg00068.html

There's one more issue I almost forgot about.  I had to apply this patch to make the agents table get created:

https://review.openstack.org/#/c/79660/

I've submitted a review upstream in neutron to correct the migrations so the tables get creates using innodb:

https://review.openstack.org/81334

I've split out the Neutron issue here:

https://bugzilla.redhat.com/show_bug.cgi?id=1085360

As far as I can tell, everything in foreman is working as expected.  It correctly installs and configures the ml2 plugin after the changes Jay made earlier in the bug.

Closing this as CLOSED ERRATA, since Jay's fix to enable install/configuration within openstack-foreman-installer was part of RHBA-2014:0213-03.  The remaining issue is being tracked against Neutron.