Bug 1061378

Summary:	Neutron ML2 DB configuration fails with an error
Product:	[Community] RDO	Reporter:	Nir Magnezi <nmagnezi>
Component:	openstack-neutron	Assignee:	RHOS Maint <rhos-maint>
Status:	CLOSED WORKSFORME	QA Contact:	Nir Magnezi <nmagnezi>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	unspecified	CC:	chrisw, jguiditt, jlibosva, oblaut, sgordon, yeylon
Target Milestone:	---	Keywords:	Regression, TestBlocker
Target Release:	---
Hardware:	Unspecified
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:
Clones:	1082194 (view as bug list)		Environment:
Last Closed:	2014-04-11 12:09:27 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1082194

Description Nir Magnezi 2014-02-04 17:43:37 UTC

Description of problem:
=======================
In order to configure an Icehoue RDO setup to work with ML2, I followed: http://openstack.redhat.com/ML2_plugin

The command: neutron-db-manage --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini upgrade head
Failed with: (ProgrammingError) (1146, "Table 'neutron_ml2.agents' doesn't exist")

Version-Release number of selected component (if applicable):
=============================================================
openstack-neutron-2014.1-0.4.b2.el6.noarch

How reproducible:
=================
Tested only once.

Steps to Reproduce:
===================
1. Install RDO Icehouse via packstack
2. Follow http://openstack.redhat.com/ML2_plugin

Actual results:
===============
stdout: 
stderr: No handlers could be found for logger "neutron.common.legacy"
INFO [alembic.migration] Context impl MySQLImpl.
INFO [alembic.migration] Will assume non-transactional DDL.
INFO [alembic.migration] Running upgrade None -> folsom
INFO [alembic.migration] Running upgrade folsom -> 2c4af419145b
INFO [alembic.migration] Running upgrade 2c4af419145b -> 5a875d0e5c 
INFO [alembic.migration] Running upgrade 5a875d0e5c -> 48b6f43f7471 
INFO [alembic.migration] Running upgrade 48b6f43f7471 -> 3cb5d900c5de 
INFO [alembic.migration] Running upgrade 3cb5d900c5de -> 1d76643bcec4 
INFO [alembic.migration] Running upgrade 1d76643bcec4 -> 2a6d0b51f4bb 
INFO [alembic.migration] Running upgrade 2a6d0b51f4bb -> 1b693c095aa3 
INFO [alembic.migration] Running upgrade 1b693c095aa3 -> 1149d7de0cfa 
INFO [alembic.migration] Running upgrade 1149d7de0cfa -> 49332180ca96 
INFO [alembic.migration] Running upgrade 49332180ca96 -> 38335592a0dc 
INFO [alembic.migration] Running upgrade 38335592a0dc -> 54c2c487e913 
INFO [alembic.migration] Running upgrade 54c2c487e913 -> 45680af419f9 
INFO [alembic.migration] Running upgrade 45680af419f9 -> 1c33fa3cd1a1 
INFO [alembic.migration] Running upgrade 1c33fa3cd1a1 -> 363468ac592c 
INFO [alembic.migration] Running upgrade 363468ac592c -> 511471cc46b 
INFO [alembic.migration] Running upgrade 511471cc46b -> 3b54bf9e29f7 
INFO [alembic.migration] Running upgrade 3b54bf9e29f7 -> 4692d074d587 
INFO [alembic.migration] Running upgrade 4692d074d587 -> 1341ed32cc1e 
INFO [alembic.migration] Running upgrade 1341ed32cc1e -> grizzly 
INFO [alembic.migration] Running upgrade grizzly -> f489cf14a79c 
INFO [alembic.migration] Running upgrade f489cf14a79c -> 176a85fc7d79 
INFO [alembic.migration] Running upgrade 176a85fc7d79 -> 32b517556ec9 
INFO [alembic.migration] Running upgrade 32b517556ec9 -> 128e042a2b68 
INFO [alembic.migration] Running upgrade 128e042a2b68 -> 5ac71e65402c 
INFO [alembic.migration] Running upgrade 5ac71e65402c -> 3cbf70257c28 
INFO [alembic.migration] Running upgrade 3cbf70257c28 -> 5918cbddab04 
INFO [alembic.migration] Running upgrade 5918cbddab04 -> 3cabb850f4a5 
INFO [alembic.migration] Running upgrade 3cabb850f4a5 -> b7a8863760e 
INFO [alembic.migration] Running upgrade b7a8863760e -> 13de305df56e 
INFO [alembic.migration] Running upgrade 13de305df56e -> 20ae61555e95 
INFO [alembic.migration] Running upgrade 20ae61555e95 -> 477a4488d3f4 
INFO [alembic.migration] Running upgrade 477a4488d3f4 -> 2032abe8edac 
INFO [alembic.migration] Running upgrade 2032abe8edac -> 52c5e4a18807 
INFO [alembic.migration] Running upgrade 52c5e4a18807 -> 557edfc53098 
INFO [alembic.migration] Running upgrade 557edfc53098 -> e6b16a30d97 
INFO [alembic.migration] Running upgrade e6b16a30d97 -> 39cf3f799352 
INFO [alembic.migration] Running upgrade 39cf3f799352 -> 52ff27f7567a 
INFO [alembic.migration] Running upgrade 52ff27f7567a -> 11c6e18605c8 
INFO [alembic.migration] Running upgrade 11c6e18605c8 -> 35c7c198ddea 
INFO [alembic.migration] Running upgrade 35c7c198ddea -> 263772d65691 
INFO [alembic.migration] Running upgrade 263772d65691 -> c88b6b5fea3 
INFO [alembic.migration] Running upgrade c88b6b5fea3 -> f9263d6df56 
INFO [alembic.migration] Running upgrade f9263d6df56 -> 569e98a8132b 
INFO [alembic.migration] Running upgrade 569e98a8132b -> 86cf4d88bd3 
INFO [alembic.migration] Running upgrade 86cf4d88bd3 -> 3c6e57a23db4 
INFO [alembic.migration] Running upgrade 3c6e57a23db4 -> 63afba73813 
INFO [alembic.migration] Running upgrade 63afba73813 -> 40dffbf4b549 
INFO [alembic.migration] Running upgrade 40dffbf4b549 -> 53bbd27ec841 
INFO [alembic.migration] Running upgrade 53bbd27ec841 -> 46a0efbd8f0 
INFO [alembic.migration] Running upgrade 46a0efbd8f0 -> 2a3bae1ceb8 
INFO [alembic.migration] Running upgrade 2a3bae1ceb8 -> 14f24494ca31 
INFO [alembic.migration] Running upgrade 14f24494ca31 -> 32a65f71af51 
INFO [alembic.migration] Running upgrade 32a65f71af51 -> 66a59a7f516 
INFO [alembic.migration] Running upgrade 66a59a7f516 -> 51b4de912379 
INFO [alembic.migration] Running upgrade 51b4de912379 -> 1efb85914233 
INFO [alembic.migration] Running upgrade 1efb85914233 -> 38fc1f6789f8 
INFO [alembic.migration] Running upgrade 38fc1f6789f8 -> 4a666eb208c2 
INFO [alembic.migration] Running upgrade 4a666eb208c2 -> 338d7508968c 
INFO [alembic.migration] Running upgrade 338d7508968c -> 3ed8f075e38a 
INFO [alembic.migration] Running upgrade 3ed8f075e38a -> 3d6fae8b70b0 
INFO [alembic.migration] Running upgrade 3d6fae8b70b0 -> 1064e98b7917 
INFO [alembic.migration] Running upgrade 1064e98b7917 -> 2528ceb28230 
INFO [alembic.migration] Running upgrade 2528ceb28230 -> 3a520dd165d0 
INFO [alembic.migration] Running upgrade 3a520dd165d0 -> 27ef74513d33 
INFO [alembic.migration] Running upgrade 27ef74513d33 -> havana 
INFO [alembic.migration] Running upgrade havana -> e197124d4b9 
INFO [alembic.migration] Running upgrade e197124d4b9 -> 1fcfc149aca4
Traceback (most recent call last):
 File "/usr/bin/neutron-db-manage", line 10, in <module>
 sys.exit(main())
 File "/usr/lib/python2.6/site-packages/neutron/db/migration/cli.py", line 143, in main
 CONF.command.func(config, CONF.command.name)
 File "/usr/lib/python2.6/site-packages/neutron/db/migration/cli.py", line 80, in do_upgrade_downgrade
 do_alembic_command(config, cmd, revision, sql=CONF.command.sql)
 File "/usr/lib/python2.6/site-packages/neutron/db/migration/cli.py", line 59, in do_alembic_command
 getattr(alembic_command, cmd)(config, *args, **kwargs)
 File "/usr/lib/python2.6/site-packages/alembic/command.py", line 124, in upgrade
 script.run_env()
 File "/usr/lib/python2.6/site-packages/alembic/script.py", line 191, in run_env
 util.load_python_file(self.dir, 'env.py')
 File "/usr/lib/python2.6/site-packages/alembic/util.py", line 186, in load_python_file
 module = imp.load_source(module_id, path, open(path, 'rb'))
 File "/usr/lib/python2.6/site-packages/neutron/db/migration/alembic_migrations/env.py", line 105, in <module>
 run_migrations_online()
 File "/usr/lib/python2.6/site-packages/neutron/db/migration/alembic_migrations/env.py", line 89, in run_migrations_online
 options=build_options())
 File "<string>", line 7, in run_migrations
 File "/usr/lib/python2.6/site-packages/alembic/environment.py", line 494, in run_migrations
 self.get_context().run_migrations(**kw)
 File "/usr/lib/python2.6/site-packages/alembic/migration.py", line 211, in run_migrations
 change(**kw)
 File "/usr/lib/python2.6/site-packages/neutron/db/migration/alembic_migrations/versions/1fcfc149aca4_agents_unique_by_type_and_host.py", line 50, in upgrade
 local_cols=['agent_type', 'host']
 File "<string>", line 7, in create_unique_constraint
 File "/usr/lib/python2.6/site-packages/alembic/operations.py", line 478, in create_unique_constraint
 schema=schema, **kw)
 File "/usr/lib/python2.6/site-packages/alembic/ddl/impl.py", line 134, in add_constraint
 self._exec(schema.AddConstraint(const))
 File "/usr/lib/python2.6/site-packages/alembic/ddl/impl.py", line 75, in _exec
 conn.execute(construct, *multiparams, **params)
 File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1449, in execute
 params)
 File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1542, in _execute_ddl
compiled
 File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1698, in _execute_context
 context)
 File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1691, in _execute_context
 context)
 File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/default.py", line 331, in do_execute
 cursor.execute(statement, parameters)
 File "/usr/lib64/python2.6/site-packages/MySQLdb/cursors.py", line 173, in execute
 self.errorhandler(self, exc, value)
 File "/usr/lib64/python2.6/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
 raise errorclass, errorvalue
sqlalchemy.exc.ProgrammingError: (ProgrammingError) (1146, "Table 'neutron_ml2.agents' doesn't exist") 'ALTER TABLE agents ADD CONSTRAINT uniq_agents0agent_type0host UNIQUE (agent_type, host)' ()
rc: 1

Expected results:
=================
Should successfully update the database.

Comment 3 Jakub Libosvar 2014-02-05 15:44:41 UTC

Can you paste here what value is set to core_plugin in /etc/neutron/neutron.conf?

Comment 4 Nir Magnezi 2014-02-06 12:21:15 UTC

(In reply to Jakub Libosvar from comment #3)
> Can you paste here what value is set to core_plugin in
> /etc/neutron/neutron.conf?

# grep core_plugin /etc/neutron/neutron.conf | grep -v "#"
core_plugin =neutron.plugins.ml2.plugin.Ml2Plugin


Re-tested this (since that setup was re-installed).
This time with: openstack-neutron-2014.1-0.5.b2.el6.noarch
The 'neutron-db-manage' command passed ok (not sure why, maybe something was changed between those two versions).

now it fails to start neutron server with the following:
2014-02-06 13:51:19.932 12409 ERROR neutron.plugins.ml2.managers [-] No type driver for tenant network_type: local. Service terminated!

Comment 7 Jakub Libosvar 2014-02-06 16:25:45 UTC

From neutron-server.log:
2014-02-06 13:51:19.930 12409 TRACE stevedore.extension Traceback (most recent call last):
2014-02-06 13:51:19.930 12409 TRACE stevedore.extension   File "/usr/lib/python2.6/site-packages/stevedore/extension.py", line 89, in _load_plugins
2014-02-06 13:51:19.930 12409 TRACE stevedore.extension     invoke_kwds,
2014-02-06 13:51:19.930 12409 TRACE stevedore.extension   File "/usr/lib/python2.6/site-packages/stevedore/named.py", line 57, in _load_one_plugin
2014-02-06 13:51:19.930 12409 TRACE stevedore.extension     ep, invoke_on_load, invoke_args, invoke_kwds,
2014-02-06 13:51:19.930 12409 TRACE stevedore.extension   File "/usr/lib/python2.6/site-packages/stevedore/extension.py", line 101, in _load_one_plugin
2014-02-06 13:51:19.930 12409 TRACE stevedore.extension     plugin = ep.load()
2014-02-06 13:51:19.930 12409 TRACE stevedore.extension   File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 1947, in load
2014-02-06 13:51:19.930 12409 TRACE stevedore.extension     if require: self.require(env, installer)
2014-02-06 13:51:19.930 12409 TRACE stevedore.extension   File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 1960, in require
2014-02-06 13:51:19.930 12409 TRACE stevedore.extension     working_set.resolve(self.dist.requires(self.extras),env,installer))
2014-02-06 13:51:19.930 12409 TRACE stevedore.extension   File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 546, in resolve
2014-02-06 13:51:19.930 12409 TRACE stevedore.extension     raise DistributionNotFound(req)
2014-02-06 13:51:19.930 12409 TRACE stevedore.extension DistributionNotFound: pyudev

Nir, do you have installed packages python-pyudev or openstack-neutron-linuxbridge. Eventually could you try to install python-pyudev to see if it fixes the problem?

I can see in specfile that pyudev is missing for ml2

Comment 8 Nir Magnezi 2014-02-16 10:11:29 UTC

(In reply to Jakub Libosvar from comment #7)
> From neutron-server.log:
> 2014-02-06 13:51:19.930 12409 TRACE stevedore.extension Traceback (most
> recent call last):
> 2014-02-06 13:51:19.930 12409 TRACE stevedore.extension   File
> "/usr/lib/python2.6/site-packages/stevedore/extension.py", line 89, in
> _load_plugins
> 2014-02-06 13:51:19.930 12409 TRACE stevedore.extension     invoke_kwds,
> 2014-02-06 13:51:19.930 12409 TRACE stevedore.extension   File
> "/usr/lib/python2.6/site-packages/stevedore/named.py", line 57, in
> _load_one_plugin
> 2014-02-06 13:51:19.930 12409 TRACE stevedore.extension     ep,
> invoke_on_load, invoke_args, invoke_kwds,
> 2014-02-06 13:51:19.930 12409 TRACE stevedore.extension   File
> "/usr/lib/python2.6/site-packages/stevedore/extension.py", line 101, in
> _load_one_plugin
> 2014-02-06 13:51:19.930 12409 TRACE stevedore.extension     plugin =
> ep.load()
> 2014-02-06 13:51:19.930 12409 TRACE stevedore.extension   File
> "/usr/lib/python2.6/site-packages/pkg_resources.py", line 1947, in load
> 2014-02-06 13:51:19.930 12409 TRACE stevedore.extension     if require:
> self.require(env, installer)
> 2014-02-06 13:51:19.930 12409 TRACE stevedore.extension   File
> "/usr/lib/python2.6/site-packages/pkg_resources.py", line 1960, in require
> 2014-02-06 13:51:19.930 12409 TRACE stevedore.extension    
> working_set.resolve(self.dist.requires(self.extras),env,installer))
> 2014-02-06 13:51:19.930 12409 TRACE stevedore.extension   File
> "/usr/lib/python2.6/site-packages/pkg_resources.py", line 546, in resolve
> 2014-02-06 13:51:19.930 12409 TRACE stevedore.extension     raise
> DistributionNotFound(req)
> 2014-02-06 13:51:19.930 12409 TRACE stevedore.extension
> DistributionNotFound: pyudev
> 
> Nir, do you have installed packages python-pyudev or
> openstack-neutron-linuxbridge. Eventually could you try to install
> python-pyudev to see if it fixes the problem?
> 
> I can see in specfile that pyudev is missing for ml2


both python-pyudev and openstack-neutron-linuxbridge were not installed.
I installed python-pyudev and now I see diffrent type of errors:

2014-02-16 12:06:45.843 13298 INFO neutron.common.config [-] Logging enabled!
2014-02-16 12:06:45.843 13298 ERROR neutron.common.legacy [-] Skipping unknown group key: firewall_driver
2014-02-16 12:06:45.848 13298 INFO neutron.common.config [-] Config paste file: /etc/neutron/api-paste.ini
2014-02-16 12:06:45.899 13298 INFO neutron.manager [-] Loading core plugin: neutron.plugins.ml2.plugin.Ml2Plugin
2014-02-16 12:06:45.991 13298 INFO neutron.plugins.ml2.managers [-] Configured type driver names: ['local', 'flat', 'vlan', 'gre', 'vxlan']
2014-02-16 12:06:45.994 13298 INFO neutron.plugins.ml2.drivers.type_flat [-] Allowable flat physical_network names: []
2014-02-16 12:06:45.997 13298 INFO neutron.plugins.ml2.drivers.type_vlan [-] Network VLAN ranges: {'int_vlan_range': [(208, 210), (185, 185)]}
2014-02-16 12:06:45.998 13298 INFO neutron.plugins.ml2.drivers.type_local [-] ML2 LocalTypeDriver initialization complete
2014-02-16 12:06:46.005 13298 INFO neutron.plugins.ml2.managers [-] Loaded type driver names: ['flat', 'vlan', 'local', 'gre', 'vxlan']
2014-02-16 12:06:46.006 13298 INFO neutron.plugins.ml2.managers [-] Registered types: ['flat', 'vlan', 'local', 'gre', 'vxlan']
2014-02-16 12:06:46.006 13298 INFO neutron.plugins.ml2.managers [-] Tenant network_types: ['local', 'vlan', 'gre', 'vxlan']
2014-02-16 12:06:46.006 13298 INFO neutron.plugins.ml2.managers [-] Configured mechanism driver names: ['openvswitch']
2014-02-16 12:06:46.008 13298 INFO neutron.plugins.ml2.managers [-] Loaded mechanism driver names: ['openvswitch']
2014-02-16 12:06:46.008 13298 INFO neutron.plugins.ml2.managers [-] Registered mechanism drivers: ['openvswitch']
2014-02-16 12:06:46.706 13298 INFO neutron.plugins.ml2.managers [-] Initializing driver for type 'flat'
2014-02-16 12:06:46.706 13298 INFO neutron.plugins.ml2.drivers.type_flat [-] ML2 FlatTypeDriver initialization complete
2014-02-16 12:06:46.707 13298 INFO neutron.plugins.ml2.managers [-] Initializing driver for type 'vlan'
2014-02-16 12:06:46.806 13298 INFO neutron.plugins.ml2.drivers.type_vlan [-] VlanTypeDriver initialization complete
2014-02-16 12:06:46.806 13298 INFO neutron.plugins.ml2.managers [-] Initializing driver for type 'local'
2014-02-16 12:06:46.806 13298 INFO neutron.plugins.ml2.managers [-] Initializing driver for type 'gre'
2014-02-16 12:06:46.807 13298 INFO neutron.plugins.ml2.drivers.type_tunnel [-] gre ID ranges: []
2014-02-16 12:06:46.809 13298 INFO neutron.plugins.ml2.managers [-] Initializing driver for type 'vxlan'
2014-02-16 12:06:46.809 13298 INFO neutron.plugins.ml2.drivers.type_tunnel [-] vxlan ID ranges: []
2014-02-16 12:06:46.812 13298 INFO neutron.plugins.ml2.managers [-] Initializing mechanism driver 'openvswitch'
2014-02-16 12:06:46.893 13298 INFO neutron.openstack.common.rpc.impl_qpid [-] Connected to AMQP server on 192.168.160.29:5672
2014-02-16 12:06:46.900 13298 INFO neutron.plugins.ml2.plugin [-] Modular L2 Plugin initialization complete
2014-02-16 12:06:46.901 13298 INFO neutron.manager [-] Loading Plugin: neutron.services.loadbalancer.plugin.LoadBalancerPlugin
2014-02-16 12:06:47.028 13298 INFO neutron.openstack.common.rpc.impl_qpid [-] Connected to AMQP server on 192.168.160.29:5672
2014-02-16 12:06:47.057 13298 INFO neutron.manager [-] Loading Plugin: neutron.services.firewall.fwaas_plugin.FirewallPlugin
2014-02-16 12:06:47.208 13298 INFO neutron.openstack.common.rpc.impl_qpid [-] Connected to AMQP server on 192.168.160.29:5672
2014-02-16 12:06:47.229 13298 INFO neutron.manager [-] Loading Plugin: neutron.services.l3_router.l3_router_plugin.L3RouterPlugin
2014-02-16 12:06:47.511 13298 INFO neutron.openstack.common.rpc.impl_qpid [-] Connected to AMQP server on 192.168.160.29:5672
2014-02-16 12:06:47.523 13298 INFO neutron.manager [-] Loading Plugin: neutron.services.l3_router.l3_router_plugin.L3RouterPlugin
2014-02-16 12:06:47.578 13298 INFO neutron.openstack.common.rpc.impl_qpid [-] Connected to AMQP server on 192.168.160.29:5672
2014-02-16 12:06:47.593 13298 ERROR neutron.service [-] Unrecoverable error: please check log for details.
2014-02-16 12:06:47.593 13298 TRACE neutron.service Traceback (most recent call last):
2014-02-16 12:06:47.593 13298 TRACE neutron.service   File "/usr/lib/python2.6/site-packages/neutron/service.py", line 99, in serve_wsgi
2014-02-16 12:06:47.593 13298 TRACE neutron.service     service.start()
2014-02-16 12:06:47.593 13298 TRACE neutron.service   File "/usr/lib/python2.6/site-packages/neutron/service.py", line 68, in start
2014-02-16 12:06:47.593 13298 TRACE neutron.service     self.wsgi_app = _run_wsgi(self.app_name)
2014-02-16 12:06:47.593 13298 TRACE neutron.service   File "/usr/lib/python2.6/site-packages/neutron/service.py", line 112, in _run_wsgi
2014-02-16 12:06:47.593 13298 TRACE neutron.service     app = config.load_paste_app(app_name)
2014-02-16 12:06:47.593 13298 TRACE neutron.service   File "/usr/lib/python2.6/site-packages/neutron/common/config.py", line 147, in load_paste_app
2014-02-16 12:06:47.593 13298 TRACE neutron.service     app = deploy.loadapp("config:%s" % config_path, name=app_name)
2014-02-16 12:06:47.593 13298 TRACE neutron.service   File "/usr/lib/python2.6/site-packages/PasteDeploy-1.5.0-py2.6.egg/paste/deploy/loadwsgi.py", line 247, in loadapp
2014-02-16 12:06:47.593 13298 TRACE neutron.service     return loadobj(APP, uri, name=name, **kw)
2014-02-16 12:06:47.593 13298 TRACE neutron.service   File "/usr/lib/python2.6/site-packages/PasteDeploy-1.5.0-py2.6.egg/paste/deploy/loadwsgi.py", line 272, in loadobj
2014-02-16 12:06:47.593 13298 TRACE neutron.service     return context.create()
2014-02-16 12:06:47.593 13298 TRACE neutron.service   File "/usr/lib/python2.6/site-packages/PasteDeploy-1.5.0-py2.6.egg/paste/deploy/loadwsgi.py", line 710, in create
2014-02-16 12:06:47.593 13298 TRACE neutron.service     return self.object_type.invoke(self)
2014-02-16 12:06:47.593 13298 TRACE neutron.service   File "/usr/lib/python2.6/site-packages/PasteDeploy-1.5.0-py2.6.egg/paste/deploy/loadwsgi.py", line 144, in invoke
2014-02-16 12:06:47.593 13298 TRACE neutron.service     **context.local_conf)
2014-02-16 12:06:47.593 13298 TRACE neutron.service   File "/usr/lib/python2.6/site-packages/PasteDeploy-1.5.0-py2.6.egg/paste/deploy/util.py", line 56, in fix_call
2014-02-16 12:06:47.593 13298 TRACE neutron.service     val = callable(*args, **kw)
2014-02-16 12:06:47.593 13298 TRACE neutron.service   File "/usr/lib/python2.6/site-packages/paste/urlmap.py", line 25, in urlmap_factory
2014-02-16 12:06:47.593 13298 TRACE neutron.service     app = loader.get_app(app_name, global_conf=global_conf)
2014-02-16 12:06:47.593 13298 TRACE neutron.service   File "/usr/lib/python2.6/site-packages/PasteDeploy-1.5.0-py2.6.egg/paste/deploy/loadwsgi.py", line 350, in get_app
2014-02-16 12:06:47.593 13298 TRACE neutron.service     name=name, global_conf=global_conf).create()
2014-02-16 12:06:47.593 13298 TRACE neutron.service   File "/usr/lib/python2.6/site-packages/PasteDeploy-1.5.0-py2.6.egg/paste/deploy/loadwsgi.py", line 710, in create
2014-02-16 12:06:47.593 13298 TRACE neutron.service     return self.object_type.invoke(self)
2014-02-16 12:06:47.593 13298 TRACE neutron.service   File "/usr/lib/python2.6/site-packages/PasteDeploy-1.5.0-py2.6.egg/paste/deploy/loadwsgi.py", line 144, in invoke
2014-02-16 12:06:47.593 13298 TRACE neutron.service     **context.local_conf)
2014-02-16 12:06:47.593 13298 TRACE neutron.service   File "/usr/lib/python2.6/site-packages/PasteDeploy-1.5.0-py2.6.egg/paste/deploy/util.py", line 56, in fix_call
2014-02-16 12:06:47.593 13298 TRACE neutron.service     val = callable(*args, **kw)
2014-02-16 12:06:47.593 13298 TRACE neutron.service   File "/usr/lib/python2.6/site-packages/neutron/auth.py", line 64, in pipeline_factory
2014-02-16 12:06:47.593 13298 TRACE neutron.service     app = loader.get_app(pipeline[-1])
2014-02-16 12:06:47.593 13298 TRACE neutron.service   File "/usr/lib/python2.6/site-packages/PasteDeploy-1.5.0-py2.6.egg/paste/deploy/loadwsgi.py", line 350, in get_app
2014-02-16 12:06:47.593 13298 TRACE neutron.service     name=name, global_conf=global_conf).create()
2014-02-16 12:06:47.593 13298 TRACE neutron.service   File "/usr/lib/python2.6/site-packages/PasteDeploy-1.5.0-py2.6.egg/paste/deploy/loadwsgi.py", line 710, in create
2014-02-16 12:06:47.593 13298 TRACE neutron.service     return self.object_type.invoke(self)
2014-02-16 12:06:47.593 13298 TRACE neutron.service   File "/usr/lib/python2.6/site-packages/PasteDeploy-1.5.0-py2.6.egg/paste/deploy/loadwsgi.py", line 146, in invoke
2014-02-16 12:06:47.593 13298 TRACE neutron.service     return fix_call(context.object, context.global_conf, **context.local_conf)
2014-02-16 12:06:47.593 13298 TRACE neutron.service   File "/usr/lib/python2.6/site-packages/PasteDeploy-1.5.0-py2.6.egg/paste/deploy/util.py", line 56, in fix_call
2014-02-16 12:06:47.593 13298 TRACE neutron.service     val = callable(*args, **kw)
2014-02-16 12:06:47.593 13298 TRACE neutron.service   File "/usr/lib/python2.6/site-packages/neutron/api/v2/router.py", line 72, in factory
2014-02-16 12:06:47.593 13298 TRACE neutron.service     return cls(**local_config)
2014-02-16 12:06:47.593 13298 TRACE neutron.service   File "/usr/lib/python2.6/site-packages/neutron/api/v2/router.py", line 76, in __init__
2014-02-16 12:06:47.593 13298 TRACE neutron.service     plugin = manager.NeutronManager.get_plugin()
2014-02-16 12:06:47.593 13298 TRACE neutron.service   File "/usr/lib/python2.6/site-packages/neutron/manager.py", line 214, in get_plugin
2014-02-16 12:06:47.593 13298 TRACE neutron.service     return cls.get_instance().plugin
2014-02-16 12:06:47.593 13298 TRACE neutron.service   File "/usr/lib/python2.6/site-packages/neutron/manager.py", line 209, in get_instance
2014-02-16 12:06:47.593 13298 TRACE neutron.service     cls._create_instance()
2014-02-16 12:06:47.593 13298 TRACE neutron.service   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/lockutils.py", line 249, in inner
2014-02-16 12:06:47.593 13298 TRACE neutron.service     return f(*args, **kwargs)
2014-02-16 12:06:47.593 13298 TRACE neutron.service   File "/usr/lib/python2.6/site-packages/neutron/manager.py", line 203, in _create_instance
2014-02-16 12:06:47.593 13298 TRACE neutron.service     cls._instance = cls()
2014-02-16 12:06:47.593 13298 TRACE neutron.service   File "/usr/lib/python2.6/site-packages/neutron/manager.py", line 128, in __init__
2014-02-16 12:06:47.593 13298 TRACE neutron.service     self._load_service_plugins()
2014-02-16 12:06:47.593 13298 TRACE neutron.service   File "/usr/lib/python2.6/site-packages/neutron/manager.py", line 184, in _load_service_plugins
2014-02-16 12:06:47.593 13298 TRACE neutron.service     plugin_inst.get_plugin_type())
2014-02-16 12:06:47.593 13298 TRACE neutron.service ValueError: (u'Multiple plugins for service %s were configured', 'L3_ROUTER_NAT')
2014-02-16 12:06:47.593 13298 TRACE neutron.service 
2014-02-16 12:06:47.625 13298 CRITICAL neutron [-] (u'Multiple plugins for service %s were configured', 'L3_ROUTER_NAT')
2014-02-16 12:06:47.625 13298 TRACE neutron Traceback (most recent call last):
2014-02-16 12:06:47.625 13298 TRACE neutron   File "/usr/bin/neutron-server", line 10, in <module>
2014-02-16 12:06:47.625 13298 TRACE neutron     sys.exit(main())
2014-02-16 12:06:47.625 13298 TRACE neutron   File "/usr/lib/python2.6/site-packages/neutron/server/__init__.py", line 43, in main
2014-02-16 12:06:47.625 13298 TRACE neutron     neutron_service = service.serve_wsgi(service.NeutronApiService)
2014-02-16 12:06:47.625 13298 TRACE neutron   File "/usr/lib/python2.6/site-packages/neutron/service.py", line 99, in serve_wsgi
2014-02-16 12:06:47.625 13298 TRACE neutron     service.start()
2014-02-16 12:06:47.625 13298 TRACE neutron   File "/usr/lib/python2.6/site-packages/neutron/service.py", line 68, in start
2014-02-16 12:06:47.625 13298 TRACE neutron     self.wsgi_app = _run_wsgi(self.app_name)
2014-02-16 12:06:47.625 13298 TRACE neutron   File "/usr/lib/python2.6/site-packages/neutron/service.py", line 112, in _run_wsgi
2014-02-16 12:06:47.625 13298 TRACE neutron     app = config.load_paste_app(app_name)
2014-02-16 12:06:47.625 13298 TRACE neutron   File "/usr/lib/python2.6/site-packages/neutron/common/config.py", line 147, in load_paste_app
2014-02-16 12:06:47.625 13298 TRACE neutron     app = deploy.loadapp("config:%s" % config_path, name=app_name)
2014-02-16 12:06:47.625 13298 TRACE neutron   File "/usr/lib/python2.6/site-packages/PasteDeploy-1.5.0-py2.6.egg/paste/deploy/loadwsgi.py", line 247, in loadapp
2014-02-16 12:06:47.625 13298 TRACE neutron     return loadobj(APP, uri, name=name, **kw)
2014-02-16 12:06:47.625 13298 TRACE neutron   File "/usr/lib/python2.6/site-packages/PasteDeploy-1.5.0-py2.6.egg/paste/deploy/loadwsgi.py", line 272, in loadobj
2014-02-16 12:06:47.625 13298 TRACE neutron     return context.create()
2014-02-16 12:06:47.625 13298 TRACE neutron   File "/usr/lib/python2.6/site-packages/PasteDeploy-1.5.0-py2.6.egg/paste/deploy/loadwsgi.py", line 710, in create
2014-02-16 12:06:47.625 13298 TRACE neutron     return self.object_type.invoke(self)
2014-02-16 12:06:47.625 13298 TRACE neutron   File "/usr/lib/python2.6/site-packages/PasteDeploy-1.5.0-py2.6.egg/paste/deploy/loadwsgi.py", line 144, in invoke
2014-02-16 12:06:47.625 13298 TRACE neutron     **context.local_conf)
2014-02-16 12:06:47.625 13298 TRACE neutron   File "/usr/lib/python2.6/site-packages/PasteDeploy-1.5.0-py2.6.egg/paste/deploy/util.py", line 56, in fix_call
2014-02-16 12:06:47.625 13298 TRACE neutron     val = callable(*args, **kw)
2014-02-16 12:06:47.625 13298 TRACE neutron   File "/usr/lib/python2.6/site-packages/paste/urlmap.py", line 25, in urlmap_factory
2014-02-16 12:06:47.625 13298 TRACE neutron     app = loader.get_app(app_name, global_conf=global_conf)
2014-02-16 12:06:47.625 13298 TRACE neutron   File "/usr/lib/python2.6/site-packages/PasteDeploy-1.5.0-py2.6.egg/paste/deploy/loadwsgi.py", line 350, in get_app
2014-02-16 12:06:47.625 13298 TRACE neutron     name=name, global_conf=global_conf).create()
2014-02-16 12:06:47.625 13298 TRACE neutron   File "/usr/lib/python2.6/site-packages/PasteDeploy-1.5.0-py2.6.egg/paste/deploy/loadwsgi.py", line 710, in create
2014-02-16 12:06:47.625 13298 TRACE neutron     return self.object_type.invoke(self)
2014-02-16 12:06:47.625 13298 TRACE neutron   File "/usr/lib/python2.6/site-packages/PasteDeploy-1.5.0-py2.6.egg/paste/deploy/loadwsgi.py", line 144, in invoke
2014-02-16 12:06:47.625 13298 TRACE neutron     **context.local_conf)
2014-02-16 12:06:47.625 13298 TRACE neutron   File "/usr/lib/python2.6/site-packages/PasteDeploy-1.5.0-py2.6.egg/paste/deploy/util.py", line 56, in fix_call
2014-02-16 12:06:47.625 13298 TRACE neutron     val = callable(*args, **kw)
2014-02-16 12:06:47.625 13298 TRACE neutron   File "/usr/lib/python2.6/site-packages/neutron/auth.py", line 64, in pipeline_factory
2014-02-16 12:06:47.625 13298 TRACE neutron     app = loader.get_app(pipeline[-1])
2014-02-16 12:06:47.625 13298 TRACE neutron   File "/usr/lib/python2.6/site-packages/PasteDeploy-1.5.0-py2.6.egg/paste/deploy/loadwsgi.py", line 350, in get_app
2014-02-16 12:06:47.625 13298 TRACE neutron     name=name, global_conf=global_conf).create()
2014-02-16 12:06:47.625 13298 TRACE neutron   File "/usr/lib/python2.6/site-packages/PasteDeploy-1.5.0-py2.6.egg/paste/deploy/loadwsgi.py", line 710, in create
2014-02-16 12:06:47.625 13298 TRACE neutron     return self.object_type.invoke(self)
2014-02-16 12:06:47.625 13298 TRACE neutron   File "/usr/lib/python2.6/site-packages/PasteDeploy-1.5.0-py2.6.egg/paste/deploy/loadwsgi.py", line 146, in invoke
2014-02-16 12:06:47.625 13298 TRACE neutron     return fix_call(context.object, context.global_conf, **context.local_conf)
2014-02-16 12:06:47.625 13298 TRACE neutron   File "/usr/lib/python2.6/site-packages/PasteDeploy-1.5.0-py2.6.egg/paste/deploy/util.py", line 56, in fix_call
2014-02-16 12:06:47.625 13298 TRACE neutron     val = callable(*args, **kw)
2014-02-16 12:06:47.625 13298 TRACE neutron   File "/usr/lib/python2.6/site-packages/neutron/api/v2/router.py", line 72, in factory
2014-02-16 12:06:47.625 13298 TRACE neutron     return cls(**local_config)
2014-02-16 12:06:47.625 13298 TRACE neutron   File "/usr/lib/python2.6/site-packages/neutron/api/v2/router.py", line 76, in __init__
2014-02-16 12:06:47.625 13298 TRACE neutron     plugin = manager.NeutronManager.get_plugin()
2014-02-16 12:06:47.625 13298 TRACE neutron   File "/usr/lib/python2.6/site-packages/neutron/manager.py", line 214, in get_plugin
2014-02-16 12:06:47.625 13298 TRACE neutron     return cls.get_instance().plugin
2014-02-16 12:06:47.625 13298 TRACE neutron   File "/usr/lib/python2.6/site-packages/neutron/manager.py", line 209, in get_instance
2014-02-16 12:06:47.625 13298 TRACE neutron     cls._create_instance()
2014-02-16 12:06:47.625 13298 TRACE neutron   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/lockutils.py", line 249, in inner
2014-02-16 12:06:47.625 13298 TRACE neutron     return f(*args, **kwargs)
2014-02-16 12:06:47.625 13298 TRACE neutron   File "/usr/lib/python2.6/site-packages/neutron/manager.py", line 203, in _create_instance
2014-02-16 12:06:47.625 13298 TRACE neutron     cls._instance = cls()
2014-02-16 12:06:47.625 13298 TRACE neutron   File "/usr/lib/python2.6/site-packages/neutron/manager.py", line 128, in __init__
2014-02-16 12:06:47.625 13298 TRACE neutron     self._load_service_plugins()
2014-02-16 12:06:47.625 13298 TRACE neutron   File "/usr/lib/python2.6/site-packages/neutron/manager.py", line 184, in _load_service_plugins
2014-02-16 12:06:47.625 13298 TRACE neutron     plugin_inst.get_plugin_type())
2014-02-16 12:06:47.625 13298 TRACE neutron ValueError: (u'Multiple plugins for service %s were configured', 'L3_ROUTER_NAT')
2014-02-16 12:06:47.625 13298 TRACE neutron 
2014-02-16 12:06:47.628 13298 INFO neutron.manager [-] Loading core plugin: neutron.plugins.ml2.plugin.Ml2Plugin
2014-02-16 12:06:47.629 13298 INFO neutron.plugins.ml2.managers [-] Configured type driver names: ['local', 'flat', 'vlan', 'gre', 'vxlan']
2014-02-16 12:06:47.629 13298 INFO neutron.plugins.ml2.drivers.type_flat [-] Allowable flat physical_network names: []
2014-02-16 12:06:47.630 13298 INFO neutron.plugins.ml2.drivers.type_vlan [-] Network VLAN ranges: {'int_vlan_range': [(208, 210), (185, 185)]}
2014-02-16 12:06:47.630 13298 INFO neutron.plugins.ml2.drivers.type_local [-] ML2 LocalTypeDriver initialization complete
2014-02-16 12:06:47.630 13298 INFO neutron.plugins.ml2.managers [-] Loaded type driver names: ['flat', 'vlan', 'local', 'gre', 'vxlan']
2014-02-16 12:06:47.631 13298 INFO neutron.plugins.ml2.managers [-] Registered types: ['flat', 'vlan', 'local', 'gre', 'vxlan']
2014-02-16 12:06:47.631 13298 INFO neutron.plugins.ml2.managers [-] Tenant network_types: ['local', 'vlan', 'gre', 'vxlan']
2014-02-16 12:06:47.631 13298 INFO neutron.plugins.ml2.managers [-] Configured mechanism driver names: ['openvswitch']
2014-02-16 12:06:47.632 13298 INFO neutron.plugins.ml2.managers [-] Loaded mechanism driver names: ['openvswitch']
2014-02-16 12:06:47.632 13298 INFO neutron.plugins.ml2.managers [-] Registered mechanism drivers: ['openvswitch']
2014-02-16 12:06:47.699 13298 ERROR neutron.openstack.common.rpc.amqp [-] Exception during message handling
2014-02-16 12:06:47.699 13298 TRACE neutron.openstack.common.rpc.amqp Traceback (most recent call last):
2014-02-16 12:06:47.699 13298 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/amqp.py", line 462, in _process_data
2014-02-16 12:06:47.699 13298 TRACE neutron.openstack.common.rpc.amqp     **args)
2014-02-16 12:06:47.699 13298 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/neutron/common/rpc.py", line 45, in dispatch
2014-02-16 12:06:47.699 13298 TRACE neutron.openstack.common.rpc.amqp     neutron_ctxt, version, method, namespace, **kwargs)
2014-02-16 12:06:47.699 13298 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2014-02-16 12:06:47.699 13298 TRACE neutron.openstack.common.rpc.amqp     result = getattr(proxyobj, method)(ctxt, **kwargs)
2014-02-16 12:06:47.699 13298 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/neutron/db/l3_rpc_base.py", line 45, in sync_routers
2014-02-16 12:06:47.699 13298 TRACE neutron.openstack.common.rpc.amqp     l3plugin = manager.NeutronManager.get_service_plugins()[
2014-02-16 12:06:47.699 13298 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/neutron/manager.py", line 218, in get_service_plugins
2014-02-16 12:06:47.699 13298 TRACE neutron.openstack.common.rpc.amqp     return cls.get_instance().service_plugins
2014-02-16 12:06:47.699 13298 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/neutron/manager.py", line 209, in get_instance
2014-02-16 12:06:47.699 13298 TRACE neutron.openstack.common.rpc.amqp     cls._create_instance()
2014-02-16 12:06:47.699 13298 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/lockutils.py", line 249, in inner
2014-02-16 12:06:47.699 13298 TRACE neutron.openstack.common.rpc.amqp     return f(*args, **kwargs)
2014-02-16 12:06:47.699 13298 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/contextlib.py", line 34, in __exit__
2014-02-16 12:06:47.699 13298 TRACE neutron.openstack.common.rpc.amqp     self.gen.throw(type, value, traceback)
2014-02-16 12:06:47.699 13298 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/lockutils.py", line 212, in lock
2014-02-16 12:06:47.699 13298 TRACE neutron.openstack.common.rpc.amqp     yield sem
2014-02-16 12:06:47.699 13298 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/lockutils.py", line 249, in inner
2014-02-16 12:06:47.699 13298 TRACE neutron.openstack.common.rpc.amqp     return f(*args, **kwargs)
2014-02-16 12:06:47.699 13298 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/neutron/manager.py", line 203, in _create_instance
2014-02-16 12:06:47.699 13298 TRACE neutron.openstack.common.rpc.amqp     cls._instance = cls()
2014-02-16 12:06:47.699 13298 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/neutron/manager.py", line 115, in __init__
2014-02-16 12:06:47.699 13298 TRACE neutron.openstack.common.rpc.amqp     plugin_provider)
2014-02-16 12:06:47.699 13298 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/neutron/manager.py", line 143, in _get_plugin_instance
2014-02-16 12:06:47.699 13298 TRACE neutron.openstack.common.rpc.amqp     return plugin_class()
2014-02-16 12:06:47.699 13298 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/neutron/plugins/ml2/plugin.py", line 104, in __init__
2014-02-16 12:06:47.699 13298 TRACE neutron.openstack.common.rpc.amqp     self.type_manager.initialize()
2014-02-16 12:06:47.699 13298 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/neutron/plugins/ml2/managers.py", line 73, in initialize
2014-02-16 12:06:47.699 13298 TRACE neutron.openstack.common.rpc.amqp     LOG.info(_("Initializing driver for type '%s'"), network_type)
2014-02-16 12:06:47.699 13298 TRACE neutron.openstack.common.rpc.amqp TypeError: 'NoneType' object is not callable

Comment 9 Jakub Libosvar 2014-04-08 13:25:56 UTC

Can you make sure you don't have twice L3RouterPlugin in service_plugins in neutron.conf? Sounds rather like a misconfiguration.

Comment 10 Nir Magnezi 2014-04-10 13:29:46 UTC

(In reply to Jakub Libosvar from comment #9)
> Can you make sure you don't have twice L3RouterPlugin in service_plugins in
> neutron.conf? Sounds rather like a misconfiguration.

That setup is no longer live..

Comment 11 Jakub Libosvar 2014-04-11 12:09:27 UTC

Closing for now since it apparently wasn't hit for two months. Could be reopened in case of re-occurrence.