openstack-foreman-installer: Failed to start neutron-server during Neutron-controller deployment (Empty version in neutron DB) . Steps: ------ - Attempt to deploy neutron-controller using foreman. Results: -------- - During deployment neutron-server failed to start. - When attempting to browse to horizon getting " Something went wrong!" Environment (puddle 2013-12-12.1 ): ----------------------------------- openstack-foreman-installer-0.0.25-1.el6ost.noarch packstack-modules-puppet-2013.2.1-0.19.dev935.el6ost.noarch puppet-3.2.4-3.el6_5.noarch openstack-neutron-2013.2-15.el6ost.noarch During: 'puppet agent -t -v' : ------------------------------ Error: Could not start Service[neutron-server]: Execution of '/sbin/service neutron-server start' returned 6: Error: /Stage[main]/Neutron::Server/Service[neutron-server]/ensure: change from stopped to running failed: Could not start Service[neutron-server]: Execution of '/sbin/service neutron-server start' returned 6: Attempting to start neutron-server service manually : ----------------------------------------------------- [root@puma01 /] service neutron-server restart Stopping neutron: [FAILED] neutron-db-check: ERROR: neutron database does not have version information. You must run the "neutron-db-manage stamp" command before starting neutron services. -------------------------------------------------------- [root@puma01 /]# neutron-db-manage stamp revision No handlers could be found for logger "neutron.common.legacy" Traceback (most recent call last): File "/usr/bin/neutron-db-manage", line 10, in <module> sys.exit(main()) File "/usr/lib/python2.6/site-packages/neutron/db/migration/cli.py", line 143, in main CONF.command.func(config, CONF.command.name) File "/usr/lib/python2.6/site-packages/neutron/db/migration/cli.py", line 86, in do_stamp sql=CONF.command.sql) File "/usr/lib/python2.6/site-packages/neutron/db/migration/cli.py", line 59, in do_alembic_command getattr(alembic_command, cmd)(config, *args, **kwargs) File "/usr/lib/python2.6/site-packages/alembic/command.py", line 214, in stamp script.run_env() File "/usr/lib/python2.6/site-packages/alembic/script.py", line 191, in run_env util.load_python_file(self.dir, 'env.py') File "/usr/lib/python2.6/site-packages/alembic/util.py", line 186, in load_python_file module = imp.load_source(module_id, path, open(path, 'rb')) File "/usr/lib/python2.6/site-packages/neutron/db/migration/alembic_migrations/env.py", line 44, in <module> importutils.import_class(class_path) File "/usr/lib/python2.6/site-packages/neutron/openstack/common/importutils.py", line 35, in import_class traceback.format_exception(*sys.exc_info()))) ImportError: Class cannot be found (['Traceback (most recent call last):\n', ' File "/usr/lib/python2.6/site-packages/neutron/openstack/common/importutils.py", line 30, in import_class\n __import__(mod_str)\n', 'ValueError: Empty module name\n']) -------------------------------------------------------------------- mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | cinder | | glance | | heat | | keystone | | mysql | | neutron | | nova | +--------------------+ mysql> use neutron; Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A mysql> show tables; +-------------------+ | Tables_in_neutron | +-------------------+ | alembic_version | +-------------------+ mysql> select * from alembic_version; Empty set (0.00 sec)
horizon.log ( Connection to neutron failed: Maximum attempts reached) : --------------------------------------- 013-12-15 15:33:44,084 19849 DEBUG openstack_auth.backend Beginning user authentication for user "admin". 2013-12-15 15:33:44,182 19849 DEBUG keystoneclient.httpclient Request returned failure status: 401 2013-12-15 15:33:44,182 19849 DEBUG keystoneclient.v2_0.client Authorization Failed. 2013-12-15 15:33:44,183 19849 DEBUG openstack_auth.backend The request you have made requires authentication. 2013-12-15 15:33:44,183 19849 WARNING openstack_auth.forms Login failed for user "admin". 2013-12-15 15:34:13,495 19849 DEBUG openstack_auth.backend Beginning user authentication for user "admin". 2013-12-15 15:34:13,873 19849 DEBUG openstack_auth.backend Authentication completed for user "admin". 2013-12-15 15:34:13,874 19849 INFO openstack_auth.forms Login successful for user "admin". 2013-12-15 15:34:13,938 19849 DEBUG openstack_dashboard.api.nova novaclient connection created using token "19ec3c6b228fb5d5fb763482632ab610" and url "http://10.35.160.11:8774/v2/52265c0ed5184b33b1e7b068970147cd" 2013-12-15 15:34:15,260 19849 DEBUG openstack_dashboard.api.nova novaclient connection created using token "19ec3c6b228fb5d5fb763482632ab610" and url "http://10.35.160.11:8774/v2/52265c0ed5184b33b1e7b068970147cd" 2013-12-15 15:34:16,504 19849 DEBUG openstack_dashboard.api.neutron neutronclient connection created using token "19ec3c6b228fb5d5fb763482632ab610" and url "http://10.35.160.11:9696/" 2013-12-15 15:34:16,504 19849 DEBUG openstack_dashboard.api.neutron user_id=42bd1edae79e44b29ce2a06038771b7a, tenant_id=52265c0ed5184b33b1e7b068970147cd 2013-12-15 15:34:16,505 19849 DEBUG neutronclient.client REQ: curl -i http://10.35.160.11:9696//v2.0/extensions.json -X GET -H "X-Auth-Token: 19ec3c6b228fb5d5fb763482632ab610" -H "Content-Type: application/json" -H "Accept: application/json" -H "User-Agent: python-neutronclient" 2013-12-15 15:34:16,506 19849 ERROR django.request Internal Server Error: /dashboard/admin/ Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/django/core/handlers/base.py", line 111, in get_response response = callback(request, *callback_args, **callback_kwargs) File "/usr/lib/python2.6/site-packages/horizon/decorators.py", line 38, in dec return view_func(request, *args, **kwargs) File "/usr/lib/python2.6/site-packages/horizon/decorators.py", line 86, in dec return view_func(request, *args, **kwargs) File "/usr/lib/python2.6/site-packages/horizon/decorators.py", line 54, in dec return view_func(request, *args, **kwargs) File "/usr/lib/python2.6/site-packages/horizon/decorators.py", line 38, in dec return view_func(request, *args, **kwargs) File "/usr/lib/python2.6/site-packages/horizon/decorators.py", line 86, in dec return view_func(request, *args, **kwargs) File "/usr/lib/python2.6/site-packages/django/views/generic/base.py", line 48, in view return self.dispatch(request, *args, **kwargs) File "/usr/lib/python2.6/site-packages/django/views/generic/base.py", line 69, in dispatch return handler(request, *args, **kwargs) File "/usr/lib/python2.6/site-packages/horizon/tables/views.py", line 155, in get handled = self.construct_tables() File "/usr/lib/python2.6/site-packages/horizon/tables/views.py", line 146, in construct_tables handled = self.handle_table(table) File "/usr/lib/python2.6/site-packages/horizon/tables/views.py", line 118, in handle_table data = self._get_data_dict() File "/usr/lib/python2.6/site-packages/horizon/tables/views.py", line 182, in _get_data_dict self._data = {self.table_class._meta.name: self.get_data()} File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/dashboards/admin/overview/views.py", line 59, in get_data data = super(GlobalOverview, self).get_data() File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/views.py", line 29, in get_data self.usage.get_limits() File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/base.py", line 178, in get_limits self.get_neutron_limits() File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/base.py", line 147, in get_neutron_limits api.neutron.is_security_group_extension_supported(self.request) File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/neutron.py", line 761, in is_security_group_extension_supported return is_extension_supported(request, 'security-group') File "/usr/lib/python2.6/site-packages/horizon/utils/memoized.py", line 33, in __call__ value = self.func(*args) File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/neutron.py", line 741, in is_extension_supported extensions = list_extensions(request) File "/usr/lib/python2.6/site-packages/horizon/utils/memoized.py", line 33, in __call__ value = self.func(*args) File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/neutron.py", line 732, in list_extensions extensions_list = neutronclient(request).list_extensions() File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 108, in with_params ret = self.function(instance, *args, **kwargs) File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 286, in list_extensions return self.get(self.extensions_path, params=_params) File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1183, in get headers=headers, params=params) File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1175, in retry_request raise exceptions.ConnectionFailed(reason=_("Maximum attempts reached")) ConnectionFailed: Connection to neutron failed: Maximum attempts reached
(In reply to Omri Hochman from comment #0) > [root@puma01 /]# neutron-db-manage stamp revision You must run this script with provided config files containing db connection string, core plugin and service plugins. Then you need to give current version of db (mostly head if you didn't upgrade from some previous version) i.e. neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini stamp head I thought the templates for puppet manifests are shared with packstack. In neutron_api.pp mainfest is 'neutron-db-manage stamp head' called before neutron-server service starts: https://review.openstack.org/#/c/60884/2/packstack/puppet/templates/neutron_api.pp (In reply to Omri Hochman from comment #1) > ConnectionFailed: Connection to neutron failed: Maximum attempts reached This happens because clients cannot reach neutron API because neutron-server is not running.
(In reply to Jakub Libosvar from comment #3) > (In reply to Omri Hochman from comment #0) > > [root@puma01 /]# neutron-db-manage stamp revision > > You must run this script with provided config files containing db connection > string, core plugin and service plugins. Then you need to give current > version of db (mostly head if you didn't upgrade from some previous version) > > i.e. > neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file > /etc/neutron/plugin.ini stamp head Thanks Jakub. this is good as a workaround, but we need to make sure foreman will be able to deploy neutron-controller out of the box without errors while starting the neutron service. As well, I could have manually enter "havana" into neutron DB. > > I thought the templates for puppet manifests are shared with packstack. In > neutron_api.pp mainfest is 'neutron-db-manage stamp head' called before > neutron-server service starts: > https://review.openstack.org/#/c/60884/2/packstack/puppet/templates/ > neutron_api.pp The version is packstack-modules-puppet-2013.2.1-0.19.dev935.el6ost.noarch and indeed shared with packstack - if that's what you mean. > > (In reply to Omri Hochman from comment #1) > > ConnectionFailed: Connection to neutron failed: Maximum attempts reached > > This happens because clients cannot reach neutron API because neutron-server > is not running. You right- it's just a side effect and to emphasize the user experience: the horizon page display " Something went wrong!"
That is correct, foreman does not share the packstack templates, only the packstack-moudles-puppet sub-rpm (which doesnt contain the templates anyway). Those templates are using environment variables and are generally incompatible with the foreman enc setup. Following the discussion in the related packstack bug [1], it really seems to me that the neutron puppet module should do the right thing here, and that adding code to packstack OR foreman is a temporary hack to get around that issue. I may have to do the same as packstack for now in the foreman/quickstack modules, depending when/if this review lands for neutron puppet [2], which I have just asked dprince to try and get through, as it got marked abandoned. Also, we have a report of this from Friday, so I am going to close this as a duplicate and copy over the relevant parts of this comment. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1037675#c10 [2] https://review.openstack.org/#/c/50162/ *** This bug has been marked as a duplicate of bug 1042933 ***