Created attachment 844890 [details] logs Description of problem: after applying workaround from bug 1047156 and running packstack again, I failed with a second error: 2014-01-03 11:32:04::ERROR::run_setup::912::root:: Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/packstack/installer/run_setup.py", line 907, in main _main(confFile) File "/usr/lib/python2.6/site-packages/packstack/installer/run_setup.py", line 573, in _main runSequences() File "/usr/lib/python2.6/site-packages/packstack/installer/run_setup.py", line 552, in runSequences controller.runAllSequences() File "/usr/lib/python2.6/site-packages/packstack/installer/setup_controller.py", line 84, in runAllSequences sequence.run(self.CONF) File "/usr/lib/python2.6/site-packages/packstack/installer/core/sequences.py", line 105, in run step.run(config=config) File "/usr/lib/python2.6/site-packages/packstack/installer/core/sequences.py", line 52, in run raise SequenceError(str(ex)) SequenceError: Error appeared during Puppet run: 10.35.160.133_nova.pp Error: /Stage[main]/Nova::Network/Nova::Manage::Floating[nova-vm-floating]/Nova_floating[nova-vm-floating]: Could not evaluate: Execution of '/usr/bin/nova-manage floating list' returned 1: 2014-01-03 11:31:54.709 25636 DEBUG nova.openstack.common.lockutils [req-2d9a83e9-7030-4459-947c-afb458f2b061 None None] Got semaphore "dbapi_backend" lock /usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py:167 You will find full trace in log /var/tmp/packstack/20140103-112554-inPmo3/manifests/10.35.160.133_nova.pp.log Version-Release number of selected component (if applicable): openstack-packstack-2013.2.1-0.25.dev936.el6.noarch How reproducible: 100% Steps to Reproduce: 1. apply fix from bug 1047156 2. run packstack with nova network 3. Actual results: we fail to install openstack Expected results: we should be able to install openstack Additional info: here is the error from the manifest log: 2014-01-03 11:31:58.499 25756 CRITICAL nova [req-c524f762-630f-4b89-bc7c-5f273b3eb00c None None] (ProgrammingError) (1146, "Table 'nova.networks' doesn't exist") 'SELECT networks.created_at AS networks_created_at, networks.updated_at AS networks_updated_at, networks.deleted_at AS networks_deleted_at, networks.deleted AS networks_deleted, networks.id AS networks_id, networks.label AS networks_label, networks.injected AS networks_injected, networks.cidr AS networks_cidr, networks.cidr_v6 AS networks_cidr_v6, networks.multi_host AS networks_multi_host, networks.gateway_v6 AS networks_gateway_v6, networks.netmask_v6 AS networks_netmask_v6, networks.netmask AS networks_netmask, networks.bridge AS networks_bridge, networks.bridge_interface AS networks_bridge_interface, networks.gateway AS networks_gateway, networks.broadcast AS networks_broadcast, networks.dns1 AS networks_dns1, networks.dns2 AS networks_dns2, networks.vlan AS networks_vlan, networks.vpn_public_address AS networks_vpn_public_address, networks.vpn_public_port AS networks_vpn_public_port, networks.vpn_private_address AS networks_vpn_private_address, networks.dhcp_start AS networks_dhcp_start, networks.rxtx_base AS networks_rxtx_base, networks.project_id AS networks_project_id, networks.priority AS networks_priority, networks.host AS networks_host, networks.uuid AS networks_uuid \nFROM networks \nWHERE networks.deleted = %s' (0,) 2014-01-03 11:31:58.499 25756 TRACE nova Traceback (most recent call last): 2014-01-03 11:31:58.499 25756 TRACE nova File "/usr/bin/nova-manage", line 10, in <module> 2014-01-03 11:31:58.499 25756 TRACE nova sys.exit(main()) 2014-01-03 11:31:58.499 25756 TRACE nova File "/usr/lib/python2.6/site-packages/nova/cmd/manage.py", line 1378, in main 2014-01-03 11:31:58.499 25756 TRACE nova ret = fn(*fn_args, **fn_kwargs) 2014-01-03 11:31:58.499 25756 TRACE nova File "/usr/lib/python2.6/site-packages/nova/cmd/manage.py", line 531, in create 2014-01-03 11:31:58.499 25756 TRACE nova net_manager.create_networks(context.get_admin_context(), **kwargs) 2014-01-03 11:31:58.499 25756 TRACE nova File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 1073, in create_networks 2014-01-03 11:31:58.499 25756 TRACE nova return self._do_create_networks(context, **kwargs) 2014-01-03 11:31:58.499 25756 TRACE nova File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 1113, in _do_create_networks 2014-01-03 11:31:58.499 25756 TRACE nova nets = self.db.network_get_all(context) 2014-01-03 11:31:58.499 25756 TRACE nova File "/usr/lib/python2.6/site-packages/nova/db/api.py", line 950, in network_get_all 2014-01-03 11:31:58.499 25756 TRACE nova return IMPL.network_get_all(context, project_only) 2014-01-03 11:31:58.499 25756 TRACE nova File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 130, in wrapper 2014-01-03 11:31:58.499 25756 TRACE nova return f(*args, **kwargs) 2014-01-03 11:31:58.499 25756 TRACE nova File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 2558, in network_get_all 2014-01-03 11:31:58.499 25756 TRACE nova project_only=project_only).all() 2014-01-03 11:31:58.499 25756 TRACE nova File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2115, in all 2014-01-03 11:31:58.499 25756 TRACE nova return list(self) 2014-01-03 11:31:58.499 25756 TRACE nova File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2227, in __iter__ 2014-01-03 11:31:58.499 25756 TRACE nova return self._execute_and_instances(context) 2014-01-03 11:31:58.499 25756 TRACE nova File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2242, in _execute_and_instances 2014-01-03 11:31:58.499 25756 TRACE nova result = conn.execute(querycontext.statement, self._params) 2014-01-03 11:31:58.499 25756 TRACE nova File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1449, in execute 2014-01-03 11:31:58.499 25756 TRACE nova params) 2014-01-03 11:31:58.499 25756 TRACE nova File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement 2014-01-03 11:31:58.499 25756 TRACE nova compiled_sql, distilled_params 2014-01-03 11:31:58.499 25756 TRACE nova File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1698, in _execute_context 2014-01-03 11:31:58.499 25756 TRACE nova context) 2014-01-03 11:31:58.499 25756 TRACE nova File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1851, in _handle_dbapi_exception 2014-01-03 11:31:58.499 25756 TRACE nova None, sys.exc_info()[2] 2014-01-03 11:31:58.499 25756 TRACE nova ProgrammingError: (ProgrammingError) (1146, "Table 'nova.networks' doesn't exist") 'SELECT networks.created_at AS networks_created_at, networks.updated_at AS networks_updated_at, networks.deleted_at AS networks_deleted_at, networks.deleted AS networks_deleted, networks.id AS networks_id, networks.label AS networks_label, networks.injected AS networks_injected, networks.cidr AS networks_cidr, networks.cidr_v6 AS networks_cidr_v6, networks.multi_host AS networks_multi_host, networks.gateway_v6 AS networks_gateway_v6, networks.netmask_v6 AS networks_netmask_v6, networks.netmask AS networks_netmask, networks.bridge AS networks_bridge, networks.bridge_interface AS networks_bridge_interface, networks.gateway AS networks_gateway, networks.broadcast AS networks_broadcast, networks.dns1 AS networks_dns1, networks.dns2 AS networks_dns2, networks.vlan AS networks_vlan, networks.vpn_public_address AS networks_vpn_public_address, networks.vpn_public_port AS networks_vpn_public_port, networks.vpn_private_address AS networks_vpn_private_address, networks.dhcp_start AS networks_dhcp_start, networks.rxtx_base AS networks_rxtx_base, networks.project_id AS networks_project_id, networks.priority AS networks_priority, networks.host AS networks_host, networks.uuid AS networks_uuid \nFROM networks \nWHERE networks.deleted = %s' (0,) 2014-01-03 11:31:58.499 25756 TRACE nova Command failed, please check log for more info ESC[0m
This might be this unresolved intermittent error? https://bugs.launchpad.net/nova/+bug/1130379 I very much doubt it's a packstack bug in any case. Is mysql up and running and accepting connections? If so, you might try to init the db manually by using the passwords from the packstack answer file with: openstack-db --service nova --rootpw ... --password ... --init If that doesn't work then perhaps resetting the DB may help: openstack-db --service nova --rootpw ... --password ... --drop packstack --anwer-file=...to resume
root@cougar06 ~]# openstack-db --service nova --rootpw 123456 --password 123456 --init Verified connectivity to MySQL. Creating 'nova' database. Updating 'nova' database password in /etc/nova/nova.conf Initializing the nova database, please wait... Traceback (most recent call last): File "/usr/bin/nova-manage", line 10, in <module> sys.exit(main()) File "/usr/lib/python2.6/site-packages/nova/cmd/manage.py", line 1326, in main logging.setup("nova") File "/usr/lib/python2.6/site-packages/nova/openstack/common/log.py", line 374, in setup _setup_logging_from_conf() File "/usr/lib/python2.6/site-packages/nova/openstack/common/log.py", line 421, in _setup_logging_from_conf filelog = logging.handlers.WatchedFileHandler(logpath) File "/usr/lib64/python2.6/logging/handlers.py", line 377, in __init__ logging.FileHandler.__init__(self, filename, mode, encoding, delay) File "/usr/lib64/python2.6/logging/__init__.py", line 827, in __init__ StreamHandler.__init__(self, self._open()) File "/usr/lib64/python2.6/logging/__init__.py", line 846, in _open stream = open(self.baseFilename, self.mode) IOError: [Errno 13] Permission denied: '/var/log/nova/nova-manage.log' ERROR 1146 (42S02) at line 1: Table 'nova.migrate_version' doesn't exist Final sanity check failed. the problem was the log permissions: [root@cougar06 ~]# ls -l /var/log/nova/nova-manage.log rw-r--r--. 1 root root 78878 Jan 3 14:01 /var/log/nova/nova-manage.log after I delete the log I was able to install. I am not sure what changed the log permission in the first place since I did not run anything manual in the system.
(In reply to Dafna Ron from comment #2) > root@cougar06 ~]# openstack-db --service nova --rootpw 123456 --password > 123456 --init > Verified connectivity to MySQL. > Creating 'nova' database. > Updating 'nova' database password in /etc/nova/nova.conf > Initializing the nova database, please wait... > Traceback (most recent call last): > File "/usr/bin/nova-manage", line 10, in <module> > sys.exit(main()) > File "/usr/lib/python2.6/site-packages/nova/cmd/manage.py", line 1326, in > main > logging.setup("nova") > File "/usr/lib/python2.6/site-packages/nova/openstack/common/log.py", line > 374, in setup > _setup_logging_from_conf() > File "/usr/lib/python2.6/site-packages/nova/openstack/common/log.py", line > 421, in _setup_logging_from_conf > filelog = logging.handlers.WatchedFileHandler(logpath) > File "/usr/lib64/python2.6/logging/handlers.py", line 377, in __init__ > logging.FileHandler.__init__(self, filename, mode, encoding, delay) > File "/usr/lib64/python2.6/logging/__init__.py", line 827, in __init__ > StreamHandler.__init__(self, self._open()) > File "/usr/lib64/python2.6/logging/__init__.py", line 846, in _open > stream = open(self.baseFilename, self.mode) > IOError: [Errno 13] Permission denied: '/var/log/nova/nova-manage.log' > ERROR 1146 (42S02) at line 1: Table 'nova.migrate_version' doesn't exist > Final sanity check failed. > > the problem was the log permissions: > > [root@cougar06 ~]# ls -l /var/log/nova/nova-manage.log > rw-r--r--. 1 root root 78878 Jan 3 14:01 /var/log/nova/nova-manage.log > > > after I delete the log I was able to install. > I am not sure what changed the log permission in the first place since I did > not run anything manual in the system. Right, it has to be nova:nova And, if you cannot reproduce the issue & think it's due to something in your n test environment, can you please close the bug as WORKSFORME, with a rationale as environment issue (or some such)?
yes, once I finish with the test day prep I can try to reproduce this and if cannot will close it. However, I must say that we need better error messaging since I think this sort of error should be printed in the log.
Marking the nova-manage permissions issue as a dupe *** This bug has been marked as a duplicate of bug 1044155 ***