Bug 1048141 - packstack: fails to install ice-house when using nova-networks
Summary: packstack: fails to install ice-house when using nova-networks
Keywords:
Status: CLOSED DUPLICATE of bug 1044155
Alias: None
Product: RDO
Classification: Community
Component: openstack-packstack
Version: unspecified
Hardware: aarch64
OS: Linux
unspecified
urgent
Target Milestone: ---
: ---
Assignee: RHOS Maint
QA Contact: Nir Magnezi
URL:
Whiteboard: storage
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-01-03 09:38 UTC by Dafna Ron
Modified: 2014-01-04 03:29 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-01-04 03:29:08 UTC


Attachments (Terms of Use)
logs (4.76 KB, application/x-gzip)
2014-01-03 09:38 UTC, Dafna Ron
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1265749 0 None None None Never

Description Dafna Ron 2014-01-03 09:38:00 UTC
Created attachment 844890 [details]
logs

Description of problem:

after applying workaround from bug 1047156 and running packstack again, I failed with a second error: 


2014-01-03 11:32:04::ERROR::run_setup::912::root:: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/packstack/installer/run_setup.py", line 907, in main
    _main(confFile)
  File "/usr/lib/python2.6/site-packages/packstack/installer/run_setup.py", line 573, in _main
    runSequences()
  File "/usr/lib/python2.6/site-packages/packstack/installer/run_setup.py", line 552, in runSequences
    controller.runAllSequences()
  File "/usr/lib/python2.6/site-packages/packstack/installer/setup_controller.py", line 84, in runAllSequences
    sequence.run(self.CONF)
  File "/usr/lib/python2.6/site-packages/packstack/installer/core/sequences.py", line 105, in run
    step.run(config=config)
  File "/usr/lib/python2.6/site-packages/packstack/installer/core/sequences.py", line 52, in run
    raise SequenceError(str(ex))
SequenceError: Error appeared during Puppet run: 10.35.160.133_nova.pp
Error: /Stage[main]/Nova::Network/Nova::Manage::Floating[nova-vm-floating]/Nova_floating[nova-vm-floating]: Could not evaluate: Execution of '/usr/bin/nova-manage floating list' returned 1: 2014-01-03 11:31:54.709 25636 DEBUG nova.openstack.common.lockutils [req-2d9a83e9-7030-4459-947c-afb458f2b061 None None] Got semaphore "dbapi_backend" lock /usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py:167
You will find full trace in log /var/tmp/packstack/20140103-112554-inPmo3/manifests/10.35.160.133_nova.pp.log


Version-Release number of selected component (if applicable):

openstack-packstack-2013.2.1-0.25.dev936.el6.noarch

How reproducible:

100%

Steps to Reproduce:
1. apply fix from bug 1047156 
2. run packstack with nova network
3.

Actual results:

we fail to install openstack

Expected results:

we should be able to install openstack 

Additional info:

here is the error from the manifest log: 

2014-01-03 11:31:58.499 25756 CRITICAL nova [req-c524f762-630f-4b89-bc7c-5f273b3eb00c None None] (ProgrammingError) (1146, "Table 'nova.networks' doesn't exist") 'SELECT networks.created_at AS networks_created_at, networks.updated_at AS networks_updated_at, networks.deleted_at AS networks_deleted_at, networks.deleted AS networks_deleted, networks.id AS networks_id, networks.label AS networks_label, networks.injected AS networks_injected, networks.cidr AS networks_cidr, networks.cidr_v6 AS networks_cidr_v6, networks.multi_host AS networks_multi_host, networks.gateway_v6 AS networks_gateway_v6, networks.netmask_v6 AS networks_netmask_v6, networks.netmask AS networks_netmask, networks.bridge AS networks_bridge, networks.bridge_interface AS networks_bridge_interface, networks.gateway AS networks_gateway, networks.broadcast AS networks_broadcast, networks.dns1 AS networks_dns1, networks.dns2 AS networks_dns2, networks.vlan AS networks_vlan, networks.vpn_public_address AS networks_vpn_public_address, networks.vpn_public_port AS networks_vpn_public_port, networks.vpn_private_address AS networks_vpn_private_address, networks.dhcp_start AS networks_dhcp_start, networks.rxtx_base AS networks_rxtx_base, networks.project_id AS networks_project_id, networks.priority AS networks_priority, networks.host AS networks_host, networks.uuid AS networks_uuid \nFROM networks \nWHERE networks.deleted = %s' (0,)
2014-01-03 11:31:58.499 25756 TRACE nova Traceback (most recent call last):
2014-01-03 11:31:58.499 25756 TRACE nova   File "/usr/bin/nova-manage", line 10, in <module>
2014-01-03 11:31:58.499 25756 TRACE nova     sys.exit(main())
2014-01-03 11:31:58.499 25756 TRACE nova   File "/usr/lib/python2.6/site-packages/nova/cmd/manage.py", line 1378, in main
2014-01-03 11:31:58.499 25756 TRACE nova     ret = fn(*fn_args, **fn_kwargs)
2014-01-03 11:31:58.499 25756 TRACE nova   File "/usr/lib/python2.6/site-packages/nova/cmd/manage.py", line 531, in create
2014-01-03 11:31:58.499 25756 TRACE nova     net_manager.create_networks(context.get_admin_context(), **kwargs)
2014-01-03 11:31:58.499 25756 TRACE nova   File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 1073, in create_networks
2014-01-03 11:31:58.499 25756 TRACE nova     return self._do_create_networks(context, **kwargs)
2014-01-03 11:31:58.499 25756 TRACE nova   File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 1113, in _do_create_networks
2014-01-03 11:31:58.499 25756 TRACE nova     nets = self.db.network_get_all(context)
2014-01-03 11:31:58.499 25756 TRACE nova   File "/usr/lib/python2.6/site-packages/nova/db/api.py", line 950, in network_get_all
2014-01-03 11:31:58.499 25756 TRACE nova     return IMPL.network_get_all(context, project_only)
2014-01-03 11:31:58.499 25756 TRACE nova   File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 130, in wrapper
2014-01-03 11:31:58.499 25756 TRACE nova     return f(*args, **kwargs)
2014-01-03 11:31:58.499 25756 TRACE nova   File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 2558, in network_get_all
2014-01-03 11:31:58.499 25756 TRACE nova     project_only=project_only).all()
2014-01-03 11:31:58.499 25756 TRACE nova   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2115, in all
2014-01-03 11:31:58.499 25756 TRACE nova     return list(self)
2014-01-03 11:31:58.499 25756 TRACE nova   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2227, in __iter__
2014-01-03 11:31:58.499 25756 TRACE nova     return self._execute_and_instances(context)
2014-01-03 11:31:58.499 25756 TRACE nova   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2242, in _execute_and_instances
2014-01-03 11:31:58.499 25756 TRACE nova     result = conn.execute(querycontext.statement, self._params)
2014-01-03 11:31:58.499 25756 TRACE nova   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1449, in execute
2014-01-03 11:31:58.499 25756 TRACE nova     params)
2014-01-03 11:31:58.499 25756 TRACE nova   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement
2014-01-03 11:31:58.499 25756 TRACE nova     compiled_sql, distilled_params
2014-01-03 11:31:58.499 25756 TRACE nova   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1698, in _execute_context
2014-01-03 11:31:58.499 25756 TRACE nova     context)
2014-01-03 11:31:58.499 25756 TRACE nova   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1851, in _handle_dbapi_exception
2014-01-03 11:31:58.499 25756 TRACE nova     None, sys.exc_info()[2]
2014-01-03 11:31:58.499 25756 TRACE nova ProgrammingError: (ProgrammingError) (1146, "Table 'nova.networks' doesn't exist") 'SELECT networks.created_at AS networks_created_at, networks.updated_at AS networks_updated_at, networks.deleted_at AS networks_deleted_at, networks.deleted AS networks_deleted, networks.id AS networks_id, networks.label AS networks_label, networks.injected AS networks_injected, networks.cidr AS networks_cidr, networks.cidr_v6 AS networks_cidr_v6, networks.multi_host AS networks_multi_host, networks.gateway_v6 AS networks_gateway_v6, networks.netmask_v6 AS networks_netmask_v6, networks.netmask AS networks_netmask, networks.bridge AS networks_bridge, networks.bridge_interface AS networks_bridge_interface, networks.gateway AS networks_gateway, networks.broadcast AS networks_broadcast, networks.dns1 AS networks_dns1, networks.dns2 AS networks_dns2, networks.vlan AS networks_vlan, networks.vpn_public_address AS networks_vpn_public_address, networks.vpn_public_port AS networks_vpn_public_port, networks.vpn_private_address AS networks_vpn_private_address, networks.dhcp_start AS networks_dhcp_start, networks.rxtx_base AS networks_rxtx_base, networks.project_id AS networks_project_id, networks.priority AS networks_priority, networks.host AS networks_host, networks.uuid AS networks_uuid \nFROM networks \nWHERE networks.deleted = %s' (0,)
2014-01-03 11:31:58.499 25756 TRACE nova 
Command failed, please check log for more info
ESC[0m

Comment 1 Pádraig Brady 2014-01-03 11:13:41 UTC
This might be this unresolved intermittent error?
https://bugs.launchpad.net/nova/+bug/1130379
I very much doubt it's a packstack bug in any case.

Is mysql up and running and accepting connections?

If so, you might try to init the db manually by using
the passwords from the packstack answer file with:

  openstack-db --service nova --rootpw ... --password ... --init

If that doesn't work then perhaps resetting the DB may help:

  openstack-db --service nova --rootpw ... --password ... --drop
  packstack --anwer-file=...to resume

Comment 2 Dafna Ron 2014-01-03 15:59:51 UTC
root@cougar06 ~]# openstack-db --service nova --rootpw 123456 --password 123456 --init
Verified connectivity to MySQL.
Creating 'nova' database.
Updating 'nova' database password in /etc/nova/nova.conf
Initializing the nova database, please wait...
Traceback (most recent call last):
  File "/usr/bin/nova-manage", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.6/site-packages/nova/cmd/manage.py", line 1326, in main
    logging.setup("nova")
  File "/usr/lib/python2.6/site-packages/nova/openstack/common/log.py", line 374, in setup
    _setup_logging_from_conf()
  File "/usr/lib/python2.6/site-packages/nova/openstack/common/log.py", line 421, in _setup_logging_from_conf
    filelog = logging.handlers.WatchedFileHandler(logpath)
  File "/usr/lib64/python2.6/logging/handlers.py", line 377, in __init__
    logging.FileHandler.__init__(self, filename, mode, encoding, delay)
  File "/usr/lib64/python2.6/logging/__init__.py", line 827, in __init__
    StreamHandler.__init__(self, self._open())
  File "/usr/lib64/python2.6/logging/__init__.py", line 846, in _open
    stream = open(self.baseFilename, self.mode)
IOError: [Errno 13] Permission denied: '/var/log/nova/nova-manage.log'
ERROR 1146 (42S02) at line 1: Table 'nova.migrate_version' doesn't exist
Final sanity check failed.

the problem was the log permissions: 

[root@cougar06 ~]# ls -l /var/log/nova/nova-manage.log
rw-r--r--. 1 root root 78878 Jan  3 14:01 /var/log/nova/nova-manage.log


after I delete the log I was able to install. 
I am not sure what changed the log permission in the first place since I did not run anything manual in the system.

Comment 3 Kashyap Chamarthy 2014-01-03 16:34:50 UTC
(In reply to Dafna Ron from comment #2)
> root@cougar06 ~]# openstack-db --service nova --rootpw 123456 --password
> 123456 --init
> Verified connectivity to MySQL.
> Creating 'nova' database.
> Updating 'nova' database password in /etc/nova/nova.conf
> Initializing the nova database, please wait...
> Traceback (most recent call last):
>   File "/usr/bin/nova-manage", line 10, in <module>
>     sys.exit(main())
>   File "/usr/lib/python2.6/site-packages/nova/cmd/manage.py", line 1326, in
> main
>     logging.setup("nova")
>   File "/usr/lib/python2.6/site-packages/nova/openstack/common/log.py", line
> 374, in setup
>     _setup_logging_from_conf()
>   File "/usr/lib/python2.6/site-packages/nova/openstack/common/log.py", line
> 421, in _setup_logging_from_conf
>     filelog = logging.handlers.WatchedFileHandler(logpath)
>   File "/usr/lib64/python2.6/logging/handlers.py", line 377, in __init__
>     logging.FileHandler.__init__(self, filename, mode, encoding, delay)
>   File "/usr/lib64/python2.6/logging/__init__.py", line 827, in __init__
>     StreamHandler.__init__(self, self._open())
>   File "/usr/lib64/python2.6/logging/__init__.py", line 846, in _open
>     stream = open(self.baseFilename, self.mode)
> IOError: [Errno 13] Permission denied: '/var/log/nova/nova-manage.log'
> ERROR 1146 (42S02) at line 1: Table 'nova.migrate_version' doesn't exist
> Final sanity check failed.
> 
> the problem was the log permissions: 
> 
> [root@cougar06 ~]# ls -l /var/log/nova/nova-manage.log
> rw-r--r--. 1 root root 78878 Jan  3 14:01 /var/log/nova/nova-manage.log

> 
> 
> after I delete the log I was able to install. 
> I am not sure what changed the log permission in the first place since I did
> not run anything manual in the system.


Right, it has to be nova:nova

And, if you cannot reproduce the issue & think it's due to something in your n test environment, can you please close the bug as WORKSFORME, with a rationale as environment issue (or some such)?

Comment 4 Dafna Ron 2014-01-03 16:38:10 UTC
yes, once I finish with the test day prep I can try to reproduce this and if cannot will close it. 
However, I must say that we need better error messaging since I think this sort of error should be printed in the log.

Comment 5 Pádraig Brady 2014-01-04 03:29:08 UTC
Marking the nova-manage permissions issue as a dupe

*** This bug has been marked as a duplicate of bug 1044155 ***


Note You need to log in before you can comment on or make changes to this bug.