Bug 1547575

Summary:	FFU: nova api is unreacheable after running deploy_steps_playbook.yaml
Product:	Red Hat OpenStack	Reporter:	Marius Cornea <mcornea>
Component:	openstack-tripleo-heat-templates	Assignee:	Sofer Athlan-Guyot <sathlang>
Status:	CLOSED ERRATA	QA Contact:	Marius Cornea <mcornea>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	13.0 (Queens)	CC:	dbecker, jschluet, mbracho, mbultel, mburns, morazi, rhel-osp-director-maint, sathlang
Target Milestone:	beta	Keywords:	Triaged
Target Release:	13.0 (Queens)	Flags:	mbultel: needinfo+ mbracho: needinfo+
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-tripleo-heat-templates-8.0.0-0.20180326192239.e59fd2c.el7ost	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-06-27 13:45:38 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Marius Cornea 2018-02-21 14:58:02 UTC

Description of problem:

FFU: nova api is unreacheable after running deploy_steps_playbook.yaml


(overcloud) [stack@undercloud-0 ~]$ nova list
ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<type 'exceptions.TypeError'> (HTTP 500) (Request-ID: req-180d6851-b14d-4499-bd44-5f17a97b7dc9)

it looks like both the cell0 and placement databases are empty:

[root@controller-0 heat-admin]# mysql -e 'show tables' nova_placement
[root@controller-0 heat-admin]# mysql -e 'show tables' nova_cell0
[root@controller-0 heat-admin]#

nova-api log:

2018-02-21 02:12:29.960 21 ERROR nova.context [req-b81326d7-06fc-4823-aed3-58041b867e7e eef382c2c4534957bcad96c64e83929a beb53df11b78484fa74a85739a13a57e - default default] Error gathering result from cell 00000000-0000-0000-0000-000000000000: ArgumentError: Could not parse rfc1738 URL from string 'nil'
2018-02-21 02:12:29.960 21 ERROR nova.context Traceback (most recent call last):
2018-02-21 02:12:29.960 21 ERROR nova.context   File "/usr/lib/python2.7/site-packages/nova/context.py", line 433, in gather_result
2018-02-21 02:12:29.960 21 ERROR nova.context     result = fn(cctxt, *args, **kwargs)
2018-02-21 02:12:29.960 21 ERROR nova.context   File "/usr/lib/python2.7/site-packages/nova/compute/multi_cell_list.py", line 239, in do_query
2018-02-21 02:12:29.960 21 ERROR nova.context     **kwargs)
2018-02-21 02:12:29.960 21 ERROR nova.context   File "/usr/lib/python2.7/site-packages/nova/compute/instance_list.py", line 82, in get_by_filters
2018-02-21 02:12:29.960 21 ERROR nova.context     **kwargs)
2018-02-21 02:12:29.960 21 ERROR nova.context   File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 802, in instance_get_all_by_filters_sort
2018-02-21 02:12:29.960 21 ERROR nova.context     sort_dirs=sort_dirs)
2018-02-21 02:12:29.960 21 ERROR nova.context   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 168, in wrapper
2018-02-21 02:12:29.960 21 ERROR nova.context     return f(*args, **kwargs)
2018-02-21 02:12:29.960 21 ERROR nova.context   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 254, in wrapped
2018-02-21 02:12:29.960 21 ERROR nova.context     with ctxt_mgr.reader.allow_async.using(context):
2018-02-21 02:12:29.960 21 ERROR nova.context   File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
2018-02-21 02:12:29.960 21 ERROR nova.context     return self.gen.next()
2018-02-21 02:12:29.960 21 ERROR nova.context   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1035, in _transaction_scope
2018-02-21 02:12:29.960 21 ERROR nova.context     context=context) as resource:
2018-02-21 02:12:29.960 21 ERROR nova.context   File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
2018-02-21 02:12:29.960 21 ERROR nova.context     return self.gen.next()
2018-02-21 02:12:29.960 21 ERROR nova.context   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 638, in _session
2018-02-21 02:12:29.960 21 ERROR nova.context     bind=self.connection, mode=self.mode)
2018-02-21 02:12:29.960 21 ERROR nova.context   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 403, in _create_session
2018-02-21 02:12:29.960 21 ERROR nova.context     self._start()
2018-02-21 02:12:29.960 21 ERROR nova.context   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 489, in _start
2018-02-21 02:12:29.960 21 ERROR nova.context     engine_args, maker_args)
2018-02-21 02:12:29.960 21 ERROR nova.context   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 513, in _setup_for_connection
2018-02-21 02:12:29.960 21 ERROR nova.context     sql_connection=sql_connection, **engine_kwargs)
2018-02-21 02:12:29.960 21 ERROR nova.context   File "/usr/lib/python2.7/site-packages/debtcollector/renames.py", line 43, in decorator
2018-02-21 02:12:29.960 21 ERROR nova.context     return wrapped(*args, **kwargs)
2018-02-21 02:12:29.960 21 ERROR nova.context   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 138, in create_engine
2018-02-21 02:12:29.960 21 ERROR nova.context     url = sqlalchemy.engine.url.make_url(sql_connection)
2018-02-21 02:12:29.960 21 ERROR nova.context   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/url.py", line 205, in make_url
2018-02-21 02:12:29.960 21 ERROR nova.context     return _parse_rfc1738_args(name_or_url)
2018-02-21 02:12:29.960 21 ERROR nova.context   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/url.py", line 254, in _parse_rfc1738_args
2018-02-21 02:12:29.960 21 ERROR nova.context     "Could not parse rfc1738 URL from string '%s'" % name)
2018-02-21 02:12:29.960 21 ERROR nova.context ArgumentError: Could not parse rfc1738 URL from string 'nil'
2018-02-21 02:12:29.960 21 ERROR nova.context 
2018-02-21 02:12:29.961 21 DEBUG oslo_concurrency.lockutils [req-b81326d7-06fc-4823-aed3-58041b867e7e eef382c2c4534957bcad96c64e83929a beb53df11b78484fa74a85739a13a57e - default default] Lock "09c02079-f255-49aa-9cfc-e996e912fd27" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:273
2018-02-21 02:12:29.962 21 DEBUG oslo_concurrency.lockutils [req-b81326d7-06fc-4823-aed3-58041b867e7e eef382c2c4534957bcad96c64e83929a beb53df11b78484fa74a85739a13a57e - default default] Lock "09c02079-f255-49aa-9cfc-e996e912fd27" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:285
2018-02-21 02:12:29.995 21 ERROR nova.api.openstack.wsgi [req-b81326d7-06fc-4823-aed3-58041b867e7e eef382c2c4534957bcad96c64e83929a beb53df11b78484fa74a85739a13a57e - default default] Unexpected exception in API method: TypeError: 'object' object is not iterable
2018-02-21 02:12:29.995 21 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
2018-02-21 02:12:29.995 21 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 788, in wrapped
2018-02-21 02:12:29.995 21 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
2018-02-21 02:12:29.995 21 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 192, in wrapper
2018-02-21 02:12:29.995 21 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
2018-02-21 02:12:29.995 21 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 192, in wrapper
2018-02-21 02:12:29.995 21 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
2018-02-21 02:12:29.995 21 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 165, in detail
2018-02-21 02:12:29.995 21 ERROR nova.api.openstack.wsgi     servers = self._get_servers(req, is_detail=True)
2018-02-21 02:12:29.995 21 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 308, in _get_servers
2018-02-21 02:12:29.995 21 ERROR nova.api.openstack.wsgi     sort_keys=sort_keys, sort_dirs=sort_dirs)
2018-02-21 02:12:29.995 21 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2419, in get_all
2018-02-21 02:12:29.995 21 ERROR nova.api.openstack.wsgi     context, filters, limit, marker, fields, sort_keys, sort_dirs)
2018-02-21 02:12:29.995 21 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/compute/instance_list.py", line 107, in get_instance_objects_sorted
2018-02-21 02:12:29.995 21 ERROR nova.api.openstack.wsgi     expected_attrs)
2018-02-21 02:12:29.995 21 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 1199, in _make_instance_list
2018-02-21 02:12:29.995 21 ERROR nova.api.openstack.wsgi     for db_inst in db_inst_list:
2018-02-21 02:12:29.995 21 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/compute/multi_cell_list.py", line 259, in get_records_sorted
2018-02-21 02:12:29.995 21 ERROR nova.api.openstack.wsgi     for i in heapq.merge(*results.values()):
2018-02-21 02:12:29.995 21 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/heapq.py", line 372, in merge
2018-02-21 02:12:29.995 21 ERROR nova.api.openstack.wsgi     for itnum, it in enumerate(map(iter, iterables)):
2018-02-21 02:12:29.995 21 ERROR nova.api.openstack.wsgi TypeError: 'object' object is not iterable
2018-02-21 02:12:29.995 21 ERROR nova.api.openstack.wsgi 
2018-02-21 02:12:29.996 21 INFO nova.api.openstack.wsgi [req-b81326d7-06fc-4823-aed3-58041b867e7e eef382c2c4534957bcad96c64e83929a beb53df11b78484fa74a85739a13a57e - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<type 'exceptions.TypeError'>

Comment 8 errata-xmlrpc 2018-06-27 13:45:38 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2086