1127213 – Nova fails to get multi-tenant data

Bug 1127213 - Nova fails to get multi-tenant data

Summary: Nova fails to get multi-tenant data

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-foreman-installer
Sub Component:
Version:	5.0 (RHEL 7)
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	high
Target Milestone:	ga
Target Release:	Installer
Assignee:	Jason Guiditta
QA Contact:	Omri Hochman
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2014-08-06 12:06 UTC by yfried
Modified:	2019-09-09 15:57 UTC (History)
CC List:	9 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2014-08-21 18:08:17 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2014:1090	0	normal	SHIPPED_LIVE	Red Hat Enterprise Linux OpenStack Platform Enhancement Advisory	2014-08-22 15:28:08 UTC

Description yfried 2014-08-06 12:06:13 UTC

Description of problem:
[root@maca25400702875 ~(keystone_yfried)]# nova list
+--------------------------------------+----------+--------+------------+-------------+------------------+
| ID                                   | Name     | Status | Task State | Power State | Networks         |
+--------------------------------------+----------+--------+------------+-------------+------------------+
| 87899f9f-25fa-49d2-87d0-9ed9b2ef3062 | cirros   | ACTIVE | -          | Running     | private=10.0.0.2 |
| f07cfe36-d404-4258-88dd-45e63aeffcbc | cirros-2 | ACTIVE | -          | Running     | private=10.0.0.3 |
+--------------------------------------+----------+--------+------------+-------------+------------------+

[root@maca25400702875 ~(keystone_admin)]# nova list --all-tenants
ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-54c9ce75-6136-4150-bfc8-0254b4c219aa)


nova-api.log:
2014-08-06 11:44:51.902 16806 ERROR nova.api.openstack [req-54c9ce75-6136-4150-bfc8-0254b4c219aa f9ac49a72e8546649872ef41375504dc e0bcdd72299b4c6f8467872f556277f6] Caught error:
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack Traceback (most recent call last):
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/api/openstack/__init__.py", line 125, in __call__
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack     return req.get_response(self.application)
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1296, in send
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack     application, catch_exc_info=False)
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1260, in call_application
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack     return resp(environ, start_response)
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/keystoneclient/middleware/auth_token.py", line 679, in __call__
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack     return self.app(env, start_response)
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack     return resp(environ, start_response)
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack     return resp(environ, start_response)
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in __call__
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack     response = self.app(environ, start_response)
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack     return resp(environ, start_response)
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 917, in __call__
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack     content_type, body, accept)
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 1009, in _process_stack
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack     request, action_args)
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 887, in post_process_extensions
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack     **action_args)
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/contrib/security_groups.py", line 601, in detail
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack     self._extend_servers(req, list(resp_obj.obj['servers']))
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/contrib/security_groups.py", line 550, in _extend_servers
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack     servers))
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/network/security_group/neutron_driver.py", line 345, in get_instances_security_groups_bindings
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack     ports = self._get_ports_from_server_list(servers, neutron)
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/network/security_group/neutron_driver.py", line 304, in _get_ports_from_server_list
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack     ports.extend(neutron.list_ports(**search_opts).get('ports'))
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 111, in with_params
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack     ret = self.function(instance, *args, **kwargs)
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 306, in list_ports
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack     **_params)
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1250, in list
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack     for r in self._pagination(collection, path, **params):
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1263, in _pagination
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack     res = self.get(path, params=params)
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1236, in get
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack     headers=headers, params=params)
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1221, in retry_request
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack     headers=headers, params=params)
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1156, in do_request
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack     resp, replybody = self.httpclient.do_request(action, method, body=body)
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 195, in do_request
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack     self.authenticate()
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 232, in authenticate
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack     raise exceptions.NoAuthURLProvided()
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack NoAuthURLProvided
2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack
2014-08-06 11:44:51.904 16806 INFO nova.api.openstack [req-54c9ce75-6136-4150-bfc8-0254b4c219aa f9ac49a72e8546649872ef41375504dc e0bcdd72299b4c6f8467872f556277f6] http://10.35.163.72:8774/v2/e0bcdd72299b4c6f8467872f556277f6/servers/detail?all_tenants=1 returned with HTTP 500
2014-08-06 11:44:51.905 16806 INFO nova.osapi_compute.wsgi.server [req-54c9ce75-6136-4150-bfc8-0254b4c219aa f9ac49a72e8546649872ef41375504dc e0bcdd72299b4c6f8467872f556277f6] 10.35.163.72 "GET /v2/e0bcdd72299b4c6f8467872f556277f6/servers/detail?all_tenants=1 HTTP/1.1" status: 500 len: 335 time: 0.1244490
2014-08-06 11:44:56.475 16806 INFO nova.osapi_compute.wsgi.server [req-adeb443f-61d6-49da-bf80-2e9419072443 f9ac49a72e8546649872ef41375504dc fb8f63dd01564e73a811bcd550c9e56d] 10.35.163.72 "GET /v2/fb8f63dd01564e73a811bcd550c9e56d/servers/detail?project_id=fb8f63dd01564e73a811bcd550c9e56d HTTP/1.1" status: 200 len: 3229 time: 0.1118150



keystone.log:
2014-08-06 11:47:30.310 21134 ERROR root [-] Failed to start the admin server
2014-08-06 11:47:30.310 21134 TRACE root Traceback (most recent call last):
2014-08-06 11:47:30.310 21134 TRACE root   File "/usr/bin/keystone-all", line 78, in serve
2014-08-06 11:47:30.310 21134 TRACE root     server.start()
2014-08-06 11:47:30.310 21134 TRACE root   File "/usr/lib/python2.7/site-packages/keystone/common/environment/eventlet_server.py", line 66, in start
2014-08-06 11:47:30.310 21134 TRACE root     backlog=backlog)
2014-08-06 11:47:30.310 21134 TRACE root   File "/usr/lib/python2.7/site-packages/eventlet/convenience.py", line 38, in listen
2014-08-06 11:47:30.310 21134 TRACE root     sock.bind(addr)
2014-08-06 11:47:30.310 21134 TRACE root   File "/usr/lib64/python2.7/socket.py", line 224, in meth
2014-08-06 11:47:30.310 21134 TRACE root     return getattr(self._sock,name)(*args)
2014-08-06 11:47:30.310 21134 TRACE root error: [Errno 98] Address already in use
2014-08-06 11:47:30.310 21134 TRACE root
2014-08-06 11:47:30.317 21134 CRITICAL keystone [-] error: [Errno 98] Address already in use

2014-08-06 11:47:31.590 21168 WARNING keystone.openstack.common.versionutils [-] Deprecated: keystone.middleware.core.XmlBodyMiddleware is deprecated as of Icehouse in favor of support for "application/json" only and may be removed in K.


Problem is solved after running "puppet agent -tv" on controller

Comment 3 Russell Bryant 2014-08-08 14:15:42 UTC

(In reply to yfried from comment #0)
> 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack     raise
> exceptions.NoAuthURLProvided()
> 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack NoAuthURLProvided

It looks like admin_auth_url wasn't properly set in the [neutron] section of nova.conf.  Maybe the forced puppet run fixed it up?

> keystone.log:
> 2014-08-06 11:47:30.310 21134 ERROR root [-] Failed to start the admin server
> 2014-08-06 11:47:30.310 21134 TRACE root Traceback (most recent call last):
> 2014-08-06 11:47:30.310 21134 TRACE root   File "/usr/bin/keystone-all",
> line 78, in serve
> 2014-08-06 11:47:30.310 21134 TRACE root     server.start()
> 2014-08-06 11:47:30.310 21134 TRACE root   File
> "/usr/lib/python2.7/site-packages/keystone/common/environment/
> eventlet_server.py", line 66, in start
> 2014-08-06 11:47:30.310 21134 TRACE root     backlog=backlog)
> 2014-08-06 11:47:30.310 21134 TRACE root   File
> "/usr/lib/python2.7/site-packages/eventlet/convenience.py", line 38, in
> listen
> 2014-08-06 11:47:30.310 21134 TRACE root     sock.bind(addr)
> 2014-08-06 11:47:30.310 21134 TRACE root   File
> "/usr/lib64/python2.7/socket.py", line 224, in meth
> 2014-08-06 11:47:30.310 21134 TRACE root     return
> getattr(self._sock,name)(*args)
> 2014-08-06 11:47:30.310 21134 TRACE root error: [Errno 98] Address already
> in use
> 2014-08-06 11:47:30.310 21134 TRACE root
> 2014-08-06 11:47:30.317 21134 CRITICAL keystone [-] error: [Errno 98]
> Address already in use

This is a problem, though also seems unrelated to the primary issue discussed in this bug to me.

Comment 4 Jason Guiditta 2014-08-08 14:52:42 UTC

(In reply to Russell Bryant from comment #3)
> (In reply to yfried from comment #0)
> > 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack     raise
> > exceptions.NoAuthURLProvided()
> > 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack NoAuthURLProvided
> 
> It looks like admin_auth_url wasn't properly set in the [neutron] section of
> nova.conf.  Maybe the forced puppet run fixed it up?

This is configured with the quick stack $auth_host param.  Scott, is staypuft setting this?  I have not seen this error using vanilla ofi, and have a running setup right now where I cannot replicate
> 
> > keystone.log:
> > 2014-08-06 11:47:30.310 21134 ERROR root [-] Failed to start the admin server
> > 2014-08-06 11:47:30.310 21134 TRACE root Traceback (most recent call last):
> > 2014-08-06 11:47:30.310 21134 TRACE root   File "/usr/bin/keystone-all",
> > line 78, in serve
> > 2014-08-06 11:47:30.310 21134 TRACE root     server.start()
> > 2014-08-06 11:47:30.310 21134 TRACE root   File
> > "/usr/lib/python2.7/site-packages/keystone/common/environment/
> > eventlet_server.py", line 66, in start
> > 2014-08-06 11:47:30.310 21134 TRACE root     backlog=backlog)
> > 2014-08-06 11:47:30.310 21134 TRACE root   File
> > "/usr/lib/python2.7/site-packages/eventlet/convenience.py", line 38, in
> > listen
> > 2014-08-06 11:47:30.310 21134 TRACE root     sock.bind(addr)
> > 2014-08-06 11:47:30.310 21134 TRACE root   File
> > "/usr/lib64/python2.7/socket.py", line 224, in meth
> > 2014-08-06 11:47:30.310 21134 TRACE root     return
> > getattr(self._sock,name)(*args)
> > 2014-08-06 11:47:30.310 21134 TRACE root error: [Errno 98] Address already
> > in use
> > 2014-08-06 11:47:30.310 21134 TRACE root
> > 2014-08-06 11:47:30.317 21134 CRITICAL keystone [-] error: [Errno 98]
> > Address already in use
> 
> This is a problem, though also seems unrelated to the primary issue
> discussed in this bug to me.

Comment 5 Scott Seago 2014-08-12 19:33:55 UTC

For HA, staypuft sets auth_host to the keystone VIP (i.e. the same IP we set for keystone_{public|private|admin}_vip -- since all VIPs are on the provisioning network for GA3). For non_ha, it's the controller IP address (i.e. provisioning IP -- subject to the recently-fixed @host.ip returning the proper IP address for the provisioning network)

Comment 6 Mike Burns 2014-08-12 19:38:25 UTC

I suspect this is due to a different bug (since fixed) where the ip address of hosts in foreman was changing host ip address.

Can you reproduce with staypuft >= 0.2.5 ?

Comment 9 yfried 2014-08-13 06:05:55 UTC

seen in version:
ruby193-rubygem-staypuft-0.1.22.el6ost

Comment 13 Alexander Chuzhoy 2014-08-18 17:30:29 UTC

Verified: rhel-osp-installer-0.1.10-2.el6ost.noarch

[root@<hostname> ~(openstack_admin)]# nova list
+--------------------------------------+---------+--------+------------+-------------+--------------------------+
| ID                                   | Name    | Status | Task State | Power State | Networks                 |
+--------------------------------------+---------+--------+------------+-------------+--------------------------+
| d251df8c-fbf6-4247-b119-918ef1c9b97a | nested1 | BUILD  | spawning   | NOSTATE     | novanetwork=<IP> |
+--------------------------------------+---------+--------+------------+-------------+--------------------------+


[root@<hostname> ~(openstack_admin)]# nova list --all-tenants
+--------------------------------------+---------+--------+------------+-------------+---------------------------------------+
| ID                                   | Name    | Status | Task State | Power State | Networks                              |
+--------------------------------------+---------+--------+------------+-------------+---------------------------------------+
| d251df8c-fbf6-4247-b119-918ef1c9b97a | nested1 | BUILD  | spawning   | NOSTATE     | novanetwork=<IP>, <IP> |
+--------------------------------------+---------+--------+------------+-------------+---------------------------------------+

Comment 14 errata-xmlrpc 2014-08-21 18:08:17 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1090.html

Note You need to log in before you can comment on or make changes to this bug.