Bug 1127213
| Summary: | Nova fails to get multi-tenant data | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | yfried |
| Component: | openstack-foreman-installer | Assignee: | Jason Guiditta <jguiditt> |
| Status: | CLOSED ERRATA | QA Contact: | Omri Hochman <ohochman> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 5.0 (RHEL 7) | CC: | mburns, morazi, oblaut, rhos-maint, sasha, sclewis, sseago, yeylon, yfried |
| Target Milestone: | ga | Keywords: | TestBlocker |
| Target Release: | Installer | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2014-08-21 18:08:17 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
(In reply to yfried from comment #0) > 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack raise > exceptions.NoAuthURLProvided() > 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack NoAuthURLProvided It looks like admin_auth_url wasn't properly set in the [neutron] section of nova.conf. Maybe the forced puppet run fixed it up? > keystone.log: > 2014-08-06 11:47:30.310 21134 ERROR root [-] Failed to start the admin server > 2014-08-06 11:47:30.310 21134 TRACE root Traceback (most recent call last): > 2014-08-06 11:47:30.310 21134 TRACE root File "/usr/bin/keystone-all", > line 78, in serve > 2014-08-06 11:47:30.310 21134 TRACE root server.start() > 2014-08-06 11:47:30.310 21134 TRACE root File > "/usr/lib/python2.7/site-packages/keystone/common/environment/ > eventlet_server.py", line 66, in start > 2014-08-06 11:47:30.310 21134 TRACE root backlog=backlog) > 2014-08-06 11:47:30.310 21134 TRACE root File > "/usr/lib/python2.7/site-packages/eventlet/convenience.py", line 38, in > listen > 2014-08-06 11:47:30.310 21134 TRACE root sock.bind(addr) > 2014-08-06 11:47:30.310 21134 TRACE root File > "/usr/lib64/python2.7/socket.py", line 224, in meth > 2014-08-06 11:47:30.310 21134 TRACE root return > getattr(self._sock,name)(*args) > 2014-08-06 11:47:30.310 21134 TRACE root error: [Errno 98] Address already > in use > 2014-08-06 11:47:30.310 21134 TRACE root > 2014-08-06 11:47:30.317 21134 CRITICAL keystone [-] error: [Errno 98] > Address already in use This is a problem, though also seems unrelated to the primary issue discussed in this bug to me. (In reply to Russell Bryant from comment #3) > (In reply to yfried from comment #0) > > 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack raise > > exceptions.NoAuthURLProvided() > > 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack NoAuthURLProvided > > It looks like admin_auth_url wasn't properly set in the [neutron] section of > nova.conf. Maybe the forced puppet run fixed it up? This is configured with the quick stack $auth_host param. Scott, is staypuft setting this? I have not seen this error using vanilla ofi, and have a running setup right now where I cannot replicate > > > keystone.log: > > 2014-08-06 11:47:30.310 21134 ERROR root [-] Failed to start the admin server > > 2014-08-06 11:47:30.310 21134 TRACE root Traceback (most recent call last): > > 2014-08-06 11:47:30.310 21134 TRACE root File "/usr/bin/keystone-all", > > line 78, in serve > > 2014-08-06 11:47:30.310 21134 TRACE root server.start() > > 2014-08-06 11:47:30.310 21134 TRACE root File > > "/usr/lib/python2.7/site-packages/keystone/common/environment/ > > eventlet_server.py", line 66, in start > > 2014-08-06 11:47:30.310 21134 TRACE root backlog=backlog) > > 2014-08-06 11:47:30.310 21134 TRACE root File > > "/usr/lib/python2.7/site-packages/eventlet/convenience.py", line 38, in > > listen > > 2014-08-06 11:47:30.310 21134 TRACE root sock.bind(addr) > > 2014-08-06 11:47:30.310 21134 TRACE root File > > "/usr/lib64/python2.7/socket.py", line 224, in meth > > 2014-08-06 11:47:30.310 21134 TRACE root return > > getattr(self._sock,name)(*args) > > 2014-08-06 11:47:30.310 21134 TRACE root error: [Errno 98] Address already > > in use > > 2014-08-06 11:47:30.310 21134 TRACE root > > 2014-08-06 11:47:30.317 21134 CRITICAL keystone [-] error: [Errno 98] > > Address already in use > > This is a problem, though also seems unrelated to the primary issue > discussed in this bug to me. For HA, staypuft sets auth_host to the keystone VIP (i.e. the same IP we set for keystone_{public|private|admin}_vip -- since all VIPs are on the provisioning network for GA3). For non_ha, it's the controller IP address (i.e. provisioning IP -- subject to the recently-fixed @host.ip returning the proper IP address for the provisioning network)
I suspect this is due to a different bug (since fixed) where the ip address of hosts in foreman was changing host ip address. Can you reproduce with staypuft >= 0.2.5 ? seen in version: ruby193-rubygem-staypuft-0.1.22.el6ost Verified: rhel-osp-installer-0.1.10-2.el6ost.noarch [root@<hostname> ~(openstack_admin)]# nova list +--------------------------------------+---------+--------+------------+-------------+--------------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+---------+--------+------------+-------------+--------------------------+ | d251df8c-fbf6-4247-b119-918ef1c9b97a | nested1 | BUILD | spawning | NOSTATE | novanetwork=<IP> | +--------------------------------------+---------+--------+------------+-------------+--------------------------+ [root@<hostname> ~(openstack_admin)]# nova list --all-tenants +--------------------------------------+---------+--------+------------+-------------+---------------------------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+---------+--------+------------+-------------+---------------------------------------+ | d251df8c-fbf6-4247-b119-918ef1c9b97a | nested1 | BUILD | spawning | NOSTATE | novanetwork=<IP>, <IP> | +--------------------------------------+---------+--------+------------+-------------+---------------------------------------+ Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2014-1090.html |
Description of problem: [root@maca25400702875 ~(keystone_yfried)]# nova list +--------------------------------------+----------+--------+------------+-------------+------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+----------+--------+------------+-------------+------------------+ | 87899f9f-25fa-49d2-87d0-9ed9b2ef3062 | cirros | ACTIVE | - | Running | private=10.0.0.2 | | f07cfe36-d404-4258-88dd-45e63aeffcbc | cirros-2 | ACTIVE | - | Running | private=10.0.0.3 | +--------------------------------------+----------+--------+------------+-------------+------------------+ [root@maca25400702875 ~(keystone_admin)]# nova list --all-tenants ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-54c9ce75-6136-4150-bfc8-0254b4c219aa) nova-api.log: 2014-08-06 11:44:51.902 16806 ERROR nova.api.openstack [req-54c9ce75-6136-4150-bfc8-0254b4c219aa f9ac49a72e8546649872ef41375504dc e0bcdd72299b4c6f8467872f556277f6] Caught error: 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack Traceback (most recent call last): 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/api/openstack/__init__.py", line 125, in __call__ 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack return req.get_response(self.application) 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/request.py", line 1296, in send 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack application, catch_exc_info=False) 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/request.py", line 1260, in call_application 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack app_iter = application(self.environ, start_response) 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__ 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack return resp(environ, start_response) 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/keystoneclient/middleware/auth_token.py", line 679, in __call__ 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack return self.app(env, start_response) 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__ 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack return resp(environ, start_response) 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__ 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack return resp(environ, start_response) 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in __call__ 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack response = self.app(environ, start_response) 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__ 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack return resp(environ, start_response) 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__ 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack resp = self.call_func(req, *args, **self.kwargs) 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack return self.func(req, *args, **kwargs) 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 917, in __call__ 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack content_type, body, accept) 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 1009, in _process_stack 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack request, action_args) 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 887, in post_process_extensions 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack **action_args) 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/contrib/security_groups.py", line 601, in detail 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack self._extend_servers(req, list(resp_obj.obj['servers'])) 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/contrib/security_groups.py", line 550, in _extend_servers 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack servers)) 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/network/security_group/neutron_driver.py", line 345, in get_instances_security_groups_bindings 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack ports = self._get_ports_from_server_list(servers, neutron) 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/network/security_group/neutron_driver.py", line 304, in _get_ports_from_server_list 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack ports.extend(neutron.list_ports(**search_opts).get('ports')) 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 111, in with_params 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack ret = self.function(instance, *args, **kwargs) 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 306, in list_ports 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack **_params) 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1250, in list 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack for r in self._pagination(collection, path, **params): 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1263, in _pagination 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack res = self.get(path, params=params) 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1236, in get 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack headers=headers, params=params) 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1221, in retry_request 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack headers=headers, params=params) 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1156, in do_request 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack resp, replybody = self.httpclient.do_request(action, method, body=body) 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 195, in do_request 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack self.authenticate() 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 232, in authenticate 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack raise exceptions.NoAuthURLProvided() 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack NoAuthURLProvided 2014-08-06 11:44:51.902 16806 TRACE nova.api.openstack 2014-08-06 11:44:51.904 16806 INFO nova.api.openstack [req-54c9ce75-6136-4150-bfc8-0254b4c219aa f9ac49a72e8546649872ef41375504dc e0bcdd72299b4c6f8467872f556277f6] http://10.35.163.72:8774/v2/e0bcdd72299b4c6f8467872f556277f6/servers/detail?all_tenants=1 returned with HTTP 500 2014-08-06 11:44:51.905 16806 INFO nova.osapi_compute.wsgi.server [req-54c9ce75-6136-4150-bfc8-0254b4c219aa f9ac49a72e8546649872ef41375504dc e0bcdd72299b4c6f8467872f556277f6] 10.35.163.72 "GET /v2/e0bcdd72299b4c6f8467872f556277f6/servers/detail?all_tenants=1 HTTP/1.1" status: 500 len: 335 time: 0.1244490 2014-08-06 11:44:56.475 16806 INFO nova.osapi_compute.wsgi.server [req-adeb443f-61d6-49da-bf80-2e9419072443 f9ac49a72e8546649872ef41375504dc fb8f63dd01564e73a811bcd550c9e56d] 10.35.163.72 "GET /v2/fb8f63dd01564e73a811bcd550c9e56d/servers/detail?project_id=fb8f63dd01564e73a811bcd550c9e56d HTTP/1.1" status: 200 len: 3229 time: 0.1118150 keystone.log: 2014-08-06 11:47:30.310 21134 ERROR root [-] Failed to start the admin server 2014-08-06 11:47:30.310 21134 TRACE root Traceback (most recent call last): 2014-08-06 11:47:30.310 21134 TRACE root File "/usr/bin/keystone-all", line 78, in serve 2014-08-06 11:47:30.310 21134 TRACE root server.start() 2014-08-06 11:47:30.310 21134 TRACE root File "/usr/lib/python2.7/site-packages/keystone/common/environment/eventlet_server.py", line 66, in start 2014-08-06 11:47:30.310 21134 TRACE root backlog=backlog) 2014-08-06 11:47:30.310 21134 TRACE root File "/usr/lib/python2.7/site-packages/eventlet/convenience.py", line 38, in listen 2014-08-06 11:47:30.310 21134 TRACE root sock.bind(addr) 2014-08-06 11:47:30.310 21134 TRACE root File "/usr/lib64/python2.7/socket.py", line 224, in meth 2014-08-06 11:47:30.310 21134 TRACE root return getattr(self._sock,name)(*args) 2014-08-06 11:47:30.310 21134 TRACE root error: [Errno 98] Address already in use 2014-08-06 11:47:30.310 21134 TRACE root 2014-08-06 11:47:30.317 21134 CRITICAL keystone [-] error: [Errno 98] Address already in use 2014-08-06 11:47:31.590 21168 WARNING keystone.openstack.common.versionutils [-] Deprecated: keystone.middleware.core.XmlBodyMiddleware is deprecated as of Icehouse in favor of support for "application/json" only and may be removed in K. Problem is solved after running "puppet agent -tv" on controller