1309135 – When spawning a VM with RHOSP 8, the API return an unexpected value "NotFound"

Bug 1309135 - When spawning a VM with RHOSP 8, the API return an unexpected value "NotFound"

Summary: When spawning a VM with RHOSP 8, the API return an unexpected value "NotFound"

Keywords:
Status:	CLOSED DUPLICATE of bug 1308422
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	python-novaclient
Sub Component:
Version:	8.0 (Liberty)
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	high
Target Milestone:	---
Target Release:	8.0 (Liberty)
Assignee:	Eoghan Glynn
QA Contact:	nlevinki
Docs Contact:
URL:
Whiteboard:
Duplicates (1):	1310847 (view as bug list)
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2016-02-17 00:11 UTC by David Hill
Modified:	2019-09-09 14:03 UTC (History)
CC List:	16 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2016-02-23 09:35:35 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Description David Hill 2016-02-17 00:11:04 UTC

Description of problem:

When spawning a VM with RHOSP 8, the API return an unexpected value "NotFound" in a brand new RHOSP 8 deployment with the latest puddles.

+ nova boot --flavor m1.micro --image 47d4b6ce-b4a8-42dd-99cb-bebaba9ece5f --nic net-id=61b78308-1fea-44dd-bf84-b81b20e3c4de test-vm
ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'keystoneclient.exceptions.NotFound'> (HTTP 500) (Request-ID: req-0c17a3ab-3e46-4472-b18f-2f6a68cc84d1)



2016-02-16 18:45:58.982 4710 ERROR nova.api.openstack.extensions [req-04fc1a75-a5df-47e7-b5f5-600c4fb95be2 659eee92f03a4a41b462c0f453612c38 200b58ef52ee41ed889b9445e9d14880 - - -] Unexpected exception in API method
2016-02-16 18:45:58.982 4710 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2016-02-16 18:45:58.982 4710 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 478, in wrapped
2016-02-16 18:45:58.982 4710 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2016-02-16 18:45:58.982 4710 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-02-16 18:45:58.982 4710 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2016-02-16 18:45:58.982 4710 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-02-16 18:45:58.982 4710 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2016-02-16 18:45:58.982 4710 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 611, in create
2016-02-16 18:45:58.982 4710 ERROR nova.api.openstack.extensions     **create_kwargs)
2016-02-16 18:45:58.982 4710 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/hooks.py", line 149, in inner
2016-02-16 18:45:58.982 4710 ERROR nova.api.openstack.extensions     rv = f(*args, **kwargs)
2016-02-16 18:45:58.982 4710 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1581, in create
2016-02-16 18:45:58.982 4710 ERROR nova.api.openstack.extensions     check_server_group_quota=check_server_group_quota)
2016-02-16 18:45:58.982 4710 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1181, in _create_instance
2016-02-16 18:45:58.982 4710 ERROR nova.api.openstack.extensions     auto_disk_config, reservation_id, max_count)
2016-02-16 18:45:58.982 4710 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 955, in _validate_and_build_base_options
2016-02-16 18:45:58.982 4710 ERROR nova.api.openstack.extensions     pci_request_info, requested_networks)
2016-02-16 18:45:58.982 4710 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1059, in create_pci_requests_for_sriov_ports
2016-02-16 18:45:58.982 4710 ERROR nova.api.openstack.extensions     neutron = get_client(context, admin=True)
2016-02-16 18:45:58.982 4710 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 237, in get_client
2016-02-16 18:45:58.982 4710 ERROR nova.api.openstack.extensions     auth_token = _ADMIN_AUTH.get_token(_SESSION)
2016-02-16 18:45:58.982 4710 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 200, in get_token
2016-02-16 18:45:58.982 4710 ERROR nova.api.openstack.extensions     return self.get_access(session).auth_token
2016-02-16 18:45:58.982 4710 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 240, in get_access
2016-02-16 18:45:58.982 4710 ERROR nova.api.openstack.extensions     self.auth_ref = self.get_auth_ref(session)
2016-02-16 18:45:58.982 4710 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v2.py", line 88, in get_auth_ref
2016-02-16 18:45:58.982 4710 ERROR nova.api.openstack.extensions     authenticated=False, log=False)
2016-02-16 18:45:58.982 4710 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 501, in post
2016-02-16 18:45:58.982 4710 ERROR nova.api.openstack.extensions     return self.request(url, 'POST', **kwargs)
2016-02-16 18:45:58.982 4710 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 337, in inner
2016-02-16 18:45:58.982 4710 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2016-02-16 18:45:58.982 4710 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 401, in request
2016-02-16 18:45:58.982 4710 ERROR nova.api.openstack.extensions     raise exceptions.from_response(resp, method, url)
2016-02-16 18:45:58.982 4710 ERROR nova.api.openstack.extensions NotFound: The resource could not be found. (HTTP 404)
2016-02-16 18:45:58.982 4710 ERROR nova.api.openstack.extensions 

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 2 Raoul Scarazzini 2016-02-17 17:35:33 UTC

Got into the same error here. Enabling debug and verbose I looked into Nova logs, finding out something interesting. Each time the nova boot fails, these messages appears into the log, referring to a 404:

2016-02-17 12:31:56.799 32277 INFO nova.osapi_compute.wsgi.server [req-f243af94-f0cf-4985-b8a6-0e3ccc869eff 6f87701d07384104a10908f06b5a9c81 01de946f33e542fc99890ccf572f5c3c - - -] 172.17.0.13 "GET /v2/01de946f33e542fc99890ccf572f5c3c HTTP/1.1" status: 404 len: 264 time: 0.0840702
2016-02-17 12:31:57.637 32277 DEBUG nova.api.openstack.wsgi [req-8005bf34-fb17-421a-9557-ad2628259c08 6f87701d07384104a10908f06b5a9c81 01de946f33e542fc99890ccf572f5c3c - - -] Returning 404 to user: Flavor m1.demo could not be found. __call__ /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:1175
2016-02-17 12:31:57.638 32277 INFO nova.osapi_compute.wsgi.server [req-8005bf34-fb17-421a-9557-ad2628259c08 6f87701d07384104a10908f06b5a9c81 01de946f33e542fc99890ccf572f5c3c - - -] 172.17.0.13 "GET /v2/01de946f33e542fc99890ccf572f5c3c/flavors/m1.demo HTTP/1.1" status: 404 len: 298 time: 0.0193071
2016-02-17 12:31:58.778 32277 DEBUG keystoneclient.session [req-24a0612d-9deb-431e-9592-d17fc8d29f99 6f87701d07384104a10908f06b5a9c81 01de946f33e542fc99890ccf572f5c3c - - -] Request returned failure status: 404 request /usr/lib/python2.7/site-packages/keystoneclient/session.py:400

I found that ID into my neutron net list, under the name of the tenant network, which is "HA network tenant 01de946f33e542fc99890ccf572f5c3c", and the message about the flavor is almost clear. Even if I'm able to do nova flavor-list, neutron net-list and so on it seems that something is borken in terms of communication.

Still investigating...

Comment 3 Raoul Scarazzini 2016-02-18 10:09:01 UTC

Possibly relevant: https://bugs.launchpad.net/tripleo/+bug/1543493

Comment 4 David Hill 2016-02-19 00:33:45 UTC

2016-02-18 18:32:34.471 28733 WARNING keystonemiddleware.auth_token [-] Authorization failed for token
2016-02-18 18:32:34.473 28733 WARNING keystonemiddleware.auth_token [-] Identity response: {"error": {"message": "Could not find token: 9a7c7818af8c41f3a2a778fdc175e25b", "code": 404, "title": "Not Found"}}
2016-02-18 18:32:34.476 28733 WARNING keystonemiddleware.auth_token [-] Authorization failed for token

Comment 5 Raoul Scarazzini 2016-02-19 11:43:39 UTC

Since I'm going to rebuild the environment and I don't know if this error will be reproducible again, I've put all the sosreport here http://file.rdu.redhat.com/~rscarazz/BZ1309135/ so to be able to consult them even if the original environment does not exists anymore.

Comment 6 Alexander Chuzhoy 2016-02-22 21:46:41 UTC

*** Bug 1310847 has been marked as a duplicate of this bug. ***

Comment 9 Raoul Scarazzini 2016-02-23 09:21:53 UTC

From my side, with the latest puddle the issue is the same, just deployed and just got the same error while booting an instance.

Comment 10 Raoul Scarazzini 2016-02-23 09:33:53 UTC

With the workaround suggested here https://bugzilla.redhat.com/show_bug.cgi?id=1309135 it works.

Comment 11 Raoul Scarazzini 2016-02-23 09:35:35 UTC


*** This bug has been marked as a duplicate of bug 1308422 ***

Note You need to log in before you can comment on or make changes to this bug.