1256827 – [Nova] Instance removal issue

Bug 1256827 - [Nova] Instance removal issue

Summary: [Nova] Instance removal issue

Keywords:
Status:	CLOSED EOL
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-nova
Sub Component:
Version:	7.0 (Kilo)
Hardware:	Unspecified
OS:	Unspecified
Priority:	medium
Severity:	medium
Target Milestone:	---
Target Release:	8.0 (Liberty)
Assignee:	Radomir Dopieralski
QA Contact:	nlevinki
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2015-08-25 14:19 UTC by Joe Talerico
Modified:	2019-09-09 16:44 UTC (History)
CC List:	11 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2017-12-14 13:31:55 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Launchpad	1543511	0	None	None	None	2016-02-09 10:26:01 UTC

Description Joe Talerico 2015-08-25 14:19:56 UTC

Description of problem:
Deleting guests (with no network) hits this traceback :

2015-08-25 09:19:53.844 25191 DEBUG nova.api.openstack.wsgi [req-bb21e3c9-f1c5-472a-bb3c-e06cdcd3598c bf111661fd1245989bab0cf6a3f813e9 e9b8a420c1a644c799c04130ac8dada6 - - -] Calling method '<bound method Controller.detail of <nova.api.openstack.compute.servers.Controller object at 0x5915ad0>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:783
2015-08-25 09:19:53.845 25191 DEBUG nova.compute.api [req-bb21e3c9-f1c5-472a-bb3c-e06cdcd3598c bf111661fd1245989bab0cf6a3f813e9 e9b8a420c1a644c799c04130ac8dada6 - - -] Searching by: {'deleted': False, u'host': u'overcloud-compute-5.localdomain'} get_all /usr/lib/python2.7/site-packages/nova/compute/api.py:1962
2015-08-25 09:19:53.938 25191 DEBUG keystoneclient.session [req-bb21e3c9-f1c5-472a-bb3c-e06cdcd3598c bf111661fd1245989bab0cf6a3f813e9 e9b8a420c1a644c799c04130ac8dada6 - - -] REQ: curl -g -i -X GET http://172.21.33.11:9696/v2.0/ports.json?device_id=751e5479-c00a-472b-8276-d66fde0de492&device_id=9b41f1ed-2a05-4880-8586-e33ddfe5dec0&device_id=dc3f1134-5a57-442c-8e1d-c55b59cb2768&device_id=595b2edd-f8a8-449b-a7fc-44847f1b7df6&device_id=c85d145b-e0e1-4ab1-bc33-d3a6e2949672&device_id=e3e79468-d630-436f-b2ba-eb6727e32b9e&device_id=6cd0c0e1-4573-49fd-a741-2937efdf4bbf&device_id=34eb08be-04a9-4f25-93d6-e5b32bf8f888&device_id=20034eff-aa7c-438d-92a9-cdb074e3a023&device_id=1028f29d-f95f-4592-b418-edfde4a84886&device_id=1552f474-d934-4132-a659-150348d9ecf4&device_id=da746630-ce3b-4b9a-9346-12377e2c96eb&device_id=78b620eb-f942-421a-9891-0a7830b480c7&device_id=ebb2716e-b821-474e-a075-3726e4f3127d&device_id=1a080d67-69ca-4b9f-9fa0-89007da09fb8&device_id=e4986cf0-3189-40ba-a408-ad8b5a2e83d3&device_id=a233273d-4d76-445b-bcbd-f487675933cf&device_id=62f97e13-4026-4ade-83ad-c78829742d76&device_id=d5f8cb48-6746-48ac-af69-8343c21691e0&device_id=665e246e-110b-457e-870a-bc49481c044b -H "User-Agent: python-neutronclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}4ec6b2e5a3f6cad2942a8daf2dfa500990b6875a" _http_log_request /usr/lib/python2.7/site-packages/keystoneclient/session.py:195
2015-08-25 09:19:53.972 25191 DEBUG keystoneclient.session [req-bb21e3c9-f1c5-472a-bb3c-e06cdcd3598c bf111661fd1245989bab0cf6a3f813e9 e9b8a420c1a644c799c04130ac8dada6 - - -] RESP: _http_log_response /usr/lib/python2.7/site-packages/keystoneclient/session.py:224
2015-08-25 09:19:53.976 25191 DEBUG keystoneclient.session [req-bb21e3c9-f1c5-472a-bb3c-e06cdcd3598c bf111661fd1245989bab0cf6a3f813e9 e9b8a420c1a644c799c04130ac8dada6 - - -] RESP: _http_log_response /usr/lib/python2.7/site-packages/keystoneclient/session.py:224
2015-08-25 09:19:53.976 25191 DEBUG neutronclient.v2_0.client [req-bb21e3c9-f1c5-472a-bb3c-e06cdcd3598c bf111661fd1245989bab0cf6a3f813e9 e9b8a420c1a644c799c04130ac8dada6 - - -] Error message: Authentication required _handle_fault_response /usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py:176
2015-08-25 09:19:53.976 25191 ERROR nova.api.openstack [req-bb21e3c9-f1c5-472a-bb3c-e06cdcd3598c bf111661fd1245989bab0cf6a3f813e9 e9b8a420c1a644c799c04130ac8dada6 - - -] Caught error: Authentication required
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack Traceback (most recent call last):
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/api/openstack/__init__.py", line 125, in __call__
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack     return req.get_response(self.application)
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1296, in send
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack     application, catch_exc_info=False)
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1260, in call_application
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack     return resp(environ, start_response)
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 634, in __call__
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack     return self._call_app(env, start_response)
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 554, in _call_app
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack     return self._app(env, _fake_start_response)
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack     return resp(environ, start_response)
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack     return resp(environ, start_response)
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in __call__
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack     response = self.app(environ, start_response)
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack     return resp(environ, start_response)
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 756, in __call__
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack     content_type, body, accept)
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 847, in _process_stack
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack     request, action_args)
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 710, in post_process_extensions
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack     **action_args)
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/contrib/security_groups.py", line 471, in detail
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack     self._extend_servers(req, list(resp_obj.obj['servers']))
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/contrib/security_groups.py", line 422, in _extend_servers
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack     servers))
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/network/security_group/neutron_driver.py", line 353, in get_instances_security_groups_bindings
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack     ports = self._get_ports_from_server_list(servers, neutron)
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/nova/network/security_group/neutron_driver.py", line 312, in _get_ports_from_server_list
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack     ports.extend(neutron.list_ports(**search_opts).get('ports'))
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 102, in with_params
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack     ret = self.function(instance, *args, **kwargs)
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 534, in list_ports
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack     **_params)
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 307, in list
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack     for r in self._pagination(collection, path, **params):
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 320, in _pagination
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack     res = self.get(path, params=params)
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 293, in get
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack     headers=headers, params=params)
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 270, in retry_request
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack     headers=headers, params=params)
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 211, in do_request
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack     self._handle_fault_response(status_code, replybody)
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 185, in _handle_fault_response
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack     exception_handler_v20(status_code, des_error_body)
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 83, in exception_handler_v20
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack     message=message)
2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack NeutronClientException: Authentication required

Then, subsequent deletes mentions :
2015-08-25 09:45:47.612 25182 INFO nova.compute.api [req-6c8701fe-c657-4bbc-b562-592d6bb8d8b9 05e4ebc87c174835a1f1ff7ad4973c9e 69f1e9392ffd4160bb0aaa0562dd31a8 - - -] [instance: 1a080d67-69ca-4b9f-9fa0-89007da09fb8] Instance is already in deleting state, ignoring this request

Until I run nova reset-state <uuid> , the guests never deletes.

Comment 4 Radomir Dopieralski 2015-09-07 09:38:22 UTC

Once a node is in the DELETING state, it will not accept further delete requests, as the task for deleting it might be queued and just waiting for its turn, and we don't want the users to keep creating new tasks. There was a patch that made it possible to re-issue a delete request even when the node is already in DELETING state, but it was rejected for the above reason.

In order to handle such transient errors properly, we would probably need to introduce a new state, such as DELETE_FAILED, so that the request could be re-issued then. That is however a large change, involving a database migration, and not possible in an update release.

Comment 5 Joe Talerico 2015-11-11 13:36:31 UTC

Can we get some update on this?

Comment 6 Joe Talerico 2015-11-11 13:36:37 UTC

Can we get some update on this?

Comment 7 Radomir Dopieralski 2015-11-23 09:56:29 UTC

As stated, handling this case properly requires some changes in how Nova tracks the state of operations like deleting. There is some effort to improve this in Mitaka, mostly focused on the migration state. At this moment I can't say whether the required changes will land in Mitaka or later. In the mean time, running reset-state after a failed delete is the only workaround I know.

Note You need to log in before you can comment on or make changes to this bug.