Bug 1256827
| Summary: | [Nova] Instance removal issue | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Joe Talerico <jtaleric> |
| Component: | openstack-nova | Assignee: | Radomir Dopieralski <rdopiera> |
| Status: | CLOSED EOL | QA Contact: | nlevinki <nlevinki> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 7.0 (Kilo) | CC: | berrange, dasmith, eglynn, kchamart, mwagner, rdopiera, sbauza, sferdjao, sgordon, srevivo, vromanso |
| Target Milestone: | --- | Keywords: | ZStream |
| Target Release: | 8.0 (Liberty) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-12-14 13:31:55 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Once a node is in the DELETING state, it will not accept further delete requests, as the task for deleting it might be queued and just waiting for its turn, and we don't want the users to keep creating new tasks. There was a patch that made it possible to re-issue a delete request even when the node is already in DELETING state, but it was rejected for the above reason. In order to handle such transient errors properly, we would probably need to introduce a new state, such as DELETE_FAILED, so that the request could be re-issued then. That is however a large change, involving a database migration, and not possible in an update release. Can we get some update on this? Can we get some update on this? As stated, handling this case properly requires some changes in how Nova tracks the state of operations like deleting. There is some effort to improve this in Mitaka, mostly focused on the migration state. At this moment I can't say whether the required changes will land in Mitaka or later. In the mean time, running reset-state after a failed delete is the only workaround I know. |
Description of problem: Deleting guests (with no network) hits this traceback : 2015-08-25 09:19:53.844 25191 DEBUG nova.api.openstack.wsgi [req-bb21e3c9-f1c5-472a-bb3c-e06cdcd3598c bf111661fd1245989bab0cf6a3f813e9 e9b8a420c1a644c799c04130ac8dada6 - - -] Calling method '<bound method Controller.detail of <nova.api.openstack.compute.servers.Controller object at 0x5915ad0>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:783 2015-08-25 09:19:53.845 25191 DEBUG nova.compute.api [req-bb21e3c9-f1c5-472a-bb3c-e06cdcd3598c bf111661fd1245989bab0cf6a3f813e9 e9b8a420c1a644c799c04130ac8dada6 - - -] Searching by: {'deleted': False, u'host': u'overcloud-compute-5.localdomain'} get_all /usr/lib/python2.7/site-packages/nova/compute/api.py:1962 2015-08-25 09:19:53.938 25191 DEBUG keystoneclient.session [req-bb21e3c9-f1c5-472a-bb3c-e06cdcd3598c bf111661fd1245989bab0cf6a3f813e9 e9b8a420c1a644c799c04130ac8dada6 - - -] REQ: curl -g -i -X GET http://172.21.33.11:9696/v2.0/ports.json?device_id=751e5479-c00a-472b-8276-d66fde0de492&device_id=9b41f1ed-2a05-4880-8586-e33ddfe5dec0&device_id=dc3f1134-5a57-442c-8e1d-c55b59cb2768&device_id=595b2edd-f8a8-449b-a7fc-44847f1b7df6&device_id=c85d145b-e0e1-4ab1-bc33-d3a6e2949672&device_id=e3e79468-d630-436f-b2ba-eb6727e32b9e&device_id=6cd0c0e1-4573-49fd-a741-2937efdf4bbf&device_id=34eb08be-04a9-4f25-93d6-e5b32bf8f888&device_id=20034eff-aa7c-438d-92a9-cdb074e3a023&device_id=1028f29d-f95f-4592-b418-edfde4a84886&device_id=1552f474-d934-4132-a659-150348d9ecf4&device_id=da746630-ce3b-4b9a-9346-12377e2c96eb&device_id=78b620eb-f942-421a-9891-0a7830b480c7&device_id=ebb2716e-b821-474e-a075-3726e4f3127d&device_id=1a080d67-69ca-4b9f-9fa0-89007da09fb8&device_id=e4986cf0-3189-40ba-a408-ad8b5a2e83d3&device_id=a233273d-4d76-445b-bcbd-f487675933cf&device_id=62f97e13-4026-4ade-83ad-c78829742d76&device_id=d5f8cb48-6746-48ac-af69-8343c21691e0&device_id=665e246e-110b-457e-870a-bc49481c044b -H "User-Agent: python-neutronclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}4ec6b2e5a3f6cad2942a8daf2dfa500990b6875a" _http_log_request /usr/lib/python2.7/site-packages/keystoneclient/session.py:195 2015-08-25 09:19:53.972 25191 DEBUG keystoneclient.session [req-bb21e3c9-f1c5-472a-bb3c-e06cdcd3598c bf111661fd1245989bab0cf6a3f813e9 e9b8a420c1a644c799c04130ac8dada6 - - -] RESP: _http_log_response /usr/lib/python2.7/site-packages/keystoneclient/session.py:224 2015-08-25 09:19:53.976 25191 DEBUG keystoneclient.session [req-bb21e3c9-f1c5-472a-bb3c-e06cdcd3598c bf111661fd1245989bab0cf6a3f813e9 e9b8a420c1a644c799c04130ac8dada6 - - -] RESP: _http_log_response /usr/lib/python2.7/site-packages/keystoneclient/session.py:224 2015-08-25 09:19:53.976 25191 DEBUG neutronclient.v2_0.client [req-bb21e3c9-f1c5-472a-bb3c-e06cdcd3598c bf111661fd1245989bab0cf6a3f813e9 e9b8a420c1a644c799c04130ac8dada6 - - -] Error message: Authentication required _handle_fault_response /usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py:176 2015-08-25 09:19:53.976 25191 ERROR nova.api.openstack [req-bb21e3c9-f1c5-472a-bb3c-e06cdcd3598c bf111661fd1245989bab0cf6a3f813e9 e9b8a420c1a644c799c04130ac8dada6 - - -] Caught error: Authentication required 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack Traceback (most recent call last): 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/api/openstack/__init__.py", line 125, in __call__ 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack return req.get_response(self.application) 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/request.py", line 1296, in send 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack application, catch_exc_info=False) 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/request.py", line 1260, in call_application 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack app_iter = application(self.environ, start_response) 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__ 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack return resp(environ, start_response) 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 634, in __call__ 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack return self._call_app(env, start_response) 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 554, in _call_app 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack return self._app(env, _fake_start_response) 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__ 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack return resp(environ, start_response) 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__ 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack return resp(environ, start_response) 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in __call__ 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack response = self.app(environ, start_response) 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__ 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack return resp(environ, start_response) 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__ 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack resp = self.call_func(req, *args, **self.kwargs) 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack return self.func(req, *args, **kwargs) 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 756, in __call__ 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack content_type, body, accept) 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 847, in _process_stack 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack request, action_args) 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 710, in post_process_extensions 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack **action_args) 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/contrib/security_groups.py", line 471, in detail 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack self._extend_servers(req, list(resp_obj.obj['servers'])) 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/contrib/security_groups.py", line 422, in _extend_servers 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack servers)) 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/network/security_group/neutron_driver.py", line 353, in get_instances_security_groups_bindings 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack ports = self._get_ports_from_server_list(servers, neutron) 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/network/security_group/neutron_driver.py", line 312, in _get_ports_from_server_list 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack ports.extend(neutron.list_ports(**search_opts).get('ports')) 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 102, in with_params 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack ret = self.function(instance, *args, **kwargs) 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 534, in list_ports 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack **_params) 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 307, in list 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack for r in self._pagination(collection, path, **params): 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 320, in _pagination 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack res = self.get(path, params=params) 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 293, in get 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack headers=headers, params=params) 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 270, in retry_request 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack headers=headers, params=params) 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 211, in do_request 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack self._handle_fault_response(status_code, replybody) 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 185, in _handle_fault_response 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack exception_handler_v20(status_code, des_error_body) 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 83, in exception_handler_v20 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack message=message) 2015-08-25 09:19:53.976 25191 TRACE nova.api.openstack NeutronClientException: Authentication required Then, subsequent deletes mentions : 2015-08-25 09:45:47.612 25182 INFO nova.compute.api [req-6c8701fe-c657-4bbc-b562-592d6bb8d8b9 05e4ebc87c174835a1f1ff7ad4973c9e 69f1e9392ffd4160bb0aaa0562dd31a8 - - -] [instance: 1a080d67-69ca-4b9f-9fa0-89007da09fb8] Instance is already in deleting state, ignoring this request Until I run nova reset-state <uuid> , the guests never deletes.