1446752 – Can't delete instance due to failure to deallocate network.

Bug 1446752 - Can't delete instance due to failure to deallocate network.

Summary: Can't delete instance due to failure to deallocate network.

Keywords:
Status:	CLOSED NOTABUG
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-nova
Sub Component:
Version:	10.0 (Newton)
Hardware:	All
OS:	Linux
Priority:	unspecified
Severity:	high
Target Milestone:	---
Target Release:	---
Assignee:	Eoghan Glynn
QA Contact:	Prasanth Anbalagan
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2017-04-28 19:06 UTC by Siggy Sigwald
Modified:	2020-06-11 13:44 UTC (History)
CC List:	13 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2017-07-17 15:47:08 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Launchpad	1656056	0	None	None	None	2017-04-28 19:06:55 UTC
OpenStack gerrit	422696	0	None	ABANDONED	neutron: re-raise 401 for a non-admin client	2020-12-07 05:56:51 UTC

Description Siggy Sigwald 2017-04-28 19:06:56 UTC

Description of problem:
Tenant consistently encounters heat stack-delete failure after about an hour, during which a server in the stack is perpetually in the DELETE IN PROGRESS state for most of the time. 

Version-Release number of selected component (if applicable):
openstack-neutron-9.2.0-2.el7ost.noarch


Actual results:
Instance goes to ERROR and can't be deleted.

Expected results:
Instance is deleted.

Additional info:
2017-04-27 17:26:45.289 7162 ERROR nova.network.neutronv2.api [req-a6dd729f-4a96-45df-953a-0f68356c16ae 74724ded789d435e835c6c8dd64da7a3 e141bc50e81c47469ce45aeab9e179b2 - - -] Neutron client was not able to generate a valid admin token, please verify Neutron admin credential located in nova.conf
2017-04-27 17:26:45.289 7162 ERROR nova.compute.manager [req-a6dd729f-4a96-45df-953a-0f68356c16ae 74724ded789d435e835c6c8dd64da7a3 e141bc50e81c47469ce45aeab9e179b2 - - -] [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f] Failed to deallocate network for instance. Error: Networking client is experiencing an unauthorized exception.
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [req-a6dd729f-4a96-45df-953a-0f68356c16ae 74724ded789d435e835c6c8dd64da7a3 e141bc50e81c47469ce45aeab9e179b2 - - -] [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f] Setting instance vm_state to ERROR
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f] Traceback (most recent call last):
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2376, in do_terminate_instance
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     self._delete_instance(context, instance, bdms, quotas)
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/nova/hooks.py", line 154, in inner
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     rv = f(*args, **kwargs)
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2339, in _delete_instance
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     quotas.rollback()
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     self.force_reraise()
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     six.reraise(self.type_, self.value, self.tb)
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2303, in _delete_instance
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     self._shutdown_instance(context, instance, bdms)
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2211, in _shutdown_instance
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     self._try_deallocate_network(context, instance, requested_networks)
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2135, in _try_deallocate_network
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     self._set_instance_obj_error_state(context, instance)
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     self.force_reraise()
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     six.reraise(self.type_, self.value, self.tb)
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2129, in _try_deallocate_network
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     self._deallocate_network(context, instance, requested_networks)
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1662, in _deallocate_network
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     context, instance, requested_networks=requested_networks)
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1157, in deallocate_for_instance
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     data = neutron.list_ports(**search_opts)
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 97, in wrapper
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     ret = obj(*args, **kwargs)
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 742, in list_ports
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     **_params)
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 97, in wrapper
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     ret = obj(*args, **kwargs)
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 375, in list
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     for r in self._pagination(collection, path, **params):
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 390, in _pagination
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     res = self.get(path, params=params)
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 97, in wrapper
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     ret = obj(*args, **kwargs)
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 360, in get
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     headers=headers, params=params)
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 97, in wrapper
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     ret = obj(*args, **kwargs)
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 337, in retry_request
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     headers=headers, params=params)
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 97, in wrapper
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     ret = obj(*args, **kwargs)
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 300, in do_request
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     self._handle_fault_response(status_code, replybody, resp)
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 113, in wrapper
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     raise exception.NeutronAdminCredentialConfigurationInvalid()
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f] NeutronAdminCredentialConfigurationInvalid: Networking client is experiencing an unauthorized exception.
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f] 
2017-04-27 17:26:45.481 7162 INFO nova.compute.manager [-] [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f] During sync_power_state the instance has a pending task (deleting). Skip.
2017-04-27 17:26:45.627 7162 INFO nova.compute.manager [req-a6dd729f-4a96-45df-953a-0f68356c16ae 74724ded789d435e835c6c8dd64da7a3 e141bc50e81c47469ce45aeab9e179b2 - - -] [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f] Successfully reverted task state from deleting on failure for instance.
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server [req-a6dd729f-4a96-45df-953a-0f68356c16ae 74724ded789d435e835c6c8dd64da7a3 e141bc50e81c47469ce45aeab9e179b2 - - -] Exception during message handling
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 188, in decorated_function
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     LOG.warning(msg, e, instance=instance)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 157, in decorated_function
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 613, in decorated_function
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2388, in terminate_instance
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     do_terminate_instance(instance, bdms)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2386, in do_terminate_instance
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     self._set_instance_obj_error_state(context, instance)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2376, in do_terminate_instance
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     self._delete_instance(context, instance, bdms, quotas)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/hooks.py", line 154, in inner
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     rv = f(*args, **kwargs)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2339, in _delete_instance
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     quotas.rollback()
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2303, in _delete_instance
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     self._shutdown_instance(context, instance, bdms)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2211, in _shutdown_instance
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     self._try_deallocate_network(context, instance, requested_networks)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2135, in _try_deallocate_network
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     self._set_instance_obj_error_state(context, instance)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2129, in _try_deallocate_network
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     self._deallocate_network(context, instance, requested_networks)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1662, in _deallocate_network
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     context, instance, requested_networks=requested_networks)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1157, in deallocate_for_instance
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     data = neutron.list_ports(**search_opts)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 97, in wrapper
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     ret = obj(*args, **kwargs)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 742, in list_ports
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     **_params)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 97, in wrapper
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     ret = obj(*args, **kwargs)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 375, in list
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     for r in self._pagination(collection, path, **params):
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 390, in _pagination
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     res = self.get(path, params=params)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 97, in wrapper
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     ret = obj(*args, **kwargs)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 360, in get
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     headers=headers, params=params)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 97, in wrapper
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     ret = obj(*args, **kwargs)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 337, in retry_request
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     headers=headers, params=params)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 97, in wrapper
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     ret = obj(*args, **kwargs)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 300, in do_request
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     self._handle_fault_response(status_code, replybody, resp)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 113, in wrapper
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     raise exception.NeutronAdminCredentialConfigurationInvalid()
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server NeutronAdminCredentialConfigurationInvalid: Networking client is experiencing an unauthorized exception.
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server

Comment 1 Red Hat Bugzilla Rules Engine 2017-04-28 19:09:18 UTC

This bugzilla has been removed from the release and needs to be reviewed and Triaged for another Target Release.

Comment 3 Dan Smith 2017-05-05 14:27:48 UTC

The reason for the failure aside, if we are unable to deallocate network for the instance, the proper behavior _is_ to set the instance to ERROR state. However you should be able to retry deleting the instance to retry deallocating the network. What happens when you try to delete the ERROR instance? Does it complain about a task_state or something?

Note You need to log in before you can comment on or make changes to this bug.