Bug 1446752

Summary: Can't delete instance due to failure to deallocate network.
Product: Red Hat OpenStack Reporter: Siggy Sigwald <ssigwald>
Component: openstack-novaAssignee: Eoghan Glynn <eglynn>
Status: CLOSED NOTABUG QA Contact: Prasanth Anbalagan <panbalag>
Severity: high Docs Contact:
Priority: unspecified    
Version: 10.0 (Newton)CC: amuller, awaugama, berrange, chrisw, dasmith, eglynn, kchamart, nyechiel, sbauza, sferdjao, sgordon, srevivo, vromanso
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-07-17 15:47:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Siggy Sigwald 2017-04-28 19:06:56 UTC 
Description of problem:
Tenant consistently encounters heat stack-delete failure after about an hour, during which a server in the stack is perpetually in the DELETE IN PROGRESS state for most of the time. 

Version-Release number of selected component (if applicable):
openstack-neutron-9.2.0-2.el7ost.noarch


Actual results:
Instance goes to ERROR and can't be deleted.

Expected results:
Instance is deleted.

Additional info:
2017-04-27 17:26:45.289 7162 ERROR nova.network.neutronv2.api [req-a6dd729f-4a96-45df-953a-0f68356c16ae 74724ded789d435e835c6c8dd64da7a3 e141bc50e81c47469ce45aeab9e179b2 - - -] Neutron client was not able to generate a valid admin token, please verify Neutron admin credential located in nova.conf
2017-04-27 17:26:45.289 7162 ERROR nova.compute.manager [req-a6dd729f-4a96-45df-953a-0f68356c16ae 74724ded789d435e835c6c8dd64da7a3 e141bc50e81c47469ce45aeab9e179b2 - - -] [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f] Failed to deallocate network for instance. Error: Networking client is experiencing an unauthorized exception.
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [req-a6dd729f-4a96-45df-953a-0f68356c16ae 74724ded789d435e835c6c8dd64da7a3 e141bc50e81c47469ce45aeab9e179b2 - - -] [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f] Setting instance vm_state to ERROR
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f] Traceback (most recent call last):
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2376, in do_terminate_instance
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     self._delete_instance(context, instance, bdms, quotas)
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/nova/hooks.py", line 154, in inner
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     rv = f(*args, **kwargs)
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2339, in _delete_instance
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     quotas.rollback()
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     self.force_reraise()
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     six.reraise(self.type_, self.value, self.tb)
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2303, in _delete_instance
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     self._shutdown_instance(context, instance, bdms)
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2211, in _shutdown_instance
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     self._try_deallocate_network(context, instance, requested_networks)
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2135, in _try_deallocate_network
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     self._set_instance_obj_error_state(context, instance)
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     self.force_reraise()
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     six.reraise(self.type_, self.value, self.tb)
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2129, in _try_deallocate_network
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     self._deallocate_network(context, instance, requested_networks)
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1662, in _deallocate_network
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     context, instance, requested_networks=requested_networks)
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1157, in deallocate_for_instance
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     data = neutron.list_ports(**search_opts)
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 97, in wrapper
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     ret = obj(*args, **kwargs)
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 742, in list_ports
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     **_params)
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 97, in wrapper
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     ret = obj(*args, **kwargs)
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 375, in list
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     for r in self._pagination(collection, path, **params):
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 390, in _pagination
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     res = self.get(path, params=params)
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 97, in wrapper
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     ret = obj(*args, **kwargs)
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 360, in get
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     headers=headers, params=params)
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 97, in wrapper
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     ret = obj(*args, **kwargs)
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 337, in retry_request
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     headers=headers, params=params)
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 97, in wrapper
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     ret = obj(*args, **kwargs)
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 300, in do_request
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     self._handle_fault_response(status_code, replybody, resp)
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 113, in wrapper
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f]     raise exception.NeutronAdminCredentialConfigurationInvalid()
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f] NeutronAdminCredentialConfigurationInvalid: Networking client is experiencing an unauthorized exception.
2017-04-27 17:26:45.386 7162 ERROR nova.compute.manager [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f] 
2017-04-27 17:26:45.481 7162 INFO nova.compute.manager [-] [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f] During sync_power_state the instance has a pending task (deleting). Skip.
2017-04-27 17:26:45.627 7162 INFO nova.compute.manager [req-a6dd729f-4a96-45df-953a-0f68356c16ae 74724ded789d435e835c6c8dd64da7a3 e141bc50e81c47469ce45aeab9e179b2 - - -] [instance: 3a8815e3-eae5-41de-aeba-2a1b4139310f] Successfully reverted task state from deleting on failure for instance.
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server [req-a6dd729f-4a96-45df-953a-0f68356c16ae 74724ded789d435e835c6c8dd64da7a3 e141bc50e81c47469ce45aeab9e179b2 - - -] Exception during message handling
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 188, in decorated_function
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     LOG.warning(msg, e, instance=instance)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 157, in decorated_function
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 613, in decorated_function
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2388, in terminate_instance
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     do_terminate_instance(instance, bdms)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2386, in do_terminate_instance
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     self._set_instance_obj_error_state(context, instance)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2376, in do_terminate_instance
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     self._delete_instance(context, instance, bdms, quotas)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/hooks.py", line 154, in inner
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     rv = f(*args, **kwargs)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2339, in _delete_instance
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     quotas.rollback()
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2303, in _delete_instance
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     self._shutdown_instance(context, instance, bdms)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2211, in _shutdown_instance
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     self._try_deallocate_network(context, instance, requested_networks)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2135, in _try_deallocate_network
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     self._set_instance_obj_error_state(context, instance)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2129, in _try_deallocate_network
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     self._deallocate_network(context, instance, requested_networks)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1662, in _deallocate_network
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     context, instance, requested_networks=requested_networks)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1157, in deallocate_for_instance
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     data = neutron.list_ports(**search_opts)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 97, in wrapper
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     ret = obj(*args, **kwargs)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 742, in list_ports
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     **_params)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 97, in wrapper
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     ret = obj(*args, **kwargs)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 375, in list
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     for r in self._pagination(collection, path, **params):
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 390, in _pagination
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     res = self.get(path, params=params)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 97, in wrapper
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     ret = obj(*args, **kwargs)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 360, in get
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     headers=headers, params=params)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 97, in wrapper
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     ret = obj(*args, **kwargs)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 337, in retry_request
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     headers=headers, params=params)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 97, in wrapper
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     ret = obj(*args, **kwargs)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 300, in do_request
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     self._handle_fault_response(status_code, replybody, resp)
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 113, in wrapper
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server     raise exception.NeutronAdminCredentialConfigurationInvalid()
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server NeutronAdminCredentialConfigurationInvalid: Networking client is experiencing an unauthorized exception.
2017-04-27 17:26:45.718 7162 ERROR oslo_messaging.rpc.server
Comment 1 Red Hat Bugzilla Rules Engine 2017-04-28 19:09:18 UTC 
This bugzilla has been removed from the release and needs to be reviewed and Triaged for another Target Release.
Comment 3 Dan Smith 2017-05-05 14:27:48 UTC 
The reason for the failure aside, if we are unable to deallocate network for the instance, the proper behavior _is_ to set the instance to ERROR state. However you should be able to retry deleting the instance to retry deallocating the network. What happens when you try to delete the ERROR instance? Does it complain about a task_state or something?