Hide Forgot
Description of problem: Performing a nova evacuation on an instance with an attached volume fails with 'ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500)' How reproducible: Every time Steps to Reproduce: 1. Launch an instance with an attached volume 2. Power off the compute node the instance was booted on 3. Attempt a nova evacuate: nova evacuate <instance_name> <destination_host> --password "${OS_PASSWORD}" Actual results: The evacuation fails with: nova-compute.log:2016-03-10 11:20:23.867 3316 ERROR nova.compute.manager [req-324d2738-6808-4e7d-9b52-88f32bc609b8 bfeb81563d9440a78d313d923693fb5a 2d1a06b20fea47ddabe215347a61a861 - - -] [instance: <uuid>] Setting instance vm_state to ERROR nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>] Traceback (most recent call last): nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6476, in _error_out_instance_on_exception nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>] yield nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3113, in rebuild_instance nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>] self._rebuild_default_impl(**kwargs) nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2942, in _rebuild_default_impl nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>] detach_block_devices(context, bdms) nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3091, in detach_block_devices nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>] destroy_bdm=False) nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4956, in _detach_volume nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>] self.volume_api.terminate_connection(context, volume_id, connector) nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>] File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 214, in wrapper nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>] res = method(self, ctx, volume_id, *args, **kwargs) nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>] File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 369, in terminate_connection nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>] connector) nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>] File "/usr/lib/python2.7/site-packages/cinderclient/v2/volumes.py", line 448, in terminate_connection nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>] {'connector': connector}) nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>] File "/usr/lib/python2.7/site-packages/cinderclient/v2/volumes.py", line 375, in _action nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>] return self.api.client.post(url, body=body) nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>] File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 118, in post nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>] return self._cs_request(url, 'POST', **kwargs) nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>] File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 112, in _cs_request nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>] return self.request(url, method, **kwargs) nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>] File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 105, in request nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>] raise exceptions.from_response(resp, body) nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>] ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-abaa35dc-c471-4613-b478-4579c2ce91d6) nova-compute.log:2016-03-10 11:20:23.867 3316 TRACE nova.compute.manager [instance: <uuid>] Expected results: The evacuation is successful Additional info: The below error is seen in the cinder api log: 2016-03-10 11:20:23.863 5316 ERROR cinder.api.middleware.fault [req-abaa35dc-c471-4613-b478-4579c2ce91d6 bfeb81563d9440a78d313d923693fb5a 2d1a06b20fea47ddabe215347a61a861 - - -] Caught error: Remote error: Remote error: VolumeBackendAPIException Bad or unexpected response from the storage volume backend API: Unable to terminate volume connection: Failed to detach iSCSI target for volume <uuid>. [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n executor_callback))\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n executor_callback)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper\n return f(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1181, in terminate_connection\n raise exception.VolumeBackendAPIException(data=err_msg)\n', u'VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Unable to terminate volume connection: Failed to detach iSCSI target for volume <uuid>.\n']. None None. 2016-03-10 11:20:23.864 5316 INFO cinder.api.middleware.fault [req-abaa35dc-c471-4613-b478-4579c2ce91d6 bfeb81563d9440a78d313d923693fb5a 2d1a06b20fea47ddabe215347a61a861 - - -] http://<ip>:8776/v2/<uuid>/volumes/<uuid>/action returned with HTTP 500 This looks like the issue reported in https://bugs.launchpad.net/nova/+bug/1522496 and there appears to be a patch already upstream for Liberty. Is it possible to backport this to OSP 7?
@eglynn : Yes, I was on leave for the past few days. I have applied for the LDAP accesses and would start working once I get them.
*** This bug has been marked as a duplicate of bug 1295681 ***