Bug 1285186 - Instance evacuate failed with ('Connection aborted.', BadStatusLine("''",)) [NEEDINFO]
Instance evacuate failed with ('Connection aborted.', BadStatusLine("''",))
Status: CLOSED WORKSFORME
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-cinder (Show other bugs)
6.0 (Juno)
Unspecified Unspecified
unspecified Severity high
: ---
: 8.0 (Liberty)
Assigned To: Eric Harney
nlevinki
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-11-25 03:14 EST by Chen
Modified: 2016-04-26 12:02 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-12-17 08:48:58 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
sgotliv: needinfo? (cchen)


Attachments (Terms of Use)

  None (edit)
Description Chen 2015-11-25 03:14:00 EST
Description of problem:

Instance evacuate failed with ('Connection aborted.', BadStatusLine("''",))

Version-Release number of selected component (if applicable):

python-cinderclient-1.1.1-1.el7ost.noarch

How reproducible:

50%

Steps to Reproduce:
1.
2.
3.

Actual results:

2015-11-24 11:45:14.606 17299 DEBUG cinderclient.client [req-a2695d2e-3ce5-4168-8028-7624b4682c7d ] Connection error: ('Connection aborted.', BadStatusLine("''",)) _cs_request /usr/lib/python2.7/site-packages/cinderclient/client.py:291
2015-11-24 11:45:14.607 17299 ERROR nova.compute.manager [req-a2695d2e-3ce5-4168-8028-7624b4682c7d None] [instance: 89547ff5-d04f-4c5d-9976-258925619888] Instance failed block device setup
2015-11-24 11:45:14.607 17299 TRACE nova.compute.manager [instance: 89547ff5-d04f-4c5d-9976-258925619888] Traceback (most recent call last):
2015-11-24 11:45:14.607 17299 TRACE nova.compute.manager [instance: 89547ff5-d04f-4c5d-9976-258925619888]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1864, in _prep_block_device
2015-11-24 11:45:14.607 17299 TRACE nova.compute.manager [instance: 89547ff5-d04f-4c5d-9976-258925619888]     do_check_attach=do_check_attach) +
2015-11-24 11:45:14.607 17299 TRACE nova.compute.manager [instance: 89547ff5-d04f-4c5d-9976-258925619888]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 407, in attach_block_devices
2015-11-24 11:45:14.607 17299 TRACE nova.compute.manager [instance: 89547ff5-d04f-4c5d-9976-258925619888]     map(_log_and_attach, block_device_mapping)
2015-11-24 11:45:14.607 17299 TRACE nova.compute.manager [instance: 89547ff5-d04f-4c5d-9976-258925619888]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 405, in _log_and_attach
2015-11-24 11:45:14.607 17299 TRACE nova.compute.manager [instance: 89547ff5-d04f-4c5d-9976-258925619888]     bdm.attach(*attach_args, **attach_kwargs)
2015-11-24 11:45:14.607 17299 TRACE nova.compute.manager [instance: 89547ff5-d04f-4c5d-9976-258925619888]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 46, in wrapped
2015-11-24 11:45:14.607 17299 TRACE nova.compute.manager [instance: 89547ff5-d04f-4c5d-9976-258925619888]     ret_val = method(obj, context, *args, **kwargs)
2015-11-24 11:45:14.607 17299 TRACE nova.compute.manager [instance: 89547ff5-d04f-4c5d-9976-258925619888]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 237, in attach
2015-11-24 11:45:14.607 17299 TRACE nova.compute.manager [instance: 89547ff5-d04f-4c5d-9976-258925619888]     connector)
2015-11-24 11:45:14.607 17299 TRACE nova.compute.manager [instance: 89547ff5-d04f-4c5d-9976-258925619888]   File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 185, in wrapper
2015-11-24 11:45:14.607 17299 TRACE nova.compute.manager [instance: 89547ff5-d04f-4c5d-9976-258925619888]     res = method(self, ctx, volume_id, *args, **kwargs)
2015-11-24 11:45:14.607 17299 TRACE nova.compute.manager [instance: 89547ff5-d04f-4c5d-9976-258925619888]   File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 349, in initialize_connection
2015-11-24 11:45:14.607 17299 TRACE nova.compute.manager [instance: 89547ff5-d04f-4c5d-9976-258925619888]     connector)
2015-11-24 11:45:14.607 17299 TRACE nova.compute.manager [instance: 89547ff5-d04f-4c5d-9976-258925619888]   File "/usr/lib/python2.7/site-packages/cinderclient/v1/volumes.py", line 321, in initialize_connection
2015-11-24 11:45:14.607 17299 TRACE nova.compute.manager [instance: 89547ff5-d04f-4c5d-9976-258925619888]     {'connector': connector})[1]['connection_info']
2015-11-24 11:45:14.607 17299 TRACE nova.compute.manager [instance: 89547ff5-d04f-4c5d-9976-258925619888]   File "/usr/lib/python2.7/site-packages/cinderclient/v1/volumes.py", line 250, in _action
2015-11-24 11:45:14.607 17299 TRACE nova.compute.manager [instance: 89547ff5-d04f-4c5d-9976-258925619888]     return self.api.client.post(url, body=body)
2015-11-24 11:45:14.607 17299 TRACE nova.compute.manager [instance: 89547ff5-d04f-4c5d-9976-258925619888]   File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 305, in post
2015-11-24 11:45:14.607 17299 TRACE nova.compute.manager [instance: 89547ff5-d04f-4c5d-9976-258925619888]     return self._cs_request(url, 'POST', **kwargs)
2015-11-24 11:45:14.607 17299 TRACE nova.compute.manager [instance: 89547ff5-d04f-4c5d-9976-258925619888]   File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 294, in _cs_request
2015-11-24 11:45:14.607 17299 TRACE nova.compute.manager [instance: 89547ff5-d04f-4c5d-9976-258925619888]     raise exceptions.ConnectionError(msg)
2015-11-24 11:45:14.607 17299 TRACE nova.compute.manager [instance: 89547ff5-d04f-4c5d-9976-258925619888] CinderConnectionFailed: Connection to cinder host failed: Unable to establish connection: ('Connection aborted.', BadStatusLine("''",))
2015-11-24 11:45:14.607 17299 TRACE nova.compute.manager [instance: 89547ff5-d04f-4c5d-9976-258925619888]
2015-11-24 11:45:14.608 17299 ERROR nova.compute.manager [req-a2695d2e-3ce5-4168-8028-7624b4682c7d None] [instance: 89547ff5-d04f-4c5d-9976-258925619888] Setting instance vm_state to ERROR
2015-11-24 11:45:14.608 17299 TRACE nova.compute.manager [instance: 89547ff5-d04f-4c5d-9976-258925619888] Traceback (most recent call last):
2015-11-24 11:45:14.608 17299 TRACE nova.compute.manager [instance: 89547ff5-d04f-4c5d-9976-258925619888]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6152, in _error_out_instance_on_exception
2015-11-24 11:45:14.608 17299 TRACE nova.compute.manager [instance: 89547ff5-d04f-4c5d-9976-258925619888]     yield
2015-11-24 11:45:14.608 17299 TRACE nova.compute.manager [instance: 89547ff5-d04f-4c5d-9976-258925619888]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2870, in rebuild_instance
2015-11-24 11:45:14.608 17299 TRACE nova.compute.manager [instance: 89547ff5-d04f-4c5d-9976-258925619888]     self._rebuild_default_impl(**kwargs)
2015-11-24 11:45:14.608 17299 TRACE nova.compute.manager [instance: 89547ff5-d04f-4c5d-9976-258925619888]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2716, in _rebuild_default_impl
2015-11-24 11:45:14.608 17299 TRACE nova.compute.manager [instance: 89547ff5-d04f-4c5d-9976-258925619888]     new_block_device_info = attach_block_devices(context, instance, bdms)
2015-11-24 11:45:14.608 17299 TRACE nova.compute.manager [instance: 89547ff5-d04f-4c5d-9976-258925619888]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1892, in _prep_block_device
2015-11-24 11:45:14.608 17299 TRACE nova.compute.manager [instance: 89547ff5-d04f-4c5d-9976-258925619888]     raise exception.InvalidBDM()
2015-11-24 11:45:14.608 17299 TRACE nova.compute.manager [instance: 89547ff5-d04f-4c5d-9976-258925619888] InvalidBDM: Block Device Mapping is Invalid.
2015-11-24 11:45:14.608 17299 TRACE nova.compute.manager [instance: 89547ff5-d04f-4c5d-9976-258925619888]

The cinderclient failed to post an API call to cinder-api, which causes the evacuation failed

Expected results:

The evacuation should succeed

Additional info:
Comment 2 Sergey Gotliv 2015-12-01 06:49:50 EST
Can you, please, provide steps to reproduce?
Comment 3 Sergey Gotliv 2015-12-01 09:18:06 EST
According to the volume logs it seems that EMC VNX driver had some issues to communicate with the storage backend. Try to understand why this host is unreachable. 

f61b5d0734744090a2fcf4f1ee4d72f6 - - -] EMC: Command Result: Error:  Destination Host Unreachable\nError:  Destination Host Unreachable\nError:  Destination Host Unreachable\nError:  Destination Host Unreachable\n\n. command_execute /usr/lib/python2.7/site-packages/cinder/volume/drivers/emc/emc_vnx_cli.py:1325
2015-11-24 11:12:45.017 26329 DEBUG cinder.volume.drivers.emc.emc_vnx_cli [req-b2feeb8f-1c4a-4169-b1b0-cbb7f37511f8 060bd4116dda49e0b1699ba3481a8ba4 f61b5d0734744090a2fcf4f1ee4d72f6 - - -] Exiting CommandLineHelper.command_execute. Spent 17.810446 sec. Return ('Error:  Destination Host Unreachable\nError:  Destination Host Unreachable\nError:  Destination Host Unreachable\nError:  Destination Host Unreachable\n\n', 0) inner /usr/lib/python2.7/site-packages/cinder/volume/drivers/emc/emc_vnx_cli.py:114

Note You need to log in before you can comment on or make changes to this bug.