Description of problem:
os-brick isn't cleaning up all multipath connections resulting in subsequent failure of 'create volume from image'
Version-Release number of selected component (if applicable):
python-os-brick-0.5.0-2.el7ost.noarch.rpm
How reproducible:
Consistently on arrays with more than four target ports
Steps to Reproduce:
1. Install cinder with iscsi backend (backend has 8 target ports on array)
2. Use cinder client to create new volume from existing glance image 3-4 times
3. Volume creates eventually fail
Actual results:
Volume create failed with the following log message:
cinder-volume 4096553 - [MetaCloud@40521 levelname="ERROR" component="cinder-volume" funcname="oslo_messaging.rpc.dispatcher" request_id="req-8c7315dc-d341-4a07-9e81-767dabcd2517" user="7da5700272094a8ea609d91a5516b9df" tenant="102051bc66434bbb8d5dda017a442095" instance="" lineno="/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py:151"] oslo_messaging.rpc.dispatcher Exception during message handling: Failed to copy image to volume: The device in the path /dev/disk/by-path/ip-10.1.19.40:3260-iscsi-iqn.2010-06.com.purestorage:flasharray.457fc932b46891ea-lun-1 is unavailable: Unable to access the backend storage via the path /dev/disk/by-path/ip-10.1.19.40:3260-iscsi-iqn.2010-06.com.purestorage:flasharray.457fc932b46891ea-lun-1.
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher executor_callback))
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher executor_callback)
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 547, in create_volume
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher _run_flow()
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 534, in _run_flow
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher flow_engine.run()
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 159, in run
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher for _state in self.run_iter():
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 223, in run_iter
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher failure.Failure.reraise_if_any(it)
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 308, in reraise_if_any
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher failures[0].reraise()
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 315, in reraise
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher six.reraise(*self._exc_info)
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 82, in _execute_task
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher result = task.execute(**arguments)
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 834, in execute
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher **volume_spec)
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 768, in _create_from_image
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher image_service
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 651, in _create_from_image_download
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher image_id, image_location, image_service)
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 534, in _copy_image_to_volume
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher raise exception.ImageCopyFailure(reason=ex)
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher ImageCopyFailure: Failed to copy image to volume: The device in the path /dev/disk/by-path/ip-10.1.19.40:3260-iscsi-iqn.2010-06.com.purestorage:flasharray.457fc932b46891ea-lun-1 is unavailable: Unable to access the backend storage via the path /dev/disk/by-path/ip-10.1.19.40:3260-iscsi-iqn.2010-06.com.purestorage:flasharray.457fc932b46891ea-lun-1.
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher
Expected results:
Volume available with specified image contents
Additional info:
Fixed upstream: https://bugs.launchpad.net/os-brick/+bug/1502999
Fix for this is available on latest python-os-brick versions (python-os-brick-0.5.0-3.el7ost onward).
*** This bug has been marked as a duplicate of bug 1372428 ***