Description of problem: os-brick isn't cleaning up all multipath connections resulting in subsequent failure of 'create volume from image' Version-Release number of selected component (if applicable): python-os-brick-0.5.0-2.el7ost.noarch.rpm How reproducible: Consistently on arrays with more than four target ports Steps to Reproduce: 1. Install cinder with iscsi backend (backend has 8 target ports on array) 2. Use cinder client to create new volume from existing glance image 3-4 times 3. Volume creates eventually fail Actual results: Volume create failed with the following log message: cinder-volume 4096553 - [MetaCloud@40521 levelname="ERROR" component="cinder-volume" funcname="oslo_messaging.rpc.dispatcher" request_id="req-8c7315dc-d341-4a07-9e81-767dabcd2517" user="7da5700272094a8ea609d91a5516b9df" tenant="102051bc66434bbb8d5dda017a442095" instance="" lineno="/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py:151"] oslo_messaging.rpc.dispatcher Exception during message handling: Failed to copy image to volume: The device in the path /dev/disk/by-path/ip-10.1.19.40:3260-iscsi-iqn.2010-06.com.purestorage:flasharray.457fc932b46891ea-lun-1 is unavailable: Unable to access the backend storage via the path /dev/disk/by-path/ip-10.1.19.40:3260-iscsi-iqn.2010-06.com.purestorage:flasharray.457fc932b46891ea-lun-1. !!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last): !!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply !!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher executor_callback)) !!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch !!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher executor_callback) !!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch !!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args) !!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper !!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher return f(*args, **kwargs) !!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 547, in create_volume !!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher _run_flow() !!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 534, in _run_flow !!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher flow_engine.run() !!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 159, in run !!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher for _state in self.run_iter(): !!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 223, in run_iter !!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher failure.Failure.reraise_if_any(it) !!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 308, in reraise_if_any !!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher failures[0].reraise() !!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 315, in reraise !!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher six.reraise(*self._exc_info) !!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 82, in _execute_task !!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher result = task.execute(**arguments) !!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 834, in execute !!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher **volume_spec) !!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 768, in _create_from_image !!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher image_service !!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 651, in _create_from_image_download !!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher image_id, image_location, image_service) !!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 534, in _copy_image_to_volume !!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher raise exception.ImageCopyFailure(reason=ex) !!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher ImageCopyFailure: Failed to copy image to volume: The device in the path /dev/disk/by-path/ip-10.1.19.40:3260-iscsi-iqn.2010-06.com.purestorage:flasharray.457fc932b46891ea-lun-1 is unavailable: Unable to access the backend storage via the path /dev/disk/by-path/ip-10.1.19.40:3260-iscsi-iqn.2010-06.com.purestorage:flasharray.457fc932b46891ea-lun-1. !!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher Expected results: Volume available with specified image contents Additional info: Fixed upstream: https://bugs.launchpad.net/os-brick/+bug/1502999
Fix for this is available on latest python-os-brick versions (python-os-brick-0.5.0-3.el7ost onward). *** This bug has been marked as a duplicate of bug 1372428 ***