Bug 1381347

Summary: os-brick isn't cleaning up iscsi multipath connections
Product: Red Hat OpenStack Reporter: Tim Clark <timotcla>
Component: python-os-brickAssignee: Gorka Eguileor <geguileo>
Status: CLOSED DUPLICATE QA Contact: Shai Revivo <srevivo>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 8.0 (Liberty)CC: aathomas, apevec, charcrou, eharney, geguileo, jdonohue, jschluet, lhh, pablo.iranzo
Target Milestone: asyncKeywords: ZStream
Target Release: 8.0 (Liberty)   
Hardware: All   
OS: All   
Whiteboard: hot
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-11-02 09:17:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On: 1372428, 1418856    
Bug Blocks: 1194008    

Description Tim Clark 2016-10-03 19:11:29 UTC 
Description of problem:
os-brick isn't cleaning up all multipath connections resulting in subsequent failure of 'create volume from image'

Version-Release number of selected component (if applicable):
python-os-brick-0.5.0-2.el7ost.noarch.rpm

How reproducible:
Consistently on arrays with more than four target ports

Steps to Reproduce:
1. Install cinder with iscsi backend (backend has 8 target ports on array)
2. Use cinder client to create new volume from existing glance image 3-4 times
3. Volume creates eventually fail

Actual results:
Volume create failed with the following log message:
cinder-volume 4096553 - [MetaCloud@40521 levelname="ERROR" component="cinder-volume" funcname="oslo_messaging.rpc.dispatcher" request_id="req-8c7315dc-d341-4a07-9e81-767dabcd2517" user="7da5700272094a8ea609d91a5516b9df" tenant="102051bc66434bbb8d5dda017a442095" instance="" lineno="/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py:151"] oslo_messaging.rpc.dispatcher Exception during message handling: Failed to copy image to volume: The device in the path /dev/disk/by-path/ip-10.1.19.40:3260-iscsi-iqn.2010-06.com.purestorage:flasharray.457fc932b46891ea-lun-1 is unavailable: Unable to access the backend storage via the path /dev/disk/by-path/ip-10.1.19.40:3260-iscsi-iqn.2010-06.com.purestorage:flasharray.457fc932b46891ea-lun-1.
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher     return f(*args, **kwargs)
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 547, in create_volume
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher     _run_flow()
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 534, in _run_flow
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher     flow_engine.run()
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 159, in run
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher     for _state in self.run_iter():
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 223, in run_iter
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher     failure.Failure.reraise_if_any(it)
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 308, in reraise_if_any
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher     failures[0].reraise()
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 315, in reraise
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher     six.reraise(*self._exc_info)
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 82, in _execute_task
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher     result = task.execute(**arguments)
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 834, in execute
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher     **volume_spec)
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 768, in _create_from_image
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher     image_service
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 651, in _create_from_image_download
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher     image_id, image_location, image_service)
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 534, in _copy_image_to_volume
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher     raise exception.ImageCopyFailure(reason=ex)
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher ImageCopyFailure: Failed to copy image to volume: The device in the path /dev/disk/by-path/ip-10.1.19.40:3260-iscsi-iqn.2010-06.com.purestorage:flasharray.457fc932b46891ea-lun-1 is unavailable: Unable to access the backend storage via the path /dev/disk/by-path/ip-10.1.19.40:3260-iscsi-iqn.2010-06.com.purestorage:flasharray.457fc932b46891ea-lun-1.
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher

Expected results:
Volume available with specified image contents

Additional info:
Fixed upstream: https://bugs.launchpad.net/os-brick/+bug/1502999
Comment 5 Gorka Eguileor 2016-11-02 09:17:02 UTC 
Fix for this is available on latest python-os-brick versions (python-os-brick-0.5.0-3.el7ost onward).

*** This bug has been marked as a duplicate of bug 1372428 ***