Bug 1381347 - os-brick isn't cleaning up iscsi multipath connections
Summary: os-brick isn't cleaning up iscsi multipath connections
Keywords:
Status: CLOSED DUPLICATE of bug 1372428
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-os-brick
Version: 8.0 (Liberty)
Hardware: All
OS: All
urgent
urgent
Target Milestone: async
: 8.0 (Liberty)
Assignee: Gorka Eguileor
QA Contact: Shai Revivo
URL:
Whiteboard: hot
Depends On: 1372428 1418856
Blocks: 1194008
TreeView+ depends on / blocked
 
Reported: 2016-10-03 19:11 UTC by Tim Clark
Modified: 2019-12-16 06:58 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-11-02 09:17:02 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 1502999 0 None None None 2016-10-03 19:11:28 UTC

Description Tim Clark 2016-10-03 19:11:29 UTC
Description of problem:
os-brick isn't cleaning up all multipath connections resulting in subsequent failure of 'create volume from image'

Version-Release number of selected component (if applicable):
python-os-brick-0.5.0-2.el7ost.noarch.rpm

How reproducible:
Consistently on arrays with more than four target ports

Steps to Reproduce:
1. Install cinder with iscsi backend (backend has 8 target ports on array)
2. Use cinder client to create new volume from existing glance image 3-4 times
3. Volume creates eventually fail

Actual results:
Volume create failed with the following log message:
cinder-volume 4096553 - [MetaCloud@40521 levelname="ERROR" component="cinder-volume" funcname="oslo_messaging.rpc.dispatcher" request_id="req-8c7315dc-d341-4a07-9e81-767dabcd2517" user="7da5700272094a8ea609d91a5516b9df" tenant="102051bc66434bbb8d5dda017a442095" instance="" lineno="/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py:151"] oslo_messaging.rpc.dispatcher Exception during message handling: Failed to copy image to volume: The device in the path /dev/disk/by-path/ip-10.1.19.40:3260-iscsi-iqn.2010-06.com.purestorage:flasharray.457fc932b46891ea-lun-1 is unavailable: Unable to access the backend storage via the path /dev/disk/by-path/ip-10.1.19.40:3260-iscsi-iqn.2010-06.com.purestorage:flasharray.457fc932b46891ea-lun-1.
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher     return f(*args, **kwargs)
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 547, in create_volume
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher     _run_flow()
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 534, in _run_flow
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher     flow_engine.run()
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 159, in run
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher     for _state in self.run_iter():
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 223, in run_iter
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher     failure.Failure.reraise_if_any(it)
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 308, in reraise_if_any
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher     failures[0].reraise()
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 315, in reraise
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher     six.reraise(*self._exc_info)
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 82, in _execute_task
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher     result = task.execute(**arguments)
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 834, in execute
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher     **volume_spec)
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 768, in _create_from_image
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher     image_service
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 651, in _create_from_image_download
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher     image_id, image_location, image_service)
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 534, in _copy_image_to_volume
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher     raise exception.ImageCopyFailure(reason=ex)
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher ImageCopyFailure: Failed to copy image to volume: The device in the path /dev/disk/by-path/ip-10.1.19.40:3260-iscsi-iqn.2010-06.com.purestorage:flasharray.457fc932b46891ea-lun-1 is unavailable: Unable to access the backend storage via the path /dev/disk/by-path/ip-10.1.19.40:3260-iscsi-iqn.2010-06.com.purestorage:flasharray.457fc932b46891ea-lun-1.
!!!NL!!! 4096553 TRACE oslo_messaging.rpc.dispatcher

Expected results:
Volume available with specified image contents

Additional info:
Fixed upstream: https://bugs.launchpad.net/os-brick/+bug/1502999

Comment 5 Gorka Eguileor 2016-11-02 09:17:02 UTC
Fix for this is available on latest python-os-brick versions (python-os-brick-0.5.0-3.el7ost onward).

*** This bug has been marked as a duplicate of bug 1372428 ***


Note You need to log in before you can comment on or make changes to this bug.