Bug 1634165 - fc rescan issue when looking at specific hbas
Summary: fc rescan issue when looking at specific hbas
Keywords:
Status: CLOSED EOL
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-os-brick
Version: 8.0 (Liberty)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: zstream
: 8.0 (Liberty)
Assignee: Alan Bishop
QA Contact: Tzach Shefi
URL:
Whiteboard:
Depends On: 1630842 1631024 1634163 1634164
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-09-28 20:36 UTC by Alan Bishop
Modified: 2020-07-08 08:47 UTC (History)
11 users (show)

Fixed In Version: python-os-brick-0.5.0-8.el7ost
Doc Type: Bug Fix
Doc Text:
Previously, the os-brick code for scanning FibreChannel HBA hosts returned an invalid value under certain circumstances. Services such as cinder and nova failed due to the invalid value. Now, the FibreChannel HBA scan code returns a valid value. Cinder and nova no longer crash when scanning FibreChannel HBA hosts.
Clone Of: 1634164
Environment:
Last Closed: 2019-05-01 20:40:47 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1774283 0 None None None 2018-09-28 20:36:33 UTC
OpenStack gerrit 589539 0 None MERGED FC fix for scanning only connected HBA's 2020-12-18 05:32:34 UTC

Description Alan Bishop 2018-09-28 20:36:33 UTC 
+++ This bug was initially created as a clone of Bug #1634164 +++

+++ This bug was initially created as a clone of Bug #1634163 +++

+++ This bug was initially created as a clone of Bug #1631024 +++

+++ This bug was initially created as a clone of Bug #1630842 +++

Description of problem:

Volumes fail to attach

Version-Release number of selected component (if applicable):
RHOSP12

Trace:
    2018-09-17 10:11:47.950 95318 INFO oslo.privsep.daemon [req-0954fa2d-af1c-4709-8efc-2ee0d814f9e7 8a3500afa82344b29a738db086b4ade7 819e088460f84ab8be9a8cab9b82b735 - default default] privsep process running with uid/gid: 0/0
    2018-09-17 10:11:47.952 95318 INFO oslo.privsep.daemon [req-0954fa2d-af1c-4709-8efc-2ee0d814f9e7 8a3500afa82344b29a738db086b4ade7 819e088460f84ab8be9a8cab9b82b735 - default default] privsep process running with capabilities (eff/prm/inh): CAP_SYS_ADMIN/CAP_SYS_ADMIN/none
    2018-09-17 10:11:47.952 95318 INFO oslo.privsep.daemon [req-0954fa2d-af1c-4709-8efc-2ee0d814f9e7 8a3500afa82344b29a738db086b4ade7 819e088460f84ab8be9a8cab9b82b735 - default default] privsep daemon running as pid 95318
    2018-09-17 10:11:49.291 1 WARNING nova.volume.cinder [req-0954fa2d-af1c-4709-8efc-2ee0d814f9e7 8a3500afa82344b29a738db086b4ade7 819e088460f84ab8be9a8cab9b82b735 - default default] The support for the Cinder API v2 is deprecated, please upgrade to Cinder API v3.
    2018-09-17 10:11:52.495 1 INFO os_brick.initiator.connectors.fibre_channel [req-0954fa2d-af1c-4709-8efc-2ee0d814f9e7 8a3500afa82344b29a738db086b4ade7 819e088460f84ab8be9a8cab9b82b735 - default default] Fibre Channel volume device not yet found. Will rescan & retry.  Try number: 0.
    2018-09-17 10:11:52.882 1 ERROR oslo.service.loopingcall [req-0954fa2d-af1c-4709-8efc-2ee0d814f9e7 8a3500afa82344b29a738db086b4ade7 819e088460f84ab8be9a8cab9b82b735 - default default] Fixed interval looping call 'os_brick.initiator.connectors.fibre_channel._wait_for_device_discovery' failed: TypeError: 'NoneType' object is not iterable
    2018-09-17 10:11:52.882 1 ERROR oslo.service.loopingcall Traceback (most recent call last):
    2018-09-17 10:11:52.882 1 ERROR oslo.service.loopingcall   File "/usr/lib/python2.7/site-packages/oslo_service/loopingcall.py", line 137, in _run_loop
    2018-09-17 10:11:52.882 1 ERROR oslo.service.loopingcall     result = func(*self.args, **self.kw)
    2018-09-17 10:11:52.882 1 ERROR oslo.service.loopingcall   File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/fibre_channel.py", line 161, in _wait_for_device_discovery
    2018-09-17 10:11:52.882 1 ERROR oslo.service.loopingcall     self._linuxfc.rescan_hosts(hbas, connection_properties)
    2018-09-17 10:11:52.882 1 ERROR oslo.service.loopingcall   File "/usr/lib/python2.7/site-packages/os_brick/initiator/linuxfc.py", line 112, in rescan_hosts
    2018-09-17 10:11:52.882 1 ERROR oslo.service.loopingcall     for hba_channel, target_id in cts:
    2018-09-17 10:11:52.882 1 ERROR oslo.service.loopingcall TypeError: 'NoneType' object is not iterable
    2018-09-17 10:11:52.882 1 ERROR oslo.service.loopingcall
    2018-09-17 10:11:52.884 1 ERROR nova.virt.block_device [req-0954fa2d-af1c-4709-8efc-2ee0d814f9e7 8a3500afa82344b29a738db086b4ade7 819e088460f84ab8be9a8cab9b82b735 - default default] [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b] Driver failed to attach volume 46ee1840-3cc1-4d12-a827-fbcfea4a824e at /dev/vdb: TypeError: 'NoneType' object is not iterable
    2018-09-17 10:11:52.884 1 ERROR nova.virt.block_device [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b] Traceback (most recent call last):
    2018-09-17 10:11:52.884 1 ERROR nova.virt.block_device [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 393, in attach
    2018-09-17 10:11:52.884 1 ERROR nova.virt.block_device [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]     device_type=self['device_type'], encryption=encryption)
    2018-09-17 10:11:52.884 1 ERROR nova.virt.block_device [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1267, in attach_volume
    2018-09-17 10:11:52.884 1 ERROR nova.virt.block_device [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]     self._connect_volume(connection_info, disk_info, instance)
    2018-09-17 10:11:52.884 1 ERROR nova.virt.block_device [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1189, in _connect_volume
    2018-09-17 10:11:52.884 1 ERROR nova.virt.block_device [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]     vol_driver.connect_volume(connection_info, disk_info, instance)
    2018-09-17 10:11:52.884 1 ERROR nova.virt.block_device [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume/fibrechannel.py", line 53, in connect_volume
    2018-09-17 10:11:52.884 1 ERROR nova.virt.block_device [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]     device_info = self.connector.connect_volume(connection_info['data'])
    2018-09-17 10:11:52.884 1 ERROR nova.virt.block_device [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]   File "/usr/lib/python2.7/site-packages/os_brick/utils.py", line 137, in trace_logging_wrapper
    2018-09-17 10:11:52.884 1 ERROR nova.virt.block_device [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]     return f(*args, **kwargs)
    2018-09-17 10:11:52.884 1 ERROR nova.virt.block_device [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
    2018-09-17 10:11:52.884 1 ERROR nova.virt.block_device [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]     return f(*args, **kwargs)
    2018-09-17 10:11:52.884 1 ERROR nova.virt.block_device [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]   File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/fibre_channel.py", line 169, in connect_volume
    2018-09-17 10:11:52.884 1 ERROR nova.virt.block_device [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]     timer.start(interval=2).wait()
    2018-09-17 10:11:52.884 1 ERROR nova.virt.block_device [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]   File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 121, in wait
    2018-09-17 10:11:52.884 1 ERROR nova.virt.block_device [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]     return hubs.get_hub().switch()
    2018-09-17 10:11:52.884 1 ERROR nova.virt.block_device [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]   File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
    2018-09-17 10:11:52.884 1 ERROR nova.virt.block_device [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]     return self.greenlet.switch()
    2018-09-17 10:11:52.884 1 ERROR nova.virt.block_device [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]   File "/usr/lib/python2.7/site-packages/oslo_service/loopingcall.py", line 137, in _run_loop
    2018-09-17 10:11:52.884 1 ERROR nova.virt.block_device [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]     result = func(*self.args, **self.kw)
    2018-09-17 10:11:52.884 1 ERROR nova.virt.block_device [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]   File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/fibre_channel.py", line 161, in _wait_for_device_discovery
    2018-09-17 10:11:52.884 1 ERROR nova.virt.block_device [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]     self._linuxfc.rescan_hosts(hbas, connection_properties)
    2018-09-17 10:11:52.884 1 ERROR nova.virt.block_device [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]   File "/usr/lib/python2.7/site-packages/os_brick/initiator/linuxfc.py", line 112, in rescan_hosts
    2018-09-17 10:11:52.884 1 ERROR nova.virt.block_device [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]     for hba_channel, target_id in cts:
    2018-09-17 10:11:52.884 1 ERROR nova.virt.block_device [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b] TypeError: 'NoneType' object is not iterable
    2018-09-17 10:11:52.884 1 ERROR nova.virt.block_device [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]
    2018-09-17 10:11:52.887 1 WARNING nova.volume.cinder [req-0954fa2d-af1c-4709-8efc-2ee0d814f9e7 8a3500afa82344b29a738db086b4ade7 819e088460f84ab8be9a8cab9b82b735 - default default] The support for the Cinder API v2 is deprecated, please upgrade to Cinder API v3.
    2018-09-17 10:11:55.823 1 ERROR nova.compute.manager [req-0954fa2d-af1c-4709-8efc-2ee0d814f9e7 8a3500afa82344b29a738db086b4ade7 819e088460f84ab8be9a8cab9b82b735 - default default] [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b] Failed to attach 46ee1840-3cc1-4d12-a827-fbcfea4a824e at /dev/vdb: TypeError: 'NoneType' object is not iterable
    2018-09-17 10:11:55.823 1 ERROR nova.compute.manager [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b] Traceback (most recent call last):
    2018-09-17 10:11:55.823 1 ERROR nova.compute.manager [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4939, in _attach_volume
    2018-09-17 10:11:55.823 1 ERROR nova.compute.manager [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]     do_driver_attach=True)
    2018-09-17 10:11:55.823 1 ERROR nova.compute.manager [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 45, in wrapped
    2018-09-17 10:11:55.823 1 ERROR nova.compute.manager [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]     ret_val = method(obj, context, *args, **kwargs)
    2018-09-17 10:11:55.823 1 ERROR nova.compute.manager [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 402, in attach
    2018-09-17 10:11:55.823 1 ERROR nova.compute.manager [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]     connector)
    2018-09-17 10:11:55.823 1 ERROR nova.compute.manager [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
    2018-09-17 10:11:55.823 1 ERROR nova.compute.manager [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]     self.force_reraise()
    2018-09-17 10:11:55.823 1 ERROR nova.compute.manager [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
    2018-09-17 10:11:55.823 1 ERROR nova.compute.manager [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]     six.reraise(self.type_, self.value, self.tb)
    2018-09-17 10:11:55.823 1 ERROR nova.compute.manager [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 393, in attach
    2018-09-17 10:11:55.823 1 ERROR nova.compute.manager [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]     device_type=self['device_type'], encryption=encryption)
    2018-09-17 10:11:55.823 1 ERROR nova.compute.manager [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1267, in attach_volume
    2018-09-17 10:11:55.823 1 ERROR nova.compute.manager [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]     self._connect_volume(connection_info, disk_info, instance)
    2018-09-17 10:11:55.823 1 ERROR nova.compute.manager [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1189, in _connect_volume
    2018-09-17 10:11:55.823 1 ERROR nova.compute.manager [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]     vol_driver.connect_volume(connection_info, disk_info, instance)
    2018-09-17 10:11:55.823 1 ERROR nova.compute.manager [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume/fibrechannel.py", line 53, in connect_volume
    2018-09-17 10:11:55.823 1 ERROR nova.compute.manager [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]     device_info = self.connector.connect_volume(connection_info['data'])
    2018-09-17 10:11:55.823 1 ERROR nova.compute.manager [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]   File "/usr/lib/python2.7/site-packages/os_brick/utils.py", line 137, in trace_logging_wrapper
    2018-09-17 10:11:55.823 1 ERROR nova.compute.manager [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]     return f(*args, **kwargs)
    2018-09-17 10:11:55.823 1 ERROR nova.compute.manager [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
    2018-09-17 10:11:55.823 1 ERROR nova.compute.manager [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]     return f(*args, **kwargs)
    2018-09-17 10:11:55.823 1 ERROR nova.compute.manager [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]   File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/fibre_channel.py", line 169, in connect_volume
    2018-09-17 10:11:55.823 1 ERROR nova.compute.manager [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]     timer.start(interval=2).wait()
    2018-09-17 10:11:55.823 1 ERROR nova.compute.manager [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]   File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 121, in wait
    2018-09-17 10:11:55.823 1 ERROR nova.compute.manager [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]     return hubs.get_hub().switch()
    2018-09-17 10:11:55.823 1 ERROR nova.compute.manager [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]   File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
    2018-09-17 10:11:55.823 1 ERROR nova.compute.manager [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]     return self.greenlet.switch()
    2018-09-17 10:11:55.823 1 ERROR nova.compute.manager [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]   File "/usr/lib/python2.7/site-packages/oslo_service/loopingcall.py", line 137, in _run_loop
    2018-09-17 10:11:55.823 1 ERROR nova.compute.manager [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]     result = func(*self.args, **self.kw)
    2018-09-17 10:11:55.823 1 ERROR nova.compute.manager [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]   File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/fibre_channel.py", line 161, in _wait_for_device_discovery
    2018-09-17 10:11:55.823 1 ERROR nova.compute.manager [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]     self._linuxfc.rescan_hosts(hbas, connection_properties)
    2018-09-17 10:11:55.823 1 ERROR nova.compute.manager [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]   File "/usr/lib/python2.7/site-packages/os_brick/initiator/linuxfc.py", line 112, in rescan_hosts
    2018-09-17 10:11:55.823 1 ERROR nova.compute.manager [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]     for hba_channel, target_id in cts:
    2018-09-17 10:11:55.823 1 ERROR nova.compute.manager [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b] TypeError: 'NoneType' object is not iterable
    2018-09-17 10:11:55.823 1 ERROR nova.compute.manager [instance: 87eab092-a72e-4892-a3f3-d6443aaeb87b]
    2018-09-17 10:11:55.825 1 WARNING nova.volume.cinder [req-0954fa2d-af1c-4709-8efc-2ee0d814f9e7 8a3500afa82344b29a738db086b4ade7 819e088460f84ab8be9a8cab9b82b735 - default default] The support for the Cinder API v2 is deprecated, please upgrade to Cinder API v3.
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server [req-0954fa2d-af1c-4709-8efc-2ee0d814f9e7 8a3500afa82344b29a738db086b4ade7 819e088460f84ab8be9a8cab9b82b735 - default default] Exception during message handling: TypeError: 'NoneType' object is not iterable
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in wrapped
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server     self.force_reraise()
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in wrapped
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 218, in decorated_function
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server     self.force_reraise()
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 206, in decorated_function
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4924, in attach_volume
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server     do_attach_volume(context, instance, driver_bdm)
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4922, in do_attach_volume
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server     bdm.destroy()
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server     self.force_reraise()
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4919, in do_attach_volume
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server     return self._attach_volume(context, instance, driver_bdm)
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4953, in _attach_volume
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server     volume_id=bdm.volume_id)
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server     self.force_reraise()
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4939, in _attach_volume
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server     do_driver_attach=True)
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 45, in wrapped
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server     ret_val = method(obj, context, *args, **kwargs)
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 402, in attach
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server     connector)
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server     self.force_reraise()
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 393, in attach
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server     device_type=self['device_type'], encryption=encryption)
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1267, in attach_volume
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server     self._connect_volume(connection_info, disk_info, instance)
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1189, in _connect_volume
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server     vol_driver.connect_volume(connection_info, disk_info, instance)
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume/fibrechannel.py", line 53, in connect_volume
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server     device_info = self.connector.connect_volume(connection_info['data'])
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/os_brick/utils.py", line 137, in trace_logging_wrapper
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/fibre_channel.py", line 169, in connect_volume
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server     timer.start(interval=2).wait()
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 121, in wait
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server     return hubs.get_hub().switch()
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server     return self.greenlet.switch()
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_service/loopingcall.py", line 137, in _run_loop
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server     result = func(*self.args, **self.kw)
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/fibre_channel.py", line 161, in _wait_for_device_discovery
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server     self._linuxfc.rescan_hosts(hbas, connection_properties)
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/os_brick/initiator/linuxfc.py", line 112, in rescan_hosts
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server     for hba_channel, target_id in cts:
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not iterable
    2018-09-17 10:11:56.849 1 ERROR oslo_messaging.rpc.server


Seems fixed upstream:
https://review.openstack.org/#/c/589539/

We need to backport it.

Regards,
Pierre-André
Comment 8 Tzach Shefi 2019-02-03 15:25:26 UTC 
Hi Alan, 

Need some clarification on bz verification steps

Does it matter which FC back end/driver I use to test this? 
"specific HBA" would this only hit some HBA's, if so which mfg/model did we hit this on? 

How do I trigger the issue which was fixed ? 
Attach FC vol to instance?

Noticed LP mentions "only connected HBA's", is this relevant for verification? 
My BM nodes all have a single (dual port) HBA, 
I can disconnect one of the dual ports if it helps.
Comment 9 Alan Bishop 2019-02-04 18:25:13 UTC 
Tzach,

I'm afraid I don't have any useful info. This BZ is basically a backport of an upstream fix for a regression introduced by a prior patch related to FC HBA scanning. You might find some clues in the commit message (see https://review.openstack.org/571910).

Gorka, do you have any thoughts on this?
Note You need to log in before you can comment on or make changes to this bug.