Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2299636

Summary: Some multipath device paths are missing after instance live migration with FC cinder backend
Product: Red Hat OpenStack Reporter: yatanaka
Component: python-os-brickAssignee: Rajat Dhasmana <rdhasman>
Status: CLOSED MIGRATED QA Contact: Yosi Ben Shimon <ybenshim>
Severity: medium Docs Contact:
Priority: high    
Version: 16.2 (Train)CC: apevec, eharney, jschluet, lhh, udesale
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2025-01-14 17:46:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description yatanaka 2024-07-24 06:06:35 UTC 
Description of problem:

After instance live-migration, cinder volume device attachments are also migrated to the destination compute node.
However, sometimes a multipath device has fewer paths than expected. 

~~~
36xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx dm-x XXXX,XXXXXXX
size=100G features='0' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
 |- 1:0:3:1 sde 8:64   active ready running
 `- 2:0:3:1 sdg 8:96   active ready running

  ==> This multipath device is corresponding to a cinder volume attached to the migrated instance
  ==> This multipath device should have 4 paths, but actually it only has 2 paths
~~~

After checking logs, I noticed that some of paths were detected automatically, but other paths were not detected automatically.
When there is no paths in a multipath device, os-brick scans all paths[1] and all paths are detected. In this case, the multipath device has all paths properly.
However, when only some paths of a multipath device are automatically detected and other paths are not, os-brick doesn't scan other paths, because os-brick doesn't rescan when there are at least one detected path[2] in a multipath device. In this case, some multipath device paths are missing.
When we manually scan all paths using `echo "- - -" >  /sys/class/scsi_host/hostX/scan` command, all paths are detected. So, there are not connectivity issues between the compute node and the backend storage.

I'm not sure why not all paths were not automatically detected, but I'm wondering if os-brick should scan all paths always even if some paths are already detected.


Version-Release number of selected component (if applicable):
The customer reported this issue on RHOSP 16.2.0 + Pure Storage FC backend 

How reproducible:
Steps to Reproduce:
1. Create an instance and attach multiple cinder volumes
2. live-migrate the instance 14 times.
3. The issue occurred 4 times out of 14 live-migrations

Actual results:
Some of multipath device paths are missing sometimes after live-migration

Expected results:
All paths of multipath devices are recognized always after live-migration

Additional info:
[1] https://github.com/openstack/os-brick/blob/train-eol/os_brick/initiator/connectors/fibre_channel.py#L236
[2] https://github.com/openstack/os-brick/blob/train-eol/os_brick/initiator/connectors/fibre_channel.py#L218-L226