1224233 – When a FC storage attached to a scsi (HBA or vHBA) with target_number!=0, the FC storage pool cannot display it with "virsh vol-list <poolName>"

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1224233 - When a FC storage attached to a scsi (HBA or vHBA) with target_number!=0, the FC storage pool cannot display it with "virsh vol-list <poolName>"

Summary: When a FC storage attached to a scsi (HBA or vHBA) with target_number!=0, the...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	libvirt
Sub Component:
Version:	7.2
Hardware:	x86_64
OS:	Linux
Priority:	high
Severity:	high
Target Milestone:	rc
Target Release:	---
Assignee:	John Ferlan
QA Contact:	Virtualization Bugs
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2015-05-22 11:13 UTC by yisun
Modified:	2015-11-19 06:36 UTC (History)
CC List:	8 users (show)
Fixed In Version:	libvirt-1.2.17-1.el7
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2015-11-19 06:36:29 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2015:2202	0	normal	SHIPPED_LIVE	libvirt bug fix and enhancement update	2015-11-19 08:17:58 UTC

Description yisun 2015-05-22 11:13:02 UTC

description:
When a FC storage attached to a scsi (HBA or vHBA) with target_number!=0, the FC storage pool cannot display it with "virsh vol-list <poolName>"

versions:
kernel-3.10.0-253.el7.x86_64
libvirt-1.2.15-2.el7.x86_64
qemu-kvm-rhev-2.3.0-1.el7.x86_64

How reproducible:
50% (when creating a npiv pool, sometimes the fc storage will always be attached to scsi target_number=0 and it's cannot be reproduced in this situation.)


repro steps:
1. create a fc pool with valid fc_host&wwpn&wwnn
1.1 find an available HBA
# ll /sys/class/fc_host/
total 0
lrwxrwxrwx. 1 root root 0 May 18 03:18 host4 -> ../../devices/pci0000:00/0000:00:0d.0/0000:04:00.0/host4/fc_host/host4
lrwxrwxrwx. 1 root root 0 May 18 03:18 host5 -> ../../devices/pci0000:00/0000:00:0d.0/0000:04:00.1/host5/fc_host/host5  <==== use this as parent HBA

1.2 create a pool xml
#cat pool.xml
<pool type='scsi'>
    <name>via_wwn</name>
    <source>
        <adapter type='fc_host' parent='scsi_host5' wwnn='2101001b32a90002' wwpn='2101001b32a90003'/>
    </source>
    <target>
        <path>/dev/disk/by-path/</path>
        <permissions>
          <mode>0755</mode>
          <owner>-1</owner>
          <group>-1</group>
        </permissions>
    </target>
</pool>

1.3 create pool
# virsh pool-create pool.xm
Pool via_wwn created from pool.xm

2. Check the newly created fc pool
2.1 use lsscsi to check the new fc sotrage
( in my test, vhba scsi_host17 created)
# lsscsi
[0:0:0:0]    disk    ATA      WDC WD2502ABYS-1 02.0  /dev/sda
[2:0:0:0]    cd/dvd  PLDS     DVD-ROM DH-16D3S SD11  /dev/sr0
[4:0:0:0]    disk    IBM      1726-4xx  FAStT  0617  -        
[4:0:1:0]    disk    IBM      1726-4xx  FAStT  0617  -        
[4:0:2:0]    disk    IBM      1726-4xx  FAStT  0617  /dev/sdb
[4:0:3:0]    disk    IBM      1726-4xx  FAStT  0617  /dev/sdc
[5:0:0:0]    disk    IBM      1726-4xx  FAStT  0617  -        
[5:0:1:0]    disk    IBM      1726-4xx  FAStT  0617  -        
[5:0:2:0]    disk    IBM      1726-4xx  FAStT  0617  /dev/sdd
[5:0:3:0]    disk    IBM      1726-4xx  FAStT  0617  /dev/sde
[17:0:0:0]   disk    IBM      1726-4xx  FAStT  0617  /dev/sdf    <========== fc disk attached to scsi x.x.0.0
[17:0:1:0]   disk    IBM      1726-4xx  FAStT  0617  /dev/sdg
[17:0:2:0]   disk    IBM      1726-4xx  FAStT  0617  -        
[17:0:3:0]   disk    IBM      1726-4xx  FAStT  0617  -  
    
2.2 check the volume in this pool
# virsh vol-list via_wwn
 Name                 Path                                    
------------------------------------------------------------------------------
 unit:0:0:0           /dev/disk/by-path/pci-0000:04:00.1-fc-0x203600a0b85b5dd4-lun-0


3.destroy and create the pool again ( again and again until the fc storage attached to a scsi host with target_number!=0)
3.1 destroy the pool
# virsh pool-destroy via_wwn
Pool via_wwn destroyed

3.2 start/create the pool again
# virsh pool-start via_wwn
Pool via_wwn started

3.3 check the fc storage
# lsscsi      <=== now vhba on scsi_host 19 created
[0:0:0:0]    disk    ATA      WDC WD2502ABYS-1 02.0  /dev/sda
[2:0:0:0]    cd/dvd  PLDS     DVD-ROM DH-16D3S SD11  /dev/sr0
[4:0:0:0]    disk    IBM      1726-4xx  FAStT  0617  -        
[4:0:1:0]    disk    IBM      1726-4xx  FAStT  0617  -        
[4:0:2:0]    disk    IBM      1726-4xx  FAStT  0617  /dev/sdb
[4:0:3:0]    disk    IBM      1726-4xx  FAStT  0617  /dev/sdc
[5:0:0:0]    disk    IBM      1726-4xx  FAStT  0617  -        
[5:0:1:0]    disk    IBM      1726-4xx  FAStT  0617  -        
[5:0:2:0]    disk    IBM      1726-4xx  FAStT  0617  /dev/sdd
[5:0:3:0]    disk    IBM      1726-4xx  FAStT  0617  /dev/sde
[19:0:0:0]   disk    IBM      1726-4xx  FAStT  0617  -        
[19:0:1:0]   disk    IBM      1726-4xx  FAStT  0617  -        
[19:0:2:0]   disk    IBM      1726-4xx  FAStT  0617  /dev/sdf  <========== fc storage attached to scsi x.x.2.0
[19:0:3:0]   disk    IBM      1726-4xx  FAStT  0617  /dev/sdg

3.4 refresh the pool
# virsh pool-refresh via_wwn
Pool via_wwn refreshed

3.5 list the vol in the pool , nothing displayed.
# virsh vol-list via_wwn
 Name                 Path                                    
------------------------------------------------------------------------------
<====================== no volume shows up here.

Actual result:
When FC storage attached on scsi (HBA/vHBA) doesn't start from target=0, it disappears in pool's volume list.

Expected result:
The volumes in the pool should always be listed no matter if it starts from target=0.

Additional info:
====libvirt debug info=======
2015-05-22 09:23:12.245+0000: 6371: debug : virStorageBackendSCSIFindLUs:476 : Found possible LU '25:0:0:0'
2015-05-22 09:23:12.245+0000: 6371: debug : processLU:401 : Processing LU 25:0:0:0
2015-05-22 09:23:12.245+0000: 6371: debug : getDeviceType:105 : Device type is 0
2015-05-22 09:23:12.245+0000: 6371: debug : processLU:418 : 25:0:0:0 is a Direct-Access LUN
2015-05-22 09:23:12.245+0000: 6371: debug : processLU:421 : Failed to find block device for this LUN
==============================

Comment 1 John Ferlan 2015-06-10 13:57:00 UTC

Any chance you can get a system into this state and then contact me so I can debug on a live system?

The debug you show is for "25:*" while the lsscsi output shows "19:*", while that's perhaps just cut-n-paste related - it is "confusing".

I'm hoping it's not timing related, but could well be a timing related problem. Since libvirt doesn't "control" when the devices appear and are "linked" to the /dev/sd* block device, it may be that 19:0:2:0 didn't exist when libvirt looked. Does refreshing the pool when this happens cause the volumes to appear? If so, then this is just timing related. If not, then digging deeper via a debug libvirtd will probably be necessary.

FWIW: On a host I have access to, I see:

# lsscsi
...
[10:0:0:0] enclosu HP MSA2012fc J200 -
[10:0:1:0] enclosu HP MSA2012fc J200 -
[10:0:2:0] enclosu HP MSA2012fc J200 -
[10:0:3:0] enclosu HP MSA2012fc J200 -
[10:0:4:0] disk DGC LUNZ 0429 /dev/sdd
[10:0:5:0] disk DGC LUNZ 0429 /dev/sde
# virsh vol-list vhbapool_host3
Name Path
------------------------------------------------------------------------------
unit:0:4:0 /dev/disk/by-path/pci-0000:10:00.0-fc-0x5006016844602198-lun-0
unit:0:5:0 /dev/disk/by-path/pci-0000:10:00.0-fc-0x5006016044602198-lun-0

So just from a quick look, I'm not completely convinced the problem statement is correct.

Comment 7 John Ferlan 2015-06-12 11:23:16 UTC

A patch has been posted upstream to resolve this issue:

http://www.redhat.com/archives/libvir-list/2015-June/msg00567.html

Comment 8 Michal Privoznik 2015-06-18 15:21:10 UTC

I've just pushed the patch upstream:

commit 5c9fc1c11fb54236576c2bff40e6e3df1e710983
Author:     John Ferlan <jferlan>
AuthorDate: Fri Jun 12 07:22:34 2015 -0400
Commit:     Michal Privoznik <mprivozn>
CommitDate: Thu Jun 18 17:14:00 2015 +0200

    scsi: Adjust return status from getBlockDevice
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1224233
    
    Currently it's not possible to determine the difference between a
    fatal memory allocation or failure to open/read the directory error
    with a perhaps less fatal, I didn't find the "block" device in the
    directory (which may be a disk entry without a block device).
    
    In the case of the latter, we shouldn't cause failure to continue
    searching in the caller (virStorageBackendSCSIFindLUs), rather we
    should allow trying reading the next directory entry.
    
    Signed-off-by: John Ferlan <jferlan>
    Signed-off-by: Michal Privoznik <mprivozn>

v1.2.16-235-g5c9fc1c

Comment 10 yisun 2015-07-16 08:59:27 UTC

verified and passed on:
libvirt-1.2.17-2.el7.x86_64
qemu-kvm-rhev-2.3.0-9.el7.x86_64
kernel-3.10.0-263.el7.x86_64

(I didn't use the latest kernel because kernel has problem to create vHBA as bug 1240912 mentioned, which will block the test. I'll keep an eye on that issue)




Steps:
1. choose a HBA as parent to create vHBA
# ll /sys/class/fc_host/
total 0
lrwxrwxrwx. 1 root root 0 Jul 15 20:58 host4 -> ../../devices/pci0000:00/0000:00:0d.0/0000:04:00.0/host4/fc_host/host4
lrwxrwxrwx. 1 root root 0 Jul 15 20:58 host5 -> ../../devices/pci0000:00/0000:00:0d.0/0000:04:00.1/host5/fc_host/host5   <==== we'll use scsi_host5 as parent HBA. 

2. create a fc pool 
# cat fc-pool.xml 
<pool type='scsi'>
<name>fc-pool</name>
<source>
<adapter  type='fc_host' parent='scsi_host5' wwnn='2101001b32a90002' wwpn='2101001b32a90002'/>
</source>
<target>
<path>/dev/disk/by-path</path>
<permissions>
<mode>0700</mode>
<owner>0</owner>
<group>0</group>
</permissions>
</target>
</pool>

# virsh pool-create fc-pool.xml 
Pool fc-pool created from fc-pool.xml

3. check the newly create vHBA connected to san storage
# lsscsi 
[0:0:0:0]    disk    ATA      WDC WD2502ABYS-1 3B04  /dev/sda 
[2:0:0:0]    cd/dvd  PLDS     DVD-ROM DH-16D3S SD11  /dev/sr0 
[4:0:0:0]    disk    IBM      1726-4xx  FAStT  0617  -        
[4:0:1:0]    disk    IBM      1726-4xx  FAStT  0617  -        
[4:0:2:0]    disk    IBM      1726-4xx  FAStT  0617  /dev/sdb 
[4:0:3:0]    disk    IBM      1726-4xx  FAStT  0617  /dev/sdc 
[5:0:0:0]    disk    IBM      1726-4xx  FAStT  0617  -        
[5:0:1:0]    disk    IBM      1726-4xx  FAStT  0617  -        
[5:0:2:0]    disk    IBM      1726-4xx  FAStT  0617  /dev/sdd 
[5:0:3:0]    disk    IBM      1726-4xx  FAStT  0617  /dev/sde 
[13:0:0:0]   disk    IBM      1726-4xx  FAStT  0617  -        
[13:0:1:0]   disk    IBM      1726-4xx  FAStT  0617  -        
[13:0:2:0]   disk    IBM      1726-4xx  FAStT  0617  /dev/sdf  <=== new and not attached to scsi lun 13:0:0:0
[13:0:3:0]   disk    IBM      1726-4xx  FAStT  0617  /dev/sdg 

4. check the vol-list can display the newly connected storage
# virsh vol-list fc-pool
 Name                 Path                                    
------------------------------------------------------------------------------
 unit:0:2:0           /dev/disk/by-path/pci-0000:04:00.1-fc-0x203600a0b85b5dd4-lun-0

#ll /dev/disk/by-path/pci-0000:04:00.1-fc-0x203600a0b85b5dd4-lun-0
lrwxrwxrwx. 1 root root 9 Jul 16 04:51 /dev/disk/by-path/pci-0000:04:00.1-fc-0x203600a0b85b5dd4-lun-0 -> ../../sdf

Comment 12 errata-xmlrpc 2015-11-19 06:36:29 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2202.html

Note You need to log in before you can comment on or make changes to this bug.