Bug 963817
Summary: | Stable SCSI host addressing | ||
---|---|---|---|
Product: | Red Hat Enterprise Linux 7 | Reporter: | Osier Yang <jyang> |
Component: | libvirt | Assignee: | John Ferlan <jferlan> |
Status: | CLOSED ERRATA | QA Contact: | Virtualization Bugs <virt-bugs> |
Severity: | unspecified | Docs Contact: | |
Priority: | unspecified | ||
Version: | 7.0 | CC: | ajia, cwei, dyuan, jferlan, lsu, mzhan, rbalakri, xuzhang, yanyang |
Target Milestone: | rc | ||
Target Release: | --- | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | libvirt-1.2.7-1.el7 | Doc Type: | Bug Fix |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2015-03-05 07:20:27 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Osier Yang
2013-05-16 15:30:10 UTC
A new series has been posted upstream: http://www.redhat.com/archives/libvir-list/2014-June/msg00448.html to replace the former series that had languished and been abandoned: http://www.redhat.com/archives/libvir-list/2013-June/msg00396.html The new code will modify the initial design a bit parsing/using XML such as: <adapter type='scsi_host'> <parentaddr unique_id='3'> <address domain='0x0000' bus='0x00' slot='0x1f' func='0x2'/> </parentaddr> </adapter> The new code will peruse the /sys/class/scsi_host directory looking for 'host#' entries with matching PCI Address and then look at the unique_id file for any matching PCI address to find a matched unique_id. Series was reviewed and code now pushed. $ git show commit ea37fb34a95df2295538b51122dd1316c765c41b Author: John Ferlan <jferlan> Date: Mon Jun 9 12:19:16 2014 -0400 getAdapterName: Lookup stable scsi_host If a parentaddr was provided in the XML, have getAdapterName lookup the stable address. This allows virStorageBackendSCSICheckPool() and virStorageBackendSCSIRefreshPool() to automagically find the scsi_host by its PCI address and unique_id $ git describe: v1.2.6-214-gea37fb3 Put my conversation with John about this bug here to reduce confusion On 10/16/2014 09:51 AM, Yang Yang wrote: > Hi John, > > Regarding bug 963817, I am not clear why the adapter identified > by parentaddr and unique_id is more stable. > > After host rebooting, the SCSI host number used for SCSI > storage pool may change while the unique_id may also change. So > maybe unique_id is unstable too, right? > > bug: https://bugzilla.redhat.com/show_bug.cgi?id=963817 > > > I inherited the code from Osier Yang who left Red Hat earlier this year so my explanation comes from my recollections of conversations with Osier about 8 or so months ago. Essentially, the goal of using the parentaddr (eg, the PCI address) is "more stable" between host reboots than the "scsi_host#" string. Someone could have added a new adapter or a kernel rebuild could have caused the tree to change. Thus as the system boots, the 'scsi_host#' string can change. So Osier investigated and determined that by providing the PCI address it was possible to have a more stable address in that regard. As for the unique_id - everything I've read indicates that value doesn't change between reboots. It's described in lkml.org: https://lkml.org/lkml/2011/8/17/274 +What: /sys/bus/scsi/devices/host*/scsi_host/host*/unique_id +Date: February, 2003 +KernelVersion: Unknown +Contact: James Bottomley <James.Bottomley> +Description: + Read only unsigned integer uniquely identifying the SCSI host + within the system. This value is assigned by the low level + driver. + Do you have an example of the same PCI Address SCSI_HOST having it's unique_id change? The "key" with using parentaddr (eg, the PCI address) is libvirt will walk the scsi_host "database" (files) starting at "/sys/class/scsi_host" looking the softlink for the PCI address of each "host#" entry. It then defines that as the scsi_host. Conversely, if you define things using scsi_host4 - that code would use the "/sys/class/host4" and whatever it points at - which can change between reboots. I'm at KVM Forum right now... Travelling for the next day... So if you have further questions I'll answer them when I'm back in the office. John My conversation with John following #8 John Ferlan wrote: > From: "Yang Yang" <yanyang> > To: "John Ferlan" <jferlan> >> From: "John Ferlan" <jferlan> >> To: "Yang Yang" <yanyang> >> Do you have an example of the same PCI Address SCSI_HOST having it's >> unique_id change? >> > > Thanks a million for your reply. > > Here are the example and steps of the same PCI Address SCSI_HOST having it's > unique_id change. > Steps: > 1. discover the HBA > # virsh nodedev-list --cap vports > scsi_host4 > scsi_host5 > > 2. create persistent vHBA based on scsi_host5 > # cat fc-pool.xml > <pool type='scsi'> > <name>fc-pool</name> > <source> > <adapter type='fc_host' parent='scsi_host5' wwnn='2101001b32a90002' wwpn='2101001b32a90003'/> > </source> > <target> > <path>/dev/disk/by-path</path> > <permissions> > <mode>0700</mode> > <owner>0</owner> > <group>0</group> > </permissions> > </target> > </pool> > > # virsh pool-define fc-pool.xml > Pool fc-pool defined from fc-pool.xml > # virsh pool-start fc-pool > Pool fc-pool started > # virsh pool-refresh fc-pool > Pool fc-pool refreshed > > # virsh nodedev-list scsi_host > scsi_host0 > scsi_host1 > scsi_host2 > scsi_host3 > scsi_host4 > scsi_host5 > scsi_host7 -------------> vHBA > > 3. define/start a scsi pool with parentaddr > > # ll /sys/class/scsi_host/ > lrwxrwxrwx. 1 root root 0 Oct 16 04:26 host5 -> ../../devices/pci0000:00/0000:00:0d.0/0000:04:00.1/host5/scsi_host/host5 > lrwxrwxrwx. 1 root root 0 Oct 16 20:02 host7 -> ../../devices/pci0000:00/0000:00:0d.0/0000:04:00.1/host5/vport-5:0-1/host7/scsi_host/host7 > > # cat /sys/class/scsi_host/host7/unique_id > 7 > > # cat scsi-pool-1.xml > <pool type='scsi'> > <name>scsi-pool-1</name> > <source> > <adapter type='scsi_host'> > <parentaddr unique_id='7'> > <address domain='0x0' bus='0x04' slot='0x0' function='0x1'/> -----> 0000:04:00.1 > </parentaddr> > </adapter> > </source> > <target> > <path>/dev/disk/by-path</path> > <permissions> > <mode>0700</mode> > <owner>0</owner> > <group>0</group> > </permissions> > </target> > </pool> > > # virsh pool-define scsi-pool-1.xml > Pool scsi-pool-1 defined from scsi-pool-1.xml > # virsh pool-start scsi-pool-1 > Pool scsi-pool-1 started > > # virsh pool-list --all > Name State Autostart > ------------------------------------------- > default active no > fc-pool active no > scsi-pool-1 active no > > 4. Reboot host > > 5. check scsi host > # virsh nodedev-list scsi_host > scsi_host0 > scsi_host1 > scsi_host2 > scsi_host3 > scsi_host4 > scsi_host5 > scsi_host6 ------> scsi_host7 disappear and scsi_host6 instead > > # virsh nodedev-dumpxml scsi_host6 > <device> > <name>scsi_host6</name> > <path>/sys/devices/pci0000:00/0000:00:0d.0/0000:04:00.1/host5/vport-5:0-0/host6</path> > <parent>scsi_host5</parent> > <capability type='scsi_host'> > <host>6</host> > <unique_id>6</unique_id> > <capability type='fc_host'> > <wwnn>2101001b32a90002</wwnn> > <wwpn>2101001b32a90003</wwpn> > <fabric_wwn>2001000dec9877c1</fabric_wwn> > </capability> > </capability> > </device> > > # ls -al /sys/class/scsi_host/ > lrwxrwxrwx. 1 root root 0 Oct 16 20:44 host5 -> ../../devices/pci0000:00/0000:00:0d.0/0000:04:00.1/host5/scsi_host/host5 > lrwxrwxrwx. 1 root root 0 Oct 16 20:48 host6 -> ../../devices/pci0000:00/0000:00:0d.0/0000:04:00.1/host5/vport-5:0-0/host6/scsi_host/host6 > > # cat /sys/class/scsi_host/host6/unique_id > 6 -----> the unique_id is also changed to 6 > > > It seems that the unique_id keeps same with the scsi host name. > e.g. > unique_id for scsi_host6 is 6 > unique_id for scsi_host7 is 7 > > So if the scsi host name changed after rebooting, the unique_id changed too. > > Thanks > Yang Yang > Pardon the delay in getting back to this - my post KVM Forum pile of things to get to is dwindling, but not yet empty! Hmm... I see; however, this is a vHBA, e.g. one of our own creation used for NPIV. That is, I think when Osier wrote the code for parentaddr and unique_id his target was a physical HBA and not something generated for use by NPIV. I'll have to think about this some more, but perhaps allowing an adapter type of 'scsi_host' of a vHBA using 'fc_host' shouldn't be allowed. In this case, if you think about it they are essentially looking at the same storage target(s), right? Perhaps once the fix for bz1146837 is in the downstream code for test I can revisit this as a bug. That change does better duplication checking if the adapter types are the same, but not so much if they are different. That is the to be defined adapter type comparisons and ensuring the "incoming" definition doesn't duplicate something already defined for an existing pool. It seems another check needs to be made with respect to if the incoming adapter_type is scsi_host, then make sure it's target isn't an already defined pool with adapter_type fc_host. Perhaps the key is the "vport-5:0-" in the resolve link file name. I'd have to come up with some sort of algorithm to have an incoming scsi_host scan/find any defined fc_host's and check for duplication. Curious can you also define a "scsi_host" of an "fc_host" adapter using: <source> <adapter type='scsi_host' name='scsi_host#'/> </source> where "#" is the scsi_host# assigned to the fc_host adapter? or is it only possible if you use the parentaddr syntax <source> <adapter type='scsi_host'> <parentaddr unique_id='7'> <address domain='0x0' bus='0x04' slot='0x0' function='0x1'/> </parentaddr> </adapter> </source> where of course unique_id and pci address matches whatever is in use for the fc_host? Tks, John Probably should file a new bug on this "condition" in any case. Whether it's fixed in 7.1 or 7.2 is TBD... BTW: My unique_id's are usually different than the host#, e.g. # find -H /sys/class/scsi_host/host*/unique_id | xargs grep '[0-9]' /sys/class/scsi_host/host0/unique_id:1 /sys/class/scsi_host/host1/unique_id:2 /sys/class/scsi_host/host2/unique_id:3 /sys/class/scsi_host/host3/unique_id:4 /sys/class/scsi_host/host4/unique_id:5 /sys/class/scsi_host/host5/unique_id:6 /sys/class/scsi_host/host7/unique_id:0 /sys/class/scsi_host/host8/unique_id:0 /sys/class/scsi_host/host9/unique_id:0 My conversation with John following #9 Yang Yang wrote: ----- Original Message ----- > From: "Yang Yang" <yanyang> > To: "John Ferlan" <jferlan> > > From: "John Ferlan" <jferlan> > > To: "Yang Yang" <yanyang> >>> From: "Yang Yang" <yanyang> >>> To: "John Ferlan" <jferlan> > > > Here are the example and steps of the same PCI Address SCSI_HOST having > > > it's > > > unique_id change. > > > > > > > Pardon the delay in getting back to this - my post KVM Forum pile of > > things to get to is dwindling, but not yet empty! > > > > Hmm... I see; however, this is a vHBA, e.g. one of our own creation used > > for NPIV. That is, I think when Osier wrote the code for parentaddr and > > unique_id his target was a physical HBA and not something generated for > > use by NPIV. > > > > I'll have to think about this some more, but perhaps allowing an adapter > > type of 'scsi_host' of a vHBA using 'fc_host' shouldn't be allowed. In > > this case, if you think about it they are essentially looking at the > > same storage target(s), right? Yes. If a scsi pool is created using an adapter of 'scsi_host' and specifying a target already defined pool using 'fc_host', the two pools looking for the same target. > > > > > Perhaps once the fix for bz1146837 is in the downstream code for test I > > can revisit this as a bug. That change does better duplication checking > > if the adapter types are the same, but not so much if they are > > different. That is the to be defined adapter type comparisons and > > ensuring the "incoming" definition doesn't duplicate something already > > defined for an existing pool. > > > > It seems another check needs to be made with respect to if the incoming > > adapter_type is scsi_host, then make sure it's target isn't an already > > defined pool with adapter_type fc_host. Perhaps the key is the > > "vport-5:0-" in the resolve link file name. I'd have to come up with > > some sort of algorithm to have an incoming scsi_host scan/find any > > defined fc_host's and check for duplication. > > > > Curious can you also define a "scsi_host" of an "fc_host" adapter using: > > > > <source> > > <adapter type='scsi_host' name='scsi_host#'/> > > </source> Yes, I can. Defining 'scsi_host' type pool specifying both name using the target already defined pool using 'fc_host' works. > > > > where "#" is the scsi_host# assigned to the fc_host adapter? > > > > or is it only possible if you use the parentaddr syntax No, both of the two methods work. > > > > <source> > > <adapter type='scsi_host'> > > <parentaddr unique_id='7'> > > <address domain='0x0' bus='0x04' slot='0x0' function='0x1'/> > > </parentaddr> > > </adapter> > > </source> > > > > where of course unique_id and pci address matches whatever is in use for > > the fc_host? > > > > Tks, > > > > John > > > > Probably should file a new bug on this "condition" in any case. Whether > > it's fixed in 7.1 or 7.2 is TBD... So I can file a bug to track the issue "Do NOT allow to define 'scsi_host' pool using the target already defined pool using 'fc_host'", right? Does it make sense that defining 'fc_host' pool using a transient vHBA ? If no, should libvirt check it and forbid defining that pool ? If yes, defining 'fc_host' pool using the target already defined pool using 'scsi_host'" should not be allowed, right ? e.g. 1. creating a transient vHBA scsi_host8 # virsh nodedev-create vHBA.xml Node device scsi_host8 created from vHBA.xml 2. define 'fc_host' pool with above vHBA 'scsi_host8' # cat fc-pool.xml <source> <adapter type='fc_host' parent='scsi_host5' wwnn='2101001b32a90002' wwpn='2101001b32a90003'/> == scsi_host8 actually </source> # virsh pool-define fc-pool.xml Pool fc-pool defined from fc-pool.xml Scsi_host# is more unstable for vHBA than physical HBA, it always changes between rebooting. Hence, does it deserve handling stable scsi host addressing for Virtual HBA ? Thanks Yang Yang My conversation with John following #10 John Ferlan wrote: ----- Original Message ----- > From: "Yang Yang" <yanyang> > To: "John Ferlan" <jferlan> <...snip...? >>> >>> Probably should file a new bug on this "condition" in any case. Whether >>> it's fixed in 7.1 or 7.2 is TBD... > > So I can file a bug to track the issue "Do NOT allow to define 'scsi_host' > pool using the target already defined pool using 'fc_host'", right? > Yes - although based on what follows maybe it's a more general disallow a scsi_host adapter for a vHBA target. That would fix all sorts of potentially weird problems. The question I don't have answered in my mind though is - is there a valid customer use case to create a scsi_host adapter pool using a vHBA target? The point of a one of these is to have something for a fc_host adapter pool. > > Does it make sense that defining 'fc_host' pool using a transient vHBA ? huh? The creation of a vHBA is done via nodedev-create. That creates a "somewhat" transient vHBA. If a 'fc_pool' is "defined" for it, then the "next reboot" results in an automatic (re)creation of the vHBA (IIRC). If someone creates a transient fc_host adapter, then I assuming it's gone on the next reboot. If someone defines a scsi_host adapter for the vHBA, then I'm assuming on the next boot, the pool doesn't have the adapter it expects (although I didn't try that) since nothing automagically creates the vHBA (there's no way to do so with the wwnn & wwpn). > If no, should libvirt check it and forbid defining that pool ? > If yes, defining 'fc_host' pool using the target already defined pool using > 'scsi_host'" should not be allowed, right ? Although your example doesn't show this path... It's possible to create a 'scsi_host' adapter for the vHBA (for what purpose, who knows). If one tries to then create a 'fc_host' adapter using the "parent=... wwnn=... wwpn=..." syntax, it's disallowed because the code knows to look for an existing 'scsi_host#' for the parent argument. What I'm not sure of (since I was testing using an older libvirt) is since the "parent=" is optional, does libvirt handle things well in this scenario. I was "borrowing' a system where libvirtd crashed (not good, but that might be fixed by my other changes). I also tested creating an fc_host pool, then attempt to create a scsi_host pool, which had the same results (libvirtd crash). > e.g. > 1. creating a transient vHBA scsi_host8 > # virsh nodedev-create vHBA.xml > Node device scsi_host8 created from vHBA.xml > > 2. define 'fc_host' pool with above vHBA 'scsi_host8' > # cat fc-pool.xml > <source> > <adapter type='fc_host' parent='scsi_host5' > wwnn='2101001b32a90002' wwpn='2101001b32a90003'/> == scsi_host8 actually > </source> > > # virsh pool-define fc-pool.xml > Pool fc-pool defined from fc-pool.xml > > > Scsi_host# is more unstable for vHBA than physical HBA, it always changes > between rebooting. Hence, does it deserve handling stable scsi host > addressing for Virtual HBA ? > True; however, the more difficult question to answer is how to determine if the target is a vHBA or a pHBA. Secondary to that is whether a customer has a use case to create a scsi_host adapter for a vHBA. I may need to borrow/spend a few testing cycles on a host where I can experiment, but will only know more once I dig deeper into the code. John Thanks John first. Since 'scsi_host' adapter will be disabled on 'fc_host' device later, I am trying to verify the one using generic usb and iscsi devices instead of Fibre Channel (v)HBA. It works well for usb device. However, it does NOT work for iscsi device since there is no pci address for iscsi device. Fortunately,'scsi_host' pool can be defined using 'name' attribute on iscsi device. As we know, it's unstable. So do you plan to handle the unstable host# for iscsi device? If it does not deserve a fix, how about highlighting it in doc to let others know 'pci address'&'unique_id' attributes are not available for iscsi device? An example for iscsi device is as following: 1. login to an iscsi server # iscsiadm -m node -T iqn.yy:server.target2 -p $ip:3260 -l Login to [iface: default, target: iqn.yy:server.target2, portal: 10.66.100.101,3260] successful. 2. # virsh nodedev-list --tree computer | +- scsi_host9 ----> iscsi device | +- scsi_target9_0_0 | +- scsi_9_0_0_0 | | | +- scsi_generic_sg2 | +- scsi_9_0_0_1 | +- block_sdc_1IET_00020001 +- scsi_generic_sg3 # virsh nodedev-list scsi_host scsi_host0 scsi_host1 scsi_host2 scsi_host3 scsi_host4 scsi_host5 scsi_host6 scsi_host9 ---> iscsi device # find -H /sys/class/scsi_host/host*/unique_id | xargs grep '[0-9]' /sys/class/scsi_host/host0/unique_id:1 /sys/class/scsi_host/host1/unique_id:2 /sys/class/scsi_host/host2/unique_id:3 /sys/class/scsi_host/host3/unique_id:4 /sys/class/scsi_host/host4/unique_id:5 /sys/class/scsi_host/host5/unique_id:6 /sys/class/scsi_host/host6/unique_id:0 /sys/class/scsi_host/host9/unique_id:0 ---> iscsi device 3. create scsi pool # virsh pool-dumpxml scsi-pool-1 <source> <adapter type='scsi_host' name='scsi_host9'/> </source> # virsh pool-create scsi-pool-1.xml Pool scsi-pool-1 created from scsi-pool-1.xml # virsh vol-list scsi-pool-1 Name Path ------------------------------------------------------------------------------ unit:0:0:1 /dev/disk/by-path/ip-10.66.100.101:3260-iscsi-iqn.yy:server.target2-lun-1 ============================================================== Steps for verifying this one using usb device are as following: # rpm -q libvirt libvirt-1.2.8-5.el7.x86_64 steps: 1. # find -H /sys/class/scsi_host/host*/unique_id | xargs grep '[0-9]' /sys/class/scsi_host/host0/unique_id:1 /sys/class/scsi_host/host1/unique_id:2 /sys/class/scsi_host/host2/unique_id:3 /sys/class/scsi_host/host3/unique_id:4 /sys/class/scsi_host/host4/unique_id:5 /sys/class/scsi_host/host5/unique_id:6 /sys/class/scsi_host/host8/unique_id:0 ---> usb, used to define 'scsi_host' pool # virsh nodedev-list --tree +- pci_0000_00_1a_0 | | | | | +- scsi_host8 | | | +- scsi_target8_0_0 | | | +- scsi_8_0_0_0 | | | +- block_sdb_Kingston_DataTraveler_G3_CCCB1011091031230081295302_0_0 | +- scsi_generic_sg1 2. define/start a scsi pool with the following xml # virsh pool-dumpxml scsi-pool <pool type='scsi'> <name>scsi-pool</name> <uuid>0a1eead8-987c-4cf6-8ec9-a573d43c9895</uuid> <capacity unit='bytes'>4013948928</capacity> <allocation unit='bytes'>4013948928</allocation> <available unit='bytes'>0</available> <source> <adapter type='scsi_host'> <parentaddr unique_id='0'> <address domain='0x0000' bus='0x00' slot='0x1a' function='0x0'/> </parentaddr> </adapter> </source> <target> <path>/dev/disk/by-path</path> <permissions> <mode>0700</mode> <owner>0</owner> <group>0</group> </permissions> </target> </pool> # virsh pool-define scsi-pool.xml # virsh pool-start scsi-pool # virsh pool-autostart scsi-pool # virsh pool-list --all Name State Autostart ------------------------------------------- default active yes scsi-pool active yes # virsh vol-list scsi-pool Name Path ------------------------------------------------------------------------------ unit:0:0:0 /dev/disk/by-path/pci-0000:00:1a.0-usb-0:1.4:1.0-scsi-0:0:0:0 # virsh pool-destroy scsi-pool Pool scsi-pool destroyed 3. Rebooting host # virsh nodedev-list --tree +- pci_0000_00_1a_0 -----> PCI address does not change | | | | | +- scsi_host6 ----> host# changed | | | +- scsi_target6_0_0 | | | +- scsi_6_0_0_0 | | | +- block_sdb_Kingston_DataTraveler_G3_CCCB1011091031230081295302_0_0 | +- scsi_generic_sg1 # find -H /sys/class/scsi_host/host*/unique_id | xargs grep '[0-9]' /sys/class/scsi_host/host0/unique_id:1 /sys/class/scsi_host/host1/unique_id:2 /sys/class/scsi_host/host2/unique_id:3 /sys/class/scsi_host/host3/unique_id:4 /sys/class/scsi_host/host4/unique_id:5 /sys/class/scsi_host/host5/unique_id:6 /sys/class/scsi_host/host6/unique_id:0 -----> changed to host6 instead of host8, but unique_id is stable # virsh pool-list --all Name State Autostart ------------------------------------------- default active yes scsi-pool active yes scsi pool is autostarted # virsh vol-list scsi-pool Name Path ------------------------------------------------------------------------------ unit:0:0:0 /dev/disk/by-path/pci-0000:00:1a.0-usb-0:1.4:1.0-scsi-0:0:0:0 4. define a vm using the volume <disk type='volume' device='disk'> <driver name='qemu' type='raw'/> <source pool='scsi-pool' volume='unit:0:0:0'/> <backingStore/> <target dev='sda' bus='scsi'/> <boot order='1'/> <alias name='scsi0-0-0-0'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> guest can be read/wrote well. Using scsi_host for an iscsi 'scsi_host#' probably should not be allowed to start. Although that might come in the fallout of bz 1159180 - not quite sure yet as I'm focused on the fc_host start/stop with parent issues right now. As described in comment #14, It works well for usb device. Although it does not work well for iscsi device, the issue may be addressed in bz 1159180. It should not prevent from verifying the bug. So mark it verified. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2015-0323.html |