Hi Ravi, 
Please try following disk xml, and please note when you use type=volume, only device='disk' is supported. 

<disk type='volume' device='disk'>
     <driver name='qemu' type='raw'/>
     <source pool='poolvhba5' volume='unit:0:0:1'/>
     <target dev='sdb' bus='scsi'/>
</disk>  

Seems there is a typo in http://wiki.libvirt.org/page/NPIV_in_libvirt#To_configure_the_LUN_as_a_pass-through_device.2C_use_the_following_XML_examples

which used a sample including    <disk type='volume' device='lun'>

(In reply to yisun from comment #2)
> Hi Ravi, 
> Please try following disk xml, and please note when you use type=volume,
> only device='disk' is supported. 
> 
> <disk type='volume' device='disk'>
>      <driver name='qemu' type='raw'/>
>      <source pool='poolvhba5' volume='unit:0:0:1'/>
>      <target dev='sdb' bus='scsi'/>
> </disk>  
> 
> Seems there is a typo in
> http://wiki.libvirt.org/page/NPIV_in_libvirt#To_configure_the_LUN_as_a_pass-
> through_device.2C_use_the_following_XML_examples
> 
> which used a sample including    <disk type='volume' device='lun'>

Hi Yisun,

Please let me know if disk type as 'volume' is supported with device as 'disk', then I have below question, please try to reply inline.

1. How Persistent reservation or pass-through will work as it will work only if device is exposed as "lun" not with "disk" ? 

2. In RHEL 7.1 after exposing FC storage to KVM guest sg_persist command will not work ? 

3. From which version of RHEL "disk_type as volume" with device as "lun" will be supported ?

Gentle reminder !! 

Please try to provide me answers to below question:

1. How Persistent reservation or pass-through will work on RHEL 7.x (KVM) as it will work only if device is exposed as "lun" not with "disk" ? 

2. Will sg_persist command work in RHEL 7.1 after exposing FC storage to KVM guest ? 

3. From which version of RHEL "disk type" as volume with device as "lun" will be supported ?

4. While closing Bug 1224641, John has replied in RHEL 6.x using vHBA created by node device driver will work.

## "I believe the /dev/sdX could be used as a path, but it won't be persistent." -John

Do you have any conformation from RedHat on above statement so that it can be recommended to any customer or user with limitation.

If we can not recommend above thing then using vHBA created via storage pool is the only way if anyone want to use Pass-thought feature on exposed shared disks from host to guest as "lun". 

Look like there is bug in in RHEL 7.X and it must be fixed; exposing shared storage disks as "lun" is a valid use case, if anyone want to use "persistent reservation" (Pass-though) feature on guest.

There should not be any restriction like host's shared disks can be always exposed as "disk".

(In reply to ravi.mishra from comment #4)
> Gentle reminder !! 
> 
> Please try to provide me answers to below question:
> 
> 1. How Persistent reservation or pass-through will work on RHEL 7.x (KVM) as
> it will work only if device is exposed as "lun" not with "disk" ? 
You need to passthrough the lun as a hostdev to kvm guest. 
Add following segment to your guest's xml
Where adapter name is the HBA or vHBA card which connected to the luns you need. 
And bus, target, unit must be set. Then you'll see the scsi storage in your guest. 

 <hostdev mode='subsystem' type='scsi' managed='no'>
      <source>
        <adapter name='scsi_host11'/>
        <address bus='0' target='2' unit='0'/>
      </source>
    </hostdev>


> 2. Will sg_persist command work in RHEL 7.1 after exposing FC storage to KVM
> guest ? 
During my try there is no error in guest with cmd
#sg_persist -k /dev/sda -vvv
Where sda is the passthrough fc disk.

> 
> 3. From which version of RHEL "disk type" as volume with device as "lun"
> will be supported ?
Not sure, if it's important to you, maybe you can open a more specific comment/bug with business justification to gain more attention. 

> 4. While closing Bug 1224641, John has replied in RHEL 6.x using vHBA
> created by node device driver will work.
> 
> ## "I believe the /dev/sdX could be used as a path, but it won't be
> persistent." -John
> 
> Do you have any conformation from RedHat on above statement so that it can
> be recommended to any customer or user with limitation.
The only recommended way to use FC luns is pool/volume, in following link  

http://wiki.libvirt.org/page/NPIV_in_libvirt

search following paragraph.
"It is recommended to define a libvirt storage pool based on the vHBA in order to preserve the vHBA configuration."

All other ways are not persistent.

But as you know, this way will pass the storage to guest as device=disk but not device=lun



> 
> If we can not recommend above thing then using vHBA created via storage pool
> is the only way if anyone want to use Pass-thought feature on exposed shared
> disks from host to guest as "lun". 
> 
> Look like there is bug in in RHEL 7.X and it must be fixed; exposing shared
> storage disks as "lun" is a valid use case, if anyone want to use
> "persistent reservation" (Pass-though) feature on guest.
> 
> There should not be any restriction like host's shared disks can be always
> exposed as "disk".

Every time NPIV questions are asked, my head starts spinning in the wrong direction, especially with all the various ways to configure things. I agree the existing RHEL7 instructions are a bit difficult to read; however, I do know the "next release" has made a concerted effort to adjust those docs (I recall reviewing them in detail, but have paged them out of my conscience).  I also recall making some code changes in the space, but nothing related to allowing exposing as a "<disk type='volume' device='lun'>"

I also note that you added "sgio='unfiltered'" to your XML - there's another private bz related to restoring some functionality around sgio for the next release. There are patches currently upstream awaiting review.

Although the upstream docs on the libvirt wiki haven't been touched for a while, I'm not certain there's a typo there. I have never tried to configure using a pass through lun using the instructions (I picked up ownership of the code, but didn't not write it)... 

In particular, with respect to using as a 'lun', you will note the instructions on the libvirt wiki page have two ways to use

#1 - As a LUN pass-through, using:

   <disk type='volume' device='lun'>
     <driver name='qemu' type='raw'/>
     <source dev='/dev/disk/by-path/pci-0000\:04\:00.1-fc-0x203400a0b85ad1d7-lun-0'/>
     <target dev='sda' bus='scsi'/>
   </disk>

#2 - As a Disk using a storage pool:

   <disk type='volume' device='disk'>
     <driver name='qemu' type='raw'/>
     <source pool='poolvhba0' volume='unit:0:2:0'/>
     <target dev='hda' bus='ide'/>
   </disk>


Attempting to use 'lun' in the #2 isn't supported as you found out. However, is there something that stops you from using the #1 option? If you do a 'virsh vol-list $pool', you will not volumes are listed by 'Name" and 'Path'. I believe if you use that path string as the "source dev='$path'" (with the prerequisite '\' before the ':' 's), then you'll be able to use 'lun'.

If that works, then I assume you'd rather have libvirt be able to allow you to use the much shorter source pool option?  Since the PCI address could change between reboots (or even pool destroy/create/start processing).

TBH: This could be an oversight of the original development effort, but without investigating deeper and trying things myself I'm never quite sure.  But if the source dev syntax works for you, then that'll help me know the feasibility of doing something for the next release.

Thanks John for knowledge sharing on KVM with FC with NPIV. 

There is one correction is immediate required in redhat-virtualization document. 
disk type as "block" will work with "lun" not with "volume" as documented:

Working:
	<disk type='block' device='lun' sgio='unfiltered'>

Not working:
	<disk type='volume' device='lun' sgio='unfiltered'>


You have asked question that "What is the blocker for me to use '#1 As a LUN pass-through' 
(vHBA created node device driver way)". 

My answer is below:
	
	After creating Virtual HBAs (vHBA) in host system (using node-device driver way or storage pool way) and soft zoning at switch 
	side to storage array (using WWPN&WWNN),  for each lun on host system start showing multi-paths. For each vHBA
	there is one path to same lun. RedHat document recommend to use stable path to lun which appears under 
	"/dev/disk/by-path" directory as a source path while attaching to KVM guest. Problem with this approach even
	after creation of multiple vHBAs on host, for each lun on host only one path appears under "/dev/disk/by-path" directory
	but under "/dev/" directory I will get multiple paths, It will be more clear with below example: 
	
	In example I have created two vHBAs using on physical HBA using node-device driver way and did storage level zoning.
	After zoning I will get below results.
	
	--------------------------------------
	BEFORE CREATITION OF vHBAs and ZONING
	--------------------------------------
	Only one path appears under "/dev/" directory 
	# /dev/sda ---> which uses physical HBA

	Under "/dev/disk/by-path" directory:
	# ls -l /dev/disk/by-path/
	lrwxrwxrwx. 1 root root  9 Jun  1 23:34 pci-0000:05:00.0-fc-0x5001438011362ad8-lun-1 -> ../../sda

	--------------------------------------
	AFTER CREATITION OF vHBAs and ZONING
	--------------------------------------
	Two more paths appears under "/dev/" directory 
	# /dev/sda ---> which uses physical HBA
	# /dev/sdaa ---> Which uses 1st vHBA created via node-device driver way.
	# /dev/sdab ---> Which uses 2nd vHBA created via node-device driver way.
	
	Under "/dev/disk/by-path" directory, still it has only one soft-link to one path. 
	# ls -l /dev/disk/by-path/
	lrwxrwxrwx. 1 root root  9 Jun  1 23:34 pci-0000:05:00.0-fc-0x5001438011362ad8-lun-1 -> ../../sda
	
	I can not give same path like "/dev/disk/by-path/pci-0000:05:00.0-fc-0x5001438011362ad8-lun-1" to two KVM guest system,
	then both guest can write on shared disk which results data integrity. (Even I have tried using same path exposed to both
	kvm guests then persistent reservation was not working) 	
	
Note: I am not getting any syntax error while using above path. (Without '\' before the ':'s)
	
	As per my understanding NPIV to work properly we need to assign two different path (which uses two vHBAs created from same physical HBA) 
	to two different KVM guest system so that pass-through (Persistent reservation) will work properly.
	
	Even I have tested and it is working as expected (Persistent reservation was working across two KVM guests hosted on same host system);
	We can use "/dev/sdaa" or "/dev/sdab" as source device while exposing as lun to KVM guests paths will appear to guests but once 
	host will reboot device name will change so path exposed to guest KVMs will also not work. 
	  	
	
## To overcome above problems it will be better if device can be exposed as "lun" and vHBA created via storage pool way. Even it is not working
but this should be supported in next release of redhat to give customer freedom to select between "lun" or "disk" with vHBA created using storage pool.

## If device can be exposed only as "lun" to KVM guests using vHBA created via node-device driver or storage pool way then which path can be used 
as "source dev='$path'". As out of multiple paths only one of them will be listed under "/dev/disk/by-path" directory. This must be clearly documented by redhat. 
 	
	
I have few question on my explanation:

1. NPIV is officially supported and tested by redhat or not with FC storage ?

2. If yes, is there any way to make device path name to be consistent across host reboot ?
(Before host reboot suppose lun path is "dev/sdaa" but after reboot it might appear as "/dev/sdaX")

Is there any way to create udev rules for all paths associated with each vHBAs to make their name to be consistent across reboots? 


# /dev/sda ---> which uses physical HBA
# /dev/sdaa ---> Which uses 1st vHBA 
# /dev/sdab ---> Which uses 2nd vHBA 

Is there any udev rule by which after reboot we can preserve the device name as it is ?

Personally all the various NPIV details aren't something I work with every day nor do I have in depth knowledge of the architecture. For me it's one of many different storage types I have at least "some" knowledge about.  I have shared access to one machine that has FC storage, but I have never touched the fabric portion nor would I know how to set it up or find out anything more than basic information. To a degree the role of libvirt is to expose what is there for usage and make a few things easier by doing the search through the hierarchy to find and expose what it knows about. The code is "freely viewable" and "patches are always welcome"...  The libvirt git repo can be cloned from  git://libvirt.org/libvirt.git and the code that does the bulk of the work is found in 'src/storage/storage_backend_scsi.c' as well as a bunch of utility work in src/util/virscsi.c and src/util/virutil.c.

I always 'fear' questions about "officially supported" - I'm an engineer, that to me is a product management type question. I also used to work at HP and I know it has a different connotation or degree of expectation there than perhaps it does here. I can say for sure things are tested as I get bz's assigned based on that testing. To what degree things are tested - I have less insight into that (perhaps I'd have more if I could find the cycles to read the test plans).

With regard to creating udev rules - that's not my expertise area (nor is it a libvirt type issue). That's a kernel issue. The libvirt code merely takes advantage of what's supplied. I just try to tread water with respect to keeping up with multiple different storage technologies.

Anyway given your output - you indicate that a /dev/sdaa and a /dev/sdab are created, but there's only one link in '/dev/disk/by-path/' to /dev/sda, thus if I'm reading this right you 'expected' to see a path to /dev/sdaa and /dev/sdab, correct?  and furthermore you'd expect that path to be the same from reboot to reboot (which I don't think it will be, hence using the 'unit:#:#:#' to access.

Whether providing/using 'lun' in addition to 'disk' is supportable is something I'd have to investigate a bit more and gain a better understanding of. The whole persistent reservations discussion is something I don't have a lot of exposure to - more details which I haven't had the cycles to dig into.


FWIW: Here's a config I use for my testing:

# virsh nodedev-dumpxml scsi_host3
<device>
  <name>scsi_host3</name>
  <path>/sys/devices/pci0000:00/0000:00:04.0/0000:10:00.0/host3</path>
  <parent>pci_0000_10_00_0</parent>
  <capability type='scsi_host'>
    <host>3</host>
    <unique_id>0</unique_id>
    <capability type='fc_host'>
      <wwnn>20000000c9848140</wwnn>
      <wwpn>10000000c9848140</wwpn>
      <fabric_wwn>2002000573de9a81</fabric_wwn>
    </capability>
    <capability type='vport_ops'>
      <max_vports>127</max_vports>
      <vports>2</vports>
    </capability>
  </capability>
</device>

# virsh nodedev-dumpxml scsi_host10
<device>
  <name>scsi_host10</name>
  <path>/sys/devices/pci0000:00/0000:00:04.0/0000:10:00.0/host3/vport-3:0-5/host10</path>
  <parent>scsi_host3</parent>
  <capability type='scsi_host'>
    <host>10</host>
    <unique_id>7</unique_id>
    <capability type='fc_host'>
      <wwnn>5001a4a93526d0a1</wwnn>
      <wwpn>5001a4ace3ee045d</wwpn>
      <fabric_wwn>2002000573de9a81</fabric_wwn>
    </capability>
  </capability>
</device>

# virsh vol-list vhbapool_host3
 Name                 Path                                    
------------------------------------------------------------------------------
 unit:0:4:0           /dev/disk/by-path/pci-0000:10:00.0-fc-0x5006016844602198-lun-0
 unit:0:5:0           /dev/disk/by-path/pci-0000:10:00.0-fc-0x5006016044602198-lun-0

# ll /dev/disk/by-path | grep "pci-0000:10"
lrwxrwxrwx. 1 root root  9 Jun 23 17:28 pci-0000:10:00.0-fc-0x217800c0ffd79b2a-lun-13 -> ../../sdg
lrwxrwxrwx. 1 root root  9 Jun 23 17:28 pci-0000:10:00.0-fc-0x5006016044602198-lun-0 -> ../../sdw
lrwxrwxrwx. 1 root root  9 Jun 23 17:28 pci-0000:10:00.0-fc-0x5006016844602198-lun-0 -> ../../sdu
lrwxrwxrwx. 1 root root  9 Jun 23 17:28 pci-0000:10:00.1-fc-0x207800c0ffd79b2a-lun-13 -> ../../sdf
lrwxrwxrwx. 1 root root  9 Jun 23 17:28 pci-0000:10:00.1-fc-0x5006016144602198-lun-0 -> ../../sdv
lrwxrwxrwx. 1 root root  9 Jun 23 17:28 pci-0000:10:00.1-fc-0x5006016944602198-lun-0 -> ../../sdt


Extracts from 'virsh nodedev-list --tree':

  +- pci_0000_00_04_0
  |   |
  |   +- pci_0000_10_00_0
  |   |   |
  |   |   +- scsi_host3
  |   |       |
  |   |       +- scsi_host10
  |   |       |   |
  |   |       |   +- scsi_target10_0_0
  |   |       |   |   |
  |   |       |   |   +- scsi_10_0_0_0
  |   |       |   |       |
  |   |       |   |       +- scsi_generic_sg11
  |   |       |   |         
  |   |       |   +- scsi_target10_0_1
  |   |       |   |   |
  |   |       |   |   +- scsi_10_0_1_0
  |   |       |   |       |
  |   |       |   |       +- scsi_generic_sg12
  |   |       |   |         
  |   |       |   +- scsi_target10_0_2
  |   |       |   |   |
  |   |       |   |   +- scsi_10_0_2_0
  |   |       |   |       |
  |   |       |   |       +- scsi_generic_sg22
  |   |       |   |         
  |   |       |   +- scsi_target10_0_3
  |   |       |   |   |
  |   |       |   |   +- scsi_10_0_3_0
  |   |       |   |       |
  |   |       |   |       +- scsi_generic_sg23
  |   |       |   |         
  |   |       |   +- scsi_target10_0_4
  |   |       |   |   |
  |   |       |   |   +- scsi_10_0_4_0
  |   |       |   |       |
  |   |       |   |       +- block_sdd_350060160c460219850060160c4602198
  |   |       |   |       +- scsi_generic_sg24
  |   |       |   |         
  |   |       |   +- scsi_target10_0_5
  |   |       |       |
  |   |       |       +- scsi_10_0_5_0
  |   |       |           |
  |   |       |           +- block_sde_350060160c460219850060160c4602198
  |   |       |           +- scsi_generic_sg25
  |   |       |             
...

lsscsi -lwigvvv
...
[10:0:0:0]   enclosu                                 -          -  /dev/sg11
  state=running queue_depth=30 scsi_level=6 type=13 device_blocked=0 timeout=0
  dir: /sys/bus/scsi/devices/10:0:0:0  [/sys/devices/pci0000:00/0000:00:04.0/0000:10:00.0/host3/vport-3:0-5/host10/rport-10:0-1/target10:0:0/10:0:0:0]
[10:0:1:0]   enclosu                                 -          -  /dev/sg12
  state=running queue_depth=30 scsi_level=6 type=13 device_blocked=0 timeout=0
  dir: /sys/bus/scsi/devices/10:0:1:0  [/sys/devices/pci0000:00/0000:00:04.0/0000:10:00.0/host3/vport-3:0-5/host10/rport-10:0-2/target10:0:1/10:0:1:0]
[10:0:2:0]   enclosu                                 -          -  /dev/sg22
  state=running queue_depth=30 scsi_level=6 type=13 device_blocked=0 timeout=0
  dir: /sys/bus/scsi/devices/10:0:2:0  [/sys/devices/pci0000:00/0000:00:04.0/0000:10:00.0/host3/vport-3:0-5/host10/rport-10:0-3/target10:0:2/10:0:2:0]
[10:0:3:0]   enclosu                                 -          -  /dev/sg23
  state=running queue_depth=30 scsi_level=6 type=13 device_blocked=0 timeout=0
  dir: /sys/bus/scsi/devices/10:0:3:0  [/sys/devices/pci0000:00/0000:00:04.0/0000:10:00.0/host3/vport-3:0-5/host10/rport-10:0-4/target10:0:3/10:0:3:0]
[10:0:4:0]   disk                                    /dev/sdd   -  /dev/sg24
  state=running queue_depth=30 scsi_level=5 type=0 device_blocked=0 timeout=30
  dir: /sys/bus/scsi/devices/10:0:4:0  [/sys/devices/pci0000:00/0000:00:04.0/0000:10:00.0/host3/vport-3:0-5/host10/rport-10:0-5/target10:0:4/10:0:4:0]
[10:0:5:0]   disk                                    /dev/sde   -  /dev/sg25
  state=running queue_depth=30 scsi_level=5 type=0 device_blocked=0 timeout=30
  dir: /sys/bus/scsi/devices/10:0:5:0  [/sys/devices/pci0000:00/0000:00:04.0/0000:10:00.0/host3/vport-3:0-5/host10/rport-10:0-6/target10:0:5/10:0:5:0]


"How" each was used/configured in a guest and the expectations vis-a-vis multiple guests and persistent reservations, etc. is something I just don't have the inherent knowledge about.
...

Thanks John, but above explanation will not solve the problem. This is major drawback in KVM as compare to VMware. In VMware NPIV is supported and documented properly. Device can be exposed to guest as "lun" and Persistent reservation is working properly.

This is very valid use case for any customer those are using KVM technology and want to use pass-though feature. Redhat team should work towards to support exposing any disk to KVM guest system as "lun" and at least guide how to configure if anyone is using NPIV technology.

Without this any customer can not implement NPIV on disk which exposed as "lun" to guest. 

There are only two possible solution: 

1. Start supporting storage pool way of exposing volume device (using 'unit:#:#:#' and unique "pool" name) to guest as "lun". Currently disk type as "volume" is works with device type as "disk". I do not know what will be side effects of supporting "lun" with disk type "volume" like pass-though will work or not.

2. Need some change in redhat kernel side so that OS will create some persistent paths for any device which exposed using vHBAs (and autostart option is set in storage pool) under "dev/disk/by-path" directly and has soft link to device like "/dev/sdxx" as it is happening with physical HBA.


Please acknowledge the below:

One correction is immediate required in redhat-virtualization document. 
disk type as "block" will work with device type as "lun" not with "volume" but in example it is incorrectly documented:

Working:
	<disk type='block' device='lun' sgio='unfiltered'>

Not working:
	<disk type='volume' device='lun' sgio='unfiltered'>

Disk type as "volume" is working with device type "disk".

Please try the above things in your test system and let me know is this documentation defect or my understanding is not correct.

I would like to know do you want me to open new bugzilla for updating "Virtulization deployment and administration guide (under section "16.7.4.3.2".) with correct example. Please got to below link 

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Virtualization_Deployment_and_Administration_Guide/sect-NPIV_storage.html#sect-NPIV_storage-Locating_LUNs_vHBAs


<disk type='volume' device='disk'> <------- Bug in this 
​     <driver name='qemu' type='raw'/>
​     <source dev='/dev/disk/by-path/pci-0000\:04\:00.1-fc-0x203400a0b85ad1d7-lun-0'/>
​     <target dev='sda' bus='scsi'/>
​   </disk>

If customer will try to apply using "virsh attach-device .." command above example xml file it will fail with below error.

# virsh attach-device --persistent node1 sda.xml
error: Failed to attach device from sda_without_pool.xml
error: unsupported configuration: disk device='lun' is only valid for block type disk source


Corrected example would be:
----------------------------
<disk type='block' device='disk'>
​     <driver name='qemu' type='raw'/>
​     <source dev='/dev/disk/by-path/pci-0000\:04\:00.1-fc-0x203400a0b85ad1d7-lun-0'/>
​     <target dev='sda' bus='scsi'/>
​   </disk>

disk type "block" is supported with "lun" not "volume".

(In reply to ravi.mishra from comment #11)
> I would like to know do you want me to open new bugzilla for updating
> "Virtulization deployment and administration guide (under section
> "16.7.4.3.2".) with correct example. Please got to below link 
> 
> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/
> html/Virtualization_Deployment_and_Administration_Guide/sect-NPIV_storage.
> html#sect-NPIV_storage-Locating_LUNs_vHBAs
> 
> 
> <disk type='volume' device='disk'> <------- Bug in this 
> ​     <driver name='qemu' type='raw'/>
> ​     <source
> dev='/dev/disk/by-path/pci-0000\:04\:00.1-fc-0x203400a0b85ad1d7-lun-0'/>
> ​     <target dev='sda' bus='scsi'/>
> ​   </disk>
> 
> If customer will try to apply using "virsh attach-device .." command above
> example xml file it will fail with below error.
> 
> # virsh attach-device --persistent node1 sda.xml
> error: Failed to attach device from sda_without_pool.xml
> error: unsupported configuration: disk device='lun' is only valid for block
> type disk source
> 
> 
> Corrected example would be:
> ----------------------------
> <disk type='block' device='disk'>
> ​     <driver name='qemu' type='raw'/>
> ​     <source
> dev='/dev/disk/by-path/pci-0000\:04\:00.1-fc-0x203400a0b85ad1d7-lun-0'/>
> ​     <target dev='sda' bus='scsi'/>
> ​   </disk>
> 
> disk type "block" is supported with "lun" not "volume".

Sorry for typo mistake, I have used "disk" instead of "lun" in above example. 

<disk type='volume' device='lun'> ----> This is not working need to update also in deployment guide and change with below:

<disk type='block' device='lun'>

Without this nobody can expose FC disk to guest KVM as "lun".

I have raised one separate bug on documentation and got acknowledgment that there is bug in that. Instead of "volume" in example "block" must be used.

https://bugzilla.redhat.com/show_bug.cgi?id=1238606

I would like to know can we use "/dev/sdax" or "/dev/sday" if we are not getting soft-link under "/dev/disk/by-path" directory, if not then there is one more bug in redhat documentation which is telling to use vHBA to expose host FC disk to KVM guest systems as "lun"?

Redhat must explain the case if you are not getting stable path under "/dev/disk/by-path" or "/dev/disk/by-id" etc then which path can be use as source dev. 

Nobody will create vHBA if host has only one guest. Only for exposing to multiple guest multiple vHBA has to be created.

Let's summarize...

It seems the libvirt wiki has been updated to be:

Using a LUN from a vHBA created by the storage pool (disk volume)

   <disk type='volume' device='disk'>
     <driver name='qemu' type='raw'/>
     <source pool='poolvhba0' volume='unit:0:2:0'/>
     <target dev='hda' bus='ide'/>
   </disk>

or

Using a LUN from a vHBA created using the node device driver (direct lun)

   <disk type='block' device='disk'>
     <driver name='qemu' type='raw'/>
     <source dev='/dev/disk/by-path/pci-0000\:04\:00.1-fc-0x203400a0b85ad1d7-lun-0'/>
     <target dev='sda' bus='scsi'/>
   </disk>

or

To configure the LUN as a pass-through device

   <disk type='block' device='lun'>
     <driver name='qemu' type='raw'/>
     <source dev='/dev/disk/by-path/pci-0000\:04\:00.1-fc-0x203400a0b85ad1d7-lun-0'/>
     <target dev='sda' bus='scsi'/>
   </disk>

or

   <disk type='volume' device='disk'>
     <driver name='qemu' type='raw'/>
     <source pool='poolvhba0' volume='unit:0:2:0'/>
     <target dev='hda' bus='ide'/>
   </disk>

  NB: No difference between this second example and first disk volume example, but I suppose it was meant to be type='volume' device='lun' (e.g. the issue you're raising).  I'd also flip-flop the pass-through examples since the 'disk' usage describes pool first and then node device, while the 'pass-through' usage describes node device first and then pool (eg, it's confusing).

Anyway, I assume these are essentially "correct" from even your multiple updates (like I noted before, it's not that easy to keep it all straight - and also why I'm using the bz kind of like a whiteboard in an office!).  I also think the pool example in the pass through should go back to 'lun', but perhaps needs a note that it's currently not working as designed which would be updated once/if patches are accepted and pushed upstream.

Your desire is to be able to have the pool pass-through example be:

   <disk type='volume' device='lun'>
     <driver name='qemu' type='raw'/>
     <source pool='poolvhba0' volume='unit:0:2:0'/>
     <target dev='sda' bus='scsi'/>
     <shareable/>
   </disk>
	
With the claim/belief that two pools using this XML would 'view' the lun as the same, it would be shareable, and be able to perform persistent reservation commands.


Beyond that other than fixing the examples provided, it seems that "today" the supported/working configurations only allow "volume/disk", "block/disk", or "block/lun", but those don't work for what you're trying to do because you want/need the persistent reservations which require the "volume/lun" configuration to work. I would hope that "volume/lun" is merely an oversight of the original set of changes or perhaps some update that allowed block/lun forgot to allow volume/lun (still have to look at the history). But now that I have a better idea (and perhaps more time to focus on this), I can work through all the "details".  Using "volume" requires an extra step to translate the pool data so that other checks "just work". It may be that the translation has "missed" a case or we're making a specific check before the translation occurs (still to be determined, but as you can see I'm now thinking...)


FWIW: While I see/understand your urgency and frustration, I would say I've been working under the assumption that things have been or are working correctly. Configuration options are complex, so I need to understand in my vernacular what's not working. I also have multiple competing issues - in various different areas. I don't focus solely on vHBA/NPIV. 

This is the first report where the claim is things aren't working as documented as well as a request for allowing 'volume/lun'. I don't regularly configure vHBA's for use, so understanding the exact steps/syntax you used is always helpful. It's why I showed you the steps/commands I used and my output.  In comment 7, I have no idea what you mean by "and soft zoning at switch side to storage array (using WWPN&WWNN)..." - it's not something I've ever done. I'll have to step through the creation processing again to perhaps understand how I can correlate your /dev/sdaa and /dev/sdab to what I see.

Thanks John for deep dive into the issue, yes you are correct looks like as of now supported/working configurations only allow "volume/disk", "block/disk", or "block/lun", but currently "volume/lun" is not working which is required for persistent-reservation to work. 

Complete problem will be solved only after supporting "volume/lun" configuration along with sgio option which can be set as "filtered" and "unfiltered" for Lun pass-though feature to work lun must be exposed with sgio option to be set as "unfiltered". 

Please got to below link and search for "sgio" then you will get below lines which talks about that "volume/lun" should work, but actually it is not working now.

https://libvirt.org/formatdomain.html

SNAPSHOT from web page.
>>>>>>>>>>>

disk
The disk element is the main container for describing disks (since 0.0.3).
type attribute since 0.0.3
Valid values are "file", "block", "dir" (since 0.7.5), "network" (since 0.8.7), or "volume" (since 1.0.5) and refer to the underlying source for the disk.
device attribute since 0.1.4
Indicates how the disk is to be exposed to the guest OS. Possible values for this attribute are "floppy", "disk", "cdrom", and "lun", defaulting to "disk".
Using "lun" (since 0.9.10) is only valid when the type is "block" or "network" for protocol='iscsi' or when the type is "volume" when using an iSCSI source pool for mode "host". Configured in this manner, the LUN behaves identically to "disk", except that generic SCSI commands from the guest are accepted and passed through to the physical device. Also note that device='lun' will only be recognized for actual raw devices, but never for individual partitions or LVM partitions (in those cases, the kernel will reject the generic SCSI commands, making it identical to device='disk').

rawio attribute since 0.9.10
Indicates whether the disk needs rawio capability. Valid settings are "yes" or "no" (default is "no"). If any one disk in a domain has rawio='yes', rawio capability will be enabled for all disks in the domain (because, in the case of QEMU, this capability can only be set on a per-process basis). This attribute is only valid when device is "lun". NB, rawio intends to confine the capability per-device, however, current QEMU implementation gives the domain process broader capability than that (per-process basis, affects all the domain disks). To confine the capability as much as possible for QEMU driver as this stage, sgio is recommended, it's more secure than rawio.
sgio attribute since 1.0.2
Indicates whether the kernel will filter unprivileged SG_IO commands for the disk, valid settings are "filtered" or "unfiltered". Defaults to "filtered". Similar to rawio, sgio is only valid for device 'lun'.
snapshot attribute since 0.9.5
Indicates the default behavior of the disk during disk snapshots: "internal" requires a file format such as qcow2 that can store both the snapshot and the data changes since the snapshot; "external" will separate the snapshot from the live data; and "no" means the disk will not participate in snapshots. Read-only disks default to "no", while the default for other disks depends on the hypervisor's capabilities. Some hypervisors allow a per-snapshot choice as well, during domain snapshot creation. Not all snapshot modes are supported; for example, snapshot='yes' with a transient disk generally does not make sense.

<<<<<<<<<<<<<<<<<<<<<<<<<

From above explanation in libvirt documentation two points are very clear:

1. Lun can be configured on "volume" type of device (As it is working for disk). Looks like some bug in code side that does not allow "volume/lun" configuration. 

2. sgio option is only valid for "lun" type device.

# pass-though to work customer need to do following things: 

1. Expose entire raw device to guest KVM not as a partitioned device or LVM volumes.
2. Option need to be set as "unfiltered" so that host kernel will not reject any unprivileged SG_IO commands requests from guest.

There must be proper steps in Virtualization deployment and administration guide Under section "16.7.4. Locating LUNs on a vHBA" under this Redhat must explain how to get stable path if there are multiple vHBAs. Without this customer will ask same question as I am asking how to find stable path for one lun came from multiple vHBAs.

In my case I have configured two vHBAs from single HBA. Created two storage pool to create persistent vHBAs.

I have one HBA name "scsi_host1" then I have created two vHBAs using storage pool namely "scsi_host5" and "scsi_host6".

Before creation of vHBAs, system has only one path to lun of wwn (60014380125989c80000800021a50000)
/dev/sdb

After creation of vHBAs two more path appears in system under /dev/ directory.
/dev/sdn
/dev/sdo

# virsh nodedev-list --tree |more 

  |   |   +- scsi_host1
  |   |       |
  |   |       +- scsi_host5
  |   |       |   |
  |   |       |   +- scsi_target5_0_0
  |   |       |       |
  |   |       |       +- scsi_5_0_0_0
  |   |       |       |   |
  |   |       |       |   +- scsi_generic_sg18
  |   |       |       |
  |   |       |       +- scsi_5_0_0_1
  |   |       |          |
  |   |       |          +- block_sdn_360014380125989c80000800021a50000
  |   |       |          +- scsi_generic_sg20
  |   |       |       
  |   |       |           
  |   |       |
  |   |       +- scsi_host6
  |   |       |   |
  |   |       |   +- scsi_target6_0_0
  |   |       |       |
  |   |       |       +- scsi_6_0_0_0
  |   |       |       |   |
  |   |       |       |   +- scsi_generic_sg19
  |   |       |       |
  |   |       |       +- scsi_6_0_0_1
  |   |       |          |
  |   |       |          +- block_sdo_360014380125989c80000800021a50000
  |   |       |          +- scsi_generic_sg21
  |   |       |       
  |   |       |       
  |   |       |
  |   |       +- scsi_target1_0_0
  |   |       |   |
  |   |       |   +- scsi_1_0_0_0
  |   |       |   |   |
  |   |       |   |   +- scsi_generic_sg2
  |   |       |   |
  |   |       |   +- scsi_1_0_0_1
  |   |       |      |
  |   |       |      +- block_sdb_360014380125989c80000800021a50000
  |   |       |      +- scsi_generic_sg4
 

#systool -c fc_host -v |more

Class = "fc_host"

  Class Device = "host1"
  Class Device path = "/sys/devices/pci0000:00/0000:00:09.0/0000:08:00.0/host1/fc_host/host1"
    dev_loss_tmo        = "16"
    fabric_name         = "0x100000051e0220b0"
    issue_lip           = <store method only>
    max_npiv_vports     = "127"
    node_name           = "0x50014380120dc535"
    npiv_vports_inuse   = "2"
    port_id             = "0x020800"
    port_name           = "0x50014380120dc534"
    port_state          = "Online"
    port_type           = "NPort (fabric via point-to-point)"
    speed               = "4 Gbit"
    supported_classes   = "Class 3"
    supported_speeds    = "1 Gbit, 2 Gbit, 4 Gbit"
    symbolic_name       = "HPAE312A FW:v7.03.00 DVR:v8.07.00.08.07.1-k2"
    system_hostname     = ""
    tgtid_bind_type     = "wwpn (World Wide Port Name)"
    uevent              =
    vport_create        = <store method only>
    vport_delete        = <store method only>

    Device = "host1"
    Device path = "/sys/devices/pci0000:00/0000:00:09.0/0000:08:00.0/host1"
      fw_dump             =
      nvram               = "ISP "
      optrom_ctl          = <store method only>
      optrom              =
      reset               = <store method only>
      sfp                 = ""
      uevent              = "DEVTYPE=scsi_host"
      vpd                 = "▒+"


  Class Device = "host5"
  Class Device path = "/sys/devices/pci0000:00/0000:00:09.0/0000:08:00.0/host1/vport-1:0-2/host7/fc_host/host5"
    dev_loss_tmo        = "16"
    fabric_name         = "0x100000051e0220b0"
    issue_lip           = <store method only>
    node_name           = "0x50014380120dc543"
    port_id             = "0x020803"
    port_name           = "0x50014380120dc542"
    port_state          = "Online"
    port_type           = "NPIV VPORT"
    speed               = "4 Gbit"
    supported_classes   = "Class 3"
    symbolic_name       = "HPAE312A FW:v7.03.00 DVR:v8.07.00.08.07.1-k2"
    system_hostname     = ""
    tgtid_bind_type     = "wwpn (World Wide Port Name)"
    uevent              =

    Device = "host5"
    Device path = "/sys/devices/pci0000:00/0000:00:09.0/0000:08:00.0/host1/vport-1:0-2/host5"
      uevent              = "DEVTYPE=scsi_host"


  Class Device = "host6"
  Class Device path = "/sys/devices/pci0000:00/0000:00:09.0/0000:08:00.0/host1/vport-1:0-3/host8/fc_host/host6"
    dev_loss_tmo        = "16"
    fabric_name         = "0x100000051e0220b0"
    issue_lip           = <store method only>
    node_name           = "0x50014380120dc545"
    port_id             = "0x020804"
    port_name           = "0x50014380120dc544"
    port_state          = "Online"
    port_type           = "NPIV VPORT"
    speed               = "4 Gbit"
    supported_classes   = "Class 3"
    symbolic_name       = "HPAE312A FW:v7.03.00 DVR:v8.07.00.08.07.1-k2"
    system_hostname     = ""
    tgtid_bind_type     = "wwpn (World Wide Port Name)"
    uevent              =

    Device = "host6"
    Device path = "/sys/devices/pci0000:00/0000:00:09.0/0000:08:00.0/host1/vport-1:0-3/host6"
      uevent              = "DEVTYPE=scsi_host"

# virsh vol-list poolvhba5 --details
 Name        Path                                                            Type   Capacity  Allocation
---------------------------------------------------------------------------------------------------------
 unit:0:0:1  /dev/disk/by-path/pci-0000:08:00.0-fc-0x5001438011362ad8-lun-1  block  2.00 GiB    2.00 GiB

#virsh vol-list poolvhba6 --details
 Name        Path                                                            Type   Capacity  Allocation
---------------------------------------------------------------------------------------------------------
 unit:0:0:1  /dev/disk/by-path/pci-0000:08:00.0-fc-0x5001438011362ad8-lun-1  block  2.00 GiB    2.00 GiB

# ls /sys/bus/scsi/devices/5:*/block/
/sys/bus/scsi/devices/5:0:0:1/block/:
sdn

# ls /sys/bus/scsi/devices/6:*/block/
/sys/bus/scsi/devices/5:0:0:1/block/:
sdo


If I follow the steps mentioned in "Procedure 16.8. Locating a vHBA LUN" in Virtulization and deployment guide of RHEL 7.1. 

"Determine the stable path to the LUN."

# ls -l /dev/disk/by-path/ | grep sdb
lrwxrwxrwx. 1 root root  9 Jul  3 12:48 pci-0000:08:00.0-fc-0x5001438011362ad8-lun-1 -> ../../sdb
# ls -l /dev/disk/by-path/ | grep sdo
# ls -l /dev/disk/by-path/ | grep sdn

I will see above output, from this If i want to expose stable path for "sdn" or "sdo" then what will I give ??

If customer can use only "/dev/sdn" or "/dev/sdo" (which are not stable paths and might change after host reboot) to expose as lun to guest KVMs then that must be there in document without clarification customer will never use NPIV on KVM.

If customer does have option to use "volume/lun" then they can configure two differnet vHBA storage pools and unit name instead of block device path because of this only telling you to investigate on it.

w/r/t Comment 15/16

Good - it seems "volume/lun" is what is desired.  With respect to whether it's a bug or an intentional omission - that's still up for debate.  You'll note from the webpage you reference there are very specific types supported for 'lun'. This is also something undergoing fine tuning and patch updates within the last month or so - in upstream.

The question I have to answer or problem to solve (depending on your viewpoint) is whether a LUN exposed via the FC SCSI can be utilized in the same manner as the existing supported types. I think you'll note a couple of similarities - I'm hoping it's mostly a validation effort, but we'll see.

As for the "sgio" - yes, I'm quite aware of that. It's also only supported on one downstream platform. There's a somewhat interesting discussion ongoing on the upstream libvir-list regarding some other patches in that area, see:

http://www.redhat.com/archives/libvir-list/2015-July/msg00204.html


w/r/t Comment 17

Thanks for the details and example - it helps.

My assumption is the solution will be very similar to the manner in which an iSCSI LUN is exposed. That is the magic behind the scenes to find what relates to the particular source volume will be hidden since as you point out the path to it is not stable between reboots, e.g. something similar to:

    <disk type='volume' device='lun'>
      <driver name='qemu' type='raw'/>
      <source pool='vhbapool' volume='unit:0:0:1'/>
      <target dev='sdb' bus='scsi'/>
    </disk>

where libvirt will do the magic to convert 'unit:0:0:1' into the appropriate path when assigning the lun to the guest (such as from your example /dev/disk/by-path/pci-0000:10:00.0-fc-0x5006016844602198-lun-0) in the qemu-kvm command. 

Adjusting documentation to describe how to configure is always something I try to ensure happens.

While reviewing another issue (it's a locked bug, so not worth linking) related to unprivileged SGIO, someone pointed out perhaps a different way to access FC storage as a LUN:


1. connect to a fc storage
    1.1 root@host #echo '2101001b32a90002:2101001b32a90002' > /sys/class/fc_host/host5/vport_create
    1.2 root@host # lsscsi
...
//scsi_host6 is the new vHBA created as follow
[6:0:0:0]    disk    IBM      1726-4xx  FAStT  0617  -       
[6:0:1:0]    disk    IBM      1726-4xx  FAStT  0617  -       
[6:0:2:0]    disk    IBM      1726-4xx  FAStT  0617  /dev/sdf 
[6:0:3:0]    disk    IBM      1726-4xx  FAStT  0617  /dev/sdg

2.having a pool for device mapper:
    2.1 root@host # virsh pool-dumpxml mpath
<pool type='mpath'>
  <name>mpath</name>
  <uuid>d0243e3c-0b98-235b-8704-c104e309abf0</uuid>
  <capacity unit='bytes'>32218696704</capacity>
  <allocation unit='bytes'>32218696704</allocation>
  <available unit='bytes'>0</available>
  <source>
  </source>
  <target>
    <path>/dev/mapper</path>
    <permissions>
      <mode>0755</mode>
      <owner>-1</owner>
      <group>-1</group>
    </permissions>
  </target>
</pool>

    2.2 root@host #virsh pool-refresh mpath
    2.3 root@host # virsh vol-list mpath
Name                 Path                                    
-----------------------------------------
dm-0                 /dev/mapper/3600a0b80005adb0b0000ab2d4cae9254
dm-5                 /dev/mapper/3600a0b80005ad1d700002dde4fa32ca8    <=== this one is from vhba scsi_host6

3. having a pool for scsi_host6
    3.1 root@host #vim pool-dumpxml host6
<pool type='scsi'>
  <name>host6</name>
  <uuid>08ffaca7-da35-a822-8c8a-8c15204bb51d</uuid>
  <capacity unit='bytes'>10737418240</capacity>
  <allocation unit='bytes'>10737418240</allocation>
  <available unit='bytes'>0</available>
  <source>
    <adapter name='host6'/>
  </source>
  <target>
    <path>/dev/disk/by-id</path>
    <permissions>
      <mode>0755</mode>
      <owner>-1</owner>
      <group>-1</group>
    </permissions>
  </target>
</pool>

    3.2 root@host #virsh vol-list host6
Name                 Path                                    
-----------------------------------------
unit:0:2:0           /dev/disk/by-id/wwn-0x600a0b80005ad1d700002dde4fa32ca8




Scenario 1: passthrough the fc storage to vm using /dev/disk/by-id/xxxxxx 
steps:
1. make sure the disk with sgio='unfiltered'
    root@host #virsh dumpxml vm | grep disk -a10
....
    <disk type='block' device='lun' sgio='unfiltered'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/disk/by-id/wwn-0x600a0b80005ad1d700002dde4fa32ca8'/>
      <target dev='sda' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
...

2. start the vm and login it, use sg_persist cmd
   root@guest # sg_persist -k /dev/sda -vvv
open /dev/sda with flags=0x800
    inquiry cdb: 12 00 00 00 24 00
      duration=0 ms
  IBM       1726-4xx  FAStT   0617
  Peripheral device type: disk
open /dev/sda with flags=0x802
    Persistent Reservation In cmd: 5e 00 00 00 00 00 00 20 00 00
      duration=1 ms
    persistent reservation in: requested 8192 bytes but got 8 bytes
    persistent reserve in: response
        00 00 00 00 00 00 00 00                             
  PR generation=0x0, there are NO registered reservation keys
<======= the command works well

....

NOTE:
Step 1 is essentially what the creation of the vHBA is - it writes wwnn/wwpn to the vport_create.  The 'lsscsi' is a utility that can be installed. It's quite useful.  And yes, the naming isn't as "persistent" as you're desiring, but it does show that this type of LUN can be exposed and thus something could be done. Hopefully it's just a matter of a few adjustments in the code to recognize things properly.

(In reply to John Ferlan from comment #19)
> While reviewing another issue (it's a locked bug, so not worth linking)
> related to unprivileged SGIO, someone pointed out perhaps a different way to
> access FC storage as a LUN:
> 
> 
> 1. connect to a fc storage
>     1.1 root@host #echo '2101001b32a90002:2101001b32a90002' >
> /sys/class/fc_host/host5/vport_create
>     1.2 root@host # lsscsi
> ...
> //scsi_host6 is the new vHBA created as follow
> [6:0:0:0]    disk    IBM      1726-4xx  FAStT  0617  -       
> [6:0:1:0]    disk    IBM      1726-4xx  FAStT  0617  -       
> [6:0:2:0]    disk    IBM      1726-4xx  FAStT  0617  /dev/sdf 
> [6:0:3:0]    disk    IBM      1726-4xx  FAStT  0617  /dev/sdg
> 
> 2.having a pool for device mapper:
>     2.1 root@host # virsh pool-dumpxml mpath
> <pool type='mpath'>
>   <name>mpath</name>
>   <uuid>d0243e3c-0b98-235b-8704-c104e309abf0</uuid>
>   <capacity unit='bytes'>32218696704</capacity>
>   <allocation unit='bytes'>32218696704</allocation>
>   <available unit='bytes'>0</available>
>   <source>
>   </source>
>   <target>
>     <path>/dev/mapper</path>
>     <permissions>
>       <mode>0755</mode>
>       <owner>-1</owner>
>       <group>-1</group>
>     </permissions>
>   </target>
> </pool>
> 
>     2.2 root@host #virsh pool-refresh mpath
>     2.3 root@host # virsh vol-list mpath
> Name                 Path                                    
> -----------------------------------------
> dm-0                 /dev/mapper/3600a0b80005adb0b0000ab2d4cae9254
> dm-5                 /dev/mapper/3600a0b80005ad1d700002dde4fa32ca8    <===
> this one is from vhba scsi_host6
> 
> 3. having a pool for scsi_host6
>     3.1 root@host #vim pool-dumpxml host6
> <pool type='scsi'>
>   <name>host6</name>
>   <uuid>08ffaca7-da35-a822-8c8a-8c15204bb51d</uuid>
>   <capacity unit='bytes'>10737418240</capacity>
>   <allocation unit='bytes'>10737418240</allocation>
>   <available unit='bytes'>0</available>
>   <source>
>     <adapter name='host6'/>
>   </source>
>   <target>
>     <path>/dev/disk/by-id</path>
>     <permissions>
>       <mode>0755</mode>
>       <owner>-1</owner>
>       <group>-1</group>
>     </permissions>
>   </target>
> </pool>
> 
>     3.2 root@host #virsh vol-list host6
> Name                 Path                                    
> -----------------------------------------
> unit:0:2:0           /dev/disk/by-id/wwn-0x600a0b80005ad1d700002dde4fa32ca8
> 
> 
> 
> 
> Scenario 1: passthrough the fc storage to vm using /dev/disk/by-id/xxxxxx 
> steps:
> 1. make sure the disk with sgio='unfiltered'
>     root@host #virsh dumpxml vm | grep disk -a10
> ....
>     <disk type='block' device='lun' sgio='unfiltered'>
>       <driver name='qemu' type='raw'/>
>       <source dev='/dev/disk/by-id/wwn-0x600a0b80005ad1d700002dde4fa32ca8'/>
>       <target dev='sda' bus='scsi'/>
>       <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>     </disk>
> ...
> 
> 2. start the vm and login it, use sg_persist cmd
>    root@guest # sg_persist -k /dev/sda -vvv
> open /dev/sda with flags=0x800
>     inquiry cdb: 12 00 00 00 24 00
>       duration=0 ms
>   IBM       1726-4xx  FAStT   0617
>   Peripheral device type: disk
> open /dev/sda with flags=0x802
>     Persistent Reservation In cmd: 5e 00 00 00 00 00 00 20 00 00
>       duration=1 ms
>     persistent reservation in: requested 8192 bytes but got 8 bytes
>     persistent reserve in: response
>         00 00 00 00 00 00 00 00                             
>   PR generation=0x0, there are NO registered reservation keys
> <======= the command works well
> 
> ....
> 
> NOTE:
> Step 1 is essentially what the creation of the vHBA is - it writes wwnn/wwpn
> to the vport_create.  The 'lsscsi' is a utility that can be installed. It's
> quite useful.  And yes, the naming isn't as "persistent" as you're desiring,
> but it does show that this type of LUN can be exposed and thus something
> could be done. Hopefully it's just a matter of a few adjustments in the code
> to recognize things properly.


Thanks John,

I would like to know how to get stable path for more than one vHBA as each path need to be exposed to individual VMs.

If you explain that then my problem would be solved.

In above example you have taken very easy path by explaining about single vHBA and VM. Problem which I am facing while doing something on more than one VM.

The point of my example was to indicate it seems like it was possible to view the FC SCSI device as a LUN - no more, no less.  That was a data point and indicated to me that using an FC SCSI LUN "should" work.

Perhaps your vision of "stable path" and mine are a bit different. My view is you'd provide the XML using the "volume" and "lun" with a source pool and volume. That way no matter what happens across reboots, you'd end up with the same volume. Trying to use the /dev/sdX obviously isn't nor is the "/sys/bus/scsi/devices/*", since between reboots whatever scsi_hostN device is created as a result of the vHBA could change (e.g., the N value can change).

In any case, I have perhaps good news and bad news.  The good news is using an upstream version of libvirt with a fedora box that's connected to a Fiber Channel storage, I can define a disk. The bad news is it seems like there's something wrong if I'm using the scsi bus for the device on the guest. Works fine for virtio. I have to investigate if it's a known problem or a configuration limitation.

Assuming I have a guest, if I use the following XML:

    <disk type='volume' device='lun'>
      <driver name='qemu' type='raw'/>
      <source pool='vhbapool_host3' volume='unit:0:0:0'/>
      <target dev='vdb' bus='virtio'/>
    </disk>

Then "so far" no issues seeing the volume, using the volume, etc.; however, reading into what you want - you want a 'scsi' bus target because you have a desire to use SCSI commands with the "sgio='unfiltered'" option.

On this system:

# virsh pool-dumpxml vhbapool_host3
...
  <source>
    <adapter type='fc_host' parent='scsi_host3' managed='yes' wwnn='5001a4a93526d0a1' wwpn='5001a4ace3ee045d'/>
  </source>
...
#
# virsh vol-list vhbapool_host3
 Name                 Path                                    
------------------------------------------------------------------------------
 unit:0:0:0           /dev/disk/by-path/pci-0000:10:00.0-fc-0x5006016044602198-lun-0
 unit:0:1:0           /dev/disk/by-path/pci-0000:10:00.0-fc-0x5006016844602198-lun-0

# lsscsi -tg
...
[11:0:0:0]   disk    fc:0x50060160446021980x102000   /dev/sdd   /dev/sg11
[11:0:1:0]   disk    fc:0x50060168446021980x101f00   /dev/sde   /dev/sg12
...

or in the manner you requested:

# ls /sys/bus/scsi/devices/11:*/block/
/sys/bus/scsi/devices/11:0:0:0/block/:
sdd

/sys/bus/scsi/devices/11:0:1:0/block/:
sde
# 
# ls -l /dev/disk/by-path | grep sdd
lrwxrwxrwx. 1 root root  9 Jul 15 13:28 pci-0000:10:00.0-fc-0x5006016044602198-lun-0 -> ../../sdd
# ls -l /dev/disk/by-path | grep sde
lrwxrwxrwx. 1 root root  9 Jul 15 13:28 pci-0000:10:00.0-fc-0x5006016844602198-lun-0 -> ../../sde

So it seems some of the "connections" you're needing are fixed; however, whether they work in RHEL is still a bit of an unknown to me as I don't have that environment to test in right now.

If I continue to pump data, here's scsi_host11 which is what serves the block devices shown above:

# virsh nodedev-dumpxml scsi_host11
<device>
  <name>scsi_host11</name>
  <path>/sys/devices/pci0000:00/0000:00:04.0/0000:10:00.0/host3/vport-3:0-6/host11</path>
  <parent>scsi_host3</parent>
  <capability type='scsi_host'>
    <host>11</host>
    <unique_id>8</unique_id>
    <capability type='fc_host'>
      <wwnn>5001a4a93526d0a1</wwnn>
      <wwpn>5001a4ace3ee045d</wwpn>
      <fabric_wwn>2002000573de9a81</fabric_wwn>
    </capability>
  </capability>
</device>

If I shutdown the guest, modify the XML to add the following:

    <disk type='volume' device='lun'>
      <driver name='qemu' type='raw'/>
      <source pool='vhbapool_host3' volume='unit:0:0:0'/>
      <target dev='vdb' bus='virtio'/>
    </disk>

while the guest boots/starts up, there's something wrong in a lower layer than libvirt because shortly after guest boot there's some sort of lockup:

    <disk type='volume' device='lun'>
      <driver name='qemu' type='raw'/>
      <source pool='vhbapool_host3' volume='unit:0:1:0'/>
      <target dev='sdb' bus='scsi'/>
      <shareable/>
    </disk>

I tried this method because attempting to 'virsh attach-device $dom sdb.xml' of :

<disk type='volume' device='lun'>
  <driver name='qemu' type='raw'/>
  <source pool='vhbapool_host3' volume='unit:0:1:0'/>
  <target dev='sdb' bus='scsi'/>
  <shareable/>
</disk>

doesn't allow the device to be seen.

So while not the answer you want - there is progress between what you're perhaps testing with (RHEL7.1, libvirt 1.2.8) and the next release.

FWIW: my qemu command is:

# ps -ef | grep 0x5006016
root     10390     1 99 13:28 ?        00:07:05 /usr/bin/qemu-system-x86_64 -machine accel=kvm -name f22-vol-lun-fc -S -machine pc-i440fx-1.6,accel=kvm,usb=off -m 1536 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid 114975bf-a57a-44a5-b2f2-2e4a4b8189bb -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/f22-vol-lun-fc.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device lsi,id=scsi0,bus=pci.0,addr=0x5 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -device usb-ccid,id=ccid0 -drive file=/home/jferlan/vm-images/f22.img,if=none,id=drive-virtio-disk0,format=raw -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/dev/disk/by-path/pci-0000:10:00.0-fc-0x5006016044602198-lun-0,if=none,id=drive-virtio-disk1,format=raw -device virtio-blk-pci,scsi=on,bus=pci.0,addr=0x9,drive=drive-virtio-disk1,id=virtio-disk1 -drive file=/dev/disk/by-path/pci-0000:10:00.0-fc-0x5006016844602198-lun-0,if=none,id=drive-scsi0-0-1,format=raw,cache=off -device scsi-block,bus=scsi0.0,scsi-id=1,drive=drive-scsi0-0-1,id=scsi0-0-1 -netdev tap,fd=26,id=hostnet0,vhost=on,vhostfd=27 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:5d:f6:26,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/f22-vol-lun-fc.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel1,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=com.redhat.spice.0 -chardev pty,id=charconsole0 -device virtconsole,chardev=charconsole0,id=console0 -device usb-tablet,id=input0 -spice port=5903,addr=127.0.0.1,disable-ticketing,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16,bus=pci.0,addr=0x2 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -msg timestamp=on
#

You note the /dev/disk/by-path/pci* for the two volumes is there - one is virtio and one is scsi.

Just wanted to post what I've discovered and will hope to keep you up to date as I find out more.

Thanks John, for detailed report. 

I have tried the same thing which you have explained but in my system (RHEL7.1 + libvirt version 1.2.8). Exposing a host device as lun to guest using virtio bus is also not happening.

# virsh vol-list  poolvhba6
 Name                 Path
------------------------------------------------------------------------------
 unit:0:0:1           /dev/disk/by-path/pci-0000:08:00.0-fc-0x5001438011362ad8-lun-1
 unit:0:0:2           /dev/disk/by-path/pci-0000:08:00.0-fc-0x5001438011362ad8-lun-2


In my system poolvhba6 creates node device driver scsi_host8.


# virsh pool-dumpxml poolvhba6
<pool type='scsi'>
  <name>poolvhba6</name>
  <uuid>3ea4f45c-0ced-46d3-8a21-45e84e989a36</uuid>
  <capacity unit='bytes'>12884901888</capacity>
  <allocation unit='bytes'>12884901888</allocation>
  <available unit='bytes'>0</available>
  <source>
    <adapter type='fc_host' parent='scsi_host1' managed='yes' wwnn='50014380120dc545' wwpn='50014380120dc544'/>
  </source>
  <target>
    <path>/dev/disk/by-path</path>
    <permissions>
      <mode>0700</mode>
      <owner>0</owner>
      <group>0</group>
    </permissions>
  </target>
</pool>

# virsh nodedev-dumpxml scsi_host8
<device>
  <name>scsi_host8</name>
  <path>/sys/devices/pci0000:00/0000:00:09.0/0000:08:00.0/host1/vport-1:0-3/host8</path>
  <parent>scsi_host1</parent>
  <capability type='scsi_host'>
    <host>8</host>
    <unique_id>8</unique_id>
    <capability type='fc_host'>
      <wwnn>50014380120dc545</wwnn>
      <wwpn>50014380120dc544</wwpn>
      <fabric_wwn>100000051e0220b0</fabric_wwn>
    </capability>
  </capability>
</device>

When I tried the below XML, I still got the same error as I got with bus as "scsi".

# cat sda_with_pool_6.xml
<disk type='volume' device='lun'>
<driver name='qemu' type='raw'/>
<source pool='poolvhba6' volume='unit:0:0:2'/>
<target dev='vda' bus='virtio'/>
<shareable/>
</disk>

# virsh attach-device --persistent guest1 sda_with_pool_6.xml
error: Failed to attach device from sda_with_pool_6.xml
error: unsupported configuration: disk device='lun' is only valid for block type disk source

Then I tried with bus as "scsi", which as usual failed with same error.

# cat sda_with_pool_6_scsi.xml
<disk type='volume' device='lun'>
<driver name='qemu' type='raw'/>
<source pool='poolvhba6' volume='unit:0:0:2'/>
<target dev='sda' bus='scsi'/>
<shareable/>
</disk>

# virsh attach-device --persistent guest1 sda_with_pool_6_scsi.xml
error: Failed to attach device from sda_with_pool_6_scsi.xml
error: unsupported configuration: disk device='lun' is only valid for block type disk source


In short, I would like to inform you that in my system "volume/lun" is not working either I use "virtio" or "scsi" as bus.

One more thing I observed that sgio='unfiltered' option is working with bus as "scsi" as well as "virtio" when i used "block/lun" instead of "volume/lun" type of configuration in XML.

Today, I have raised new bugzilla on documentation (Virtualization deployment and administration guide).


1243723: Unable to expose FC disk as lun to multiple guest using NPIV

I looked at the 7.1 sources and just coming back with - this should work for you...   Then it dawned on me what the difference is...

Remove the "--persistent" flag?  Does that work?  That is - is the device added to your live system (you can use the --live flag too). 

See bug 1228007 - I realize it indicates 'iSCSI' but it's the same code path - it's trying to save into the config; however, the config path didn't translate the source pool.

Long story short is the "details" are in the storage pool and only when it's needed is that referenced. 

I'm aware of the other bugs that have been filed... I'm getting pinged by the writer.

Thanks John for root causing the problem. 

I tried attaching device "volume/lun" configuration in XML using "--live" option it was successfully attached and sgio option is also honored by "volume/lun" configuration. "sg_persist" command is working as expected.


Yes, you are correct there is bug in libvirt (virsh attach-device command ) because of that "volume/lun" was not working with "--persistent/--config" option. 

Only request is to document this issue until it got fixed. 

As "volume/lun" configuration is working even with sgio option must be documented so that customer can refer it.

I have exposed two paths using two vHBA to two different VM (storage side NPIV configuration is also done). To expose I have used below XMLs.

# cat sda_with_pool_5_scsi.xml
<disk type='volume' device='lun'>
<driver name='qemu' type='raw'/>
<source pool='poolvhba5' volume='unit:0:0:1'/>
<target dev='sda' bus='scsi'/>
<shareable/>
</disk>

# cat sda_with_pool_6_scsi.xml
<disk type='volume' device='lun'>
<driver name='qemu' type='raw'/>
<source pool='poolvhba6' volume='unit:0:0:1'/>
<target dev='sda' bus='scsi'/>
<shareable/>
</disk>

I have used below command to attach device to kVM guests.

# virsh attach-device --live guest1 disk_pool_guest1.xml
virsh attach-device --live guest1 disk_pool_guest1.xml
Device attached successfully


# virsh attach-device --live guest2 disk_pool_guest2.xml
virsh attach-device --live guest2 disk_pool_guest2.xml
Device attached successfully

----------------------------------------------------------------------
From guest1, I have registered and reserved the key on device expose using 1st vHBA (pool_5 storage pool).
----------------------------------------------------------------------

# sg_persist --out --no-inquiry --register --param-sark=0x34 --device=/dev/sd
# sg_persist -k /dev/sda
  HP        HSV300            0953
  Peripheral device type: disk
  PR generation=0x41, 1 registered reservation key follows:
    0x34
# sg_persist -r /dev/sda
  HP        HSV300            0953
  Peripheral device type: disk
  PR generation=0x41, there is NO reservation held

# /usr/bin/sg_persist --out --reserve --param-rk=0x34 --prout-type=5 --device=/dev/sda
  HP        HSV300            0953
  Peripheral device type: disk

# sg_persist -r /dev/sda
  HP        HSV300            0953
  Peripheral device type: disk
  PR generation=0x329, Reservation follows:
    Key=0x34
    scope: LU_SCOPE,  type: Write Exclusive, registrants only

----------------------------------------------------------------------
From guest2, I was trying to do some I/O  on device expose using 2nd vHBA (pool_6 storage pool), it must be failed as key is registered with "guest1"
----------------------------------------------------------------------
 # sg_persist -r /dev/sda
  HP        HSV300            0953
  Peripheral device type: disk
  PR generation=0x329, Reservation follows:
    Key=0x34
    scope: LU_SCOPE,  type: Write Exclusive, registrants only

# sg_dd if=/dev/zero blk_sgio=1 count=1000 bs=512 of=/dev/sda
1000+0 records in
1000+0 records out


Issue with this look like, even device has correct key but other KVM guest can do I/O on device that may lead data corruption that must be fixed. This problem was not appearing if I use "block/lun" combination. 

Looks like in "volume/lun" combination persistent reservation is not happening as expected.

Ravi -

w/r/t Comment 26 and documenting - I think bug 1228007 is sufficient as a way to "document" that "--persistent" or "--config" don't work. I did update the title to help someone "searching" for it to "Using virsh attach-device fail to hotplug iscsi or fc lun volume with --config/--persistent option".

w/r/t Comment 27 - I have asked for testing to try this within an updated environment (eg, RHEL 7.2, recent QEMU, and libvirt).  If it still fails, then I'll clone this bug and get it triaged appropriately. If it succeeds, then we know something has been fixed in the next release, which I think should suffice.

Regardless of what happens with Comment 27, I believe I will end up duplicating this bug to bug 1228007 since the documentation issues are being handled separately. There are upstream patches on the list to clarify usage of the FC/NPIV device for a disk volume lun and another to move where the error message is displayed to help clarify which of the paths caused the failure to config/use a device as a lun.  If you're curious, see:

http://www.redhat.com/archives/libvir-list/2015-July/msg00705.html

(In reply to ravi.mishra from comment #27)
> I have exposed two paths using two vHBA to two different VM (storage side
> NPIV configuration is also done). To expose I have used below XMLs.
> 
> # cat sda_with_pool_5_scsi.xml
> <disk type='volume' device='lun'>
> <driver name='qemu' type='raw'/>
> <source pool='poolvhba5' volume='unit:0:0:1'/>
> <target dev='sda' bus='scsi'/>
> <shareable/>
> </disk>
> 
> # cat sda_with_pool_6_scsi.xml
> <disk type='volume' device='lun'>
> <driver name='qemu' type='raw'/>
> <source pool='poolvhba6' volume='unit:0:0:1'/>
> <target dev='sda' bus='scsi'/>
> <shareable/>
> </disk>
> 
> I have used below command to attach device to kVM guests.
> 
> # virsh attach-device --live guest1 disk_pool_guest1.xml
> virsh attach-device --live guest1 disk_pool_guest1.xml
> Device attached successfully
> 
> 
> # virsh attach-device --live guest2 disk_pool_guest2.xml
> virsh attach-device --live guest2 disk_pool_guest2.xml
> Device attached successfully
> 
> ----------------------------------------------------------------------
> From guest1, I have registered and reserved the key on device expose using
> 1st vHBA (pool_5 storage pool).
> ----------------------------------------------------------------------
> 
> # sg_persist --out --no-inquiry --register --param-sark=0x34 --device=/dev/sd
> # sg_persist -k /dev/sda
>   HP        HSV300            0953
>   Peripheral device type: disk
>   PR generation=0x41, 1 registered reservation key follows:
>     0x34
> # sg_persist -r /dev/sda
>   HP        HSV300            0953
>   Peripheral device type: disk
>   PR generation=0x41, there is NO reservation held
> 
> # /usr/bin/sg_persist --out --reserve --param-rk=0x34 --prout-type=5
> --device=/dev/sda
>   HP        HSV300            0953
>   Peripheral device type: disk
> 
> # sg_persist -r /dev/sda
>   HP        HSV300            0953
>   Peripheral device type: disk
>   PR generation=0x329, Reservation follows:
>     Key=0x34
>     scope: LU_SCOPE,  type: Write Exclusive, registrants only
> 
> ----------------------------------------------------------------------
> From guest2, I was trying to do some I/O  on device expose using 2nd vHBA
> (pool_6 storage pool), it must be failed as key is registered with "guest1"
> ----------------------------------------------------------------------
>  # sg_persist -r /dev/sda
>   HP        HSV300            0953
>   Peripheral device type: disk
>   PR generation=0x329, Reservation follows:
>     Key=0x34
>     scope: LU_SCOPE,  type: Write Exclusive, registrants only
> 
> # sg_dd if=/dev/zero blk_sgio=1 count=1000 bs=512 of=/dev/sda
> 1000+0 records in
> 1000+0 records out
> 
> 
> Issue with this look like, even device has correct key but other KVM guest
> can do I/O on device that may lead data corruption that must be fixed. This
> problem was not appearing if I use "block/lun" combination. 
> 
> Looks like in "volume/lun" combination persistent reservation is not
> happening as expected.

Hi Ravi, 
I'm trying to reproduce what you mentioned in comment 27. 
You mentioned "I have exposed two paths using two vHBA to two different VM"
you used 2 vhbas? so each vhab has different wwpn/wwnn number right? otherwise one vhba cannot be "online". 
So I guess you have 2 different wwpn/wwnn and set it to 2 different vHBAs, but both of them connected to the same backend FC lun. Is that so?

Yes you are correct, I have created two vHBAs and both of them connected to the same backend FC lun. 

In my case, I have one HBA name "host1" and created two vHBA "host10" and "host8", below is the details.

# systool -c fc_host -v |grep -e "Class Device" -e "node_name" -e "port_name" -e "Class Device path" -e "port_state" -e "port_type"
  Class Device = "host1"
  Class Device path = "/sys/devices/pci0000:00/0000:00:09.0/0000:08:00.0/host1/fc_host/host1"
    node_name           = "0x50014380120dc535"
    port_name           = "0x50014380120dc534"
    port_state          = "Online"
    port_type           = "NPort (fabric via point-to-point)"
  Class Device = "host10"
  Class Device path = "/sys/devices/pci0000:00/0000:00:09.0/0000:08:00.0/host1/vport-1:0-5/host10/fc_host/host10"
    node_name           = "0x50014380120dc543"
    port_name           = "0x50014380120dc542"
    port_state          = "Online"
    port_type           = "NPIV VPORT"
   Class Device = "host8"
  Class Device path = "/sys/devices/pci0000:00/0000:00:09.0/0000:08:00.0/host1/vport-1:0-3/host8/fc_host/host8"
    node_name           = "0x50014380120dc545"
    port_name           = "0x50014380120dc544"
    port_state          = "Online"
    port_type           = "NPIV VPORT"


Note: I have done the similar setup using Node device driver way (block/lun combination), in which I have used "source dev" as non-persistent device path name like "/dev/sdxx" etc (I know it is not recommended way), with sgio option to set as "unfiltered". Persistent reservation worked as expect.

But when I have used storage pool way (volume/lun combination) and instead of "source dev" start using "source pool" and "unit" with sgio option to set as "unfiltered", then Persistent reservation does not work which I have already explained in my previous comment.

(In reply to ravi.mishra from comment #31)
> Yes you are correct, I have created two vHBAs and both of them connected to
> the same backend FC lun. 
> 
> In my case, I have one HBA name "host1" and created two vHBA "host10" and
> "host8", below is the details.
> 
> # systool -c fc_host -v |grep -e "Class Device" -e "node_name" -e
> "port_name" -e "Class Device path" -e "port_state" -e "port_type"
>   Class Device = "host1"
>   Class Device path =
> "/sys/devices/pci0000:00/0000:00:09.0/0000:08:00.0/host1/fc_host/host1"
>     node_name           = "0x50014380120dc535"
>     port_name           = "0x50014380120dc534"
>     port_state          = "Online"
>     port_type           = "NPort (fabric via point-to-point)"
>   Class Device = "host10"
>   Class Device path =
> "/sys/devices/pci0000:00/0000:00:09.0/0000:08:00.0/host1/vport-1:0-5/host10/
> fc_host/host10"
>     node_name           = "0x50014380120dc543"
>     port_name           = "0x50014380120dc542"
>     port_state          = "Online"
>     port_type           = "NPIV VPORT"
>    Class Device = "host8"
>   Class Device path =
> "/sys/devices/pci0000:00/0000:00:09.0/0000:08:00.0/host1/vport-1:0-3/host8/
> fc_host/host8"
>     node_name           = "0x50014380120dc545"
>     port_name           = "0x50014380120dc544"
>     port_state          = "Online"
>     port_type           = "NPIV VPORT"
> 
> 
> Note: I have done the similar setup using Node device driver way (block/lun
> combination), in which I have used "source dev" as non-persistent device
> path name like "/dev/sdxx" etc (I know it is not recommended way), with sgio
> option to set as "unfiltered". Persistent reservation worked as expect.
> 
> But when I have used storage pool way (volume/lun combination) and instead
> of "source dev" start using "source pool" and "unit" with sgio option to set
> as "unfiltered", then Persistent reservation does not work which I have
> already explained in my previous comment.

Thanks for the confirmation, that's clear. I'll have a try ASAP. cause I do not have experience/access to FC SAN storage side operation (it's maintained by IT team. I'll need their help during the test) 
Do I need to specify something to IT guys? 
Or only ask them to mapping two wwnn/wwpns to the same LUN and that's all?
Thanks for your patience.

You have to tell IT team that you want storage zoing and vHBA's wwnn and wwpn must be inserted into switch along with storage array's wwn. Like below example.

#:admin> zoneshow host4*

zone:  host4ibtest4
                50:01:43:80:12:0d:c5:34; 50:01:43:80:12:0d:c5:35;
                50:01:43:80:11:36:2a:d8
zone:  host4vp1zone
                50:01:43:80:12:0d:c5:42; 50:01:43:80:12:0d:c5:43;
                50:01:43:80:12:0d:c5:34; 50:01:43:80:12:0d:c5:35;
                50:01:43:80:11:36:2a:d8
zone:  host4vp2zone
                50:01:43:80:12:0d:c5:44; 50:01:43:80:12:0d:c5:45;
                50:01:43:80:12:0d:c5:34; 50:01:43:80:12:0d:c5:35;
                50:01:43:80:11:36:2a:d8


In above example: 

phy HBA's wwnn: 50:01:43:80:12:0d:c5:34
phy HBA's wwpn: 50:01:43:80:12:0d:c5:35

1st vHBA wwnn: 50:01:43:80:12:0d:c5:42
1st vHBA wwpn: 50:01:43:80:12:0d:c5:43

2nd vHBA wwnn: 50:01:43:80:12:0d:c5:44
2nd vHBA wwnn: 50:01:43:80:12:0d:c5:45

Storage Array's wwn: 50:01:43:80:11:36:2a:d8


Apart from this all switches and storage array must be NPIV capable. 
Please cross check the support matrix before configure.

jsut inform you guys there is a kernel bug blocks craeting vhba, bz 1240912
I need to reproduce it on recent kernel anyway. So will waiting for it's fixed.

I am working on reproducing comment 27 today, but it's not reproducible with:
HOST versions -
kernel-3.10.0-316.el7.x86_64
libvirt-1.2.17-8.el7.x86_64
sg3_utils-1.37-5.el7.x86_64

GUEST versions -
kernel-3.10.0-295.el7.x86_64
sg3_utils-1.37-5.el7.x86_64

=====
as https://bugzilla.redhat.com/show_bug.cgi?id=1254316#c2 talked, sg_persist doesn't work well with mpath pool. That should be indicated here. 

During my test, the problem is not reproduced. When VM1 reserves the lun, VM2 cannot write to it. 

I used 2 pools each has 1 volume, both of the volumes pointing to a same backend fc lun. 

pools' xml as follow:

pool1 - 
<pool type='scsi'>
  <name>npiv_pool1</name>
  <uuid>75fa4b39-5e96-4574-b289-799f0b4b78ee</uuid>
  <capacity unit='bytes'>10737418240</capacity>
  <allocation unit='bytes'>10737418240</allocation>
  <available unit='bytes'>0</available>
  <source>
    <adapter type='fc_host' parent='scsi_host5' managed='yes' wwnn='2101001b32a9da4e' wwpn='2101001b32a90000'/>
  </source>
  <target>
    <path>/dev</path>
    <permissions>
      <mode>0755</mode>
    </permissions>
  </target>
</pool>

pool2 -
<pool type='scsi'>
  <name>npiv_pool2</name>
  <uuid>2a52d736-63cf-4d79-bda5-ff768f48d294</uuid>
  <capacity unit='bytes'>10737418240</capacity>
  <allocation unit='bytes'>10737418240</allocation>
  <available unit='bytes'>0</available>
  <source>
    <adapter type='fc_host' parent='scsi_host5' managed='yes' wwnn='2101001b32a9da4e' wwpn='2101001b32a90001'/>
  </source>
  <target>
    <path>/dev</path>
    <permissions>
      <mode>0755</mode>
    </permissions>
  </target>
</pool>


disks' xml in vms

VM1
    <disk type='volume' device='lun'>
      <driver name='qemu' type='raw'/>
      <source pool='npiv_pool1' volume='unit:0:2:0'/>
      <backingStore/>
      <target dev='sdb' bus='scsi'/>
      <shareable/>
      <alias name='scsi0-0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>

VM2
    <disk type='volume' device='lun'>
      <driver name='qemu' type='raw'/>
      <source pool='npiv_pool2' volume='unit:0:0:0'/>
      <backingStore/>
      <target dev='sdb' bus='scsi'/>
      <shareable/>
      <alias name='scsi0-0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>


start both vms

In VM1:

# sg_persist --out --no-inquiry --register --param-sark=0x34 --device=/dev/sdb
# sg_persist -k /dev/sdb
  IBM       1726-4xx  FAStT   0617
  Peripheral device type: disk
  PR generation=0x3, 1 registered reservation key follows:
    0x34

# sg_persist -r /dev/sdb
  HP        HSV300            0953
  Peripheral device type: disk
  PR generation=0x41, there is NO reservation held

# sg_persist --out --reserve --param-rk=0x34 --prout-type=5 --device=/dev/sdb

# sg_persist -r /dev/sdb
  IBM       1726-4xx  FAStT   0617
  Peripheral device type: disk
  PR generation=0x3, Reservation follows:
    Key=0x34
    scope: LU_SCOPE,  type: Write Exclusive, registrants only


#  sg_dd if=/dev/zero blk_sgio=1 count=1000 bs=512 of=/dev/sdb
1000+0 records in
1000+0 records out
<==== io operation correctly as expected 


In VM2:
# sg_persist -r /dev/sdb
  IBM       1726-4xx  FAStT   0617
  Peripheral device type: disk
  PR generation=0x3, Reservation follows:
    Key=0x34
    scope: LU_SCOPE,  type: Write Exclusive, registrants only

# sg_dd if=/dev/zero blk_sgio=1 count=1000 bs=512 of=/dev/sdb
writing: SCSI status: Check Condition 
 Fixed format, current;  Sense key: Aborted Command
 Additional sense: I/O process terminated
Aborted command, continuing (w)
writing: SCSI status: Check Condition 
 Fixed format, current;  Sense key: Aborted Command
 Additional sense: I/O process terminated
sg_write failed, seek=0
Some error occurred,  remaining block count=1000
128+0 records in
0+0 records out

<== IO operation failed as expected.

And test with <hostdev> it's not reproduced either. 

VM1 xml:
...
    <hostdev mode='subsystem' type='scsi' managed='no'>
      <source>
        <adapter name='scsi_host8'/>
        <address bus='0' target='2' unit='0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </hostdev>


VM2 xml:
...
    <hostdev mode='subsystem' type='scsi' managed='no'>
      <source>
        <adapter name='scsi_host9'/>
        <address bus='0' target='0' unit='0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </hostdev>


login VM1:
# sg_dd if=/dev/zero blk_sgio=1 count=1000 bs=512 of=/dev/sdb
1000+0 records in
1000+0 records out

login VM2:
# sg_dd if=/dev/zero blk_sgio=1 count=1000 bs=512 of=/dev/sdb
writing: SCSI status: Reservation Conflict 
sg_write failed, seek=0
Some error occurred,  remaining block count=1000
128+0 records in
0+0 records out
1 unrecovered error(s)

Hi John,

I would like to know that both VM1 and VM2 are hosted on same Host system or residing on two different host. 

If you have created pools on two different host and exposed to their respective VMs then behavior is expected and correct. 

But I am interested in a case where both pools created on same host and exposed to both VMs (having same host), and backed FC disks has been configured with NPIV.

Please give me clarity on above points.

w/r/t Comment 39 regarding mpath:

mpath is a totally separate issue and I don't believe is relevant to what HP was looking for here. What I'm not clear on since I don't know that much about mpath is whether those paths are persistent between reboots - if not, certainly doesn't fit the bill HP was requiring. In any case, mpath is tracked elsewhere and this bug was more related to block/lun and volume/lun using FC/SCSI storage.

w/r/t Comment 41 - I did not perform the testing, but as I read Comment 39 I assumed the testing was done on the same host. This was the one piece I was holding out on duplicating this bz to bug 1228007 since that was the root cause of libvirt command failure. Even though there was a lot of other activity, other bz's were generated and handled separately.  

In any case, I think the test results are *good news* - there are so many variables at play here. Thanks for taking the time to test and the persistence regarding the kernel issue that crept up. Thanks also to Ravi for his persistence in making sure things not only worked correct, but were better documented.

Before I close/clone, I'll wait for my assumption regarding testing being done on one test to be validate (eg, the needsinfo...)

yes, I used one host to create vhbas, pools and vms.

Duplicating to bug 1228007 since that resolves the original problem which was that the lun couldn't be attached (the key being that use of --persistent had an issue).  Everything else this bug spawned was still good...

# virsh attach-device --persistent guest1 sda1.xml
error: Failed to attach device from sda1.xml
error: unsupported configuration: disk device='lun' is only valid for block type disk source
#

*** This bug has been marked as a duplicate of bug 1228007 ***

(In reply to yisun from comment #39)
> I am working on reproducing comment 27 today, but it's not reproducible with:
> HOST versions -
> kernel-3.10.0-316.el7.x86_64
> libvirt-1.2.17-8.el7.x86_64
> sg3_utils-1.37-5.el7.x86_64
> 
> GUEST versions -
> kernel-3.10.0-295.el7.x86_64
> sg3_utils-1.37-5.el7.x86_64
> 
> =====
> as https://bugzilla.redhat.com/show_bug.cgi?id=1254316#c2 talked, sg_persist
> doesn't work well with mpath pool. That should be indicated here. 
> 
> During my test, the problem is not reproduced. When VM1 reserves the lun,
> VM2 cannot write to it. 
> 
> I used 2 pools each has 1 volume, both of the volumes pointing to a same
> backend fc lun. 
> 
> pools' xml as follow:
> 
> pool1 - 
> <pool type='scsi'>
>   <name>npiv_pool1</name>
>   <uuid>75fa4b39-5e96-4574-b289-799f0b4b78ee</uuid>
>   <capacity unit='bytes'>10737418240</capacity>
>   <allocation unit='bytes'>10737418240</allocation>
>   <available unit='bytes'>0</available>
>   <source>
>     <adapter type='fc_host' parent='scsi_host5' managed='yes'
> wwnn='2101001b32a9da4e' wwpn='2101001b32a90000'/>
>   </source>
>   <target>
>     <path>/dev</path>
>     <permissions>
>       <mode>0755</mode>
>     </permissions>
>   </target>
> </pool>
> 
> pool2 -
> <pool type='scsi'>
>   <name>npiv_pool2</name>
>   <uuid>2a52d736-63cf-4d79-bda5-ff768f48d294</uuid>
>   <capacity unit='bytes'>10737418240</capacity>
>   <allocation unit='bytes'>10737418240</allocation>
>   <available unit='bytes'>0</available>
>   <source>
>     <adapter type='fc_host' parent='scsi_host5' managed='yes'
> wwnn='2101001b32a9da4e' wwpn='2101001b32a90001'/>
>   </source>
>   <target>
>     <path>/dev</path>
>     <permissions>
>       <mode>0755</mode>
>     </permissions>
>   </target>
> </pool>
> 
> 
> disks' xml in vms
> 
> VM1
>     <disk type='volume' device='lun'>
>       <driver name='qemu' type='raw'/>
>       <source pool='npiv_pool1' volume='unit:0:2:0'/>
>       <backingStore/>
>       <target dev='sdb' bus='scsi'/>
>       <shareable/>
>       <alias name='scsi0-0-0-0'/>
>       <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>     </disk>
> 
> VM2
>     <disk type='volume' device='lun'>
>       <driver name='qemu' type='raw'/>
>       <source pool='npiv_pool2' volume='unit:0:0:0'/>
>       <backingStore/>
>       <target dev='sdb' bus='scsi'/>
>       <shareable/>
>       <alias name='scsi0-0-0-0'/>
>       <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>     </disk>
> 
> 
> start both vms
> 
> In VM1:
> 
> # sg_persist --out --no-inquiry --register --param-sark=0x34
> --device=/dev/sdb
> # sg_persist -k /dev/sdb
>   IBM       1726-4xx  FAStT   0617
>   Peripheral device type: disk
>   PR generation=0x3, 1 registered reservation key follows:
>     0x34
> 
> # sg_persist -r /dev/sdb
>   HP        HSV300            0953
>   Peripheral device type: disk
>   PR generation=0x41, there is NO reservation held
> 
> # sg_persist --out --reserve --param-rk=0x34 --prout-type=5 --device=/dev/sdb
> 
> # sg_persist -r /dev/sdb
>   IBM       1726-4xx  FAStT   0617
>   Peripheral device type: disk
>   PR generation=0x3, Reservation follows:
>     Key=0x34
>     scope: LU_SCOPE,  type: Write Exclusive, registrants only
> 
> 
> #  sg_dd if=/dev/zero blk_sgio=1 count=1000 bs=512 of=/dev/sdb
> 1000+0 records in
> 1000+0 records out
> <==== io operation correctly as expected 
> 
> 
> In VM2:
> # sg_persist -r /dev/sdb
>   IBM       1726-4xx  FAStT   0617
>   Peripheral device type: disk
>   PR generation=0x3, Reservation follows:
>     Key=0x34
>     scope: LU_SCOPE,  type: Write Exclusive, registrants only
> 
> # sg_dd if=/dev/zero blk_sgio=1 count=1000 bs=512 of=/dev/sdb
> writing: SCSI status: Check Condition 
>  Fixed format, current;  Sense key: Aborted Command
>  Additional sense: I/O process terminated
> Aborted command, continuing (w)
> writing: SCSI status: Check Condition 
>  Fixed format, current;  Sense key: Aborted Command
>  Additional sense: I/O process terminated
> sg_write failed, seek=0
> Some error occurred,  remaining block count=1000
> 128+0 records in
> 0+0 records out
> 
> <== IO operation failed as expected.

Hi,

Sorry I have reopened this bug but I am not able to reproduce as you have added as test log with below combination, it is like solution will work only if guest and host both OS has Redhat 7.X OS version, if yes please let me know, will this work for RHEL 7.0 OS as host and guest OS:

Curently i am using host OS as RHEL7.2 and two Guest OS of RHEL6.5 and i am seeing even after doing reservation by one VM, second VM is also able to do I/O which is unexpected.

HOST versions -
kernel-3.10.0-302.el7.x86_64
libvirt-1.2.17-3.el7.x86_64
sg3_utils-1.37-5.el7.x86_64
OS version: 
NAME="Red Hat Enterprise Linux Server"
VERSION="7.2 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="7.2"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.2 Beta (Maipo)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.2:beta:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.2
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.2 Beta"

GUEST versions -
kernel-2.6.32-431.el6.x86_64
sg3_utils-1.28-5.el6.x86_64
OS version:
Red Hat Enterprise Linux Server release 6.5 (Santiago)

-Ravi

The question raised here has nothing to do with the original query nor what the duplicated to bug. 

The "proper mechanic" should have been to open a new bz to ask your question. I am marking this one closed as duplicate again.

As far as I'm concerned, there isn't an "easy answer" to the question though as the persistent reservation feature is based on specific RHEL kernel code being present for 'sg_persist' to work right. Also based on another bug (1210832), it's not working as expected on/in a device mapper environment (that bug is marked private so you probably won't be able to read it).  As I dig further through that I tripped across bz 841577 which indicates some underlying function is "fixed" as of kernel-2.6.32-345.el6. Of course your kernel is later than that one, but doesn't mean there weren't other issues.

If you want to pursue this further, a new bz would need to be created including the details of your host and guest environments. With enough information someone from the RHEL6 kernel team should be able to answer whether or not the feature you are attempting to use will work on your referenced kernel.

*** This bug has been marked as a duplicate of bug 1228007 ***