Bug 1000159

Summary: Unable to find iSCSI passthrough lun
Product: Red Hat Enterprise Linux 7 Reporter: John Ferlan <jferlan>
Component: libvirtAssignee: John Ferlan <jferlan>
Status: CLOSED CURRENTRELEASE QA Contact: Virtualization Bugs <virt-bugs>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.0CC: acathrow, chhu, cwei, dyuan, jferlan, pbonzini, weizhan
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-1.1.1-3.el7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-06-13 12:39:38 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 923843    
Bug Blocks:    

Description John Ferlan 2013-08-22 20:20:40 UTC
Description of problem:

Commit id '1f49b05a' added 'virDomainDiskSourceIsBlockType', but the change to call the method from qemuSetUnprivSGIO() missed the '!'


Steps to Reproduce:

Assuming proper setup of iSCSI with the following in the targets.conf file:

<target iqn.2013-07.com.example:iscsi>
backing-store /home/tgtd/iscsi/disk1
backing-store /home/tgtd/iscsi/disk2
incominguser libvirt redhat
</target>


1.  The following xml example would exhibit the problem

       <disk type='network' device='lun'>
          <driver name='qemu' type='raw'/>
          <source protocol='iscsi' name='iqn.2013-07.com.example:iscsi/1'>
            <host name='example.com' port='3260'/>
          </source>
          <target dev='sda' bus='scsi'/>
        </disk>


2. The attempt to start the domain would fail with:

   error: Failed to start domain iscsilun
   error: Unable to get device ID 'iqn.2013-07.com.example:iscsi/1': No such fi
    


Expected results:

start should succeed

Comment 3 chhu 2013-09-10 09:10:19 UTC
Tried to verify this bug with packages:
libvirt-1.1.1-4.el7.x86_64
qemu-kvm-1.5.3-2.el7.x86_64
kernel-3.10.0-14.el7.x86_64

Test steps:
1. setup of iSCSI with the following in the targets.conf file on machine A:

<target iqn.2013-10.66.7.108>
    backing-store /dev/sdc
</target>
# service tgtd start

2. access to iscsi on another test machine B successfully.
# service iscsid status
# iscsiadm --mode discovery --type sendtargets --portal 10.66.7.108
10.66.7.108:3260,1 iqn.2013-10.66.7.108

# iscsiadm -m node -T iqn.2013-10.66.7.108 -p 10.66.7.108:3260 --login
Logging in to [iface: default, target: iqn.2013-10.66.7.108, portal: 10.66.7.108,3260] (multiple)
Login to [iface: default, target: iqn.2013-10.66.7.108, portal: 10.66.7.108,3260] successful.

# iscsiadm -m node -T iqn.2013-10.66.7.108 -p 10.66.7.108:3260 --logout
Logging out of session [sid: 7, target: iqn.2013-10.66.7.108, portal: 10.66.7.108,3260]

3. start a domain on machine B.
# virsh define rh7-qcow2.xml
# virsh start rh7-qcow2
# virsh edit rh7-qcow2
add below xml:
    <disk type='network' device='lun'>
      <driver name='qemu' type='raw'/>
      <source protocol='iscsi' name='iqn.2013-10.66.7.108/0'>
        <host name='10.66.7.108' port='3260'/>
      </source>
      <target dev='sda' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
# virsh destroy rh7-qcow2

4. try to start the domain
# virsh start rh7-qcow2
error: Failed to start domain rh7-qcow2
error: internal error: process exited while connecting to monitor: char device redirected to /dev/pts/3 (label charserial0)
qemu-kvm: -drive file=iscsi://10.66.7.108:3260/iqn.2013-10.66.7.108/0,if=none,id=drive-scsi0-0-0-0,format=raw: could not open disk image iscsi://10.66.7.108:3260/iqn.2013-10.66.7.108/0: Operation not supported

This error is depends On bug:923843, right ?

related bug:
Bug 923843 - include userspace iSCSI initiator in block driver whitelist
Bug 957295 - support direct connection to an iSCSI LUN using pool/volume sources

Comment 4 John Ferlan 2013-09-10 17:28:55 UTC
I do not believe the error you are seeing is a result of the libiscsi bug which I believe is referenced by those BZ's, but I'm not 100% sure either.  The one issue I'm aware of that is fixed by changes to libiscsi result from a "%3A" being placed into the resulting name rather than a ":".

I would say my hangup deals with the missing "Optional ":" prefixing a storage target name specified by the naming authority." from the IQN as described by https://en.wikipedia.org/wiki/ISCSI.

That is your targets.conf has:

<target iqn.2013-10.66.7.108>

while mine has

<target iqn.2013-07.com.example:iscsi>

The difference being the ":iscsi" that I have on my name.  Whether that's something the mentioned BZ's resolve - I have no idea.

Can you at least try adding it and see if you get different results?  If the results are having a "%3a" inserted into the name passed to kvm and rather than colon right at that particular spot and an error as a result of that, then I'd be more inclined to believe they are at least related. That is, perhaps not having the ":" has caused grief to the existing code... If that works, then perhaps Paolo can pinged to confirm the findings.

Another suggestion (from my email folder of tricks) would be to install 'iscsi-utils' on machine B and try running 'iscsi-ls -s iscsi://10.66.7.108:3260' to further make sure you can see the storage (assuming that 10.66.7.108 is machine A's IP address).

Comment 5 chhu 2013-09-11 02:39:14 UTC
(In reply to John Ferlan from comment #4)
> I do not believe the error you are seeing is a result of the libiscsi bug
> which I believe is referenced by those BZ's, but I'm not 100% sure either. 
> The one issue I'm aware of that is fixed by changes to libiscsi result from
> a "%3A" being placed into the resulting name rather than a ":".
> 
> I would say my hangup deals with the missing "Optional ":" prefixing a
> storage target name specified by the naming authority." from the IQN as
> described by https://en.wikipedia.org/wiki/ISCSI.
> 
> That is your targets.conf has:
> 
> <target iqn.2013-10.66.7.108>
> 
> while mine has
> 
> <target iqn.2013-07.com.example:iscsi>
> 
> The difference being the ":iscsi" that I have on my name.  Whether that's
> something the mentioned BZ's resolve - I have no idea.
> 
> Can you at least try adding it and see if you get different results?  If the
> results are having a "%3a" inserted into the name passed to kvm and rather
> than colon right at that particular spot and an error as a result of that,
> then I'd be more inclined to believe they are at least related. That is,
> perhaps not having the ":" has caused grief to the existing code... If that
> works, then perhaps Paolo can pinged to confirm the findings.
> 

Thanks John!

Yes, the results are having a "%3a" inserted into the name if I add ":", so I didn't use ":" to avoid the "%3a" issue.

# virsh start rh7-qcow2
error: Failed to start domain rh7-qcow2
error: internal error: process exited while connecting to monitor: char device redirected to /dev/pts/3 (label charserial0)
qemu-kvm: -drive file=iscsi://10.66.7.108:3260/iqn.2013-10.com.example%3Aiscsi/0,if=none,id=drive-scsi0-0-0-0,format=raw: could not open disk image iscsi://10.66.7.108:3260/iqn.2013-10.com.example%3Aiscsi/0: Operation not supported

More details:
1. setup of iSCSI with the following in the targets.conf file on machine A:

<target iqn.2013-10.com.example:iscsi>
     backing-store dev/sdb
</target>
# service tgtd restart

# tgt-admin --show
Target 1: iqn.2013-10.com.example:iscsi
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
    LUN information:
        LUN: 0
            Type: controller
            SCSI ID: IET     00010000
            SCSI SN: beaf10
.......

2. access to iscsi on another test machine B successfully. (I use iscsiadm here but not iscsi-ls, as I have not found iscsi-utils package for rhel yet)

# iscsiadm --mode discovery --type sendtargets --portal 10.66.7.108
10.66.7.108:3260,1 iqn.2013-10.com.example:iscsi

3. start a domain on machine B with below lines in the xml:
    <disk type='network' device='lun'>
      <driver name='qemu' type='raw'/>
      <source protocol='iscsi' name='iqn.2013-10.com.example:iscsi/0'>
        <host name='10.66.7.108' port='3260'/>
      </source>
      <target dev='sda' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>

# virsh start rh7-qcow2
error: Failed to start domain rh7-qcow2
error: internal error: process exited while connecting to monitor: char device redirected to /dev/pts/3 (label charserial0)
qemu-kvm: -drive file=iscsi://10.66.7.108:3260/iqn.2013-10.com.example%3Aiscsi/0,if=none,id=drive-scsi0-0-0-0,format=raw: could not open disk image iscsi://10.66.7.108:3260/iqn.2013-10.com.example%3Aiscsi/0: Operation not supported

> Another suggestion (from my email folder of tricks) would be to install
> 'iscsi-utils' on machine B and try running 'iscsi-ls -s
> iscsi://10.66.7.108:3260' to further make sure you can see the storage
> (assuming that 10.66.7.108 is machine A's IP address).

Comment 6 John Ferlan 2013-09-11 10:57:02 UTC
Not quite sure why there was a needinfo left on me, but if you want the iscsi-ls tool, find the package 'libiscsi-utils' for RHEL:

# yum whatprovides iscsi-ls
Loaded plugins: auto-update-debuginfo, langpacks, refresh-packagekit
libiscsi-utils-1.7.0-4.fc19.x86_64 : iSCSI Client Utilities
Repo        : fedora
Matched from:
Filename    : /usr/bin/iscsi-ls



libiscsi-utils-1.7.0-5.fc19.x86_64 : iSCSI Client Utilities
Repo        : updates
Matched from:
Filename    : /usr/bin/iscsi-ls
...


In any case, as I understand the idea behind at least one of the bz's mentioned is to ensure the latest libiscsi changes are included in RHEL7. 

I have to assume there was more than one issue in that parsing code, but I haven't followed it that closely either.

Comment 7 chhu 2013-09-12 09:15:56 UTC
Thanks John!

Just update my latest test result here, will update it when the latest libiscsi changes are included in RHEL7.

Tested with packages:
libvirt-1.1.1-4.el7.x86_64
qemu-kvm-1.5.3-2.el7.x86_64
kernel-3.10.0-14.el7.x86_64
libiscsi-1.7.0-6.el7.x86_64
libiscsi-utils-1.7.0-6.el7.x86_64

1. setup of iSCSI with the following in the targets.conf file on machine A:

<target iqn.2013-10.com.example:iscsi>
     backing-store dev/sdb
</target>
# service tgtd restart

# tgt-admin --show
Target 1: iqn.2013-10.com.example:iscsi
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
    LUN information:
        LUN: 0
            Type: controller
            SCSI ID: IET     00010000
            SCSI SN: beaf10
......

2. make sure the storage can be seen on machine B:
# iscsi-ls -s iscsi://10.66.7.108:3260
Target:iqn.2013-10.com.example:iscsi Portal:10.66.7.108:3260,1
Lun:0    Type:STORAGE_ARRAY_CONTROLLER

3. start a domain on machine B with below lines in the xml:
    <disk type='network' device='lun'>
      <driver name='qemu' type='raw'/>
      <source protocol='iscsi' name='iqn.2013-10.com.example:iscsi/0'>
        <host name='10.66.7.108' port='3260'/>
      </source>
      <target dev='sda' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>

# virsh start rh7-qcow2
error: Failed to start domain rh7-qcow2
error: internal error: process exited while connecting to monitor: char device redirected to /dev/pts/1 (label charserial0)
qemu-kvm: -drive file=iscsi://10.66.7.108:3260/iqn.2013-10.com.example%3Aiscsi/0,if=none,id=drive-scsi0-0-0-0,format=raw: could not open disk image iscsi://10.66.7.108:3260/iqn.2013-10.com.example%3Aiscsi/0: Operation not supported

Comment 8 chhu 2013-09-16 06:23:37 UTC
Hi, Paolo

As the test results described in comment 7, the results are having a "%3a" inserted into the name passed to kvm. Do you know which version of libiscsi package will solve the "%3a" issue ?

Comment 9 chhu 2013-09-16 06:32:35 UTC
(In reply to chhu from comment #8)
> Hi, Paolo
> 
> As the test results described in comment 7, the results are having a "%3a"
> inserted into the name passed to kvm. Do you know which version of libiscsi
> package will solve the "%3a" issue ?

After the bug below is fixed ?
Bug 957293 - support libiscsi for SCSI passthrough devices
https://bugzilla.redhat.com/show_bug.cgi?id=957293

Comment 10 Paolo Bonzini 2013-09-16 13:35:45 UTC
The libiscsi package already has this bug fixed since libiscsi-1.7.0-6.el7.

The "not supported" error is due to bug 923843.

Comment 12 chhu 2013-09-22 02:32:56 UTC
According to comment 7,8,10. There is still "not supported" error due to bug 923843.

Comment 13 chhu 2013-10-29 08:54:13 UTC
Verified with packages:
libvirt-1.1.1-10.el7.x86_64
qemu-kvm-1.5.3-10.el7.x86_64

Test steps:
1. create a domain with xml below:
    <disk type='network' device='lun'>
      <driver name='qemu' type='raw'/>
      <source protocol='iscsi' name='iqn.2013-10.com.example:iscsi/1'>
        <host name='hostname' port='3260'/>
      </source>
      <target dev='sda' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>

# virsh create r7-qcow2.xml
Domain r7-qcow2 created from r7-qcow2.xml

# ps -ef|grep qemu-kvm| grep iscsi
qemu      7033     1  3 16:41 ?        00:00:17 /usr/libexec/qemu-kvm -name r7-qcow2 -S -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off -m 1024
.......
-drive file=iscsi://10.66.7.108:3260/iqn.2013-10.com.example%3Aiscsi/1,if=none,id=drive-scsi0-0-0-0,format=raw
......

2. login to the guest can mount/read/write the iscsi disk

Test results:
current command work well, change the bug status to verified.

Comment 14 chhu 2013-10-30 01:41:53 UTC
Test with package:
libiscsi-1.9.0-2.el7.x86_64

The %3A problem is still existed. As it's the problem about libiscsi, according to comment10, it should be fixed in libiscsi-1.7.0-6.el7, so update the information here.

# ps -ef|grep qemu-kvm| grep iscsi
qemu      7033     1  3 16:41 ?        00:00:17 /usr/libexec/qemu-kvm -name r7-qcow2 -S -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off -m 1024
.......
-drive file=iscsi://10.66.7.108:3260/iqn.2013-10.com.example%3Aiscsi/1,if=none,id=drive-scsi0-0-0-0,format=raw

Comment 15 Ludek Smid 2014-06-13 12:39:38 UTC
This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.