Bug 1172015

Summary: Failed to start multi-iqn based storage pool
Product: Red Hat Enterprise Linux 7 Reporter: Yang Yang <yanyang>
Component: libvirtAssignee: John Ferlan <jferlan>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 7.1CC: dyuan, jferlan, rbalakri, tzheng, xuzhang, yanyang
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: libvirt-1.2.13-1.el7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-11-19 05:57:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
libvirtd debug log none

Description Yang Yang 2014-12-09 08:59:14 UTC
Created attachment 966147 [details]
libvirtd debug log

Description of problem:
Multi-iqn based storage pool startup failed as failure against updating target's db with the new interface.

Version-Release number of selected component (if applicable):
libvirt-1.2.8-10.el7.x86_64
3.10.0-212.el7.x86_64
iscsi-initiator-utils-6.2.0.873-24.el7.x86_64
scsi-target-utils-1.0.46-3.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.setup an iscsi server
# tgtadm --lld iscsi --mode target --op show
Target 2: iqn.yy:server.host1
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
    LUN information:
        LUN: 0
            Type: controller
            SCSI ID: IET     00020000
            SCSI SN: beaf20
            Size: 0 MB, Block size: 1
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            SWP: No
            Thin-provisioning: No
            Backing store type: null
            Backing store path: None
            Backing store flags: 
        LUN: 1
            Type: disk
            SCSI ID: IET     00020001
            SCSI SN: beaf21
            Size: 105 MB, Block size: 512
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: Yes
            SWP: No
            Thin-provisioning: No
            Backing store type: rdwr
            Backing store path: /tmp/iscsi1
            Backing store flags: 
    Account information:
    ACL information:
        ALL
Target 3: iqn.yy:server.host2
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
    LUN information:
        LUN: 0
            Type: controller
            SCSI ID: IET     00030000
            SCSI SN: beaf30
            Size: 0 MB, Block size: 1
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            SWP: No
            Thin-provisioning: No
            Backing store type: null
            Backing store path: None
            Backing store flags: 
        LUN: 1
            Type: disk
            SCSI ID: IET     00030001
            SCSI SN: beaf31
            Size: 105 MB, Block size: 512
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: Yes
            SWP: No
            Thin-provisioning: No
            Backing store type: rdwr
            Backing store path: /tmp/iscsi2
            Backing store flags: 
    Account information:
    ACL information:
        ALL

2. discover target, check iface and session

# iscsiadm -m iface
default tcp,<empty>,<empty>,<empty>,<empty>
iser iser,<empty>,<empty>,<empty>,<empty>

# iscsiadm -m session
iscsiadm: No active sessions.


3. create pool with the following xml

# cat multiiqn.xml 
<pool type="iscsi">
  <name>multiiqn</name>
<source>
    <host name="$ip"/>
    <device path="iqn.yy:server.host1"/>
<initiator>
    <iqn name="iqn.yy:server.host2"/>
    </initiator>
    </source>  
<target>
  <path>/dev/disk/by-path</path>
  </target>
</pool>

# virsh pool-create multiiqn.xml 
error: Failed to create pool from scsi-pool.xml
error: internal error: Child process (iscsiadm --mode node --portal 10.66.4.201:3260,1 --targetname iqn.yy:server.host1 --login --interface libvirt-iface-11be4e1f) unexpected exit status 21: 2014-12-09 07:26:25.380+0000: 3401: debug : virFileClose:99 : Closed fd 24
2014-12-09 07:26:25.380+0000: 3401: debug : virFileClose:99 : Closed fd 26
2014-12-09 07:26:25.380+0000: 3401: debug : virFileClose:99 : Closed fd 22
iscsiadm: No records found

4. check iface again
# iscsiadm -m iface
default tcp,<empty>,<empty>,<empty>,<empty>
iser iser,<empty>,<empty>,<empty>,<empty>
libvirt-iface-11be4e1f tcp,<empty>,<empty>,<empty>,iqn.yy:server.host2


Actual results:
Multi-iqn based pool creation failed

Expected results:


Additional info:
libvirt debug log
2014-12-09 07:26:25.374+0000: 1114: error : virCommandWait:2522 : internal error: Child process (iscsiadm --mode node --portal 10.66.4.201:3260,1 --targetname iqn.yy:server.host1 --op update --name node.startup --value manual --interface libvirt-iface-11be4e1f) unexpected exit status 21: 2014-12-09 07:26:25.373+0000: 3397: debug : virFileClose:99 : Closed fd 25

2014-12-09 07:26:25.378+0000: 1114: error : virCommandWait:2522 : internal error: Child process (iscsiadm --mode node --portal 10.66.4.201:3260,1 --targetname iqn.yy:server.host2 --op update --name node.startup --value manual --interface libvirt-iface-11be4e1f) unexpected exit status 21: 2014-12-09 07:26:25.376+0000: 3399: debug : virFileClose:99 : Closed fd 25
2014-12-09 07:26:25.376+0000: 3399: debug : virFileClose:99 : Closed fd 27
2014-12-09 07:26:25.376+0000: 3399: debug : virFileClose:99 : Closed fd 22

Comment 2 John Ferlan 2014-12-09 23:24:35 UTC
After quite a bit of research, it seems commit id '59446096' refactored things just enough to change the order of the


    const char *const cmdsendtarget[] = {
        ISCSIADM, "--mode", "discovery", "--type", "sendtargets",
        "--portal", portal, NULL

To occur before the :

     const char *const cmdargv1[] = {
         ISCSIADM, "--mode", "iface", "--interface",
         &temp_ifacename[0], "--op", "new", NULL
     };

and

     const char *const cmdargv2[] = {
         ISCSIADM, "--mode", "iface", "--interface", &temp_ifacename[0],
         "--op", "update", "--name", "iface.initiatorname", "--value",
         initiatoriqn, NULL
     };

which causes any subsequent action to fail, such as --login or the virISCSITargetAutologin code which disables autologin.


Working up a patch now for this issue for upstream.

Comment 3 John Ferlan 2014-12-10 00:06:54 UTC
This bug has been around since libvirt v0.8.6 when the refactoring occurred.

So given that and given that it hasn't been noticed until now, I'm inclined to move the bug to rhel 7.2.  It just doesn't seem this is a highly sought after configuration.

Do you have a good reason to request this for 7.1?  (that's the needsinfo).

Comment 4 Yang Yang 2014-12-10 01:45:34 UTC
(In reply to John Ferlan from comment #3)
> This bug has been around since libvirt v0.8.6 when the refactoring occurred.
> 
> So given that and given that it hasn't been noticed until now, I'm inclined
> to move the bug to rhel 7.2.  It just doesn't seem this is a highly sought
> after configuration.
> 
> Do you have a good reason to request this for 7.1?  (that's the needsinfo).

It's okay for me to move it to rhel 7.2.

Comment 5 John Ferlan 2014-12-10 12:26:46 UTC
Patch is pushed upstream:

commit 7e71533bfb1cb3637eeb1166491e89dce72f2e3a
Author: John Ferlan <jferlan>
Date:   Mon Dec 8 08:06:57 2014 -0500

    virstoragefile: Have virStorageFileResize use safezero
    
    Currently virStorageFileResize() function uses build conditionals to
    choose either the posix_fallocate() or syscall(SYS_fallocate) with no
    fallback in order to preallocate the space in the newly resized file.
    
    Since the safezero code has the same HAVE_POSIX_FALLOCATE, modify the
    code paths in order to allow the resize logic to make use of safezero
    to more or less unify the look/feel of the code paths.
    
    Add a new boolean (resize) to safezero() to make the optional decision
    whether to try syscall(SYS_fallocate) if the posix_fallocate fails because
    HAVE_POSIX_FALLOCATE is not defined (eg, return -1 and errno == 0).
    
    Create a local safezero_sys_fallocate in order to handle the resize
    code paths that support that.  If not present, the set errno = ENOSYS
    in order to allow the caller to handle the failure scenarios
    
    Signed-off-by: John Ferlan <jferlan>


v1.2.11-rc1-31-g7e71533

Comment 8 Yang Yang 2015-05-13 10:27:28 UTC
Verified on libvirt-1.2.15-1.el7.x86_64

# iscsiadm -m discovery -t sendtargets -p localhost
[::1]:3260,1 iqn.2001-01.com.virttest:logical-pool.target
[::1]:3260,1 iqn.2001-01.com.virttest:logical-pool.target2

# cat multiiqn.xml 
<pool type="iscsi">
  <name>multiiqn</name>
<source>
    <host name="localhost"/>
    <device path="iqn.2001-01.com.virttest:logical-pool.target"/>
<initiator>
    <iqn name="iqn.2001-01.com.virttest:logical-pool.target2"/>
    </initiator>
    </source>  
<target>
  <path>/dev/disk/by-path</path>
  </target>
</pool>

#virsh pool-create multiiqn.xml

# virsh pool-list iscsi
 Name                 State      Autostart 
-------------------------------------------
 multiiqn             active     no     

# virsh vol-list multiiqn
 Name                 Path                                    
------------------------------------------------------------------------------
 unit:0:0:1           /dev/disk/by-path/ip-::1:3260-iscsi-iqn.2001-01.com.virttest:logical-pool.target-lun-1

# iscsiadm -m iface
default tcp,<empty>,<empty>,<empty>,<empty>
iser iser,<empty>,<empty>,<empty>,<empty>
libvirt-iface-15aac223 tcp,<empty>,<empty>,<empty>,iqn.2001-01.com.virttest:logical-pool.target2

Comment 10 errata-xmlrpc 2015-11-19 05:57:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2202.html