Bug 1174053

Summary: libvirtd crash when try to cold plug a network iscsi hostdev which guest already have a iscsi hostdev
Product: Red Hat Enterprise Linux 7 Reporter: Luyao Huang <lhuang>
Component: libvirtAssignee: John Ferlan <jferlan>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 7.1CC: dyuan, hliu, honzhang, jferlan, lcheng, lmiksik, mzhan, rbalakri, xuzhang
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: libvirt-1.2.8-11.el7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-03-05 07:48:09 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Luyao Huang 2014-12-15 02:25:26 UTC 
Description of problem:
libvirtd crash when try to cold plug a network iscsi hostdev which guest already have a iscsi hostdev

Version-Release number of selected component (if applicable):
libvirt-1.2.8-10.el7.x86_64

How reproducible:
100%

Steps to Reproduce:

1.prepare a guest xml like this:
# virsh dumpxml test4
    <hostdev mode='subsystem' type='scsi' managed='no' rawio='yes'>
      <source>
        <adapter name='scsi_host0'/>
        <address bus='0' target='0' unit='0'/>
      </source>
      <address type='drive' controller='0' bus='0' target='4' unit='8'/>
    </hostdev>

2.prepare a hostdev xml；
# cat hostnetiscsi.xml
    <hostdev mode='subsystem' type='scsi'>
      <source protocol='iscsi' name='iqn.2014-08.com.example:iscsi-nopool/1'>   
        <host name='test1' port='3260'/>
      </source>
<readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </hostdev>

3.cold plug:
# virsh attach-device test4 hostnetiscsi.xml --config
error: Failed to attach device from hostnetiscsi.xml
error: End of file while reading data: Input/output error
error: Failed to reconnect to the hypervisor
Actual results:


Expected results:
libvirtd crash

Additional info:
no crash

back trace:

Thread 11 (Thread 0x7f84fea57700 (LWP 7978)):
#0  0x00007f850d6bc307 in virDomainHostdevMatchSubsysSCSIiSCSI (second=0x7f84e800a130, first=0x7f84e8002490) at conf/domain_conf.c:10889
#1  virDomainHostdevMatchSubsys (b=0x7f84e800a130, a=0x7f84e8002490) at conf/domain_conf.c:10911
#2  virDomainHostdevMatch (b=0x7f84e800a130, a=0x7f84e8002490) at conf/domain_conf.c:10973
#3  virDomainHostdevFind (def=def@entry=0x7f84e80098f0, match=match@entry=0x7f84e8002490, found=0x7f84fea56990, found@entry=0x0) at conf/domain_conf.c:10998
#4  0x00007f84f6a10560 in qemuDomainAttachDeviceConfig (dev=0x7f84e8002970, dev=0x7f84e8002970, vmdef=0x7f84e80098f0, qemuCaps=0x7f84f0226270) at qemu/qemu_driver.c:7223
#5  qemuDomainAttachDeviceFlags (dom=0x7f84e8000a60, xml=<optimized out>, flags=2) at qemu/qemu_driver.c:7554
#6  0x00007f850d733f27 in virDomainAttachDeviceFlags (domain=domain@entry=0x7f84e8000a60,
    xml=0x7f84e8000940 "    <hostdev mode='subsystem' type='scsi'>\n      <source protocol='iscsi' name='iqn.2014-08.com.example:iscsi-nopool/1'>\n        <host name='test1' port='3260'/>\n      </source>\n<readonly/>\n      <add"..., flags=2) at libvirt.c:10445
#7  0x00007f850e1c4e63 in remoteDispatchDomainAttachDeviceFlags (server=<optimized out>, msg=<optimized out>, args=0x7f84e8000900, rerr=0x7f84fea56c80, client=<optimized out>) at remote_dispatch.h:2537
#8  remoteDispatchDomainAttachDeviceFlagsHelper (server=<optimized out>, client=<optimized out>, msg=<optimized out>, rerr=0x7f84fea56c80, args=0x7f84e8000900, ret=<optimized out>) at remote_dispatch.h:2515
#9  0x00007f850d792222 in virNetServerProgramDispatchCall (msg=0x7f850e4af0c0, client=0x7f850e4af130, server=0x7f850e4a3400, prog=0x7f850e4ae0f0) at rpc/virnetserverprogram.c:437
#10 virNetServerProgramDispatch (prog=0x7f850e4ae0f0, server=server@entry=0x7f850e4a3400, client=0x7f850e4af130, msg=0x7f850e4af0c0) at rpc/virnetserverprogram.c:307
#11 0x00007f850e1e03ed in virNetServerProcessMsg (msg=<optimized out>, prog=<optimized out>, client=<optimized out>, srv=0x7f850e4a3400) at rpc/virnetserver.c:172
#12 virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x7f850e4a3400) at rpc/virnetserver.c:193
#13 0x00007f850d6960f5 in virThreadPoolWorker (opaque=opaque@entry=0x7f850e484e20) at util/virthreadpool.c:145
#14 0x00007f850d695a8e in virThreadHelper (data=<optimized out>) at util/virthread.c:197
#15 0x00007f850aeefdf3 in start_thread () from /lib64/libpthread.so.0
#16 0x00007f850a80605d in clone () from /lib64/libc.so.6
Comment 1 Luyao Huang 2014-12-15 02:27:04 UTC 
I make a mistake in this place:
Actual results:


Expected results:
libvirtd crash

Additional info:
no crash

this should be :

Actual results:
libvirtd crash

Expected results:
no crash

Additional info:
Comment 2 John Ferlan 2014-12-15 13:06:15 UTC 

Pushed upstream:

commit 5fc1c51743643a02c0306d6c17fe86e9013ce342
Author: Luyao Huang <lhuang>
Date:   Mon Dec 15 11:09:39 2014 +0800

    conf: Fix libvirtd crash matching hostdev XML
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1174053
    
    Introduced by commit id '17bddc46f' - fix a libvirtd crash when
    matching a network iscsi hostdev with a host iscsi hostdev.
    
    When we use attach-device to coldplug a network iscsi hostdev,
    libvirt will check if there is already a device in XML. But if
    the 'b' is a host iscsi hostdev and 'a' is a network iscsi hostdev,
    then libvirtd will crash in virDomainHostdevMatchSubsysSCSIiSCSI
    because 'b' doesn't have a hostname.
    
    Add a check in virDomainHostdevMatchSubsys, if the a's protocol
    and b's protocol is not the same.
    
    Following is the backtrace:
    
    0  0x00007f850d6bc307 in virDomainHostdevMatchSubsysSCSIiSCSI at conf/domain
    1  virDomainHostdevMatchSubsys at conf/domain_conf.c:10911
    2  virDomainHostdevMatch at conf/domain_conf.c:10973
    3  virDomainHostdevFind at conf/domain_conf.c:10998
    4  0x00007f84f6a10560 in qemuDomainAttachDeviceConfig at qemu/qemu_driver.c:
    5  qemuDomainAttachDeviceFlags at qemu/qemu_driver.c:7554
    
    Signed-off-by: Luyao Huang <lhuang>


git describe: v1.2.11-11-g5fc1c51

and POSTed to rhvirt-patches:

http://post-office.corp.redhat.com/archives/rhvirt-patches/2014-December/msg00312.html
Comment 4 lcheng 2014-12-26 06:44:17 UTC 
I can reproduce this issue with libvirt-1.2.8-9.el7.x86_64 and qemu-kvm-rhev-2.1.2-17.el7.x86_64.


Steps:

[root@intel-q9550-4-1 ~]# rpm -q libvirt qemu-kvm-rhev
libvirt-1.2.8-9.el7.x86_64
qemu-kvm-rhev-2.1.2-17.el7.x86_64

[root@intel-q9550-4-1 ~]# virsh dumpxml a | grep hostdev -A10
    <hostdev mode='subsystem' type='scsi' managed='no' rawio='yes'>
      <source>
        <adapter name='scsi_host7'/>
        <address bus='0' target='0' unit='0'/>
      </source>
      <address type='drive' controller='0' bus='0' target='4' unit='8'/>
    </hostdev>
...

[root@intel-q9550-4-1 ~]# cat hostscsi.xml 
    <hostdev mode='subsystem' type='scsi'>
      <source protocol='iscsi' name='iqn.st:tape:sttarget1/1'>   
        <host name='127.0.0.1' port='3260'/>
      </source>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </hostdev>

[root@intel-q9550-4-1 ~]# virsh attach-device a hostscsi.xml --config
error: Failed to attach device from hostscsi.xml
error: End of file while reading data: Input/output error
error: Failed to reconnect to the hypervisor


==================================================================


Verify it as follows. The result is expected.

Steps:

On host:
[root@intel-q9550-4-1 ~]# rpm -q libvirt qemu-kvm-rhev
libvirt-1.2.8-11.el7.x86_64
qemu-kvm-rhev-2.1.2-17.el7.x86_64

[root@intel-q9550-4-1 ~]# virsh dumpxml a
...
    <hostdev mode='subsystem' type='scsi' managed='no' rawio='yes'>
      <source>
        <adapter name='scsi_host7'/>
        <address bus='0' target='0' unit='0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='drive' controller='0' bus='0' target='4' unit='8'/>
    </hostdev>
...

[root@intel-q9550-4-1 ~]# cat hostscsi.xml 
    <hostdev mode='subsystem' type='scsi'>
      <source protocol='iscsi' name='iqn.st:tape:sttarget1/1'>   
        <host name='127.0.0.1' port='3260'/>
      </source>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </hostdev>

[root@intel-q9550-4-1 ~]# virsh attach-device a hostscsi.xml --config
Device attached successfully

[root@intel-q9550-4-1 ~]# virsh start a
Domain a started

[root@intel-q9550-4-1 ~]# virsh dumpxml a
...
    <hostdev mode='subsystem' type='scsi' managed='no' rawio='yes'>
      <source>
        <adapter name='scsi_host7'/>
        <address bus='0' target='0' unit='0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='drive' controller='0' bus='0' target='4' unit='8'/>
    </hostdev>
    <hostdev mode='subsystem' type='scsi' managed='no'>
      <source protocol='iscsi' name='iqn.st:tape:sttarget1/1'>
        <host name='127.0.0.1' port='3260'/>
      </source>
      <readonly/>
      <alias name='hostdev1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </hostdev>
...


On guest:
[root@localhost ~]# cat /proc/scsi/scsi
Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
  Vendor: ATA      Model: QEMU HARDDISK    Rev: 2.1.
  Type:   Direct-Access                    ANSI  SCSI revision: 05
Host: scsi2 Channel: 00 Id: 00 Lun: 00
  Vendor: IET      Model: VIRTUAL-TAPE     Rev: 0001
  Type:   Sequential-Access                ANSI  SCSI revision: 05
Host: scsi2 Channel: 00 Id: 04 Lun: 08
  Vendor: IET      Model: Controller       Rev: 0001
  Type:   RAID                             ANSI  SCSI revision: 05
Comment 6 errata-xmlrpc 2015-03-05 07:48:09 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-0323.html