1174053 – libvirtd crash when try to cold plug a network iscsi hostdev which guest already have a iscsi hostdev

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1174053 - libvirtd crash when try to cold plug a network iscsi hostdev which guest already have a iscsi hostdev

Summary: libvirtd crash when try to cold plug a network iscsi hostdev which guest alre...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	libvirt
Sub Component:
Version:	7.1
Hardware:	x86_64
OS:	Linux
Priority:	high
Severity:	high
Target Milestone:	rc
Target Release:	---
Assignee:	John Ferlan
QA Contact:	Virtualization Bugs
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2014-12-15 02:25 UTC by Luyao Huang
Modified:	2015-03-05 07:48 UTC (History)
CC List:	9 users (show)
Fixed In Version:	libvirt-1.2.8-11.el7
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2015-03-05 07:48:09 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHSA-2015:0323	0	normal	SHIPPED_LIVE	Low: libvirt security, bug fix, and enhancement update	2015-03-05 12:10:54 UTC

Description Luyao Huang 2014-12-15 02:25:26 UTC

Description of problem:
libvirtd crash when try to cold plug a network iscsi hostdev which guest already have a iscsi hostdev

Version-Release number of selected component (if applicable):
libvirt-1.2.8-10.el7.x86_64

How reproducible:
100%

Steps to Reproduce:

1.prepare a guest xml like this:
# virsh dumpxml test4
    <hostdev mode='subsystem' type='scsi' managed='no' rawio='yes'>
      <source>
        <adapter name='scsi_host0'/>
        <address bus='0' target='0' unit='0'/>
      </source>
      <address type='drive' controller='0' bus='0' target='4' unit='8'/>
    </hostdev>

2.prepare a hostdev xml；
# cat hostnetiscsi.xml
    <hostdev mode='subsystem' type='scsi'>
      <source protocol='iscsi' name='iqn.2014-08.com.example:iscsi-nopool/1'>   
        <host name='test1' port='3260'/>
      </source>
<readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </hostdev>

3.cold plug:
# virsh attach-device test4 hostnetiscsi.xml --config
error: Failed to attach device from hostnetiscsi.xml
error: End of file while reading data: Input/output error
error: Failed to reconnect to the hypervisor
Actual results:


Expected results:
libvirtd crash

Additional info:
no crash

back trace:

Thread 11 (Thread 0x7f84fea57700 (LWP 7978)):
#0  0x00007f850d6bc307 in virDomainHostdevMatchSubsysSCSIiSCSI (second=0x7f84e800a130, first=0x7f84e8002490) at conf/domain_conf.c:10889
#1  virDomainHostdevMatchSubsys (b=0x7f84e800a130, a=0x7f84e8002490) at conf/domain_conf.c:10911
#2  virDomainHostdevMatch (b=0x7f84e800a130, a=0x7f84e8002490) at conf/domain_conf.c:10973
#3  virDomainHostdevFind (def=def@entry=0x7f84e80098f0, match=match@entry=0x7f84e8002490, found=0x7f84fea56990, found@entry=0x0) at conf/domain_conf.c:10998
#4  0x00007f84f6a10560 in qemuDomainAttachDeviceConfig (dev=0x7f84e8002970, dev=0x7f84e8002970, vmdef=0x7f84e80098f0, qemuCaps=0x7f84f0226270) at qemu/qemu_driver.c:7223
#5  qemuDomainAttachDeviceFlags (dom=0x7f84e8000a60, xml=<optimized out>, flags=2) at qemu/qemu_driver.c:7554
#6  0x00007f850d733f27 in virDomainAttachDeviceFlags (domain=domain@entry=0x7f84e8000a60,
    xml=0x7f84e8000940 "    <hostdev mode='subsystem' type='scsi'>\n      <source protocol='iscsi' name='iqn.2014-08.com.example:iscsi-nopool/1'>\n        <host name='test1' port='3260'/>\n      </source>\n<readonly/>\n      <add"..., flags=2) at libvirt.c:10445
#7  0x00007f850e1c4e63 in remoteDispatchDomainAttachDeviceFlags (server=<optimized out>, msg=<optimized out>, args=0x7f84e8000900, rerr=0x7f84fea56c80, client=<optimized out>) at remote_dispatch.h:2537
#8  remoteDispatchDomainAttachDeviceFlagsHelper (server=<optimized out>, client=<optimized out>, msg=<optimized out>, rerr=0x7f84fea56c80, args=0x7f84e8000900, ret=<optimized out>) at remote_dispatch.h:2515
#9  0x00007f850d792222 in virNetServerProgramDispatchCall (msg=0x7f850e4af0c0, client=0x7f850e4af130, server=0x7f850e4a3400, prog=0x7f850e4ae0f0) at rpc/virnetserverprogram.c:437
#10 virNetServerProgramDispatch (prog=0x7f850e4ae0f0, server=server@entry=0x7f850e4a3400, client=0x7f850e4af130, msg=0x7f850e4af0c0) at rpc/virnetserverprogram.c:307
#11 0x00007f850e1e03ed in virNetServerProcessMsg (msg=<optimized out>, prog=<optimized out>, client=<optimized out>, srv=0x7f850e4a3400) at rpc/virnetserver.c:172
#12 virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x7f850e4a3400) at rpc/virnetserver.c:193
#13 0x00007f850d6960f5 in virThreadPoolWorker (opaque=opaque@entry=0x7f850e484e20) at util/virthreadpool.c:145
#14 0x00007f850d695a8e in virThreadHelper (data=<optimized out>) at util/virthread.c:197
#15 0x00007f850aeefdf3 in start_thread () from /lib64/libpthread.so.0
#16 0x00007f850a80605d in clone () from /lib64/libc.so.6

Comment 1 Luyao Huang 2014-12-15 02:27:04 UTC

I make a mistake in this place:
Actual results:


Expected results:
libvirtd crash

Additional info:
no crash

this should be :

Actual results:
libvirtd crash

Expected results:
no crash

Additional info:

Comment 2 John Ferlan 2014-12-15 13:06:15 UTC


Pushed upstream:

commit 5fc1c51743643a02c0306d6c17fe86e9013ce342
Author: Luyao Huang <lhuang>
Date:   Mon Dec 15 11:09:39 2014 +0800

    conf: Fix libvirtd crash matching hostdev XML
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1174053
    
    Introduced by commit id '17bddc46f' - fix a libvirtd crash when
    matching a network iscsi hostdev with a host iscsi hostdev.
    
    When we use attach-device to coldplug a network iscsi hostdev,
    libvirt will check if there is already a device in XML. But if
    the 'b' is a host iscsi hostdev and 'a' is a network iscsi hostdev,
    then libvirtd will crash in virDomainHostdevMatchSubsysSCSIiSCSI
    because 'b' doesn't have a hostname.
    
    Add a check in virDomainHostdevMatchSubsys, if the a's protocol
    and b's protocol is not the same.
    
    Following is the backtrace:
    
    0  0x00007f850d6bc307 in virDomainHostdevMatchSubsysSCSIiSCSI at conf/domain
    1  virDomainHostdevMatchSubsys at conf/domain_conf.c:10911
    2  virDomainHostdevMatch at conf/domain_conf.c:10973
    3  virDomainHostdevFind at conf/domain_conf.c:10998
    4  0x00007f84f6a10560 in qemuDomainAttachDeviceConfig at qemu/qemu_driver.c:
    5  qemuDomainAttachDeviceFlags at qemu/qemu_driver.c:7554
    
    Signed-off-by: Luyao Huang <lhuang>


git describe: v1.2.11-11-g5fc1c51

and POSTed to rhvirt-patches:

http://post-office.corp.redhat.com/archives/rhvirt-patches/2014-December/msg00312.html

Comment 4 lcheng 2014-12-26 06:44:17 UTC

I can reproduce this issue with libvirt-1.2.8-9.el7.x86_64 and qemu-kvm-rhev-2.1.2-17.el7.x86_64.


Steps:

[root@intel-q9550-4-1 ~]# rpm -q libvirt qemu-kvm-rhev
libvirt-1.2.8-9.el7.x86_64
qemu-kvm-rhev-2.1.2-17.el7.x86_64

[root@intel-q9550-4-1 ~]# virsh dumpxml a | grep hostdev -A10
    <hostdev mode='subsystem' type='scsi' managed='no' rawio='yes'>
      <source>
        <adapter name='scsi_host7'/>
        <address bus='0' target='0' unit='0'/>
      </source>
      <address type='drive' controller='0' bus='0' target='4' unit='8'/>
    </hostdev>
...

[root@intel-q9550-4-1 ~]# cat hostscsi.xml 
    <hostdev mode='subsystem' type='scsi'>
      <source protocol='iscsi' name='iqn.st:tape:sttarget1/1'>   
        <host name='127.0.0.1' port='3260'/>
      </source>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </hostdev>

[root@intel-q9550-4-1 ~]# virsh attach-device a hostscsi.xml --config
error: Failed to attach device from hostscsi.xml
error: End of file while reading data: Input/output error
error: Failed to reconnect to the hypervisor


==================================================================


Verify it as follows. The result is expected.

Steps:

On host:
[root@intel-q9550-4-1 ~]# rpm -q libvirt qemu-kvm-rhev
libvirt-1.2.8-11.el7.x86_64
qemu-kvm-rhev-2.1.2-17.el7.x86_64

[root@intel-q9550-4-1 ~]# virsh dumpxml a
...
    <hostdev mode='subsystem' type='scsi' managed='no' rawio='yes'>
      <source>
        <adapter name='scsi_host7'/>
        <address bus='0' target='0' unit='0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='drive' controller='0' bus='0' target='4' unit='8'/>
    </hostdev>
...

[root@intel-q9550-4-1 ~]# cat hostscsi.xml 
    <hostdev mode='subsystem' type='scsi'>
      <source protocol='iscsi' name='iqn.st:tape:sttarget1/1'>   
        <host name='127.0.0.1' port='3260'/>
      </source>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </hostdev>

[root@intel-q9550-4-1 ~]# virsh attach-device a hostscsi.xml --config
Device attached successfully

[root@intel-q9550-4-1 ~]# virsh start a
Domain a started

[root@intel-q9550-4-1 ~]# virsh dumpxml a
...
    <hostdev mode='subsystem' type='scsi' managed='no' rawio='yes'>
      <source>
        <adapter name='scsi_host7'/>
        <address bus='0' target='0' unit='0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='drive' controller='0' bus='0' target='4' unit='8'/>
    </hostdev>
    <hostdev mode='subsystem' type='scsi' managed='no'>
      <source protocol='iscsi' name='iqn.st:tape:sttarget1/1'>
        <host name='127.0.0.1' port='3260'/>
      </source>
      <readonly/>
      <alias name='hostdev1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </hostdev>
...


On guest:
[root@localhost ~]# cat /proc/scsi/scsi
Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
  Vendor: ATA      Model: QEMU HARDDISK    Rev: 2.1.
  Type:   Direct-Access                    ANSI  SCSI revision: 05
Host: scsi2 Channel: 00 Id: 00 Lun: 00
  Vendor: IET      Model: VIRTUAL-TAPE     Rev: 0001
  Type:   Sequential-Access                ANSI  SCSI revision: 05
Host: scsi2 Channel: 00 Id: 04 Lun: 08
  Vendor: IET      Model: Controller       Rev: 0001
  Type:   RAID                             ANSI  SCSI revision: 05

Comment 6 errata-xmlrpc 2015-03-05 07:48:09 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-0323.html

Note You need to log in before you can comment on or make changes to this bug.