Bug 1597550

Summary: libvirtd crashed when hot plug a scsi hostdev
Product: Red Hat Enterprise Linux 7 Reporter: yisun
Component: libvirtAssignee: Peter Krempa <pkrempa>
Status: CLOSED ERRATA QA Contact: yisun
Severity: high Docs Contact:
Priority: high    
Version: 7.6CC: pkrempa, yisun
Target Milestone: rcKeywords: Automation, Regression
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: libvirt-4.5.0-2.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-10-30 09:56:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description yisun 2018-07-03 08:20:34 UTC
Description of problem:
libvirtd crashed when hot plug a scsi hostdev

Version-Release number of selected component (if applicable):
libvirt-4.4.0-2.virtcov.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1. prepare a hostdev xm (you don't really need a existing device on host, just xml is enough)
# cat hostdev.xml 
<hostdev mode="subsystem" type="scsi"><source name="iqn.2018-06.com.virttest:img0.target/0" protocol="iscsi"><host name="127.0.0.1" port="3260" /></source></hostdev>

2. having a running vm
# virsh list
 Id    Name                           State
----------------------------------------------------
...
 13    pc                             running

3. do the hot plug
# virsh attach-device pc hostdev.xml
error: Disconnected from qemu:///system due to end of file
error: Failed to attach device from hostdev.xml
error: End of file while reading data: Input/output error

Actual results:
libvirt crashed.


Additional info:
The same auto case passed on rhel7.5, so set as regression.
gdb info as follow:
(gdb) c
Continuing.
Detaching after fork from child process 5892.
Detaching after fork from child process 5893.
Detaching after fork from child process 5894.
Detaching after fork from child process 5895.

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f43d3d6a700 (LWP 5350)]
0x00007f43bba3aba7 in qemuDomainAttachHostSCSIDevice (hostdev=0x7f439c000bc0, vm=<optimized out>, driver=0x7f43640f2270) at qemu/qemu_hotplug.c:2397
2397	        secinfo = srcPriv->secinfo;
(gdb) c
Continuing.
[Thread 0x7f43d556d700 (LWP 5347) exited]
[Thread 0x7f43d3d6a700 (LWP 5350) exited]
[Thread 0x7f43d456b700 (LWP 5349) exited]
[Thread 0x7f43d3569700 (LWP 5351) exited]
[Thread 0x7f43d2d68700 (LWP 5352) exited]
[Thread 0x7f43d2567700 (LWP 5353) exited]
[Thread 0x7f43d1d66700 (LWP 5354) exited]
[Thread 0x7f43d1565700 (LWP 5355) exited]
[Thread 0x7f43d0d64700 (LWP 5356) exited]
[Thread 0x7f43bb4b4700 (LWP 5357) exited]
[Thread 0x7f43bacb3700 (LWP 5358) exited]
[Thread 0x7f43ba4b2700 (LWP 5359) exited]
[Thread 0x7f43b9cb1700 (LWP 5360) exited]
[Thread 0x7f43b94b0700 (LWP 5361) exited]
[Thread 0x7f43abfff700 (LWP 5401) exited]
[Thread 0x7f43aa7fc700 (LWP 5422) exited]
[Thread 0x7f43e5f8c8c0 (LWP 5346) exited]

Program terminated with signal SIGSEGV, Segmentation fault.
The program no longer exists.

Comment 3 Peter Krempa 2018-07-03 08:36:00 UTC
srcPriv is not allocated when no authentication is present

Comment 4 yisun 2018-07-03 08:37:34 UTC
seems srcPriv->secinfo is NULL when we didn't provide <auth> part in hostdev xml, with following xml, no crash happens:

<hostdev mode="subsystem" type="scsi"><source name="iqn.2018-06.com.virttest:img0.target/0" protocol="iscsi"><host name="127.0.0.1" port="3260" /><auth username='myuser'><secret type='iscsi' usage='libvirtiscsi'/></auth></source></hostdev>

And this is not reproduced with hostdev xml pointing to a local scsi device as follow:
<hostdev mode="subsystem" rawio="yes" type="scsi"><source><adapter name="scsi_host31" /><address bus="0" target="0" unit="0" /></source></hostdev>

Comment 5 Peter Krempa 2018-07-03 14:38:37 UTC
Fixed upstream by:

commit 33a475056fdd76c030528982e422bae79c0a0e4a 
Author: Peter Krempa <pkrempa>
Date:   Tue Jul 3 10:45:34 2018 +0200

    qemu: hotplug: Don't access srcPriv when it's not allocated
    
    The private data of a virStorageSource which is backing an iSCSI hostdev
    may be NULL if no authentication is present. The code handling the
    hotplug would attempt to extract the authentication info stored in
    'secinfo' without checking if it is allocated which resulted in a crash.
    
    Here we opt the easy way to check if srcPriv is not NULL so that we
    don't duplicate all the logic which selects whether the disk source has
    a secret.

Comment 8 yisun 2018-08-17 07:23:04 UTC
Verified with:libvirt-4.5.0-6.el7.x86_64

Steps:
1. # cat hostdev.iscsi 
<hostdev mode="subsystem" type="scsi">
    <source name="iqn.2016-03.com.virttest:logical-pool.target/0" protocol="iscsi">
        <host name="10.73.73.57" port="3260" />
    </source>
</hostdev>


2. # virsh list
 Id    Name                           State
----------------------------------------------------
 10    vm1                            running


3. # virsh attach-device vm1 hostdev.iscsi 
Device attached successfully

4. login vm and check new block device added.

Comment 10 errata-xmlrpc 2018-10-30 09:56:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:3113