Bug 1665416

Summary: libvirt crashed when starting guest with hostdev device
Product: Red Hat Enterprise Linux 8 Reporter: liunana <nanliu>
Component: libvirtAssignee: Libvirt Maintainers <libvirt-maint>
Status: CLOSED DUPLICATE QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: high    
Version: 8.0CC: chayang, fjin, jdenemar, jinzhao, juzhang, rbalakri, virt-maint, xuzhang, yafu, yalzhang, zhguo
Target Milestone: rcKeywords: Regression
Target Release: 8.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-01-11 11:45:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
more journalctl log none

Description liunana 2019-01-11 10:17:12 UTC
Description of problem:
libvirt start guest failed with mdev device


Version-Release number of selected component (if applicable):
(host)
kernel-4.18.0-57.el8.x86_64
qemu-kvm-2.12.0-53.module+el8+2637+096d5272.x86_64
libvirt-4.5.0-17.module+el8+2625+db702f9d.x86_64
selinux-policy-targeted-3.14.1-49.el8.noarch


How reproducible:
6/6


Steps to Reproduce:
1. install a guest
2. start guest with medv device:

    <hostdev mode='subsystem' type='mdev' managed='no' model='vfio-pci' display='on'>
      <source>
        <address uuid='5bb18984-3286-4242-87d3-e867450ce3c2'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>

3.

Actual results:
guest can't be started with error logs[1], meanwhile journalctl get some error logs,I will attach them in comments.


Expected results:
guest can be start normally without any error


Additional info:
[1]
# virsh start rhel76 
error: Disconnected from qemu:///system due to end of file
error: Failed to start domain rhel76
error: End of file while reading data: Input/output error

Comment 1 yafu 2019-01-11 10:24:43 UTC
It's a regression. Can not reproduced with ibvirt-4.5.0-16.el8.x86_64.

Comment 2 yafu 2019-01-11 10:26:21 UTC
The backtrace of libvirtd is as follows:
(gdb) t a a bt

Thread 17 (Thread 0x7f7c77185700 (LWP 14339)):
#0  0x00007f7c8a6bf48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f7c8d83ea6a in virCondWait () from /lib64/libvirt.so.0
#2  0x00007f7c8d83f5a3 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3  0x00007f7c8d83e7cc in virThreadHelper () from /lib64/libvirt.so.0
#4  0x00007f7c8a6b92de in start_thread () from /lib64/libpthread.so.0
#5  0x00007f7c8a3e9a63 in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7f7c6f7fe700 (LWP 14347)):
#0  0x00007f7c8a6bf48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f7c8d83ea6a in virCondWait () from /lib64/libvirt.so.0
#2  0x00007f7c8d83f554 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3  0x00007f7c8d83e7cc in virThreadHelper () from /lib64/libvirt.so.0
#4  0x00007f7c8a6b92de in start_thread () from /lib64/libpthread.so.0
#5  0x00007f7c8a3e9a63 in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7f7c76183700 (LWP 14342)):
#0  0x00007f7c8a6bf48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f7c8d83ea6a in virCondWait () from /lib64/libvirt.so.0
#2  0x00007f7c8d83f5a3 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3  0x00007f7c8d83e7cc in virThreadHelper () from /lib64/libvirt.so.0
--Type <RET> for more, q to quit, c to continue without paging--
#4  0x00007f7c8a6b92de in start_thread () from /lib64/libpthread.so.0
#5  0x00007f7c8a3e9a63 in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7f7c6effd700 (LWP 14348)):
#0  0x00007f7c8a6bf48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f7c8d83ea6a in virCondWait () from /lib64/libvirt.so.0
#2  0x00007f7c8d83f554 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3  0x00007f7c8d83e7cc in virThreadHelper () from /lib64/libvirt.so.0
#4  0x00007f7c8a6b92de in start_thread () from /lib64/libpthread.so.0
#5  0x00007f7c8a3e9a63 in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7f7c6ffff700 (LWP 14346)):
#0  0x00007f7c8a6bf48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f7c8d83ea6a in virCondWait () from /lib64/libvirt.so.0
#2  0x00007f7c8d83f554 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3  0x00007f7c8d83e7cc in virThreadHelper () from /lib64/libvirt.so.0
#4  0x00007f7c8a6b92de in start_thread () from /lib64/libpthread.so.0
#5  0x00007f7c8a3e9a63 in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7f7c427fc700 (LWP 14352)):
#0  0x00007f7c8a6bf48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f7c8d83ea6a in virCondWait () from /lib64/libvirt.so.0
--Type <RET> for more, q to quit, c to continue without paging--
#2  0x00007f7c8d83f5a3 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3  0x00007f7c8d83e7cc in virThreadHelper () from /lib64/libvirt.so.0
#4  0x00007f7c8a6b92de in start_thread () from /lib64/libpthread.so.0
#5  0x00007f7c8a3e9a63 in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7f7c43fff700 (LWP 14349)):
#0  0x00007f7c8a6bf48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f7c8d83ea6a in virCondWait () from /lib64/libvirt.so.0
#2  0x00007f7c8d83f5a3 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3  0x00007f7c8d83e7cc in virThreadHelper () from /lib64/libvirt.so.0
#4  0x00007f7c8a6b92de in start_thread () from /lib64/libpthread.so.0
#5  0x00007f7c8a3e9a63 in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7f7c437fe700 (LWP 14350)):
#0  0x00007f7c8a6bf48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f7c8d83ea6a in virCondWait () from /lib64/libvirt.so.0
#2  0x00007f7c8d83f5a3 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3  0x00007f7c8d83e7cc in virThreadHelper () from /lib64/libvirt.so.0
#4  0x00007f7c8a6b92de in start_thread () from /lib64/libpthread.so.0
#5  0x00007f7c8a3e9a63 in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7f7c41ffb700 (LWP 14353)):
--Type <RET> for more, q to quit, c to continue without paging--
#0  0x00007f7c8a6bf48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f7c8d83ea6a in virCondWait () from /lib64/libvirt.so.0
#2  0x00007f7c8d83f5a3 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3  0x00007f7c8d83e7cc in virThreadHelper () from /lib64/libvirt.so.0
#4  0x00007f7c8a6b92de in start_thread () from /lib64/libpthread.so.0
#5  0x00007f7c8a3e9a63 in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f7c42ffd700 (LWP 14351)):
#0  0x00007f7c8a6bf48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f7c8d83ea6a in virCondWait () from /lib64/libvirt.so.0
#2  0x00007f7c8d83f5a3 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3  0x00007f7c8d83e7cc in virThreadHelper () from /lib64/libvirt.so.0
#4  0x00007f7c8a6b92de in start_thread () from /lib64/libpthread.so.0
#5  0x00007f7c8a3e9a63 in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7f7c1b9e2700 (LWP 14392)):
#0  0x00007f7c8a6bf48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f7c8d83ea6a in virCondWait () from /lib64/libvirt.so.0
#2  0x00007f7c48ef20e4 in udevEventHandleThread () from /usr/lib64/libvirt/connection-driver/libvirt_driver_nodedev.so
#3  0x00007f7c8d83e7fa in virThreadHelper () from /lib64/libvirt.so.0
#4  0x00007f7c8a6b92de in start_thread () from /lib64/libpthread.so.0
#5  0x00007f7c8a3e9a63 in clone () from /lib64/libc.so.6
--Type <RET> for more, q to quit, c to continue without paging--

Thread 6 (Thread 0x7f7c74980700 (LWP 14345)):
#0  0x00007f7c8a6bf48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f7c8d83ea6a in virCondWait () from /lib64/libvirt.so.0
#2  0x00007f7c8d83f554 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3  0x00007f7c8d83e7cc in virThreadHelper () from /lib64/libvirt.so.0
#4  0x00007f7c8a6b92de in start_thread () from /lib64/libpthread.so.0
#5  0x00007f7c8a3e9a63 in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7f7c75181700 (LWP 14344)):
#0  0x00007f7c8a6bf48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f7c8d83ea6a in virCondWait () from /lib64/libvirt.so.0
#2  0x00007f7c8d83f554 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3  0x00007f7c8d83e7cc in virThreadHelper () from /lib64/libvirt.so.0
#4  0x00007f7c8a6b92de in start_thread () from /lib64/libpthread.so.0
#5  0x00007f7c8a3e9a63 in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7f7c75982700 (LWP 14343)):
#0  0x00007f7c8a6bf48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f7c8d83ea6a in virCondWait () from /lib64/libvirt.so.0
#2  0x00007f7c8d83f5a3 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3  0x00007f7c8d83e7cc in virThreadHelper () from /lib64/libvirt.so.0
--Type <RET> for more, q to quit, c to continue without paging--
#4  0x00007f7c8a6b92de in start_thread () from /lib64/libpthread.so.0
#5  0x00007f7c8a3e9a63 in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f7c8e32d380 (LWP 14338)):
#0  0x00007f7c8a3de7e1 in poll () from /lib64/libc.so.6
#1  0x00007f7c8d7e1c55 in virEventPollRunOnce () from /lib64/libvirt.so.0
#2  0x00007f7c8d7e07d5 in virEventRunDefaultImpl () from /lib64/libvirt.so.0
#3  0x00007f7c8d90edad in virNetDaemonRun () from /lib64/libvirt.so.0
#4  0x000055c5c6ca3756 in main ()

Thread 2 (Thread 0x7f7c76984700 (LWP 14340)):
#0  0x00007f7c8a6bf48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f7c8d83ea6a in virCondWait () from /lib64/libvirt.so.0
#2  0x00007f7c8d83f5a3 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3  0x00007f7c8d83e7cc in virThreadHelper () from /lib64/libvirt.so.0
#4  0x00007f7c8a6b92de in start_thread () from /lib64/libpthread.so.0
#5  0x00007f7c8a3e9a63 in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f7c6e183700 (LWP 14341)):
#0  0x00007f7c8d828646 in virSCSIDeviceGetAdapterId () from /lib64/libvirt.so.0
#1  0x00007f7c8d8288d2 in virSCSIDeviceGetDevName () from /lib64/libvirt.so.0
#2  0x00007f7c4837333b in qemuGetHostdevPath () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
--Type <RET> for more, q to quit, c to continue without paging--
#3  0x00007f7c48376dfe in qemuSetUnprivSGIO () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#4  0x00007f7c48364e08 in qemuHostdevPrepareSCSIDevices () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5  0x00007f7c48365015 in qemuHostdevPrepareDomainDevices () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#6  0x00007f7c4837f0d5 in qemuProcessPrepareHost () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#7  0x00007f7c4838551f in qemuProcessStart () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#8  0x00007f7c483e4ded in qemuDomainObjStart.constprop.49 () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#9  0x00007f7c483e5439 in qemuDomainCreateWithFlags () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#10 0x00007f7c8d9d13c7 in virDomainCreate () from /lib64/libvirt.so.0
#11 0x000055c5c6cd033e in remoteDispatchDomainCreateHelper ()
#12 0x00007f7c8d909074 in virNetServerProgramDispatch () from /lib64/libvirt.so.0
#13 0x00007f7c8d90f54c in virNetServerHandleJob () from /lib64/libvirt.so.0
#14 0x00007f7c8d83f4c0 in virThreadPoolWorker () from /lib64/libvirt.so.0
#15 0x00007f7c8d83e7cc in virThreadHelper () from /lib64/libvirt.so.0
#16 0x00007f7c8a6b92de in start_thread () from /lib64/libpthread.so.0
#17 0x00007f7c8a3e9a63 in clone () from /lib64/libc.so.6

Comment 3 liunana 2019-01-11 10:32:43 UTC
Created attachment 1519999 [details]
more journalctl log

Comment 4 Jiri Denemark 2019-01-11 11:45:24 UTC

*** This bug has been marked as a duplicate of bug 1665244 ***