Bug 1446455
| Summary: | libvirtd crashes on domain shutdown after reconnecting to a domain with a mediated host device attached | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Erik Skultety <eskultet> |
| Component: | libvirt | Assignee: | Erik Skultety <eskultet> |
| Status: | CLOSED ERRATA | QA Contact: | zhe peng <zpeng> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 7.3 | CC: | dyuan, jdenemar, rbalakri, xuzhang |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | libvirt-3.2.0-5.el7 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-08-02 00:08:25 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
> Steps to Reproduce:
> 1. Prepare a domain with a mediated host device
And here I meant to post the relevant domain XML snippet...
...
<hostdev mode='subsystem' type='mdev' managed='no' model='vfio-pci'>
<source>
<address uuid='e1aae5c9-85f8-437b-8a93-8615780b14ca'/>
</source>
</hostdev>
...
fixed upstream by:
commit 92e30a4dace54d06433f763e1acba0a81bb5c82e
Refs: v3.3.0-rc2-9-g92e30a4da
Author: Erik Skultety <eskultet>
AuthorDate: Fri Apr 28 09:24:31 2017 +0200
Commit: Erik Skultety <eskultet>
CommitDate: Thu May 4 08:05:03 2017 +0200
mdev: Fix daemon crash on domain shutdown after reconnect
The problem resides in virHostdevUpdateActiveMediatedDevices which gets
called during qemuProcessReconnect. The issue here is that
virMediatedDeviceListAdd takes a pointer to the item to be added to the
list to which VIR_APPEND_ELEMENT is used, which also clears the pointer.
However, in this case only the local copy of the pointer got cleared,
leaving the original pointing to valid memory. To sum it up, during
cleanup phase, the original pointer is freed and the daemon crashes
basically any time it would access it.
Backtrace:
0x00007ffff3ccdeba in __strcmp_sse2_unaligned
0x00007ffff72a444a in virMediatedDeviceListFindIndex
0x00007ffff7241446 in virHostdevReAttachMediatedDevices
0x00007fffc60215d9 in qemuHostdevReAttachMediatedDevices
0x00007fffc60216dc in qemuHostdevReAttachDomainDevices
0x00007fffc6046e6f in qemuProcessStop
0x00007fffc6091596 in processMonitorEOFEvent
0x00007fffc6091793 in qemuProcessEventHandler
0x00007ffff7294bf5 in virThreadPoolWorker
0x00007ffff7294184 in virThreadHelper
0x00007ffff3fdc3c4 in start_thread () from /lib64/libpthread.so.0
0x00007ffff3d269cf in clone () from /lib64/libc.so.6
Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1446455
Signed-off-by: Erik Skultety <eskultet>
Reviewed-by: Laine Stump <laine>
I can reproduce this with build: libvirt-3.2.0-4.el7.x86_64
verify with build:
libvirt-3.2.0-6.el7.x86_64
step:
1. Prepare a domain with a mediated host device
....
<hostdev mode='subsystem' type='mdev' managed='no' model='vfio-pci'>
<source>
<address uuid='9ea4eb07-c4a8-41d1-bd4a-850d1b0bef74'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</hostdev>
....
2. start the VM
3. restart libvirtd
4. shutdown the VM
virsh # shutdown rhel7
Domain rhel7 is being shutdown
virsh # list --all
Id Name State
----------------------------------------------------
- rhel7 shut off
virsh #
no crash occured, move to verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:1846 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:1846 |
Description of problem: Having a running domain with a mediated host device attached, restarting libvirtd (thus reconnecting to the VM's qemu process) and then trying to shutdown the mentioned domain will result in a crash. Version-Release number of selected component (if applicable): libvirt-3.2.0 How reproducible: always Steps to Reproduce: 1. Prepare a domain with a mediated host device 2. start the VM 3. restart libvirtd (which will then reconnect to the qemu process) 4. shutdown the VM 5. daemon crashes virsh # error: Disconnected from qemu:///system due to keepalive timeout Actual results: libvirtd crashes Expected results: libvirtd doesn't crash Additional info: The reason why the daemon crashes is due to accessing a pointer which has previously been freed during the reconnect phase in 'virHostdevReAttachMediatedDevices', refer to the traceback. #0 0x00007ffff3ccdeba in __strcmp_sse2_unaligned () from /lib64/libc.so.6 #1 0x00007ffff72a444a in virMediatedDeviceListFindIndex (list=0x7fff9c076590, dev=0x7fff9c050980) at util/virmdev.c:388 #2 0x00007ffff72a4491 in virMediatedDeviceListFind (list=0x7fff9c076590, dev=0x7fff9c050980) at util/virmdev.c:401 #3 0x00007ffff7241446 in virHostdevReAttachMediatedDevices (mgr=0x7fff9c07d0f0, drv_name=0x7fffc60e6dd2 "QEMU", dom_name=0x7fff9c0e6c60 "f24-test", hostdevs=0x7fff9c196420, nhostdevs=2) at util/virhostdev.c:2055 #4 0x00007fffc60215d9 in qemuHostdevReAttachMediatedDevices (driver=0x7fff9c08d420, name=0x7fff9c0e6c60 "f24-test", hostdevs=0x7fff9c196420, nhostdevs=2) at qemu/qemu_hostdev.c:457 #5 0x00007fffc60216dc in qemuHostdevReAttachDomainDevices (driver=0x7fff9c08d420, def=0x7fff9c04bbf0) at qemu/qemu_hostdev.c:480 #6 0x00007fffc6046e6f in qemuProcessStop (driver=0x7fff9c08d420, vm=0x7fff9c197d80, reason=VIR_DOMAIN_SHUTOFF_SHUTDOWN, asyncJob=QEMU_ASYNC_JOB_NONE, flags=0) at qemu/qemu_process.c:6306 #7 0x00007fffc6091596 in processMonitorEOFEvent (driver=0x7fff9c08d420, vm=0x7fff9c197d80) at qemu/qemu_driver.c:4566 #8 0x00007fffc6091793 in qemuProcessEventHandler (data=0x555555875870, opaque=0x7fff9c08d420) at qemu/qemu_driver.c:4611 #9 0x00007ffff7294bf5 in virThreadPoolWorker (opaque=0x5555558697a0) at util/virthreadpool.c:167 #10 0x00007ffff7294184 in virThreadHelper (data=0x555555869340) at util/virthread.c:206 #11 0x00007ffff3fdc3c4 in start_thread () from /lib64/libpthread.so.0 #12 0x00007ffff3d269cf in clone () from /lib64/libc.so.6