Bug 983344
| Summary: | QEMU core dump and host will reboot when do hot-unplug a virtio-blk disk which use the switch behind switch | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Sibiao Luo <sluo> | ||||||
| Component: | qemu-kvm | Assignee: | Markus Armbruster <armbru> | ||||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Virtualization Bugs <virt-bugs> | ||||||
| Severity: | medium | Docs Contact: | |||||||
| Priority: | medium | ||||||||
| Version: | 7.0 | CC: | armbru, chayang, flang, hhuang, juzhang, michen, pbonzini, qzhang, rbalakri, virt-maint, xfu | ||||||
| Target Milestone: | rc | Keywords: | Security, SecurityTracking | ||||||
| Target Release: | --- | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | qemu-kvm-1.5.3-43.el7 | Doc Type: | Release Note | ||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2015-01-19 11:58:18 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Bug Depends On: | |||||||||
| Bug Blocks: | 1012633, 1058321 | ||||||||
| Attachments: |
|
||||||||
|
Description
Sibiao Luo
2013-07-11 03:15:17 UTC
(gdb) bt
#0 0x00007f4ad6ef4a19 in raise () from /lib64/libc.so.6
#1 0x00007f4ad6ef6128 in abort () from /lib64/libc.so.6
#2 0x00007f4adb7faf03 in kvm_io_ioeventfd_del (listener=<optimized out>, section=0x7fff882bbb80,
match_data=<optimized out>, data=0, e=<optimized out>) at /usr/src/debug/qemu-1.5.1/kvm-all.c:862
#3 0x00007f4adb800620 in address_space_add_del_ioeventfds (fds_old_nb=38, fds_old=0x7f4ac000a460, fds_new_nb=37,
fds_new=0x7f4add8af750, as=0x7f4adc4c5e00 <address_space_io>) at /usr/src/debug/qemu-1.5.1/memory.c:603
#4 address_space_update_ioeventfds (as=0x7f4adc4c5e00 <address_space_io>) at /usr/src/debug/qemu-1.5.1/memory.c:649
#5 address_space_update_topology (as=0x7f4adc4c5e00 <address_space_io>) at /usr/src/debug/qemu-1.5.1/memory.c:730
#6 memory_region_transaction_commit () at /usr/src/debug/qemu-1.5.1/memory.c:750
#7 0x00007f4adb6f66cf in pci_unregister_io_regions (pci_dev=0x7f4adce4b440) at hw/pci/pci.c:889
#8 pci_unregister_device (dev=<optimized out>) at hw/pci/pci.c:900
#9 0x00007f4adb6b0154 in device_unrealize (dev=0x7f4adce4b440, errp=0x7fff882bbc70) at hw/core/qdev.c:191
#10 0x00007f4adb6b1824 in device_set_realized (obj=0x7f4adce4b440, value=<optimized out>, err=0x0) at hw/core/qdev.c:715
#11 0x00007f4adb76e53e in property_set_bool (obj=0x7f4adce4b440, v=<optimized out>, opaque=0x7f4adce2fc40,
name=<optimized out>, errp=0x0) at qom/object.c:1301
#12 0x00007f4adb770e07 in object_property_set_qobject (obj=0x7f4adce4b440, value=<optimized out>,
name=0x7f4adb8e094d "realized", errp=0x0) at qom/qom-qobject.c:24
#13 0x00007f4adb76fda0 in object_property_set_bool (obj=obj@entry=0x7f4adce4b440, value=value@entry=false,
name=name@entry=0x7f4adb8e094d "realized", errp=errp@entry=0x0) at qom/object.c:852
#14 0x00007f4adb6afe5b in device_unparent (obj=0x7f4adce4b440) at hw/core/qdev.c:798
#15 0x00007f4adb76f93a in object_unparent (obj=0x7f4adce4b440) at qom/object.c:372
#16 0x00007f4adb6b09cd in qdev_free (dev=<optimized out>) at hw/core/qdev.c:286
#17 0x00007f4adb6f911c in pcie_cap_slot_hotplug (qdev=<optimized out>, pci_dev=0x7f4adce4b440, state=
PCI_HOTPLUG_DISABLED) at hw/pci/pcie.c:254
#18 0x00007f4adb6b084f in qdev_unplug (dev=0x7f4adce4b440, errp=errp@entry=0x7fff882bbe80) at hw/core/qdev.c:219
#19 0x00007f4adb75c092 in qmp_device_del (id=<optimized out>, errp=errp@entry=0x7fff882bbe80) at qdev-monitor.c:628
#20 0x00007f4adb68e86b in hmp_device_del (mon=0x7f4adca8bc90, qdict=<optimized out>) at hmp.c:1177
#21 0x00007f4adb80aca9 in handle_user_command (mon=mon@entry=0x7f4adca8bc90, cmdline=<optimized out>)
at /usr/src/debug/qemu-1.5.1/monitor.c:3999
#22 0x00007f4adb80afab in monitor_command_cb (mon=0x7f4adca8bc90, cmdline=<optimized out>, opaque=<optimized out>)
at /usr/src/debug/qemu-1.5.1/monitor.c:4615
#23 0x00007f4adb7717c0 in readline_handle_byte (rs=0x7f4adcce7410, ch=<optimized out>) at readline.c:374
#24 0x00007f4adb80af14 in monitor_read (opaque=<optimized out>, buf=<optimized out>, size=<optimized out>)
at /usr/src/debug/qemu-1.5.1/monitor.c:4601
#25 0x00007f4adb75f81b in qemu_chr_be_write (len=<optimized out>, buf=0x7fff882bbfc0 "\r", s=0x7f4adc9ecfd0)
at qemu-char.c:167
#26 fd_chr_read (chan=<optimized out>, cond=<optimized out>, opaque=0x7f4adc9ecfd0) at qemu-char.c:801
#27 0x00007f4adacbae06 in g_main_context_dispatch () from /lib64/libglib-2.0.so.0
#28 0x00007f4adb737eba in glib_pollfds_poll () at main-loop.c:187
#29 os_host_main_loop_wait (timeout=<optimized out>) at main-loop.c:232
#30 main_loop_wait (nonblocking=<optimized out>) at main-loop.c:464
#31 0x00007f4adb638609 in main_loop () at vl.c:2029
#32 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4419
(gdb) bt full
#0 0x00007f4ad6ef4a19 in raise () from /lib64/libc.so.6
No symbol table info available.
#1 0x00007f4ad6ef6128 in abort () from /lib64/libc.so.6
No symbol table info available.
#2 0x00007f4adb7faf03 in kvm_io_ioeventfd_del (listener=<optimized out>, section=0x7fff882bbb80,
match_data=<optimized out>, data=0, e=<optimized out>) at /usr/src/debug/qemu-1.5.1/kvm-all.c:862
fd = <optimized out>
r = <optimized out>
#3 0x00007f4adb800620 in address_space_add_del_ioeventfds (fds_old_nb=38, fds_old=0x7f4ac000a460, fds_new_nb=37,
fds_new=0x7f4add8af750, as=0x7f4adc4c5e00 <address_space_io>) at /usr/src/debug/qemu-1.5.1/memory.c:603
_listener = 0x7f4adbcaaa00 <kvm_io_listener>
iold = 0
inew = 0
fd = 0x7f4ac000a460
section = {mr = 0x0, address_space = 0x7f4adc4c5e00 <address_space_io>, offset_within_region = 0, size = 2,
offset_within_address_space = 49168, readonly = false}
#4 address_space_update_ioeventfds (as=0x7f4adc4c5e00 <address_space_io>) at /usr/src/debug/qemu-1.5.1/memory.c:649
fr = <optimized out>
tmp = {start = {lo = 61456, hi = 0}, size = {lo = 2, hi = 0}}
i = <optimized out>
ioeventfd_nb = <optimized out>
ioeventfds = <optimized out>
#5 address_space_update_topology (as=0x7f4adc4c5e00 <address_space_io>) at /usr/src/debug/qemu-1.5.1/memory.c:730
old_view = <optimized out>
new_view = <optimized out>
#6 memory_region_transaction_commit () at /usr/src/debug/qemu-1.5.1/memory.c:750
as = 0x7f4adc4c5e00 <address_space_io>
#7 0x00007f4adb6f66cf in pci_unregister_io_regions (pci_dev=0x7f4adce4b440) at hw/pci/pci.c:889
r = 0x7f4adce4b548
i = 0
#8 pci_unregister_device (dev=<optimized out>) at hw/pci/pci.c:900
pci_dev = 0x7f4adce4b440
__func__ = "pci_unregister_device"
pc = 0x7f4adcde5c10
#9 0x00007f4adb6b0154 in device_unrealize (dev=0x7f4adce4b440, errp=0x7fff882bbc70) at hw/core/qdev.c:191
rc = <optimized out>
dc = <optimized out>
#10 0x00007f4adb6b1824 in device_set_realized (obj=0x7f4adce4b440, value=<optimized out>, err=0x0) at hw/core/qdev.c:715
dev = 0x7f4adce4b440
__func__ = "device_set_realized"
dc = 0x7f4adcde5c10
local_err = 0x0
#11 0x00007f4adb76e53e in property_set_bool (obj=0x7f4adce4b440, v=<optimized out>, opaque=0x7f4adce2fc40,
name=<optimized out>, errp=0x0) at qom/object.c:1301
prop = 0x7f4adce2fc40
value = false
local_err = 0x0
#12 0x00007f4adb770e07 in object_property_set_qobject (obj=0x7f4adce4b440, value=<optimized out>,
name=0x7f4adb8e094d "realized", errp=0x0) at qom/qom-qobject.c:24
mi = 0x7f4add8a75c0
#13 0x00007f4adb76fda0 in object_property_set_bool (obj=obj@entry=0x7f4adce4b440, value=value@entry=false,
name=name@entry=0x7f4adb8e094d "realized", errp=errp@entry=0x0) at qom/object.c:852
qbool = 0x7f4adce48a40
#14 0x00007f4adb6afe5b in device_unparent (obj=0x7f4adce4b440) at hw/core/qdev.c:798
dev = 0x7f4adce4b440
__func__ = "device_unparent"
bus = <optimized out>
event_data = <optimized out>
have_realized = true
#15 0x00007f4adb76f93a in object_unparent (obj=0x7f4adce4b440) at qom/object.c:372
No locals.
#16 0x00007f4adb6b09cd in qdev_free (dev=<optimized out>) at hw/core/qdev.c:286
No locals.
#17 0x00007f4adb6f911c in pcie_cap_slot_hotplug (qdev=<optimized out>, pci_dev=0x7f4adce4b440, state=
PCI_HOTPLUG_DISABLED) at hw/pci/pcie.c:254
d = 0x7f4adce43350
__func__ = "pcie_cap_slot_hotplug"
exp_cap = 0x7f4adce44520 "\020\200b\001"
sltsta = <optimized out>
__PRETTY_FUNCTION__ = "pcie_cap_slot_hotplug"
#18 0x00007f4adb6b084f in qdev_unplug (dev=0x7f4adce4b440, errp=errp@entry=0x7fff882bbe80) at hw/core/qdev.c:219
dc = <optimized out>
__func__ = "qdev_unplug"
__PRETTY_FUNCTION__ = "qdev_unplug"
#19 0x00007f4adb75c092 in qmp_device_del (id=<optimized out>, errp=errp@entry=0x7fff882bbe80) at qdev-monitor.c:628
dev = <optimized out>
#20 0x00007f4adb68e86b in hmp_device_del (mon=0x7f4adca8bc90, qdict=<optimized out>) at hmp.c:1177
id = <optimized out>
err = 0x0
#21 0x00007f4adb80aca9 in handle_user_command (mon=mon@entry=0x7f4adca8bc90, cmdline=<optimized out>)
at /usr/src/debug/qemu-1.5.1/monitor.c:3999
qdict = 0x7f4add8a65a0
cmd = 0x7f4adbcab280 <mon_cmds+1664>
__PRETTY_FUNCTION__ = "handle_user_command"
#22 0x00007f4adb80afab in monitor_command_cb (mon=0x7f4adca8bc90, cmdline=<optimized out>, opaque=<optimized out>)
at /usr/src/debug/qemu-1.5.1/monitor.c:4615
No locals.
#23 0x00007f4adb7717c0 in readline_handle_byte (rs=0x7f4adcce7410, ch=<optimized out>) at readline.c:374
No locals.
#24 0x00007f4adb80af14 in monitor_read (opaque=<optimized out>, buf=<optimized out>, size=<optimized out>)
at /usr/src/debug/qemu-1.5.1/monitor.c:4601
old_mon = 0x0
i = <optimized out>
#25 0x00007f4adb75f81b in qemu_chr_be_write (len=<optimized out>, buf=0x7fff882bbfc0 "\r", s=0x7f4adc9ecfd0)
at qemu-char.c:167
No locals.
#26 fd_chr_read (chan=<optimized out>, cond=<optimized out>, opaque=0x7f4adc9ecfd0) at qemu-char.c:801
chr = 0x7f4adc9ecfd0
s = 0x7f4adc9ed0a0
len = <optimized out>
buf = "\r\000\000\000\000\000\000\000\250\211\272\336J\177\000\000\000\020", '\000' <repeats 22 times>, "芺\336J\177\000\000\000\020", '\000' <repeats 22 times>, "\210\241\272\336J\177\000\000\000\020", '\000' <repeats 22 times>, "H\275\337\334J\177\000\000\000\020", '\000' <repeats 22 times>, "(\242\272\336J\177\000\000\000\060", '\000' <repeats 22 times>, "X\255\307\337J\177\000\000\000\020", '\000' <repeats 14 times>, "\240\300+\210\377\177\000\000aD"...
status = <optimized out>
bytes_read = 1
#27 0x00007f4adacbae06 in g_main_context_dispatch () from /lib64/libglib-2.0.so.0
No symbol table info available.
#28 0x00007f4adb737eba in glib_pollfds_poll () at main-loop.c:187
context = 0x7f4adc9ec000
pfds = <optimized out>
#29 os_host_main_loop_wait (timeout=<optimized out>) at main-loop.c:232
ret = 1
spin_counter = 0
#30 main_loop_wait (nonblocking=<optimized out>) at main-loop.c:464
ret = 1
timeout = 4294967295
#31 0x00007f4adb638609 in main_loop () at vl.c:2029
nonblocking = <optimized out>
last_io = 1
#32 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4419
i = <optimized out>
snapshot = 0
linux_boot = <optimized out>
icount_option = 0x0
initrd_filename = <optimized out>
kernel_filename = <optimized out>
kernel_cmdline = <optimized out>
boot_devices = '\000' <repeats 32 times>
ds = <optimized out>
cyls = 0
heads = 0
secs = 0
translation = 0
hda_opts = <optimized out>
opts = <optimized out>
machine_opts = <optimized out>
olist = <optimized out>
optind = 80
optarg = 0x7fff882be7b7 "virtio-blk-pci,scsi=on,bus=downstream2,addr=0x0,drive=drive-data-disk1,id=data-disk1"
loadvm = 0x0
machine = 0x7f4adbca8f60 <pc_q35_machine_rhel700>
cpu_model = 0x7fff882bdfe3 "SandyBridge"
vga_model = 0x7f4adb9072ef "cirrus"
pid_file = 0x0
incoming = 0x0
show_vnc_port = 0
defconfig = <optimized out>
userconfig = false
log_mask = 0x0
log_file = 0x0
mem_trace = {malloc = 0x7f4adb7a1060 <malloc_and_trace>, realloc = 0x7f4adb7a1020 <realloc_and_trace>,
free = 0x7f4adb7a0fe0 <free_and_trace>, calloc = 0x0, try_malloc = 0x0, try_realloc = 0x0}
trace_events = 0x0
trace_file = 0x0
__PRETTY_FUNCTION__ = "main"
args = {ram_size = 4294967296, boot_device = 0x7f4adb8dd7a6 "cad", kernel_filename = 0x0,
kernel_cmdline = 0x7f4adb923c90 "", initrd_filename = 0x0, cpu_model = 0x7fff882bdfe3 "SandyBridge"}
(gdb)
I also tried the PCIE bridge that can do hot-unplug successfully, did not met any core dump, both guest and host work well after do hot-unplug. # /usr/libexec/qemu-kvm -S -M q35 -cpu SandyBridge -enable-kvm...-device pci-bridge,bus=pcie.0,id=bridge1,chassis_nr=1,addr=0x3 -drive file=/home/my-data-disk2.qcow3,if=none,id=drive-data-disk2,format=qcow2,cache=none,aio=native,werror=stop,rerror=stop,serial="QEMU-DISK3" -device virtio-blk-pci,scsi=on,bus=bridge1,addr=0x6,drive=drive-data-disk2,id=data-disk2 (qemu) info block drive-system-disk: removable=0 io-status=ok file=/home/RHEL-7.0-20130628.0-Server-x86_64.qcow3 ro=0 drv=qcow2 encrypted=0 bps=0 bps_rd=0 bps_wr=0 iops=0 iops_rd=0 iops_wr=0 drive-data-disk2: removable=0 io-status=ok file=/home/my-data-disk2.qcow3 ro=0 drv=qcow2 encrypted=0 bps=0 bps_rd=0 bps_wr=0 iops=0 iops_rd=0 iops_wr=0 (qemu) device_del data-disk2 (qemu) drive_del drive-data-disk2 (qemu) info block drive-system-disk: removable=0 io-status=ok file=/home/RHEL-7.0-20130628.0-Server-x86_64.qcow3 ro=0 drv=qcow2 encrypted=0 bps=0 bps_rd=0 bps_wr=0 iops=0 iops_rd=0 iops_wr=0 Best Regards, sluo (In reply to Sibiao Luo from comment #2) > I also tried the PCIE bridge that can do hot-unplug successfully, did not > met any core dump, both guest and host work well after do hot-unplug. > sorry, it's pci birdge not pcie bridge, thx. It can do hot-unplug a virtio-blk disk which use the switch behind switch successfully and work well. # /usr/libexec/qemu-kvm -S -M q35 -cpu SandyBridge -enable-kvm ...-device ioh3420,bus=pcie.0,id=root.2,slot=3 -device x3130-upstream,bus=root.2,id=upstream2 -device xio3130-downstream,bus=upstream2,id=downstream2,chassis=3 (qemu) drive_add pci_addr=auto file=/home/my-data-disk1.qcow3,if=none,id=drive-data-disk,format=qcow2,werror=stop,rerror=stop OK (qemu) device_add virtio-blk-pci,scsi=on,bus=downstream2,addr=0x0,drive=drive-data-disk,id=data-disk (qemu) info block drive-system-disk: removable=0 io-status=ok file=/home/RHEL-7.0-20130628.0-Server-x86_64.qcow3 ro=0 drv=qcow2 encrypted=0 bps=0 bps_rd=0 bps_wr=0 iops=0 iops_rd=0 iops_wr=0 ide1-cd0: removable=1 locked=0 tray-open=0 [not inserted] floppy0: removable=1 locked=0 tray-open=0 [not inserted] sd0: removable=1 locked=0 tray-open=0 [not inserted] drive-data-disk: removable=0 io-status=ok file=/home/my-data-disk1.qcow3 ro=0 drv=qcow2 encrypted=0 bps=0 bps_rd=0 bps_wr=0 iops=0 iops_rd=0 iops_wr=0 (qemu) # nc -U /tmp/ttyS0 [ 356.774643] pciehp 0000:09:00.0:pcie24: Card present on Slot(0-2) [ 357.887193] pci 0000:0a:00.0: BAR 1: assigned [mem 0xc0400000-0xc0400fff] [ 357.889314] pci 0000:0a:00.0: BAR 0: assigned [io 0x1000-0x103f] [ 357.890410] pcieport 0000:09:00.0: PCI bridge to [bus 0a] [ 357.891371] pcieport 0000:09:00.0: bridge window [io 0x1000-0x1fff] [ 357.902919] pcieport 0000:09:00.0: bridge window [mem 0xc0400000-0xc05fffff] [ 357.910516] pcieport 0000:09:00.0: bridge window [mem 0xc0600000-0xc07fffff 64bit pref] [ 357.925532] pci 0000:0a:00.0: no hotplug settings from platform [ 357.926960] pci 0000:0a:00.0: using default PCI settings [ 357.928151] virtio-pci 0000:0a:00.0: enabling device (0000 -> 0003) [ 357.935934] ACPI: PCI Interrupt Link [GSIF] enabled at IRQ 21 [ 357.941661] vdb: unknown partition table Best Regards, sluo (In reply to Sibiao Luo from comment #4) > It can do hot-unplug a virtio-blk disk which use the switch behind switch > successfully and work well. It can do *hot-plug* a virtio-blk disk which use the switch behind switch successfully and work well. Here's how I understand the bug report. In addition to the primary test case that exposes the bug, there are two additional test cases that work find. Test case 1 (comment#0): hot unplug of virtio-blk-pci data disk cold-plugged into a certain *pcie* port ("switch behind switch") crashes qemu instantly and reliably. Sometimes, host reboot follows a few minutes later (that part is astonishing). Test case 2 (comment#2+3): similar to 1., but use a *pci* bridge instead of pcie, works fine. Test case 3 (comment#4+5): start qemu-kvm just like in 1., except don't cold-plug the virtio-blk-pci data disk, hot plug it instead. Works fine. (In reply to Markus Armbruster from comment #6) > Here's how I understand the bug report. In addition to the primary > test case that exposes the bug, there are two additional test cases > that work find. > > Test case 1 (comment#0): hot unplug of virtio-blk-pci data disk > cold-plugged into a certain *pcie* port ("switch behind switch") > crashes qemu instantly and reliably. Sometimes, host reboot follows a > few minutes later (that part is astonishing). *PCIE* + "switch behind switch" + hot-unplug=========>crash > Test case 2 (comment#2+3): similar to 1., but use a *pci* bridge > instead of pcie, works fine. *PCI* + "switch behind switch" + hot-unplug =========>good > Test case 3 (comment#4+5): start qemu-kvm just like in 1., except > don't cold-plug the virtio-blk-pci data disk, hot plug it instead. > Works fine. *PCIE* + "switch behind switch" + hot-plug =========>good Best Regards, sluo The bug is in the virtio-pci implementation and was introduced by the virtio refactoring.
The bus structure is as follows
virtio-blk-pci
\
\ virtio bus
\
virtio-blk-device
When the virtio-blk-pci device is deleted, the virtio-blk-device is removed first, because removal is done in post-order. Later, the virtio-blk-device is accessed again but proxy->vdev->vq is now a dangling pointer and it is not valid anymore. Hence, kvm_set_ioeventfd_pio fails.
CVE requested for this, see bug 1012633. Also set this as a security tracking bug and set flags accordingly. I backported the patches mentioned in comment#9. They fix the qemu-kvm crash, but now I my guest kernel-3.10.0-71.el7.x86_64 complains: [ 37.674257] ------------[ cut here ]------------ [ 37.674296] WARNING: at drivers/virtio/virtio.c:158 virtio_dev_remove+0x74/0x80 [virtio]() [ 37.674301] Modules linked in: ip6t_rpfilter ip6t_REJECT ipt_REJECT xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw iptable_filter ip_tables sg kvm_amd kvm pcspkr serio_raw i2c_i801 lpc_ich mfd_core mperf shpchp xfs libcrc32c sr_mod cdrom cirrus virtio_net virtio_blk syscopyarea sysfillrect sysimgblt drm_kms_helper ahci ttm libahci drm libata virtio_pci virtio_ring virtio i2c_core dm_mirror dm_region_hash dm_log dm_mod [ 37.674374] CPU: 0 PID: 22 Comm: kworker/0:1 Not tainted 3.10.0-71.el7.x86_64 #1 [ 37.674380] Hardware name: Red Hat KVM, BIOS Bochs 01/01/2011 [ 37.674393] Workqueue: pciehp-0 pciehp_power_thread [ 37.674397] 0000000000000009 ffff8802443dbb30 ffffffff815bd8c4 ffff8802443dbb68 [ 37.674405] ffffffff81059c61 ffff880241b0c400 ffff880241b0c408 ffffffffa0024000 [ 37.674411] ffff880244312098 0000000000000000 ffff8802443dbb78 ffffffff81059d3a [ 37.674418] Call Trace: [ 37.674431] [<ffffffff815bd8c4>] dump_stack+0x19/0x1b [ 37.674443] [<ffffffff81059c61>] warn_slowpath_common+0x61/0x80 [ 37.674454] [<ffffffff81059d3a>] warn_slowpath_null+0x1a/0x20 [ 37.674465] [<ffffffffa00220e4>] virtio_dev_remove+0x74/0x80 [virtio] [ 37.674476] [<ffffffff8139535f>] __device_release_driver+0x7f/0xf0 [ 37.674484] [<ffffffff813953f3>] device_release_driver+0x23/0x30 [ 37.674491] [<ffffffff81394b88>] bus_remove_device+0x108/0x180 [ 37.674498] [<ffffffff81391485>] device_del+0x135/0x1d0 [ 37.674505] [<ffffffff8139153e>] device_unregister+0x1e/0x60 [ 37.674516] [<ffffffffa00224b6>] unregister_virtio_device+0x16/0x30 [virtio] [ 37.674527] [<ffffffffa004c56b>] virtio_pci_remove+0x2b/0x70 [virtio_pci] [ 37.674537] [<ffffffff812d252b>] pci_device_remove+0x3b/0xb0 [ 37.674546] [<ffffffff8139535f>] __device_release_driver+0x7f/0xf0 [ 37.674553] [<ffffffff813953f3>] device_release_driver+0x23/0x30 [ 37.674560] [<ffffffff81394b88>] bus_remove_device+0x108/0x180 [ 37.674567] [<ffffffff81391485>] device_del+0x135/0x1d0 [ 37.674576] [<ffffffff812cc064>] pci_stop_bus_device+0x94/0xa0 [ 37.674583] [<ffffffff812cc152>] pci_stop_and_remove_bus_device+0x12/0x20 [ 37.674591] [<ffffffff812e4bd8>] pciehp_unconfigure_device+0xa8/0x1b0 [ 37.674599] [<ffffffff812e4538>] pciehp_disable_slot+0x68/0x200 [ 37.674607] [<ffffffff812e4753>] pciehp_power_thread+0x83/0xf0 [ 37.674616] [<ffffffff8107862b>] process_one_work+0x17b/0x460 [ 37.674623] [<ffffffff810793db>] worker_thread+0x11b/0x400 [ 37.674631] [<ffffffff810792c0>] ? rescuer_thread+0x3e0/0x3e0 [ 37.674638] [<ffffffff8107fb90>] kthread+0xc0/0xd0 [ 37.674646] [<ffffffff8107fad0>] ? kthread_create_on_node+0x110/0x110 [ 37.674653] [<ffffffff815cd66c>] ret_from_fork+0x7c/0xb0 [ 37.674659] [<ffffffff8107fad0>] ? kthread_create_on_node+0x110/0x110 [ 37.674665] ---[ end trace f5de3b0770382ce3 ]--- Paolo, is this expected? No idea if this is expected. It definitely deserves a BZ, but a better question could be, is this a regression from RHEL6 and/or PIIX? That is: - does a RHEL6 kernel complain? - does the RHEL7 kernel complain with "-M pc"? - if the answer to the second question is "yes", does a RHEL7 kernel complain on top of RHEL6 qemu? Same test, except for the guest: * kernel-2.6.32-279.el6.x86_64 crashes hard * kernel-3.3.0-0.20.el7.x86_64[*] crashes hard I can't answer your other questions, because I don't know how to reproduce without -M q35 (see comment#6). [*] Look what the cat dragged in! Created attachment 854354 [details]
tail of dmesg from crashed kernel-2.6.32-279.el6.x86_64
Created attachment 854355 [details]
tail of dmesg from crashed kernel-3.3.0-0.20.el7.x86_64
I reproduced the guest kernel oops from comment#11 under latest upstream QEMU. I'd file a BZ against kernel if I had a reproducer that doesn't involve patching qemu-kvm first, or compiling upstream QEMU. Perhaps we should just commit the backport, then file the kernel BZ (assuming it still reproduces then). Paolo, what do you think? I agree. Regarding the other questions, I meant with "-M pc" and a "normal" setup without PCIe switches. Hot unplugging the device works fine when it's plugged straight into i440FX's PCI bus. I tried to test with a pci-bridge device in between, but I can't get devices on the bridge recognized by the guest. Not a PCI guy... to comment 18: this warning implies fails to read status from device or that driver failed to reset device. I think we should tread carefully here: could be that delete does not wait for guest ack properly? Fix included in qemu-kvm-1.5.3-43.el7 Reproduce this bug as follow version: Host: # uname -r 3.10.0-84.el7.x86_64 # rpm -q qemu-kvm qemu-kvm-1.5.3-37.el7.x86_64 # rpm -q seabios seabios-1.7.2.2-11.el7.x86_64 Guest:rhel7 Steps: 1.Boot guest with the CLI like comment0 2.Hotunplug data disk (qemu) device_del data-disk1 Results:Core dump (gdb) bt #0 0x00007ffff2c99989 in raise () from /lib64/libc.so.6 #1 0x00007ffff2c9b098 in abort () from /lib64/libc.so.6 #2 0x00005555557d06c3 in kvm_io_ioeventfd_del (listener=<optimized out>, section=0x7fffffffc230, match_data=<optimized out>, data=0, e=<optimized out>) at /usr/src/debug/qemu-1.5.3/kvm-all.c:844 #3 0x00005555557d5f30 in address_space_add_del_ioeventfds (fds_old_nb=38, fds_old=0x7fffd802b870, fds_new_nb=37, fds_new=0x555556bb4370, as=0x5555564beaa0 <address_space_io>) at /usr/src/debug/qemu-1.5.3/memory.c:604 #4 address_space_update_ioeventfds (as=0x5555564beaa0 <address_space_io>) at /usr/src/debug/qemu-1.5.3/memory.c:650 #5 address_space_update_topology (as=0x5555564beaa0 <address_space_io>) at /usr/src/debug/qemu-1.5.3/memory.c:731 #6 memory_region_transaction_commit () at /usr/src/debug/qemu-1.5.3/memory.c:751 #7 0x00005555556c2b6f in pci_unregister_io_regions (pci_dev=0x555556973180) at hw/pci/pci.c:894 #8 pci_unregister_device (dev=<optimized out>) at hw/pci/pci.c:905 #9 0x000055555567ed74 in device_unrealize (dev=0x555556973180, errp=0x7fffffffc320) at hw/core/qdev.c:191 #10 0x0000555555680404 in device_set_realized (obj=0x555556973180, value=<optimized out>, err=0x0) at hw/core/qdev.c:709 #11 0x0000555555740b5e in property_set_bool (obj=0x555556973180, v=<optimized out>, opaque=0x555556973b30, name=<optimized out>, errp=0x0) at qom/object.c:1302 #12 0x0000555555743717 in object_property_set_qobject (obj=0x555556973180, value=<optimized out>, ... Verify this bug as follow version: Host: # uname -r 3.10.0-84.el7.x86_64 # rpm -q qemu-kvm qemu-kvm-1.5.3-45.el7.x86_64 # rpm -q seabios seabios-1.7.2.2-11.el7.x86_64 Guest:rhel7 3.10.0-48.el7.x86_64 Steps as same as reproduce Results: Not hit qemu core dump problem, but will hit guest kernel call trace. [ 84.204412] ------------[ cut here ]------------ [ 84.204983] WARNING: at drivers/virtio/virtio.c:158 virtio_dev_remove+0x74/0x80 [virtio]() [ 84.205899] Modules linked in: nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_MASQUERADE ip6t_REJECT ipt_REJECT xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw iptable_filter ip_tables sg crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd virtio_console serio_raw pcspkr lpc_ich mfd_core i2c_i801 shpchp microcode mperf xfs libcrc32c virtio_blk cirrus syscopyarea virtio_net ahci libahci sysfillrect sysimgblt drm_kms_helper ttm virtio_pci [ 84.214738] drm libata virtio_ring virtio i2c_core dm_mirror dm_region_hash dm_log dm_mod [ 84.215679] CPU: 0 PID: 43 Comm: kworker/0:1 Not tainted 3.10.0-48.el7.x86_64 #1 [ 84.216454] Hardware name: Red Hat KVM, BIOS Bochs 01/01/2011 [ 84.217071] Workqueue: pciehp-0 pciehp_power_thread [ 84.217601] 0000000000000009 ffff8801796c5b30 ffffffff815abfff ffff8801796c5b68 [ 84.218453] ffffffff810589d1 ffff880176f09800 ffff880176f09808 ffffffffa0024000 [ 84.219304] ffff8801795c8098 0000000000000000 ffff8801796c5b78 ffffffff81058aaa [ 84.220155] Call Trace: [ 84.220435] [<ffffffff815abfff>] dump_stack+0x19/0x1b [ 84.220976] [<ffffffff810589d1>] warn_slowpath_common+0x61/0x80 [ 84.221620] [<ffffffff81058aaa>] warn_slowpath_null+0x1a/0x20 [ 84.222249] [<ffffffffa00220e4>] virtio_dev_remove+0x74/0x80 [virtio] [ 84.222936] [<ffffffff8138cfaf>] __device_release_driver+0x7f/0xf0 [ 84.223569] [<ffffffff8138d043>] device_release_driver+0x23/0x30 [ 84.224191] [<ffffffff8138c814>] bus_remove_device+0xf4/0x170 [ 84.224775] [<ffffffff813891e5>] device_del+0x135/0x1d0 [ 84.225324] [<ffffffff8138929e>] device_unregister+0x1e/0x60 [ 84.225903] [<ffffffffa00224b6>] unregister_virtio_device+0x16/0x30 [virtio] [ 84.226620] [<ffffffffa00fb56b>] virtio_pci_remove+0x2b/0x70 [virtio_pci] [ 84.227322] [<ffffffff812cab7b>] pci_device_remove+0x3b/0xb0 [ 84.227900] [<ffffffff8138cfaf>] __device_release_driver+0x7f/0xf0 [ 84.228531] [<ffffffff8138d043>] device_release_driver+0x23/0x30 [ 84.229155] [<ffffffff8138c814>] bus_remove_device+0xf4/0x170 [ 84.229732] [<ffffffff813891e5>] device_del+0x135/0x1d0 [ 84.230283] [<ffffffff812c46e4>] pci_stop_bus_device+0x94/0xa0 [ 84.230877] [<ffffffff812c47d2>] pci_stop_and_remove_bus_device+0x12/0x20 [ 84.231566] [<ffffffff812dd108>] pciehp_unconfigure_device+0xa8/0x1b0 [ 84.232230] [<ffffffff812dca68>] pciehp_disable_slot+0x68/0x200 [ 84.232837] [<ffffffff812dcc83>] pciehp_power_thread+0x83/0xf0 [ 84.237879] [<ffffffff8107738b>] process_one_work+0x17b/0x460 [ 84.238521] [<ffffffff8107813b>] worker_thread+0x11b/0x400 [ 84.239090] [<ffffffff81078020>] ? rescuer_thread+0x3e0/0x3e0 [ 84.239761] [<ffffffff8107e8f0>] kthread+0xc0/0xd0 [ 84.240295] [<ffffffff8107e830>] ? kthread_create_on_node+0x110/0x110 [ 84.240963] [<ffffffff815bb6ec>] ret_from_fork+0x7c/0xb0 [ 84.241501] [<ffffffff8107e830>] ? kthread_create_on_node+0x110/0x110 [ 84.242246] ---[ end trace e2fb0b185e17a1f1 ]--- As above test , this bug fixed, but hit guest kernel problem. Hi, Markus as you said comment18,you will file a BZ against kernel , what's the Bug ID? thanks fang lang (In reply to Markus Armbruster from comment #25) > Bug 1058321. Hi Markus thanks for your help,according to comment24 test, so we can verify this bug now. |