RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 983344 - QEMU core dump and host will reboot when do hot-unplug a virtio-blk disk which use the switch behind switch
Summary: QEMU core dump and host will reboot when do hot-unplug a virtio-blk disk whic...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm
Version: 7.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: ---
Assignee: Markus Armbruster
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks: CVE-2013-4377 1058321
TreeView+ depends on / blocked
 
Reported: 2013-07-11 03:15 UTC by Sibiao Luo
Modified: 2015-01-19 11:58 UTC (History)
11 users (show)

Fixed In Version: qemu-kvm-1.5.3-43.el7
Doc Type: Release Note
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-01-19 11:58:18 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
tail of dmesg from crashed kernel-2.6.32-279.el6.x86_64 (5.44 KB, text/x-vhdl)
2014-01-23 12:24 UTC, Markus Armbruster 		no flags Details
tail of dmesg from crashed kernel-3.3.0-0.20.el7.x86_64 (4.13 KB, text/plain)
2014-01-23 12:25 UTC, Markus Armbruster 		no flags Details
View All

Description Sibiao Luo 2013-07-11 03:15:17 UTC 
Description of problem:
boot guest with a virtio-blk data disk which use  the switch behind switch, then do hot-unplug to it, QEMU will core dump.

Version-Release number of selected component (if applicable):
host info:
3.10.0-1.el7.x86_64
qemu-kvm-1.5.1-2.el7.x86_64
guest info:
win8 64bit
rhel7: 3.10.0-1.el7.x86_64

How reproducible:
always

Steps to Reproduce:
1.boot guest with a virtio-blk data disk which use  the switch behind switch.
e.g:...-device ioh3420,bus=pcie.0,id=root.2,slot=3 -device x3130-upstream,bus=root.2,id=upstream2 -device xio3130-downstream,bus=upstream2,id=downstream2,chassis=3 -drive file=/home/my-data-disk1.qcow3,if=none,id=drive-data-disk1,format=qcow2,cache=none,aio=native,werror=stop,rerror=stop,serial="QEMU-DISK2" -device virtio-blk-pci,scsi=on,bus=downstream2,addr=0x0,drive=drive-data-disk1,id=data-disk1
2.do hot-unplug to the data disk.
(qemu) device_del data-disk1
3.

Actual results:
after step 2, qemu core dump, and after a few minutes the host will reboot, i will attach the bt log later.
(qemu) device_del data-disk1
Aborted (core dumped)

Expected results:
it should do hot-unplug successfully and both guest/host work well.

Additional info:
# /usr/libexec/qemu-kvm -S -M q35 -cpu SandyBridge -enable-kvm -m 4096 -smp 4,sockets=2,cores=2,threads=1 -no-kvm-pit-reinjection -name sluo -uuid 355a2475-4e03-4cdd-bf7b-5d6a59edaa61 -rtc base=localtime,clock=host,driftfix=slew -device pci-bridge,bus=pcie.0,id=bridge1,chassis_nr=1,addr=0x3 -device virtio-serial-pci,id=virtio-serial0,max_ports=16,vectors=0,bus=bridge1,addr=0x4 -chardev socket,id=channel1,path=/tmp/helloworld1,server,nowait -device virtserialport,chardev=channel1,name=com.redhat.rhevm.vdsm,bus=virtio-serial0.0,id=port1 -chardev socket,id=channel2,path=/tmp/helloworld2,server,nowait -device virtserialport,chardev=channel2,name=com.redhat.rhevm.vdsm,bus=virtio-serial0.0,id=port2 -device ioh3420,bus=pcie.0,id=root.0,slot=1 -device x3130-upstream,bus=root.0,id=upstream0 -device xio3130-downstream,bus=upstream0,id=downstream0,chassis=1 -drive file=/home/RHEL-7.0-20130628.0-Server-x86_64.qcow3,if=none,id=drive-system-disk,format=qcow2,cache=none,aio=native,werror=stop,rerror=stop,serial="QEMU-DISK1" -device virtio-blk-pci,scsi=off,bus=downstream0,addr=0x0,drive=drive-system-disk,id=system-disk,bootindex=1 -device virtio-balloon-pci,id=ballooning,bus=downstream0,addr=0x5 -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -device ioh3420,bus=pcie.0,id=root.1,slot=2 -device x3130-upstream,bus=root.1,id=upstream1 -device xio3130-downstream,bus=upstream1,id=downstream1,chassis=2 -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device virtio-net-pci,netdev=hostnet0,id=virtio-net-pci0,mac=08:2e:5f:0a:0d:b1,bus=downstream1,addr=0x0,bootindex=2 -k en-us -boot menu=on -qmp tcp:0:4444,server,nowait -serial unix:/tmp/ttyS0,server,nowait -vnc :1 -spice port=5931,disable-ticketing -monitor stdio -device ioh3420,bus=pcie.0,id=root.2,slot=3 -device x3130-upstream,bus=root.2,id=upstream2 -device xio3130-downstream,bus=upstream2,id=downstream2,chassis=3 -drive file=/home/my-data-disk1.qcow3,if=none,id=drive-data-disk1,format=qcow2,cache=none,aio=native,werror=stop,rerror=stop,serial="QEMU-DISK2" -device virtio-blk-pci,scsi=on,bus=downstream2,addr=0x0,drive=drive-data-disk1,id=data-disk1
Comment 1 Sibiao Luo 2013-07-11 03:18:36 UTC 
(gdb) bt
#0  0x00007f4ad6ef4a19 in raise () from /lib64/libc.so.6
#1  0x00007f4ad6ef6128 in abort () from /lib64/libc.so.6
#2  0x00007f4adb7faf03 in kvm_io_ioeventfd_del (listener=<optimized out>, section=0x7fff882bbb80, 
    match_data=<optimized out>, data=0, e=<optimized out>) at /usr/src/debug/qemu-1.5.1/kvm-all.c:862
#3  0x00007f4adb800620 in address_space_add_del_ioeventfds (fds_old_nb=38, fds_old=0x7f4ac000a460, fds_new_nb=37, 
    fds_new=0x7f4add8af750, as=0x7f4adc4c5e00 <address_space_io>) at /usr/src/debug/qemu-1.5.1/memory.c:603
#4  address_space_update_ioeventfds (as=0x7f4adc4c5e00 <address_space_io>) at /usr/src/debug/qemu-1.5.1/memory.c:649
#5  address_space_update_topology (as=0x7f4adc4c5e00 <address_space_io>) at /usr/src/debug/qemu-1.5.1/memory.c:730
#6  memory_region_transaction_commit () at /usr/src/debug/qemu-1.5.1/memory.c:750
#7  0x00007f4adb6f66cf in pci_unregister_io_regions (pci_dev=0x7f4adce4b440) at hw/pci/pci.c:889
#8  pci_unregister_device (dev=<optimized out>) at hw/pci/pci.c:900
#9  0x00007f4adb6b0154 in device_unrealize (dev=0x7f4adce4b440, errp=0x7fff882bbc70) at hw/core/qdev.c:191
#10 0x00007f4adb6b1824 in device_set_realized (obj=0x7f4adce4b440, value=<optimized out>, err=0x0) at hw/core/qdev.c:715
#11 0x00007f4adb76e53e in property_set_bool (obj=0x7f4adce4b440, v=<optimized out>, opaque=0x7f4adce2fc40, 
    name=<optimized out>, errp=0x0) at qom/object.c:1301
#12 0x00007f4adb770e07 in object_property_set_qobject (obj=0x7f4adce4b440, value=<optimized out>, 
    name=0x7f4adb8e094d "realized", errp=0x0) at qom/qom-qobject.c:24
#13 0x00007f4adb76fda0 in object_property_set_bool (obj=obj@entry=0x7f4adce4b440, value=value@entry=false, 
    name=name@entry=0x7f4adb8e094d "realized", errp=errp@entry=0x0) at qom/object.c:852
#14 0x00007f4adb6afe5b in device_unparent (obj=0x7f4adce4b440) at hw/core/qdev.c:798
#15 0x00007f4adb76f93a in object_unparent (obj=0x7f4adce4b440) at qom/object.c:372
#16 0x00007f4adb6b09cd in qdev_free (dev=<optimized out>) at hw/core/qdev.c:286
#17 0x00007f4adb6f911c in pcie_cap_slot_hotplug (qdev=<optimized out>, pci_dev=0x7f4adce4b440, state=
    PCI_HOTPLUG_DISABLED) at hw/pci/pcie.c:254
#18 0x00007f4adb6b084f in qdev_unplug (dev=0x7f4adce4b440, errp=errp@entry=0x7fff882bbe80) at hw/core/qdev.c:219
#19 0x00007f4adb75c092 in qmp_device_del (id=<optimized out>, errp=errp@entry=0x7fff882bbe80) at qdev-monitor.c:628
#20 0x00007f4adb68e86b in hmp_device_del (mon=0x7f4adca8bc90, qdict=<optimized out>) at hmp.c:1177
#21 0x00007f4adb80aca9 in handle_user_command (mon=mon@entry=0x7f4adca8bc90, cmdline=<optimized out>)
    at /usr/src/debug/qemu-1.5.1/monitor.c:3999
#22 0x00007f4adb80afab in monitor_command_cb (mon=0x7f4adca8bc90, cmdline=<optimized out>, opaque=<optimized out>)
    at /usr/src/debug/qemu-1.5.1/monitor.c:4615
#23 0x00007f4adb7717c0 in readline_handle_byte (rs=0x7f4adcce7410, ch=<optimized out>) at readline.c:374
#24 0x00007f4adb80af14 in monitor_read (opaque=<optimized out>, buf=<optimized out>, size=<optimized out>)
    at /usr/src/debug/qemu-1.5.1/monitor.c:4601
#25 0x00007f4adb75f81b in qemu_chr_be_write (len=<optimized out>, buf=0x7fff882bbfc0 "\r", s=0x7f4adc9ecfd0)
    at qemu-char.c:167
#26 fd_chr_read (chan=<optimized out>, cond=<optimized out>, opaque=0x7f4adc9ecfd0) at qemu-char.c:801
#27 0x00007f4adacbae06 in g_main_context_dispatch () from /lib64/libglib-2.0.so.0
#28 0x00007f4adb737eba in glib_pollfds_poll () at main-loop.c:187
#29 os_host_main_loop_wait (timeout=<optimized out>) at main-loop.c:232
#30 main_loop_wait (nonblocking=<optimized out>) at main-loop.c:464
#31 0x00007f4adb638609 in main_loop () at vl.c:2029
#32 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4419
(gdb) bt full
#0  0x00007f4ad6ef4a19 in raise () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007f4ad6ef6128 in abort () from /lib64/libc.so.6
No symbol table info available.
#2  0x00007f4adb7faf03 in kvm_io_ioeventfd_del (listener=<optimized out>, section=0x7fff882bbb80, 
    match_data=<optimized out>, data=0, e=<optimized out>) at /usr/src/debug/qemu-1.5.1/kvm-all.c:862
        fd = <optimized out>
        r = <optimized out>
#3  0x00007f4adb800620 in address_space_add_del_ioeventfds (fds_old_nb=38, fds_old=0x7f4ac000a460, fds_new_nb=37, 
    fds_new=0x7f4add8af750, as=0x7f4adc4c5e00 <address_space_io>) at /usr/src/debug/qemu-1.5.1/memory.c:603
        _listener = 0x7f4adbcaaa00 <kvm_io_listener>
        iold = 0
        inew = 0
        fd = 0x7f4ac000a460
        section = {mr = 0x0, address_space = 0x7f4adc4c5e00 <address_space_io>, offset_within_region = 0, size = 2, 
          offset_within_address_space = 49168, readonly = false}
#4  address_space_update_ioeventfds (as=0x7f4adc4c5e00 <address_space_io>) at /usr/src/debug/qemu-1.5.1/memory.c:649
        fr = <optimized out>
        tmp = {start = {lo = 61456, hi = 0}, size = {lo = 2, hi = 0}}
        i = <optimized out>
        ioeventfd_nb = <optimized out>
        ioeventfds = <optimized out>
#5  address_space_update_topology (as=0x7f4adc4c5e00 <address_space_io>) at /usr/src/debug/qemu-1.5.1/memory.c:730
        old_view = <optimized out>
        new_view = <optimized out>
#6  memory_region_transaction_commit () at /usr/src/debug/qemu-1.5.1/memory.c:750
        as = 0x7f4adc4c5e00 <address_space_io>
#7  0x00007f4adb6f66cf in pci_unregister_io_regions (pci_dev=0x7f4adce4b440) at hw/pci/pci.c:889
        r = 0x7f4adce4b548
        i = 0
#8  pci_unregister_device (dev=<optimized out>) at hw/pci/pci.c:900
        pci_dev = 0x7f4adce4b440
        __func__ = "pci_unregister_device"
        pc = 0x7f4adcde5c10
#9  0x00007f4adb6b0154 in device_unrealize (dev=0x7f4adce4b440, errp=0x7fff882bbc70) at hw/core/qdev.c:191
        rc = <optimized out>
        dc = <optimized out>
#10 0x00007f4adb6b1824 in device_set_realized (obj=0x7f4adce4b440, value=<optimized out>, err=0x0) at hw/core/qdev.c:715
        dev = 0x7f4adce4b440
        __func__ = "device_set_realized"
        dc = 0x7f4adcde5c10
        local_err = 0x0
#11 0x00007f4adb76e53e in property_set_bool (obj=0x7f4adce4b440, v=<optimized out>, opaque=0x7f4adce2fc40, 
    name=<optimized out>, errp=0x0) at qom/object.c:1301
        prop = 0x7f4adce2fc40
        value = false
        local_err = 0x0
#12 0x00007f4adb770e07 in object_property_set_qobject (obj=0x7f4adce4b440, value=<optimized out>, 
    name=0x7f4adb8e094d "realized", errp=0x0) at qom/qom-qobject.c:24
        mi = 0x7f4add8a75c0
#13 0x00007f4adb76fda0 in object_property_set_bool (obj=obj@entry=0x7f4adce4b440, value=value@entry=false, 
    name=name@entry=0x7f4adb8e094d "realized", errp=errp@entry=0x0) at qom/object.c:852
        qbool = 0x7f4adce48a40
#14 0x00007f4adb6afe5b in device_unparent (obj=0x7f4adce4b440) at hw/core/qdev.c:798
        dev = 0x7f4adce4b440
        __func__ = "device_unparent"
        bus = <optimized out>
        event_data = <optimized out>
        have_realized = true
#15 0x00007f4adb76f93a in object_unparent (obj=0x7f4adce4b440) at qom/object.c:372
No locals.
#16 0x00007f4adb6b09cd in qdev_free (dev=<optimized out>) at hw/core/qdev.c:286
No locals.
#17 0x00007f4adb6f911c in pcie_cap_slot_hotplug (qdev=<optimized out>, pci_dev=0x7f4adce4b440, state=
    PCI_HOTPLUG_DISABLED) at hw/pci/pcie.c:254
        d = 0x7f4adce43350
        __func__ = "pcie_cap_slot_hotplug"
        exp_cap = 0x7f4adce44520 "\020\200b\001"
        sltsta = <optimized out>
        __PRETTY_FUNCTION__ = "pcie_cap_slot_hotplug"
#18 0x00007f4adb6b084f in qdev_unplug (dev=0x7f4adce4b440, errp=errp@entry=0x7fff882bbe80) at hw/core/qdev.c:219
        dc = <optimized out>
        __func__ = "qdev_unplug"
        __PRETTY_FUNCTION__ = "qdev_unplug"
#19 0x00007f4adb75c092 in qmp_device_del (id=<optimized out>, errp=errp@entry=0x7fff882bbe80) at qdev-monitor.c:628
        dev = <optimized out>
#20 0x00007f4adb68e86b in hmp_device_del (mon=0x7f4adca8bc90, qdict=<optimized out>) at hmp.c:1177
        id = <optimized out>
        err = 0x0
#21 0x00007f4adb80aca9 in handle_user_command (mon=mon@entry=0x7f4adca8bc90, cmdline=<optimized out>)
    at /usr/src/debug/qemu-1.5.1/monitor.c:3999
        qdict = 0x7f4add8a65a0
        cmd = 0x7f4adbcab280 <mon_cmds+1664>
        __PRETTY_FUNCTION__ = "handle_user_command"
#22 0x00007f4adb80afab in monitor_command_cb (mon=0x7f4adca8bc90, cmdline=<optimized out>, opaque=<optimized out>)
    at /usr/src/debug/qemu-1.5.1/monitor.c:4615
No locals.
#23 0x00007f4adb7717c0 in readline_handle_byte (rs=0x7f4adcce7410, ch=<optimized out>) at readline.c:374
No locals.
#24 0x00007f4adb80af14 in monitor_read (opaque=<optimized out>, buf=<optimized out>, size=<optimized out>)
    at /usr/src/debug/qemu-1.5.1/monitor.c:4601
        old_mon = 0x0
        i = <optimized out>
#25 0x00007f4adb75f81b in qemu_chr_be_write (len=<optimized out>, buf=0x7fff882bbfc0 "\r", s=0x7f4adc9ecfd0)
    at qemu-char.c:167
No locals.
#26 fd_chr_read (chan=<optimized out>, cond=<optimized out>, opaque=0x7f4adc9ecfd0) at qemu-char.c:801
        chr = 0x7f4adc9ecfd0
        s = 0x7f4adc9ed0a0
        len = <optimized out>
        buf = "\r\000\000\000\000\000\000\000\250\211\272\336J\177\000\000\000\020", '\000' <repeats 22 times>, "芺\336J\177\000\000\000\020", '\000' <repeats 22 times>, "\210\241\272\336J\177\000\000\000\020", '\000' <repeats 22 times>, "H\275\337\334J\177\000\000\000\020", '\000' <repeats 22 times>, "(\242\272\336J\177\000\000\000\060", '\000' <repeats 22 times>, "X\255\307\337J\177\000\000\000\020", '\000' <repeats 14 times>, "\240\300+\210\377\177\000\000aD"...
        status = <optimized out>
        bytes_read = 1
#27 0x00007f4adacbae06 in g_main_context_dispatch () from /lib64/libglib-2.0.so.0
No symbol table info available.
#28 0x00007f4adb737eba in glib_pollfds_poll () at main-loop.c:187
        context = 0x7f4adc9ec000
        pfds = <optimized out>
#29 os_host_main_loop_wait (timeout=<optimized out>) at main-loop.c:232
        ret = 1
        spin_counter = 0
#30 main_loop_wait (nonblocking=<optimized out>) at main-loop.c:464
        ret = 1
        timeout = 4294967295
#31 0x00007f4adb638609 in main_loop () at vl.c:2029
        nonblocking = <optimized out>
        last_io = 1
#32 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4419
        i = <optimized out>
        snapshot = 0
        linux_boot = <optimized out>
        icount_option = 0x0
        initrd_filename = <optimized out>
        kernel_filename = <optimized out>
        kernel_cmdline = <optimized out>
        boot_devices = '\000' <repeats 32 times>
        ds = <optimized out>
        cyls = 0
        heads = 0
        secs = 0
        translation = 0
        hda_opts = <optimized out>
        opts = <optimized out>
        machine_opts = <optimized out>
        olist = <optimized out>
        optind = 80
        optarg = 0x7fff882be7b7 "virtio-blk-pci,scsi=on,bus=downstream2,addr=0x0,drive=drive-data-disk1,id=data-disk1"
        loadvm = 0x0
        machine = 0x7f4adbca8f60 <pc_q35_machine_rhel700>
        cpu_model = 0x7fff882bdfe3 "SandyBridge"
        vga_model = 0x7f4adb9072ef "cirrus"
        pid_file = 0x0
        incoming = 0x0
        show_vnc_port = 0
        defconfig = <optimized out>
        userconfig = false
        log_mask = 0x0
        log_file = 0x0
        mem_trace = {malloc = 0x7f4adb7a1060 <malloc_and_trace>, realloc = 0x7f4adb7a1020 <realloc_and_trace>, 
          free = 0x7f4adb7a0fe0 <free_and_trace>, calloc = 0x0, try_malloc = 0x0, try_realloc = 0x0}
        trace_events = 0x0
        trace_file = 0x0
        __PRETTY_FUNCTION__ = "main"
        args = {ram_size = 4294967296, boot_device = 0x7f4adb8dd7a6 "cad", kernel_filename = 0x0, 
          kernel_cmdline = 0x7f4adb923c90 "", initrd_filename = 0x0, cpu_model = 0x7fff882bdfe3 "SandyBridge"}
(gdb)
Comment 2 Sibiao Luo 2013-07-11 04:28:10 UTC 
I also tried the PCIE bridge that can do hot-unplug successfully, did not met any core dump, both guest and host work well after do hot-unplug.

# /usr/libexec/qemu-kvm -S -M q35 -cpu SandyBridge -enable-kvm...-device pci-bridge,bus=pcie.0,id=bridge1,chassis_nr=1,addr=0x3 -drive file=/home/my-data-disk2.qcow3,if=none,id=drive-data-disk2,format=qcow2,cache=none,aio=native,werror=stop,rerror=stop,serial="QEMU-DISK3" -device virtio-blk-pci,scsi=on,bus=bridge1,addr=0x6,drive=drive-data-disk2,id=data-disk2
(qemu) info block
drive-system-disk: removable=0 io-status=ok file=/home/RHEL-7.0-20130628.0-Server-x86_64.qcow3 ro=0 drv=qcow2 encrypted=0 bps=0 bps_rd=0 bps_wr=0 iops=0 iops_rd=0 iops_wr=0
drive-data-disk2: removable=0 io-status=ok file=/home/my-data-disk2.qcow3 ro=0 drv=qcow2 encrypted=0 bps=0 bps_rd=0 bps_wr=0 iops=0 iops_rd=0 iops_wr=0
(qemu) device_del data-disk2
(qemu) drive_del drive-data-disk2
(qemu) info block
drive-system-disk: removable=0 io-status=ok file=/home/RHEL-7.0-20130628.0-Server-x86_64.qcow3 ro=0 drv=qcow2 encrypted=0 bps=0 bps_rd=0 bps_wr=0 iops=0 iops_rd=0 iops_wr=0

Best Regards,
sluo
Comment 3 Sibiao Luo 2013-07-11 04:32:24 UTC 
(In reply to Sibiao Luo from comment #2)
> I also tried the PCIE bridge that can do hot-unplug successfully, did not
> met any core dump, both guest and host work well after do hot-unplug.
> 
sorry, it's pci birdge not pcie bridge, thx.
Comment 4 Sibiao Luo 2013-07-11 09:56:01 UTC 
It can do hot-unplug a virtio-blk disk which use the switch behind switch successfully and work well.
# /usr/libexec/qemu-kvm -S -M q35 -cpu SandyBridge -enable-kvm ...-device ioh3420,bus=pcie.0,id=root.2,slot=3 -device x3130-upstream,bus=root.2,id=upstream2 -device xio3130-downstream,bus=upstream2,id=downstream2,chassis=3

(qemu) drive_add pci_addr=auto file=/home/my-data-disk1.qcow3,if=none,id=drive-data-disk,format=qcow2,werror=stop,rerror=stop
OK
(qemu) device_add virtio-blk-pci,scsi=on,bus=downstream2,addr=0x0,drive=drive-data-disk,id=data-disk
(qemu) info block
drive-system-disk: removable=0 io-status=ok file=/home/RHEL-7.0-20130628.0-Server-x86_64.qcow3 ro=0 drv=qcow2 encrypted=0 bps=0 bps_rd=0 bps_wr=0 iops=0 iops_rd=0 iops_wr=0
ide1-cd0: removable=1 locked=0 tray-open=0 [not inserted]
floppy0: removable=1 locked=0 tray-open=0 [not inserted]
sd0: removable=1 locked=0 tray-open=0 [not inserted]
drive-data-disk: removable=0 io-status=ok file=/home/my-data-disk1.qcow3 ro=0 drv=qcow2 encrypted=0 bps=0 bps_rd=0 bps_wr=0 iops=0 iops_rd=0 iops_wr=0
(qemu) 

# nc -U /tmp/ttyS0 
[  356.774643] pciehp 0000:09:00.0:pcie24: Card present on Slot(0-2)
[  357.887193] pci 0000:0a:00.0: BAR 1: assigned [mem 0xc0400000-0xc0400fff]
[  357.889314] pci 0000:0a:00.0: BAR 0: assigned [io  0x1000-0x103f]
[  357.890410] pcieport 0000:09:00.0: PCI bridge to [bus 0a]
[  357.891371] pcieport 0000:09:00.0:   bridge window [io  0x1000-0x1fff]
[  357.902919] pcieport 0000:09:00.0:   bridge window [mem 0xc0400000-0xc05fffff]
[  357.910516] pcieport 0000:09:00.0:   bridge window [mem 0xc0600000-0xc07fffff 64bit pref]
[  357.925532] pci 0000:0a:00.0: no hotplug settings from platform
[  357.926960] pci 0000:0a:00.0: using default PCI settings
[  357.928151] virtio-pci 0000:0a:00.0: enabling device (0000 -> 0003)
[  357.935934] ACPI: PCI Interrupt Link [GSIF] enabled at IRQ 21
[  357.941661]  vdb: unknown partition table

Best Regards,
sluo
Comment 5 Sibiao Luo 2013-07-11 09:57:01 UTC 
(In reply to Sibiao Luo from comment #4)
> It can do hot-unplug a virtio-blk disk which use the switch behind switch
> successfully and work well.
It can do *hot-plug* a virtio-blk disk which use the switch behind switch successfully and work well.
Comment 6 Markus Armbruster 2013-09-13 07:22:03 UTC 
Here's how I understand the bug report.  In addition to the primary
test case that exposes the bug, there are two additional test cases
that work find.

Test case 1 (comment#0): hot unplug of virtio-blk-pci data disk
cold-plugged into a certain *pcie* port ("switch behind switch")
crashes qemu instantly and reliably.  Sometimes, host reboot follows a
few minutes later (that part is astonishing).

Test case 2 (comment#2+3): similar to 1., but use a *pci* bridge
instead of pcie, works fine.

Test case 3 (comment#4+5): start qemu-kvm just like in 1., except
don't cold-plug the virtio-blk-pci data disk, hot plug it instead.
Works fine.
Comment 7 Sibiao Luo 2013-09-13 07:24:41 UTC 
(In reply to Markus Armbruster from comment #6)
> Here's how I understand the bug report.  In addition to the primary
> test case that exposes the bug, there are two additional test cases
> that work find.
> 
> Test case 1 (comment#0): hot unplug of virtio-blk-pci data disk
> cold-plugged into a certain *pcie* port ("switch behind switch")
> crashes qemu instantly and reliably.  Sometimes, host reboot follows a
> few minutes later (that part is astonishing).

 *PCIE* + "switch behind switch" + hot-unplug=========>crash

> Test case 2 (comment#2+3): similar to 1., but use a *pci* bridge
> instead of pcie, works fine.

 *PCI* + "switch behind switch" + hot-unplug =========>good

> Test case 3 (comment#4+5): start qemu-kvm just like in 1., except
> don't cold-plug the virtio-blk-pci data disk, hot plug it instead.
> Works fine.

 *PCIE* + "switch behind switch" + hot-plug  =========>good

Best Regards,
sluo
Comment 8 Paolo Bonzini 2013-09-20 09:52:19 UTC 
The bug is in the virtio-pci implementation and was introduced by the virtio refactoring.

The bus structure is as follows

   virtio-blk-pci
    \
     \ virtio bus
      \
       virtio-blk-device

When the virtio-blk-pci device is deleted, the virtio-blk-device is removed first, because removal is done in post-order.  Later, the virtio-blk-device is accessed again but proxy->vdev->vq is now a dangling pointer and it is not valid anymore.  Hence, kvm_set_ioeventfd_pio fails.
Comment 9 Paolo Bonzini 2013-09-20 15:00:13 UTC 
Patches at http://thread.gmane.org/gmane.comp.emulators.qemu/234440
Comment 10 Vincent Danen 2013-09-26 18:43:22 UTC 
CVE requested for this, see bug 1012633.  Also set this as a security tracking bug and set flags accordingly.
Comment 11 Markus Armbruster 2014-01-23 10:34:43 UTC 
I backported the patches mentioned in comment#9.  They fix the
qemu-kvm crash, but now I my guest kernel-3.10.0-71.el7.x86_64
complains:

[   37.674257] ------------[ cut here ]------------
[   37.674296] WARNING: at drivers/virtio/virtio.c:158 virtio_dev_remove+0x74/0x80 [virtio]()
[   37.674301] Modules linked in: ip6t_rpfilter ip6t_REJECT ipt_REJECT xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw iptable_filter ip_tables sg kvm_amd kvm pcspkr serio_raw i2c_i801 lpc_ich mfd_core mperf shpchp xfs libcrc32c sr_mod cdrom cirrus virtio_net virtio_blk syscopyarea sysfillrect sysimgblt drm_kms_helper ahci ttm libahci drm libata virtio_pci virtio_ring virtio i2c_core dm_mirror dm_region_hash dm_log dm_mod
[   37.674374] CPU: 0 PID: 22 Comm: kworker/0:1 Not tainted 3.10.0-71.el7.x86_64 #1
[   37.674380] Hardware name: Red Hat KVM, BIOS Bochs 01/01/2011
[   37.674393] Workqueue: pciehp-0 pciehp_power_thread
[   37.674397]  0000000000000009 ffff8802443dbb30 ffffffff815bd8c4 ffff8802443dbb68
[   37.674405]  ffffffff81059c61 ffff880241b0c400 ffff880241b0c408 ffffffffa0024000
[   37.674411]  ffff880244312098 0000000000000000 ffff8802443dbb78 ffffffff81059d3a
[   37.674418] Call Trace:
[   37.674431]  [<ffffffff815bd8c4>] dump_stack+0x19/0x1b
[   37.674443]  [<ffffffff81059c61>] warn_slowpath_common+0x61/0x80
[   37.674454]  [<ffffffff81059d3a>] warn_slowpath_null+0x1a/0x20
[   37.674465]  [<ffffffffa00220e4>] virtio_dev_remove+0x74/0x80 [virtio]
[   37.674476]  [<ffffffff8139535f>] __device_release_driver+0x7f/0xf0
[   37.674484]  [<ffffffff813953f3>] device_release_driver+0x23/0x30
[   37.674491]  [<ffffffff81394b88>] bus_remove_device+0x108/0x180
[   37.674498]  [<ffffffff81391485>] device_del+0x135/0x1d0
[   37.674505]  [<ffffffff8139153e>] device_unregister+0x1e/0x60
[   37.674516]  [<ffffffffa00224b6>] unregister_virtio_device+0x16/0x30 [virtio]
[   37.674527]  [<ffffffffa004c56b>] virtio_pci_remove+0x2b/0x70 [virtio_pci]
[   37.674537]  [<ffffffff812d252b>] pci_device_remove+0x3b/0xb0
[   37.674546]  [<ffffffff8139535f>] __device_release_driver+0x7f/0xf0
[   37.674553]  [<ffffffff813953f3>] device_release_driver+0x23/0x30
[   37.674560]  [<ffffffff81394b88>] bus_remove_device+0x108/0x180
[   37.674567]  [<ffffffff81391485>] device_del+0x135/0x1d0
[   37.674576]  [<ffffffff812cc064>] pci_stop_bus_device+0x94/0xa0
[   37.674583]  [<ffffffff812cc152>] pci_stop_and_remove_bus_device+0x12/0x20
[   37.674591]  [<ffffffff812e4bd8>] pciehp_unconfigure_device+0xa8/0x1b0
[   37.674599]  [<ffffffff812e4538>] pciehp_disable_slot+0x68/0x200
[   37.674607]  [<ffffffff812e4753>] pciehp_power_thread+0x83/0xf0
[   37.674616]  [<ffffffff8107862b>] process_one_work+0x17b/0x460
[   37.674623]  [<ffffffff810793db>] worker_thread+0x11b/0x400
[   37.674631]  [<ffffffff810792c0>] ? rescuer_thread+0x3e0/0x3e0
[   37.674638]  [<ffffffff8107fb90>] kthread+0xc0/0xd0
[   37.674646]  [<ffffffff8107fad0>] ? kthread_create_on_node+0x110/0x110
[   37.674653]  [<ffffffff815cd66c>] ret_from_fork+0x7c/0xb0
[   37.674659]  [<ffffffff8107fad0>] ? kthread_create_on_node+0x110/0x110
[   37.674665] ---[ end trace f5de3b0770382ce3 ]---

Paolo, is this expected?
Comment 13 Paolo Bonzini 2014-01-23 10:55:17 UTC 
No idea if this is expected.  It definitely deserves a BZ, but a better question could be, is this a regression from RHEL6 and/or PIIX?  That is:

- does a RHEL6 kernel complain?

- does the RHEL7 kernel complain with "-M pc"?

- if the answer to the second question is "yes", does a RHEL7 kernel complain on top of RHEL6 qemu?
Comment 14 Markus Armbruster 2014-01-23 12:23:12 UTC 
Same test, except for the guest:

* kernel-2.6.32-279.el6.x86_64 crashes hard

* kernel-3.3.0-0.20.el7.x86_64[*] crashes hard

I can't answer your other questions, because I don't know how to reproduce without -M q35 (see comment#6).


[*] Look what the cat dragged in!
Comment 15 Markus Armbruster 2014-01-23 12:24:56 UTC 
Created attachment 854354 [details]
tail of dmesg from crashed kernel-2.6.32-279.el6.x86_64
Comment 16 Markus Armbruster 2014-01-23 12:25:40 UTC 
Created attachment 854355 [details]
tail of dmesg from crashed kernel-3.3.0-0.20.el7.x86_64
Comment 17 Markus Armbruster 2014-01-23 12:32:31 UTC 
I reproduced the guest kernel oops from comment#11 under latest upstream QEMU.
Comment 18 Markus Armbruster 2014-01-23 12:35:02 UTC 
I'd file a BZ against kernel if I had a reproducer that doesn't
involve patching qemu-kvm first, or compiling upstream QEMU.  Perhaps
we should just commit the backport, then file the kernel BZ (assuming
it still reproduces then).  Paolo, what do you think?
Comment 19 Paolo Bonzini 2014-01-23 12:41:55 UTC 
I agree.  Regarding the other questions, I meant with "-M pc" and a "normal" setup without PCIe switches.
Comment 20 Markus Armbruster 2014-01-23 13:22:21 UTC 
Hot unplugging the device works fine when it's plugged straight into
i440FX's PCI bus.

I tried to test with a pci-bridge device in between, but I can't get
devices on the bridge recognized by the guest.  Not a PCI guy...
Comment 21 Michael S. Tsirkin 2014-01-23 14:36:14 UTC 
to comment 18:
this warning implies fails to read status from device
or that driver failed to reset device.

I think we should tread carefully here: could be that
delete does not wait for guest ack properly?
Comment 22 Miroslav Rezanina 2014-01-27 10:59:08 UTC 
Fix included in qemu-kvm-1.5.3-43.el7
Comment 24 langfang 2014-02-10 09:12:49 UTC 
Reproduce this bug as follow version:
Host:
# uname -r
3.10.0-84.el7.x86_64
# rpm -q qemu-kvm
qemu-kvm-1.5.3-37.el7.x86_64
# rpm -q seabios
seabios-1.7.2.2-11.el7.x86_64

Guest:rhel7

Steps:
1.Boot guest with the CLI like comment0
2.Hotunplug data disk
(qemu) device_del data-disk1

Results:Core dump
(gdb) bt
#0  0x00007ffff2c99989 in raise () from /lib64/libc.so.6
#1  0x00007ffff2c9b098 in abort () from /lib64/libc.so.6
#2  0x00005555557d06c3 in kvm_io_ioeventfd_del (listener=<optimized out>, section=0x7fffffffc230, 
    match_data=<optimized out>, data=0, e=<optimized out>)
    at /usr/src/debug/qemu-1.5.3/kvm-all.c:844
#3  0x00005555557d5f30 in address_space_add_del_ioeventfds (fds_old_nb=38, fds_old=0x7fffd802b870, 
    fds_new_nb=37, fds_new=0x555556bb4370, as=0x5555564beaa0 <address_space_io>)
    at /usr/src/debug/qemu-1.5.3/memory.c:604
#4  address_space_update_ioeventfds (as=0x5555564beaa0 <address_space_io>)
    at /usr/src/debug/qemu-1.5.3/memory.c:650
#5  address_space_update_topology (as=0x5555564beaa0 <address_space_io>)
    at /usr/src/debug/qemu-1.5.3/memory.c:731
#6  memory_region_transaction_commit () at /usr/src/debug/qemu-1.5.3/memory.c:751
#7  0x00005555556c2b6f in pci_unregister_io_regions (pci_dev=0x555556973180) at hw/pci/pci.c:894
#8  pci_unregister_device (dev=<optimized out>) at hw/pci/pci.c:905
#9  0x000055555567ed74 in device_unrealize (dev=0x555556973180, errp=0x7fffffffc320)
    at hw/core/qdev.c:191
#10 0x0000555555680404 in device_set_realized (obj=0x555556973180, value=<optimized out>, err=0x0)
    at hw/core/qdev.c:709
#11 0x0000555555740b5e in property_set_bool (obj=0x555556973180, v=<optimized out>, 
    opaque=0x555556973b30, name=<optimized out>, errp=0x0) at qom/object.c:1302
#12 0x0000555555743717 in object_property_set_qobject (obj=0x555556973180, value=<optimized out>, 
...

Verify this bug as follow version:
Host:
# uname -r
3.10.0-84.el7.x86_64
# rpm -q qemu-kvm
qemu-kvm-1.5.3-45.el7.x86_64
# rpm -q seabios
seabios-1.7.2.2-11.el7.x86_64

Guest:rhel7
3.10.0-48.el7.x86_64

Steps as same as reproduce

Results:
Not hit qemu core dump problem, but will hit guest kernel call trace.
[   84.204412] ------------[ cut here ]------------
[   84.204983] WARNING: at drivers/virtio/virtio.c:158 virtio_dev_remove+0x74/0x80 [virtio]()
[   84.205899] Modules linked in: nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_MASQUERADE ip6t_REJECT ipt_REJECT xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw iptable_filter ip_tables sg crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd virtio_console serio_raw pcspkr lpc_ich mfd_core i2c_i801 shpchp microcode mperf xfs libcrc32c virtio_blk cirrus syscopyarea virtio_net ahci libahci sysfillrect sysimgblt drm_kms_helper ttm virtio_pci
[   84.214738]  drm libata virtio_ring virtio i2c_core dm_mirror dm_region_hash dm_log dm_mod
[   84.215679] CPU: 0 PID: 43 Comm: kworker/0:1 Not tainted 3.10.0-48.el7.x86_64 #1
[   84.216454] Hardware name: Red Hat KVM, BIOS Bochs 01/01/2011
[   84.217071] Workqueue: pciehp-0 pciehp_power_thread
[   84.217601]  0000000000000009 ffff8801796c5b30 ffffffff815abfff ffff8801796c5b68
[   84.218453]  ffffffff810589d1 ffff880176f09800 ffff880176f09808 ffffffffa0024000
[   84.219304]  ffff8801795c8098 0000000000000000 ffff8801796c5b78 ffffffff81058aaa
[   84.220155] Call Trace:
[   84.220435]  [<ffffffff815abfff>] dump_stack+0x19/0x1b
[   84.220976]  [<ffffffff810589d1>] warn_slowpath_common+0x61/0x80
[   84.221620]  [<ffffffff81058aaa>] warn_slowpath_null+0x1a/0x20
[   84.222249]  [<ffffffffa00220e4>] virtio_dev_remove+0x74/0x80 [virtio]
[   84.222936]  [<ffffffff8138cfaf>] __device_release_driver+0x7f/0xf0
[   84.223569]  [<ffffffff8138d043>] device_release_driver+0x23/0x30
[   84.224191]  [<ffffffff8138c814>] bus_remove_device+0xf4/0x170
[   84.224775]  [<ffffffff813891e5>] device_del+0x135/0x1d0
[   84.225324]  [<ffffffff8138929e>] device_unregister+0x1e/0x60
[   84.225903]  [<ffffffffa00224b6>] unregister_virtio_device+0x16/0x30 [virtio]
[   84.226620]  [<ffffffffa00fb56b>] virtio_pci_remove+0x2b/0x70 [virtio_pci]
[   84.227322]  [<ffffffff812cab7b>] pci_device_remove+0x3b/0xb0
[   84.227900]  [<ffffffff8138cfaf>] __device_release_driver+0x7f/0xf0
[   84.228531]  [<ffffffff8138d043>] device_release_driver+0x23/0x30
[   84.229155]  [<ffffffff8138c814>] bus_remove_device+0xf4/0x170
[   84.229732]  [<ffffffff813891e5>] device_del+0x135/0x1d0
[   84.230283]  [<ffffffff812c46e4>] pci_stop_bus_device+0x94/0xa0
[   84.230877]  [<ffffffff812c47d2>] pci_stop_and_remove_bus_device+0x12/0x20
[   84.231566]  [<ffffffff812dd108>] pciehp_unconfigure_device+0xa8/0x1b0
[   84.232230]  [<ffffffff812dca68>] pciehp_disable_slot+0x68/0x200
[   84.232837]  [<ffffffff812dcc83>] pciehp_power_thread+0x83/0xf0
[   84.237879]  [<ffffffff8107738b>] process_one_work+0x17b/0x460
[   84.238521]  [<ffffffff8107813b>] worker_thread+0x11b/0x400
[   84.239090]  [<ffffffff81078020>] ? rescuer_thread+0x3e0/0x3e0
[   84.239761]  [<ffffffff8107e8f0>] kthread+0xc0/0xd0
[   84.240295]  [<ffffffff8107e830>] ? kthread_create_on_node+0x110/0x110
[   84.240963]  [<ffffffff815bb6ec>] ret_from_fork+0x7c/0xb0
[   84.241501]  [<ffffffff8107e830>] ? kthread_create_on_node+0x110/0x110
[   84.242246] ---[ end trace e2fb0b185e17a1f1 ]---

As above test , this bug fixed, but hit guest kernel problem.

Hi, Markus
   as you said comment18,you will  file a BZ against kernel , what's the Bug ID?

thanks 
fang lang
Comment 25 Markus Armbruster 2014-02-10 09:33:07 UTC 
Bug 1058321.
Comment 26 langfang 2014-02-11 02:07:30 UTC 
(In reply to Markus Armbruster from comment #25)
> Bug 1058321.

Hi Markus
   thanks for your help,according to comment24 test, so we can verify this bug now.
Note You need to log in before you can comment on or make changes to this bug.