Bug 1151306

Summary: Qemu coredumpd after reboot a mq enabled but host-tap deleted guest.
Product: Red Hat Enterprise Linux 7 Reporter: Qian Guo <qiguo>
Component: qemu-kvm-rhevAssignee: jason wang <jasowang>
Status: CLOSED WONTFIX QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact: jason wang <jasowang>
Priority: medium    
Version: 7.1CC: hhuang, jasowang, juzhang, knoel, michen, virt-maint, weliao
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-12-26 04:29:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1401400    

Description Qian Guo 2014-10-10 05:09:28 UTC
Description of problem:
Eanbled mq in guest, and then delete the tap device in host, then reboot guest, qemu coredumped

Version-Release number of selected component (if applicable):
qemu-kvm-rhev-2.1.2-1.el7.x86_64
# uname -r
3.10.0-183.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Boot guest with mq
/usr/libexec/qemu-kvm -cpu SandyBridge -enable-kvm -m 4G -smp 4,sockets=1,cores=4,threads=1 -name test -rtc base=localtime,clock=host,driftfix=slew -k en-us -boot menu=on -spice disable-ticketing,port=5901 -vga qxl -usb -device usb-tablet -monitor stdio -drive file=/home/rhel7u1cp1.qcow2,if=none,id=drive-system-disk,media=disk,format=qcow2,aio=native,werror=stop,rerror=stop -device virtio-blk-pci,drive=drive-system-disk,id=system-disk,addr=0x3 -qmp unix:/tmp/q1,server,nowait -netdev tap,id=hostnet0,vhost=on,queues=4,script=/etc/qemu-ifup -device virtio-net-pci,mq=on,vectors=10,netdev=hostnet0,id=vnet0,mac=52:54:00:1a:2b:02 -serial unix:/tmp/s2,server,nowait

2.In guest enable mq
# ethtool -l eth0
Channel parameters for eth0:
Pre-set maximums:
RX:		0
TX:		0
Other:		0
Combined:	4
Current hardware settings:
RX:		0
TX:		0
Other:		0
Combined:	4


3.Delete tap device in host
# ip link delete tap0

4.Reboot guest.

Actual results:

Then when guest try to boot

qemu coredumped
(gdb) bt
#0  0x00007ffff1ea1989 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007ffff1ea3098 in __GI_abort () at abort.c:90
#2  0x00007ffff1e9a8f6 in __assert_fail_base (fmt=0x7ffff1fea3e8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", 
    assertion=assertion@entry=0x55555589dfbd "!r", 
    file=file@entry=0x55555589dcb8 "/builddir/build/BUILD/qemu-2.1.2/hw/net/virtio-net.c", line=line@entry=437, 
    function=function@entry=0x55555589e5d0 <__PRETTY_FUNCTION__.28404> "virtio_net_set_queues") at assert.c:92
#3  0x00007ffff1e9a9a2 in __GI___assert_fail (assertion=assertion@entry=0x55555589dfbd "!r", 
    file=file@entry=0x55555589dcb8 "/builddir/build/BUILD/qemu-2.1.2/hw/net/virtio-net.c", line=line@entry=437, 
    function=function@entry=0x55555589e5d0 <__PRETTY_FUNCTION__.28404> "virtio_net_set_queues") at assert.c:101
#4  0x000055555567618a in virtio_net_set_queues (n=0x5555563fac38) at /usr/src/debug/qemu-2.1.2/hw/net/virtio-net.c:437
#5  0x00005555556768ab in virtio_net_set_multiqueue (multiqueue=1, n=0x5555563fac38)
    at /usr/src/debug/qemu-2.1.2/hw/net/virtio-net.c:1336
#6  virtio_net_set_features (vdev=<optimized out>, features=955252707) at /usr/src/debug/qemu-2.1.2/hw/net/virtio-net.c:522
#7  0x000055555567e04b in virtio_set_features (vdev=0x5555563fac38, val=955252707)
    at /usr/src/debug/qemu-2.1.2/hw/virtio/virtio.c:971
#8  0x000055555565479a in access_with_adjusted_size (addr=addr@entry=4, value=value@entry=0x7fffe8fbcaf0, size=size@entry=4, 
    access_size_min=<optimized out>, access_size_max=<optimized out>, access=0x555555654910 <memory_region_write_accessor>, 
    mr=0x5555563faaa8) at /usr/src/debug/qemu-2.1.2/memory.c:481
#9  0x0000555555659367 in memory_region_dispatch_write (size=4, data=955252707, addr=4, mr=0x5555563faaa8)
    at /usr/src/debug/qemu-2.1.2/memory.c:1143
#10 io_mem_write (mr=mr@entry=0x5555563faaa8, addr=4, val=<optimized out>, size=4) at /usr/src/debug/qemu-2.1.2/memory.c:1976
#11 0x0000555555624723 in address_space_rw (as=0x555555cb0480 <address_space_io>, addr=addr@entry=49220, 
    buf=0x7ffff7ec1000 "\343\377\357\070", len=len@entry=4, is_write=is_write@entry=true) at /usr/src/debug/qemu-2.1.2/exec.c:2088
#12 0x0000555555653c20 in kvm_handle_io (count=1, size=4, direction=<optimized out>, data=<optimized out>, port=49220)
    at /usr/src/debug/qemu-2.1.2/kvm-all.c:1600
#13 kvm_cpu_exec (cpu=cpu@entry=0x55555637c9c0) at /usr/src/debug/qemu-2.1.2/kvm-all.c:1737
#14 0x0000555555642dc2 in qemu_kvm_cpu_thread_fn (arg=0x55555637c9c0) at /usr/src/debug/qemu-2.1.2/cpus.c:874
#15 0x00007ffff6ba2df3 in start_thread (arg=0x7fffe8fbd700) at pthread_create.c:308
#16 0x00007ffff1f623dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
(gdb) bt ful
#0  0x00007ffff1ea1989 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
        resultvar = 0
        pid = 14549
        selftid = 14570
#1  0x00007ffff1ea3098 in __GI_abort () at abort.c:90
        save_stage = 2
        act = {__sigaction_handler = {sa_handler = 0x7fffffffe4c2, sa_sigaction = 0x7fffffffe4c2}, sa_mask = {__val = {
              140737253373233, 93824995679416, 437, 140737102203072, 140737252016723, 4, 140737102203040, 98398095760, 
              18446744073709550960, 8, 0, 0, 0, 21474836480, 140737354051584, 140737253385192}}, sa_flags = 1435099069, 
          sa_restorer = 0x55555589e5d0 <__PRETTY_FUNCTION__.28404>}
        sigs = {__val = {32, 0 <repeats 15 times>}}
#2  0x00007ffff1e9a8f6 in __assert_fail_base (fmt=0x7ffff1fea3e8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", 
    assertion=assertion@entry=0x55555589dfbd "!r", 
    file=file@entry=0x55555589dcb8 "/builddir/build/BUILD/qemu-2.1.2/hw/net/virtio-net.c", line=line@entry=437, 
    function=function@entry=0x55555589e5d0 <__PRETTY_FUNCTION__.28404> "virtio_net_set_queues") at assert.c:92
        str = 0x7fffe0000b50 ""
        total = 4096
#3  0x00007ffff1e9a9a2 in __GI___assert_fail (assertion=assertion@entry=0x55555589dfbd "!r", 
    file=file@entry=0x55555589dcb8 "/builddir/build/BUILD/qemu-2.1.2/hw/net/virtio-net.c", line=line@entry=437, 
    function=function@entry=0x55555589e5d0 <__PRETTY_FUNCTION__.28404> "virtio_net_set_queues") at assert.c:101
No locals.
#4  0x000055555567618a in virtio_net_set_queues (n=0x5555563fac38) at /usr/src/debug/qemu-2.1.2/hw/net/virtio-net.c:437
        i = 1
#5  0x00005555556768ab in virtio_net_set_multiqueue (multiqueue=1, n=0x5555563fac38)
    at /usr/src/debug/qemu-2.1.2/hw/net/virtio-net.c:1336
        vdev = 0x5555563fac38
        i = <optimized out>
        max = <optimized out>
#6  virtio_net_set_features (vdev=<optimized out>, features=955252707) at /usr/src/debug/qemu-2.1.2/hw/net/virtio-net.c:522
        n = 0x5555563fac38
        __func__ = "virtio_net_set_features"
        i = <optimized out>
#7  0x000055555567e04b in virtio_set_features (vdev=0x5555563fac38, val=955252707)
    at /usr/src/debug/qemu-2.1.2/hw/virtio/virtio.c:971
---Type <return> to continue, or q <return> to quit---
        qbus = <optimized out>
        __func__ = "virtio_set_features"
        vbusk = 0x55555614edf0
        k = 0x5555561563d0
        supported_features = <optimized out>
        bad = false
#8  0x000055555565479a in access_with_adjusted_size (addr=addr@entry=4, value=value@entry=0x7fffe8fbcaf0, size=size@entry=4, 
    access_size_min=<optimized out>, access_size_max=<optimized out>, access=0x555555654910 <memory_region_write_accessor>, 
    mr=0x5555563faaa8) at /usr/src/debug/qemu-2.1.2/memory.c:481
        access_mask = 4294967295
        access_size = 4
        i = <optimized out>
#9  0x0000555555659367 in memory_region_dispatch_write (size=4, data=955252707, addr=4, mr=0x5555563faaa8)
    at /usr/src/debug/qemu-2.1.2/memory.c:1143
No locals.
#10 io_mem_write (mr=mr@entry=0x5555563faaa8, addr=4, val=<optimized out>, size=4) at /usr/src/debug/qemu-2.1.2/memory.c:1976
No locals.
#11 0x0000555555624723 in address_space_rw (as=0x555555cb0480 <address_space_io>, addr=addr@entry=49220, 
    buf=0x7ffff7ec1000 "\343\377\357\070", len=len@entry=4, is_write=is_write@entry=true) at /usr/src/debug/qemu-2.1.2/exec.c:2088
        l = 4
        ptr = <optimized out>
        val = 955252707
        addr1 = 4
        mr = 0x5555563faaa8
        error = false
#12 0x0000555555653c20 in kvm_handle_io (count=1, size=4, direction=<optimized out>, data=<optimized out>, port=49220)
    at /usr/src/debug/qemu-2.1.2/kvm-all.c:1600
        i = 0
        ptr = <optimized out>
#13 kvm_cpu_exec (cpu=cpu@entry=0x55555637c9c0) at /usr/src/debug/qemu-2.1.2/kvm-all.c:1737
        run = 0x7ffff7ec0000
        ret = <optimized out>
        run_ret = <optimized out>
#14 0x0000555555642dc2 in qemu_kvm_cpu_thread_fn (arg=0x55555637c9c0) at /usr/src/debug/qemu-2.1.2/cpus.c:874
---Type <return> to continue, or q <return> to quit---
        cpu = 0x55555637c9c0
        r = <optimized out>
#15 0x00007ffff6ba2df3 in start_thread (arg=0x7fffe8fbd700) at pthread_create.c:308
        __res = <optimized out>
        pd = 0x7fffe8fbd700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140737102206720, -4482216436985367725, 1, 140737102207424, 140737102206720, 
                140737488345680, 4482249389558068051, 4482236741904495443}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 
              0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
#16 0x00007ffff1f623dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.


Expected results:

works well
Additional info:

If did not enable mq inside guest, won't hit this issue.

Comment 2 Qian Guo 2014-10-10 05:28:02 UTC
When qemu-kvm coredumpd, it prints 
"
qemu-kvm: could not disable queue
qemu-kvm: /builddir/build/BUILD/qemu-2.1.2/hw/net/virtio-net.c:437: virtio_net_set_queues: Assertion `!r' failed.
"
first

So this should be a copy from bug 1004275 and just filed to the qemu-kvm-rhev component.

Comment 3 jason wang 2014-10-11 05:38:50 UTC
Less priority, it's hard to recover from host mis-configuration.

Comment 4 Ronen Hod 2014-10-21 11:26:03 UTC
Deferring.
QEMU cannot recover from a non-functioning (deleted) interface. Need to think what can be done.

Comment 5 jason wang 2014-11-07 08:27:44 UTC
More investigation: this could be fixed by both side:
- Qemu, warn instead of abort(), this will help the case that SELinux prevent qemu from adding/removing queues.
- Kernel, do not allow the device to be deleted before file descriptors/sockets were gone. This may prevent tap to be deleted when qemu is running. This seems a regression when multiqueue was introduced.

Comment 6 jason wang 2014-11-27 06:55:16 UTC
Confirm with Michael:

This could be addressed after virtio 1.0 is merged. It has a specific device status field (DEVICE_NEEDS_RESET). So we qemu detects misconfig like this bug, it can set DEVICE_NEEDS_RESET filed and issue an config change interrupt for driver to disable the device.

Comment 8 jason wang 2014-12-15 05:32:28 UTC
Lower severity as host mis-configuration.

Comment 10 jason wang 2015-08-10 05:27:58 UTC
Low priority will try to fix it in 7.3.

Comment 11 jason wang 2015-09-24 09:17:59 UTC
*** Bug 1265903 has been marked as a duplicate of this bug. ***

Comment 16 jason wang 2016-12-26 05:47:33 UTC
*** Bug 1242383 has been marked as a duplicate of this bug. ***