Bug 1264021 - Guest hit "virtio_net virtio1: output.0:id 0 is not a head!" after some set_link test of vhost-user interface via qemu
Guest hit "virtio_net virtio1: output.0:id 0 is not a head!" after some set_l...
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev (Show other bugs)
7.2
Unspecified Unspecified
medium Severity medium
: rc
: ---
Assigned To: jason wang
Virtualization Bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-09-17 06:35 EDT by Qian Guo
Modified: 2016-04-18 17:32 EDT (History)
12 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-04-18 17:32:42 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Qian Guo 2015-09-17 06:35:29 EDT
Description of problem:
Guest with vhost-user, set_link the netdev off, and then set link on the peer and netdev, then hit the error in guest, when try to shutdown guest, hit qemu crash.

Version-Release number of selected component (if applicable):
qemu-kvm-rhev-2.3.0-23.el7.x86_64
kernel-3.10.0-316.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Boot guest with vhost-user and virtio-net-pci:
/usr/libexec/qemu-kvm \
-name rhel71-br0-1 \
-S \
-machine pc-i440fx-rhel7.2.0,accel=kvm,usb=off \
-cpu IvyBridge \
-m 4096 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 \
-object memory-backend-file,prealloc=yes,mem-path=/dev/hugepages/qemu,share=on,size=4096M,id=ram-node0 \
-numa node,nodeid=0,cpus=0-3,memdev=ram-node0 \
-uuid 7907dc99-aa2d-4a3e-9fca-6087f3cfbbe6 \
-no-user-config -nodefaults  \
-rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=discard \
-no-hpet -no-shutdown -boot menu=on \
-drive file=/home/rhel72cp1.qcow2,if=none,id=drive-virtio-disk0,format=qcow2 \
-device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \
-chardev socket,id=charnet0,path=/var/run/openvswitch/vhost-user1 \
-netdev type=vhost-user,id=hostnet0,chardev=charnet0 \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:01:0b:11,bus=pci.0,addr=0x9  \
-chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 \
-spice port=5901,disable-ticketing,seamless-migration=on \
-device qxl,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16,bus=pci.0,addr=0xe  \
-msg timestamp=on -monitor stdio \


2.In guest, ping host:
# ping 192.168.1.1

3.During ping, do followings via hmp:
(qemu) set_link hostnet0 off

(qemu) set_link net0 on
(qemu) set_link hostnet0 on

Actual results:
1.After step3, guest hit the error:
# dmesg
[   52.240327] virtio_net virtio1: output.0:id 0 is not a head!
[   52.240342] net ens9: Unexpected TXQ (0) queue failure: -5
[   52.487191] net ens9: Unexpected TXQ (0) queue failure: -5
[   53.242224] net ens9: Unexpected TXQ (0) queue failure: -5
[   53.487212] net ens9: Unexpected TXQ (0) queue failure: -5
[   54.244047] net ens9: Unexpected TXQ (0) queue failure: -5
[   54.487206] net ens9: Unexpected TXQ (0) queue failure: -5


And the network can not be resumed.


2.Try to shutdown guest, hit crash:
(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x00005555557a2d61 in qemu_deliver_packet (sender=<optimized out>, flags=<optimized out>, 
    data=data@entry=0x55555695e8a8 "", size=<optimized out>, opaque=0x555556a06180) at net/net.c:514
#2  0x00005555557a45ea in qemu_net_queue_deliver (size=<optimized out>, data=<optimized out>, 
    flags=<optimized out>, sender=<optimized out>, queue=0x555556928cf0) at net/queue.c:157
#3  qemu_net_queue_flush (queue=0x555556928cf0) at net/queue.c:254
#4  0x00005555557a25fc in qemu_flush_or_purge_queued_packets (nc=0x555556a06180, purge=<optimized out>)
    at net/net.c:543
#5  0x00005555557a2679 in net_vm_change_state_handler (opaque=0x555556a06180, running=1452665000, state=80)
    at net/net.c:1218
#6  0x00005555556d6f12 in vm_state_notify (running=running@entry=0, state=state@entry=RUN_STATE_SHUTDOWN)
    at vl.c:1517
#7  0x000055555560f992 in do_vm_stop (state=RUN_STATE_SHUTDOWN) at /usr/src/debug/qemu-2.3.0/cpus.c:603
#8  vm_stop (state=state@entry=RUN_STATE_SHUTDOWN) at /usr/src/debug/qemu-2.3.0/cpus.c:1297
#9  0x00005555555e8591 in main_loop_should_exit () at vl.c:1774
#10 main_loop () at vl.c:1822
#11 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4394
(gdb) bt ful
#0  0x0000000000000000 in ?? ()
No symbol table info available.
#1  0x00005555557a2d61 in qemu_deliver_packet (sender=<optimized out>, flags=<optimized out>, 
    data=data@entry=0x55555695e8a8 "", size=<optimized out>, opaque=0x555556a06180) at net/net.c:514
        nc = 0x555556a06180
        ret = <optimized out>
#2  0x00005555557a45ea in qemu_net_queue_deliver (size=<optimized out>, data=<optimized out>, 
    flags=<optimized out>, sender=<optimized out>, queue=0x555556928cf0) at net/queue.c:157
        ret = -1
#3  qemu_net_queue_flush (queue=0x555556928cf0) at net/queue.c:254
        packet = 0x55555695e880
        ret = <optimized out>
#4  0x00005555557a25fc in qemu_flush_or_purge_queued_packets (nc=0x555556a06180, purge=<optimized out>)
    at net/net.c:543
No locals.
#5  0x00005555557a2679 in net_vm_change_state_handler (opaque=0x555556a06180, running=1452665000, state=80)
    at net/net.c:1218
No locals.
#6  0x00005555556d6f12 in vm_state_notify (running=running@entry=0, state=state@entry=RUN_STATE_SHUTDOWN)
    at vl.c:1517
        e = <optimized out>
        next = 0x5555569e3f60
#7  0x000055555560f992 in do_vm_stop (state=RUN_STATE_SHUTDOWN) at /usr/src/debug/qemu-2.3.0/cpus.c:603
        ret = 0
#8  vm_stop (state=state@entry=RUN_STATE_SHUTDOWN) at /usr/src/debug/qemu-2.3.0/cpus.c:1297
No locals.
#9  0x00005555555e8591 in main_loop_should_exit () at vl.c:1774
        r = <optimized out>
---Type <return> to continue, or q <return> to quit---   
#10 main_loop () at vl.c:1822
        nonblocking = <optimized out>
        last_io = 1
#11 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4394
        i = <optimized out>
        snapshot = <optimized out>
        linux_boot = <optimized out>
        initrd_filename = <optimized out>
        kernel_filename = <optimized out>
        kernel_cmdline = <optimized out>
        boot_order = 0x55555587a1c7 "cad"
        boot_once = 0x0
        cyls = <optimized out>
        heads = <optimized out>
        secs = <optimized out>
        translation = <optimized out>
        hda_opts = <optimized out>
        opts = <optimized out>
        machine_opts = <optimized out>
        icount_opts = <optimized out>
        olist = <optimized out>
        optind = 52
        optarg = 0x5555569e3560 "pc-i440fx-rhel7.2.0"
        loadvm = <optimized out>
        machine_class = <optimized out>
        cpu_model = <optimized out>
        vga_model = 0x0
        qtest_chrdev = <optimized out>
---Type <return> to continue, or q <return> to quit---
        qtest_log = <optimized out>
        pid_file = <optimized out>
        incoming = <optimized out>
        show_vnc_port = <optimized out>
        defconfig = <optimized out>
        userconfig = 56
        log_mask = <optimized out>
        log_file = <optimized out>
        mem_trace = {malloc = 0x5555556d5990 <malloc_and_trace>, realloc = 0x5555556d5970 <realloc_and_trace>, 
          free = 0x5555556d5960 <free_and_trace>, calloc = 0x0, try_malloc = 0x0, try_realloc = 0x0}
        trace_events = <optimized out>
        trace_file = <optimized out>
        maxram_size = <optimized out>
        ram_slots = <optimized out>
        vmstate_dump_file = <optimized out>
        main_loop_err = 0x0
        __func__ = "main"
(gdb) 

Expected results:
Set link work well, and no crash.

Additional info:
Not regression, test with qemu-kvm-rhev-2.1.2-23.el7.x86_64(rhel7.1GA), hit too.
Comment 3 jason wang 2015-09-22 21:05:06 EDT
Is this issue reproducible with upstream qemu.git? (And defer it to 7.3 first).
Comment 4 Qian Guo 2015-09-28 04:52:58 EDT
(In reply to jason wang from comment #3)
> Is this issue reproducible with upstream qemu.git? (And defer it to 7.3
> first).

I will test it when I got the environment, since I have not an available dpdk supported nic an hands :)

Will keep the needinfo left, and I will update here once I got the result with upstream.

Thanks,
Qian
Comment 7 Amnon Ilan 2016-04-13 16:00:32 EDT
QE, 

Can you please try this again with latest 7.3?

Thanks,
Amnon
Comment 8 FuXiangChun 2016-04-15 06:34:59 EDT
(In reply to Amnon Ilan from comment #7)
> QE, 
> 
> Can you please try this again with latest 7.3?
> 
> Thanks,
> Amnon

Amnon,

QE re-tested this bug with the latest qemu-kvm-rhev and ovs-dpdk. 

qemu-kvm-rhev-2.5.0-4.el7.x86_64
openvswitch-dpdk-2.5.0-3.el7.x86_64

result: this problem is gone.  qemu-kvm process and guest work well.





host info:
# ovs-vsctl show
f5486a94-833f-4f00-9ff1-b2d9b3655241
    Bridge "ovsbr1"
        Port "ovsbr1"
            Interface "ovsbr1"
                type: internal
        Port "vhost-user4"
            Interface "vhost-user4"
                type: dpdkvhostuser
        Port "vhost-user2"
            Interface "vhost-user2"
                type: dpdkvhostuser
    Bridge "ovsbr0"
        Port "ovsbr0"
            Interface "ovsbr0"
                type: internal
        Port "vhost-user1"
            Interface "vhost-user1"
                type: dpdkvhostuser
        Port "vhost-user3"
            Interface "vhost-user3"
                type: dpdkvhostuser

qemu-kvm-rhev command line:

/usr/libexec/qemu-kvm 
.......
-chardev socket,id=charnet1,path=/var/run/openvswitch/vhost-user1 -netdev type=vhost-user,id=hostnet1,chardev=charnet1,queues=4 -device virtio-net-pci,mq=on,vectors=10,netdev=hostnet1,id=net1,mac=52:54:00:b2:0a:90,bus=pci.0,addr=0x6 \

-chardev socket,id=charnet2,path=/var/run/openvswitch/vhost-user2 -netdev type=vhost-user,id=hostnet2,chardev=charnet2,queues=4 -device virtio-net-pci,mq=on,vectors=10,netdev=hostnet2,id=net2,mac=52:54:00:b2:0a:91,bus=pci.0,addr=0x8 \
....
Comment 9 Amnon Ilan 2016-04-18 17:32:42 EDT
Based on comment#8 closing as CURRENTRELEASE.

Note You need to log in before you can comment on or make changes to this bug.