Bug 843797

Summary: qemu-kvm core dumps when virtio-net(w/ tx=timer and vhost=on) RHEL.6(w/ msi-x enabled) guest shutting down
Product: Red Hat Enterprise Linux 6 Reporter: Xiaoqing Wei <xwei>
Component: qemu-kvmAssignee: Amos Kong <akong>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.4CC: acathrow, ailan, akong, areis, bsarathy, juzhang, lnovich, michen, minovotn, mkenneth, qzhang, rhod, shuang, sradvan, tburke, virt-maint, xfu, zhzhang
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: qemu-kvm-0.12.1.2-2.385.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-11-21 05:50:01 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
guest serial output
none
gdb detail output none

Description Xiaoqing Wei 2012-07-27 11:52:03 UTC 
Description of problem:

qemu-kvm core dumps when virtio-net(w/ tx=timer and vhost=on) RHEL.6(w/ msi-x enabled) guest shutting down

Version-Release number of selected component (if applicable):

qemu-kvm-rhev-0.12.1.2-2.298.el6_3.x86_64
2.6.32-279.el6.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Boot a virtio-net  (-device virtio-net-pci,tx=timer -netdev tap,vhost=on) with RHEL guest with MSI-X enabled(eg: RHEL.6.3)
qemu-kvm -monitor stdio -nodefaults -chardev socket,id=serial_id_20120726-200643-igwg,path=/tmp/serial-20120726-200643-igwg,server,nowait -device isa-serial,chardev=serial_id_20120726-200643-igwg -device ich9-usb-uhci1,id=usb1,bus=pci.0,addr=0x4 -drive file='/home/staf-kvm-devel/autotest-devel/client/tests/kvm/images/RHEL-Server-6.3-64-virtio.qcow2',if=none,id=drive-virtio-disk1,media=disk,cache=none,boot=off,snapshot=off,format=qcow2,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk1,id=virtio-disk1 -m 4096 -smp 2,cores=1,threads=1,sockets=2 -cpu 'Penryn' -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 -vnc :0 -vga std -rtc base=utc,clock=host,driftfix=slew -M rhel6.3.0 -boot order=cdn,once=c,menu=off    -no-kvm-pit-reinjection -bios /usr/share/seabios/bios-pm.bin -enable-kvm -S \
\
\
\ -device virtio-net-pci,netdev=idBO9VqI,mac=9a:7a:eb:5b:bd:17,id=ndev00idBO9VqI,bus=pci.0,addr=0x3,tx=timer  \
\
-netdev tap,id=idBO9VqI,vhost=on

2. shutdown the guest by typing 'poweroff'
3. 
  
Actual results:
qemu-kvm core dumps

Expected results:
guest shutdown successfully, not qemu-kvm core dump

Additional info:

1) booting && shutting down a MSI-X ENABLED (vhost=on ) W2k8r2 guest Works well
2) booting && shutting down a MSI-X ENABLED (vhost=off) RHEL.6 guest Works well
3) booting && shutting down a MSI-X DISABLED(vhost=on ) RHEL.6 guest Works well
Comment 1 Xiaoqing Wei 2012-07-27 11:54:26 UTC 
Created attachment 600751 [details]
guest serial output
Comment 2 Xiaoqing Wei 2012-07-30 07:14:22 UTC 
Created attachment 601135 [details]
gdb detail output

detail gdb info attached, but for whom'd like to have quick glance

(gdb) bt
#0  0x00007f1e44caff67 in qemu_mod_timer (ts=0x7f1e45e0ed70, expire_time=71917049194)
    at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:1284
#1  0x00007f1e44cc5e55 in virtio_net_handle_tx_timer (vdev=<value optimized out>, vq=0x7f1e4721e740)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/virtio-net.c:756
#2  0x00007f1e44cc62e0 in virtio_pci_set_host_notifier_internal (proxy=0x7f1e45e13010, n=1, 
    assign=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/virtio-pci.c:224
#3  0x00007f1e44cca821 in vhost_dev_disable_notifiers (hdev=0x7f1e45c52f40, vdev=0x7f1e472125c0)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/vhost.c:677
#4  0x00007f1e44cc9fac in vhost_net_stop (net=0x7f1e45c52f40, dev=0x7f1e472125c0)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/vhost_net.c:202
#5  0x00007f1e44cc44eb in virtio_net_set_status (vdev=0x7f1e472125c0, status=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/virtio-net.c:133
#6  0x00007f1e44d0f65d in qemu_del_vlan_client (vc=0x7f1e45c41880) at net.c:329
#7  0x00007f1e44d0f6d9 in net_cleanup () at net.c:1358
#8  0x00007f1e44cb2297 in main (argc=20, argv=<value optimized out>, envp=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:6432
Comment 3 Dor Laor 2012-07-30 07:16:03 UTC 
Michael, is the combination of timer and vhost=on is relevant?
IIRC we should just fail it.
Comment 4 Amos Kong 2012-10-31 08:31:38 UTC 
Talked with mst, reassign this to me.
Comment 13 FuXiangChun 2013-06-25 10:12:19 UTC 
re-test this issue with fixed qemu-kvm-0.12.1.2-2.376.el6.x86_64

still get the same result with comment 2. 

(gdb) bt
#0  0x00007ffff7ddf9a7 in qemu_mod_timer (ts=0x7ffff9c99fa0, expire_time=59258125675)
    at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:1286
#1  0x00007ffff7df60a5 in virtio_net_handle_tx_timer (vdev=0x7ffff9caf860, vq=0x7ffff9cbb9e0)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/virtio-net.c:752
#2  0x00007ffff7df6520 in virtio_pci_set_host_notifier_internal (proxy=0x7ffff88af5e0, n=1, 
    assign=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/virtio-pci.c:224
#3  0x00007ffff7dface1 in vhost_dev_disable_notifiers (hdev=0x7ffff86ef0e0, vdev=0x7ffff9caf860)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/vhost.c:677
#4  0x00007ffff7dfa46c in vhost_net_stop (net=0x7ffff86ef0e0, dev=0x7ffff9caf860)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/vhost_net.c:202
#5  0x00007ffff7df472b in virtio_net_set_status (vdev=0x7ffff9caf860, status=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/virtio-net.c:133
#6  0x00007ffff7e448ad in qemu_del_vlan_client (vc=0x7ffff86dda20) at /usr/src/debug/qemu-kvm-0.12.1.2/net.c:329
#7  0x00007ffff7e44929 in net_cleanup () at /usr/src/debug/qemu-kvm-0.12.1.2/net.c:1363
#8  0x00007ffff7de1e7b in main (argc=36, argv=<value optimized out>, envp=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:6527
(gdb) q


So reopen this bug.
Comment 20 FuXiangChun 2013-08-02 01:01:26 UTC 
Verify this bug with qemu-kvm-0.12.1.2-2.382.el6.x86_64

steps:

1./usr/libexec/qemu-kvm -monitor stdio -nodefaults -chardev socket,id=serial_id_20120726-200643-igwg,path=/tmp/serial-20120726-200643-igwg,server,nowait -device isa-serial,chardev=serial_id_20120726-200643-igwg -device ich9-usb-uhci1,id=usb1,bus=pci.0,addr=0x4 -drive file=/home/RHEL-Server-6.3-64-virtio.qcow2,if=none,id=drive-virtio-disk1,media=disk,cache=none,boot=off,snapshot=off,format=qcow2,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk1,id=virtio-disk1 -m 4096 -smp 2,cores=1,threads=1,sockets=2 -cpu 'SandyBridge' -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 -vnc :0 -vga std -rtc base=utc,clock=host,driftfix=slew -M rhel6.3.0 -boot order=cdn,once=c,menu=off    -no-kvm-pit-reinjection -enable-kvm -device virtio-net-pci,netdev=idBO9VqI,mac=9a:7a:eb:5b:bd:17,id=ndev00idBO9VqI,bus=pci.0,addr=0x3,tx=timer -netdev tap,id=idBO9VqI,vhost=on

2. poweroff in guest

result:
guest shutdown successfully, not qemu-kvm core dump

so I think this bug is fixed
Comment 25 zhonglinzhang 2013-08-27 02:56:35 UTC 
reproduce with 2.6.32-358.el6.x86_64 kernel and qemu-kvm-0.12.1.2-2.355.el6.x86_64

Steps to Reproduce:
  1. boot a guest with gdb tools:
gdb /usr/libexec/qemu-kvm

  2. (gdb) run -M pc -cpu SandyBridge  -enable-kvm -m 4G -smp 4,sockets=1,cores=2,threads=2 -name scalability-test -rtc base=localtime,clock=host,driftfix=slew  -k en-us  -boot menu=on -spice disable-ticketing,port=5931 -vga qxl -monitor stdio -device virtio-balloon-pci,id=ballooning  -qmp tcp:0:7777,server,nowait -serial unix:/tmp/ttyS0,server,nowait -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0        -drive file=/home/RHEL-Server-6.5-64.qcow2,if=none,id=drive-system-disk,media=disk,format=qcow2,aio=native,werror=stop,rerror=stop -device virtio-blk-pci,scsi=off,bus=pci.0,drive=drive-system-disk,id=system-disk,bootindex=1,addr=0x5      -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device virtio-net-pci,netdev=hostnet0,id=virtio-net-pci0,mac=00:22:15:27:54:8d,bus=pci.0,addr=0x9,tx=timer

  3. remote spice://$host_ip:5931

  4. in guest: shutdown -h now

Actual results: qemu-kvm core dump
Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7ddf9a7 in ?? ()
(gdb) bt
#0  0x00007ffff7ddf9a7 in ?? ()
#1  0x00007ffff7df60a5 in ?? ()
#2  0x00007ffff7df6520 in ?? ()
#3  0x00007ffff7dface1 in ?? ()
#4  0x00007ffff7dfa46c in ?? ()
#5  0x00007ffff7df472b in ?? ()
#6  0x00007ffff7e448bd in ?? ()
#7  0x00007ffff7e44939 in ?? ()
#8  0x00007ffff7de1e7b in main ()


Verify with 2.6.32-414.el6.x86_64 kernel and qemu-kvm-0.12.1.2-2.398.el6.x86_64

steps as above

Actual results: 
qemu-kvm quit normally.
Program exited normally.


About comment #22
gdb /usr/libexec/qemu-kvm
(gdb) run -chardev pipe
Starting program: /usr/libexec/qemu-kvm -chardev pipe
[Thread debugging using libthread_db enabled]
qemu-kvm: -chardev pipe: chardev: no id specified

Program exited with code 01.


Based above information, so I think the bug has been fixed.
Comment 28 errata-xmlrpc 2013-11-21 05:50:01 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-1553.html