Bug 1120541 - qemu-kvm crashed when doing iofuzz testing
Summary: qemu-kvm crashed when doing iofuzz testing
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: qemu-kvm
Version: 6.6
Hardware: Unspecified
OS: Unspecified
high
unspecified
Target Milestone: rc
: ---
Assignee: Kevin Wolf
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks: 1123372
TreeView+ depends on / blocked
 
Reported: 2014-07-17 06:41 UTC by ShupingCui
Modified: 2016-05-16 04:07 UTC (History)
13 users (show)

Fixed In Version: qemu-kvm-0.12.1.2-2.440.el6
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1123372 (view as bug list)
Environment:
Last Closed: 2014-10-14 07:02:42 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2014:1490 normal SHIPPED_LIVE qemu-kvm bug fix and enhancement update 2014-10-14 01:28:27 UTC

Description ShupingCui 2014-07-17 06:41:26 UTC
Description of problem:
run iofuzz(sub test of autotest) test with rhel7.0 guest on rhel6 host, qemu-kvm crashed

Version-Release number of selected component (if applicable):
# uname -r
2.6.32-489.el6.x86_64
# rpm -qa | grep qemu-kvm
qemu-kvm-rhev-0.12.1.2-2.430.el6.x86_64

How reproducible:
2/10

Steps to Reproduce:
1. boot rhel7.0 guest on rhel6 host
/usr/bin/qemu-kvm \
    -name 'virt-tests-vm1' \
    -M rhel6.5.0  \
    -nodefaults  \
    -vga qxl  \
    -global qxl-vga.vram_size=33554432 \
    -device AC97,bus=pci.0,addr=03  \
    -device ich9-usb-uhci1,id=usb1,bus=pci.0,addr=04 \
    -drive id=drive_image1,if=none,cache=none,snapshot=off,aio=native,file=/root/tests/staf-kvm-devel/autotest-devel/client/tests/virt/shared/data/images/RHEL-Server-7.0-64-virtio.qcow2 \
    -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0,addr=05 \
    -device virtio-net-pci,mac=9a:a7:a8:a9:aa:ab,id=idSWv9BE,vectors=4,netdev=iddyBrTR,bus=pci.0,addr=06  \
    -netdev tap,id=iddyBrTR,vhost=on,vhostfd=23,fd=22  \
    -m 4096  \
    -smp 1,maxcpus=1,cores=1,threads=1,sockets=2  \
    -cpu 'Opteron_G2' \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -spice port=3000,password=123456,addr=0,image-compression=auto_glz,zlib-glz-wan-compression=auto,streaming-video=all,agent-mouse=on,playback-compression=on,ipv4  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot order=cdn,once=c,menu=off  \
    -no-kvm-pit-reinjection \
    -enable-kvm

2. run iofuzz test in rhel7.0 guest

Actual results:
qemu-kvm crashed

(gdb) bt full
#0  0x00007f795bf6c99b in bdrv_acct_done (bs=0x0, cookie=0x7f795e548ce8) at /usr/src/debug/qemu-kvm-0.12.1.2/block.c:4575
        __PRETTY_FUNCTION__ = "bdrv_acct_done"
#1  0x00007f795c0a436b in ide_flush_cb (opaque=0x7f795e548a48, ret=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/ide/core.c:844
        s = 0x7f795e548a48
#2  0x00007f795bf571b7 in kvm_handle_io (env=0x7f795d0fd370) at /usr/src/debug/qemu-kvm-0.12.1.2/kvm-all.c:145
        i = <value optimized out>
        ptr = <value optimized out>
#3  kvm_run (env=0x7f795d0fd370) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1061
        r = <value optimized out>
        kvm = 0x7f795cf6eef0
        run = 0x7f795be9c000
        fd = 40
#4  0x00007f795bf573a9 in kvm_cpu_exec (env=0x7f795d0fd370) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1756
        r = <value optimized out>
#5  0x00007f795bf5829d in kvm_main_loop_cpu (_env=0x7f795d0fd370) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2018
        run_cpu = <value optimized out>
#6  ap_main_loop (_env=0x7f795d0fd370) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2074
        env = 0x7f795d0fd370
        signals = {__val = {18446744067267100671, 18446744073709551615 <repeats 15 times>}}
        data = <value optimized out>
#7  0x00007f795b8669d1 in start_thread (arg=0x7f7951153700) at pthread_create.c:301
        __res = <value optimized out>
        pd = 0x7f7951153700
        now = <value optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140159028115200, -1267492048906720153, 140159205487456, 140159028115904, 0, 3, 1196361663721794663, 
                1196340938515442791}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <value optimized out>
        pagesize_m1 = <value optimized out>
        sp = <value optimized out>
        freesize = <value optimized out>
#8  0x00007f7958a91b6d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115


Expected results:
qemu-kvm no crash

Additional info:

Comment 3 Kevin Wolf 2014-07-22 12:59:00 UTC
Confirmed by code inspection.

When sending a FLUSH command to a slot without a drive (s->bs == NULL),
ide_flush_cache() directly calls ide_flush_cb() to return completion. However,
the latter doesn't care to check for s->bs == NULL before using it, so it
crashes.

Upstream still contains the same bug.

Comment 5 Jeff Nelson 2014-08-20 20:59:13 UTC
Fix included in qemu-kvm-0.12.1.2-2.440.el6

Comment 7 ShupingCui 2014-08-25 02:22:42 UTC
Re-test 10 times with qemu-kvm-0.12.1.2-2.440.el6.x86_64, not fount this issue, qemu no crash.


Thanks,
Shuping

Comment 8 Qunfang Zhang 2014-08-26 02:07:31 UTC
Although we found another new bug about iofuzz testing [1], but the bt log is different with this bug. So we will verify this bug first and track the new issue in bug 1133393. 

[1] Bug 1133393 - qemu core dump on iofuzz test

Comment 10 Xiaoqing Wei 2014-09-03 05:53:42 UTC
Hello Kevin,

I met a similar core dump on version -442, no knowing whether identical or not, could you pls help to have a look ?

(gdb) bt full
#0  0x00007fd072101347 in ide_set_sector (s=0x7fd0781d1a48, sector_num=-1)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/ide/core.c:367
        cyl = <value optimized out>
        r = <value optimized out>
#1  0x00007fd0721039bc in ide_exec_cmd (bus=0x7fd0781d19f0, val=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/hw/ide/core.c:1319
        s = 0x7fd0781d1a48
        n = <value optimized out>
        lba48 = <value optimized out>
#2  0x00007fd071fb12b7 in kvm_handle_io (env=0x7fd07318b180) at /usr/src/debug/qemu-kvm-0.12.1.2/kvm-all.c:145
        i = <value optimized out>
        ptr = <value optimized out>
#3  kvm_run (env=0x7fd07318b180) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1061
        r = <value optimized out>
        kvm = 0x7fd072f30150
        run = 0x7fd071da4000
        fd = 56
#4  0x00007fd071fb14a9 in kvm_cpu_exec (env=0x7fd07318b180) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1756
        r = <value optimized out>
#5  0x00007fd071fb239d in kvm_main_loop_cpu (_env=0x7fd07318b180) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2018
        run_cpu = <value optimized out>
#6  ap_main_loop (_env=0x7fd07318b180) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2074
        env = 0x7fd07318b180
        signals = {__val = {18446744067267100671, 18446744073709551615 <repeats 15 times>}}
        data = <value optimized out>
#7  0x00007fd0718bf9d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#8  0x00007fd06eac9ccd in clone () from /lib64/libc.so.6
No symbol table info available.



cmd as below:
/usr/bin/qemu-kvm \
    -S  \
    -name 'virt-tests-vm1' \
    -M rhel6.6.0  \
    -nodefaults  \
    -vga cirrus \
    -device AC97,bus=pci.0,addr=03  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/tmp/monitor-qmpmonitor1-20140903-093209-ciEUVoMr,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=serial_id_serial0,path=/tmp/serial-serial0-20140903-093209-ciEUVoMr,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -chardev socket,id=seabioslog_id_20140903-093209-ciEUVoMr,path=/tmp/seabios-20140903-093209-ciEUVoMr,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20140903-093209-ciEUVoMr,iobase=0x402 \
    -device ich9-usb-uhci1,id=usb1,bus=pci.0,addr=04 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=05 \
    -drive id=drive_image1,if=none,cache=none,snapshot=off,aio=native,file=/usr/local/staf/test/RHEV/kvm/autotest-devel/client/tests/virt/shared/data/images/RHEL-Server-6.6-64-virtio.qcow2 \
    -device scsi-hd,id=image1,drive=drive_image1 \
    -device virtio-net-pci,mac=9a:19:1a:1b:1c:1d,id=idry3TQe,vectors=4,netdev=idmCrt2c,bus=pci.0,addr=06  \
    -netdev tap,id=idmCrt2c,vhost=on,vhostfd=28,fd=27  \
    -m 16384  \
    -smp 16,maxcpus=16,cores=4,threads=2,sockets=2  \
    -cpu 'Westmere' \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot order=cdn,once=c,menu=off  \
    -no-kvm-pit-reinjection \
    -enable-kvm



Regards,
Xiaoqing.

Comment 11 Qunfang Zhang 2014-09-03 05:57:47 UTC
Hi, Kevin

Could you help check comment 10?  We verified this bug on comment 7, and the issue could not be reproduced.  Currently we are executing a round of bug re-verifcation work, and hit the issue in comment 10. 

Thanks,
Qunfang

Comment 12 Kevin Wolf 2014-09-03 13:19:22 UTC
Comment 10 describes a separate bug, so please file a new BZ for it. Unfortunately
the crash message at the beginning has been truncated from the report, so I can't
see what really happened. I suppose it's a divison by zero. If the core dump is
still available, can you also paste the result of 'p *s' into the description
of the new BZ, please?

Comment 13 Xiaoqing Wei 2014-09-03 14:33:07 UTC
(In reply to Kevin Wolf from comment #12)

Hello Kevin,

As finding another core dump, I filed two bz as below:
1136878 	qemu core dump on iofuzz test on qemu version 442 

1136894 	qemu core dump on iofuzz test on qemu version 442(virtio-net-vhost) 


Thank you for the explain

Comment 14 errata-xmlrpc 2014-10-14 07:02:42 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1490.html


Note You need to log in before you can comment on or make changes to this bug.