Bug 751937 - qxl triggers assert during iofuzz test
qxl triggers assert during iofuzz test
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm (Show other bugs)
7.0
Unspecified Unspecified
medium Severity medium
: rc
: 7.0
Assigned To: Gerd Hoffmann
Virtualization Bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2011-11-08 00:17 EST by Xiaoqing Wei
Modified: 2015-04-28 13:17 EDT (History)
18 users (show)

See Also:
Fixed In Version: qemu-kvm-1.5.3-51.el7
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-06-13 06:26:38 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
gdb bt full (13.37 KB, text/plain)
2011-11-08 20:58 EST, Xiaoqing Wei
no flags Details
autotest log (37.15 KB, text/plain)
2014-02-18 22:36 EST, mazhang
no flags Details
autotest debug info (13.47 KB, text/plain)
2014-03-09 22:26 EDT, mazhang
no flags Details

  None (edit)
Description Xiaoqing Wei 2011-11-08 00:17:33 EST
Description of problem:
qemu-kvm core dumps during iofuzz test

Version-Release number of selected component (if applicable):
qemu-kvm-0.12.1.2-2.209.el6.x86_64

How reproducible:
1 / 5

Steps to Reproduce:


    KVM iofuzz test:
    1) Log into a guest
    2) Enumerate all IO port ranges through /proc/ioports
    3) On each port of the range:
        * Read it
        * Write 0 to it
        * Write a random value to a random port on a random order

cmd line:
/home/staf-kvm-devel/autotest-devel/client/tests/kvm/qemu -name 'vm1' -chardev socket,id=qmp_monitor_id_qmpmonitor1,path=/tmp/monitor-qmpmonitor1-20111105-012707-cwg7,server,nowait -mon chardev=qmp_monitor_id_qmpmonitor1,mode=control -chardev socket,id=serial_id_20111105-012707-cwg7,path=/tmp/serial-20111105-012707-cwg7,server,nowait \
\
-device isa-serial,chardev=serial_id_20111105-012707-cwg7 -drive file='RHEL-Server-6.1-64-virtio.qcow2',index=0,if=none,id=drive-virtio-disk1,media=disk,cache=none,format=qcow2,aio=native \
-device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk1,id=virtio-disk1 \
-device virtio-net-pci,netdev=idQxQNpv,mac=9a:f8:52:c1:72:27,id=ndev00idQxQNpv,bus=pci.0,addr=0x3 \
-netdev tap,id=idQxQNpv,vhost=on,fd=21 \
\
-m 2048 -smp 2,cores=1,threads=1,sockets=2 -cpu cpu64-rhel6,+sse2,+x2apic \
-spice port=8000,disable-ticketing -vga qxl -rtc base=utc,clock=host,driftfix=slew \
-boot order=cdn,once=c,menu=off -no-kvm-pit-reinjection -M rhel6.2.0 \
-device intel-hda -device hda-duplex -global qxl.debug=1 \
-global qxl.output=1 -usb -device usb-tablet -enable-kvm


Actual results:
qemu-kvm core dumps

Expected results:

qemu-kvm works fine
Additional info:
gdb output:

Program terminated with signal 11, Segmentation fault.
#0  bdrv_read (bs=0x0, sector_num=878539882841, buf=0x35c1800 "\377\377\377\377", nb_sectors=1) at block.c:958
958  
BlockDriver *drv = bs->drv;

(gdb) #0  bdrv_read (bs=0x0, sector_num=878539882841, buf=0x35c1800 "\377\377\377\377", nb_sectors=1) at block.c:958
#1  0x0000000000439a41 in ide_sector_read (s=0x2b6f8c0) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/ide/core.c:386
#2  0x000000000042ca73 in kvm_handle_io (env=0x2b72b20) at /usr/src/debug/qemu-kvm-0.12.1.2/kvm-all.c:574
#3  kvm_run (env=0x2b72b20) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1036
#4  0x000000000042cc59 in kvm_cpu_exec (env=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1730
#5  0x000000000042da9e in kvm_main_loop_cpu (_env=0x2b72b20) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1991
#6  ap_main_loop (_env=0x2b72b20) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2041
#7  0x00000030148077f1 in start_thread (arg=0x7f690bd82700) at pthread_create.c:301
#8  0x00000030140e570d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
Comment 3 Xiaoqing Wei 2011-11-08 20:58:33 EST
Created attachment 532435 [details]
gdb bt full
Comment 15 Gerd Hoffmann 2013-03-20 10:45:29 EDT
Please retest, qxl upstream got a bunch of robustness patches for this over time and RHEL-7 should be in pretty good shape.
Comment 16 Xiaoqing Wei 2013-03-20 23:13:27 EDT
(In reply to comment #15)
> Please retest, qxl upstream got a bunch of robustness patches for this over
> time and RHEL-7 should be in pretty good shape.

Hi Gerd,

could you pls tell which version ?

is the 
spice-server-0.12.2-1.el7.x86_64
qemu-img-1.4.0-1.el7.x86_64
and use the RHEL-7(compose 0306.0, latest one)
contain the fix ?

or you mean clone the git://qemu.org and compile ?

Thx
Comment 17 Gerd Hoffmann 2013-03-21 03:36:10 EDT
latest rhel-7 compose is fine, qemu 1.4 has the fixes.
Comment 18 Xiaoqing Wei 2013-03-29 00:58:30 EDT
(In reply to comment #17)
> latest rhel-7 compose is fine, qemu 1.4 has the fixes.

Hi, 1.4 still fails: no knowing whether same BZ.

(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x00007fb3c20f5942 in access_with_adjusted_size (addr=addr@entry=0, value=value@entry=0x7fb3a7ffeae8, size=1, access_size_min=<optimized out>, 
    access_size_max=<optimized out>, access=access@entry=0x7fb3c20f5f60 <memory_region_write_accessor>, opaque=opaque@entry=0x7fb3c3426128)
    at /usr/src/debug/qemu-1.4.0/memory.c:364
#2  0x00007fb3c20f6fb7 in memory_region_iorange_write (iorange=<optimized out>, offset=0, width=1, data=0) at /usr/src/debug/qemu-1.4.0/memory.c:439
#3  0x00007fb3c20f3c22 in kvm_handle_io (count=1, size=1, direction=1, data=<optimized out>, port=45064) at /usr/src/debug/qemu-1.4.0/kvm-all.c:1426
#4  kvm_cpu_exec (env=env@entry=0x7fb3c33988d0) at /usr/src/debug/qemu-1.4.0/kvm-all.c:1581
#5  0x00007fb3c209d871 in qemu_kvm_cpu_thread_fn (arg=0x7fb3c33988d0) at /usr/src/debug/qemu-1.4.0/cpus.c:759
#6  0x00007fb3c030fd15 in start_thread (arg=0x7fb3a7fff700) at pthread_create.c:308
#7  0x00007fb3bca3f46d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:114
(gdb) 

Host: 
ipxe-bootimgs-20120328-2.gitaac9718.el7.noarch
qemu-kvm-1.4.0-1.el7.x86_64
spice-server-0.12.2-1.el7.x86_64
3.8.0-0.40.el7.x86_64
Comment 19 Xiaoqing Wei 2013-03-31 22:16:07 EDT
yet another core dump, on same host.

Using host libthread_db library "/lib64/libthread_db.so.1".

warning: Skipping deprecated .gdb_index section in /usr/lib/debug/lib64/libkeyutils.so.1.4.debug.
Do "set use-deprecated-index-sections on" before the file is read
to use the section anyway.
Core was generated by `/home/staf-kvm-devel/autotest-devel/client/tests/kvm/qemu -S -name vm1 -nodefau'.
Program terminated with signal 8, Arithmetic exception.
#0  0x00007faa644a7ff4 in ide_set_sector (s=0x7faa67549c18, sector_num=-1) at hw/ide/core.c:488
488	        cyl = sector_num / (s->heads * s->sectors);
(gdb) bt
#0  0x00007faa644a7ff4 in ide_set_sector (s=0x7faa67549c18, sector_num=-1) at hw/ide/core.c:488
#1  0x00007faa644a965b in ide_exec_cmd (bus=<optimized out>, val=<optimized out>) at hw/ide/core.c:1266
#2  0x00007faa645e6f13 in memory_region_iorange_write (iorange=<optimized out>, offset=375, width=1, data=<optimized out>)
    at /usr/src/debug/qemu-1.4.0/memory.c:430
#3  0x00007faa645e3c22 in kvm_handle_io (count=1, size=1, direction=1, data=<optimized out>, port=375) at /usr/src/debug/qemu-1.4.0/kvm-all.c:1426
#4  kvm_cpu_exec (env=env@entry=0x7faa674db160) at /usr/src/debug/qemu-1.4.0/kvm-all.c:1581
#5  0x00007faa6458d871 in qemu_kvm_cpu_thread_fn (arg=0x7faa674db160) at /usr/src/debug/qemu-1.4.0/cpus.c:759
#6  0x00007faa627ffd15 in start_thread (arg=0x7faa55b16700) at pthread_create.c:308
#7  0x00007faa5ef2f46d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:114
(gdb) 2
Comment 20 Gerd Hoffmann 2013-04-02 08:07:29 EDT
> #3  0x00007fb3c20f3c22 in kvm_handle_io (count=1, size=1, direction=1,
> data=<optimized out>, port=45064) at /usr/src/debug/qemu-1.4.0/kvm-all.c:1426

port=45064 (0xb008).  That isn't qxl but the piix-pm, please open a new bug.
Comment 21 Gerd Hoffmann 2013-04-02 08:09:17 EDT
> Program terminated with signal 8, Arithmetic exception.
> #0  0x00007faa644a7ff4 in ide_set_sector (s=0x7faa67549c18, sector_num=-1)
> at hw/ide/core.c:488
> 488	        cyl = sector_num / (s->heads * s->sectors);
> (gdb) bt

That isn't qxl too, please open a new bug for it.
Comment 22 Xiaoqing Wei 2013-04-03 02:14:59 EDT
(In reply to comment #20)
> > #3  0x00007fb3c20f3c22 in kvm_handle_io (count=1, size=1, direction=1,
> > data=<optimized out>, port=45064) at /usr/src/debug/qemu-1.4.0/kvm-all.c:1426
> 
> port=45064 (0xb008).  That isn't qxl but the piix-pm, please open a new bug.

Bug 947691 - piix-pm triggers assert during iofuzz test
Comment 23 Xiaoqing Wei 2013-04-03 02:15:24 EDT
(In reply to comment #21)
> > Program terminated with signal 8, Arithmetic exception.
> > #0  0x00007faa644a7ff4 in ide_set_sector (s=0x7faa67549c18, sector_num=-1)
> > at hw/ide/core.c:488
> > 488	        cyl = sector_num / (s->heads * s->sectors);
> > (gdb) bt
> 
> That isn't qxl too, please open a new bug for it.

Bug 947694 - ide triggers assert during iofuzz test
Comment 26 mazhang 2014-01-26 00:36:38 EST
Try reproduce this bug, but autotest still running after 5 days, so kill process, will change the config file and re-test it after back.

Thanks,
Mazhang.
Comment 27 mazhang 2014-02-18 22:35:10 EST
Test this bug with new qemu-kvm and kernel package.

Host:
qemu-img-1.5.3-46.el7.x86_64
qemu-kvm-common-1.5.3-46.el7.x86_64
qemu-kvm-1.5.3-46.el7.x86_64
qemu-kvm-debuginfo-1.5.3-46.el7.x86_64
ipxe-roms-qemu-20130517-3.gitc4bce43.el7.noarch
qemu-kvm-tools-1.5.3-46.el7.x86_64
kernel-3.10.0-86.el7.x86_64

Guest:
kernel-3.10.0-48.el7.x86_64

Steps:
    KVM iofuzz test:
    1) Log into a guest
    2) Enumerate all IO port ranges through /proc/ioports
    3) On each port of the range:
        * Read it
        * Write 0 to it
        * Write a random value to a random port on a random order

Result:
First time test, got bz1046890 ,VM has quit while write a random value to 49160, as 1046890#c2 mentioned not a bug.

Second time test, VM has quit while write a random value to 43328.
Autotest log:
02/19 04:34:05 INFO |   aexpect:0907| [qemu output] qemu: virtio_ioport_write: unexpected address 0x6 value 0xdf
02/19 04:38:39 INFO |   aexpect:0907| [qemu output] qemu: virtio_ioport_write: unexpected address 0x1 value 0x66
02/19 04:57:45 INFO |   aexpect:0907| [qemu output] qemu: virtio_ioport_write: unexpected address 0x2 value 0xa7
02/19 05:48:24 INFO |   aexpect:0907| [qemu output] qemu: virtio_ioport_write: unexpected address 0x13 value 0xb4
02/19 06:15:46 INFO |   aexpect:0907| [qemu output] qemu: virtio_ioport_write: unexpected address 0x6 value 0xf7
02/19 06:20:17 INFO |   aexpect:0907| [qemu output] qemu: virtio_ioport_write: unexpected address 0x0 value 0xf8
02/19 06:20:45 INFO |   aexpect:0907| [qemu output] qemu: virtio_ioport_write: unexpected address 0x1 value 0x6c
02/19 06:23:45 INFO |   aexpect:0907| [qemu output] qemu: virtio_ioport_write: unexpected address 0x1 value 0x1b
02/19 06:37:47 INFO |   aexpect:0907| [qemu output] qemu: virtio_ioport_write: unexpected address 0x0 value 0x30
02/19 06:51:35 INFO |   aexpect:0907| [qemu output] qemu: virtio_ioport_write: unexpected address 0x13 value 0x2b
02/19 07:33:59 INFO |   aexpect:0907| [qemu output] qemu: virtio_ioport_write: unexpected address 0x13 value 0x1
02/19 09:02:18 INFO |   aexpect:0907| [qemu output] qemu: Guest moved used index from 6140 to 0
02/19 09:02:18 INFO |   aexpect:0907| [qemu output] (Process terminated with status 1)
02/19 09:02:33 ERROR|      virt:0155| Test failed: TestFail: VM has quit abnormally during write: [43328, 46]
02/19 09:02:34 INFO |env_proces:0251| Video creation failed for vm virt-tests-vm1: gstreamer-python library was not found
02/19 09:02:34 ERROR|      test:0414| Exception escaping from test:
Traceback (most recent call last):
  File "/root/staf-kvm-devel/autotest-devel/client/shared/test.py", line 411, in _exec
    _call_test_function(self.execute, *p_args, **p_dargs)
  File "/root/staf-kvm-devel/autotest-devel/client/shared/test.py", line 823, in _call_test_function
    return func(*args, **dargs)
  File "/root/staf-kvm-devel/autotest-devel/client/shared/test.py", line 291, in execute
    postprocess_profiled_run, args, dargs)
  File "/root/staf-kvm-devel/autotest-devel/client/shared/test.py", line 209, in _call_run_once
    *args, **dargs)
  File "/root/staf-kvm-devel/autotest-devel/client/shared/test.py", line 313, in run_once_profiling
    self.run_once(*args, **dargs)
  File "/root/staf-kvm-devel/autotest-devel/client/tests/virt/virt.py", line 139, in run_once
    run_func(self, params, env)
  File "/root/staf-kvm-devel/autotest-devel/client/tests/virt/tests/iofuzz.py", line 132, in run_iofuzz
    fuzz(session, inst)
  File "/root/staf-kvm-devel/autotest-devel/client/tests/virt/tests/iofuzz.py", line 88, in fuzz
    "%s: %s" % (op, operand))
TestFail: VM has quit abnormally during write: [43328, 46]
Comment 28 mazhang 2014-02-18 22:36:10 EST
Created attachment 864949 [details]
autotest log
Comment 29 xhan 2014-02-19 05:05:28 EST
Met this on 
qemu-kvm-1.5.3-47.el7.x86_64
kernel-3.10.0-88.el7.x86_64

(gdb) 
#0  qxl_set_mode (d=d@entry=0x7f6c558eb470, modenr=modenr@entry=247, loadvm=loadvm@entry=0) at /usr/src/debug/qemu-1.5.3/hw/display/qxl.c:1431
#1  0x00007f6c5380c95d in ioport_write (opaque=0x7f6c558eb470, addr=6, val=247, size=1) at /usr/src/debug/qemu-1.5.3/hw/display/qxl.c:1593
#2  0x00007f6c53838993 in access_with_adjusted_size (addr=addr@entry=6, value=value@entry=0x7f6c4525ab88, size=1, access_size_min=<optimized out>, 
    access_size_max=<optimized out>, access=access@entry=0x7f6c53838eb0 <memory_region_write_accessor>, opaque=opaque@entry=0x7f6c558fcd08)
    at /usr/src/debug/qemu-1.5.3/memory.c:365
#3  0x00007f6c53839bcf in memory_region_iorange_write (iorange=<optimized out>, offset=6, width=1, data=247) at /usr/src/debug/qemu-1.5.3/memory.c:440
#4  0x00007f6c53837a52 in kvm_handle_io (count=1, size=1, direction=1, data=<optimized out>, port=49158) at /usr/src/debug/qemu-1.5.3/kvm-all.c:1519
#5  kvm_cpu_exec (env=env@entry=0x7f6c55882bf0) at /usr/src/debug/qemu-1.5.3/kvm-all.c:1671
#6  0x00007f6c537ec1c5 in qemu_kvm_cpu_thread_fn (arg=0x7f6c55882bf0) at /usr/src/debug/qemu-1.5.3/cpus.c:793
#7  0x00007f6c51647df3 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f6c4e35339d in clone () from /lib64/libc.so.6
Comment 30 Gerd Hoffmann 2014-02-19 05:46:24 EST
http://patchwork.ozlabs.org/patch/321829/
Comment 31 mazhang 2014-02-26 02:07:49 EST
Retest this bug on qemu-kvm-rhev-1.5.3-47.el7.x86_64, result see 947694#c13 .
Comment 32 Gerd Hoffmann 2014-02-26 06:33:20 EST
upstream commit 9c70434f825fd0d2e89d1aa0f872159378d0aab3 now.
Comment 33 Gerd Hoffmann 2014-02-26 06:54:55 EST
backport posted.
Comment 34 Miroslav Rezanina 2014-03-05 05:32:39 EST
Fix included in qemu-kvm-1.5.3-51.el7
Comment 35 mazhang 2014-03-09 22:26:25 EDT
Created attachment 872542 [details]
autotest debug info
Comment 36 mazhang 2014-03-09 22:27:47 EDT
Update qemu-kvm package, iofuzz without virtio devices, vm kernel crash, debug info see comment#35 .

qemu-img-1.5.3-52.el7.x86_64
qemu-kvm-common-1.5.3-52.el7.x86_64
qemu-kvm-1.5.3-52.el7.x86_64
qemu-kvm-debuginfo-1.5.3-52.el7.x86_64
ipxe-roms-qemu-20130517-3.gitc4bce43.el7.noarch
qemu-kvm-tools-1.5.3-52.el7.x86_64
Comment 37 Paolo Bonzini 2014-03-10 08:46:18 EDT
VM kernel crash during iofuzz is fine.
Comment 38 mazhang 2014-03-10 22:33:09 EDT
This time iofuzz test (in comment#36) not hit qxl problem.
Comment 41 Ludek Smid 2014-06-13 06:26:38 EDT
This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.

Note You need to log in before you can comment on or make changes to this bug.