Bug 751937
| Summary: | qxl triggers assert during iofuzz test | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Xiaoqing Wei <xwei> | ||||||||
| Component: | qemu-kvm | Assignee: | Gerd Hoffmann <kraxel> | ||||||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Virtualization Bugs <virt-bugs> | ||||||||
| Severity: | medium | Docs Contact: | |||||||||
| Priority: | medium | ||||||||||
| Version: | 7.0 | CC: | acathrow, bsarathy, fziglio, hhuang, juzhang, knoel, kraxel, lmiksik, mazhang, michen, mkenneth, pbonzini, qzhang, shuang, tburke, virt-maint, wdai, xhan | ||||||||
| Target Milestone: | rc | ||||||||||
| Target Release: | 7.0 | ||||||||||
| Hardware: | Unspecified | ||||||||||
| OS: | Unspecified | ||||||||||
| Whiteboard: | |||||||||||
| Fixed In Version: | qemu-kvm-1.5.3-51.el7 | Doc Type: | Bug Fix | ||||||||
| Doc Text: | Story Points: | --- | |||||||||
| Clone Of: | Environment: | ||||||||||
| Last Closed: | 2014-06-13 10:26:38 UTC | Type: | --- | ||||||||
| Regression: | --- | Mount Type: | --- | ||||||||
| Documentation: | --- | CRM: | |||||||||
| Verified Versions: | Category: | --- | |||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||
| Embargoed: | |||||||||||
| Attachments: |
|
||||||||||
Created attachment 532435 [details]
gdb bt full
Please retest, qxl upstream got a bunch of robustness patches for this over time and RHEL-7 should be in pretty good shape. (In reply to comment #15) > Please retest, qxl upstream got a bunch of robustness patches for this over > time and RHEL-7 should be in pretty good shape. Hi Gerd, could you pls tell which version ? is the spice-server-0.12.2-1.el7.x86_64 qemu-img-1.4.0-1.el7.x86_64 and use the RHEL-7(compose 0306.0, latest one) contain the fix ? or you mean clone the git://qemu.org and compile ? Thx latest rhel-7 compose is fine, qemu 1.4 has the fixes. (In reply to comment #17) > latest rhel-7 compose is fine, qemu 1.4 has the fixes. Hi, 1.4 still fails: no knowing whether same BZ. (gdb) bt #0 0x0000000000000000 in ?? () #1 0x00007fb3c20f5942 in access_with_adjusted_size (addr=addr@entry=0, value=value@entry=0x7fb3a7ffeae8, size=1, access_size_min=<optimized out>, access_size_max=<optimized out>, access=access@entry=0x7fb3c20f5f60 <memory_region_write_accessor>, opaque=opaque@entry=0x7fb3c3426128) at /usr/src/debug/qemu-1.4.0/memory.c:364 #2 0x00007fb3c20f6fb7 in memory_region_iorange_write (iorange=<optimized out>, offset=0, width=1, data=0) at /usr/src/debug/qemu-1.4.0/memory.c:439 #3 0x00007fb3c20f3c22 in kvm_handle_io (count=1, size=1, direction=1, data=<optimized out>, port=45064) at /usr/src/debug/qemu-1.4.0/kvm-all.c:1426 #4 kvm_cpu_exec (env=env@entry=0x7fb3c33988d0) at /usr/src/debug/qemu-1.4.0/kvm-all.c:1581 #5 0x00007fb3c209d871 in qemu_kvm_cpu_thread_fn (arg=0x7fb3c33988d0) at /usr/src/debug/qemu-1.4.0/cpus.c:759 #6 0x00007fb3c030fd15 in start_thread (arg=0x7fb3a7fff700) at pthread_create.c:308 #7 0x00007fb3bca3f46d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:114 (gdb) Host: ipxe-bootimgs-20120328-2.gitaac9718.el7.noarch qemu-kvm-1.4.0-1.el7.x86_64 spice-server-0.12.2-1.el7.x86_64 3.8.0-0.40.el7.x86_64 yet another core dump, on same host.
Using host libthread_db library "/lib64/libthread_db.so.1".
warning: Skipping deprecated .gdb_index section in /usr/lib/debug/lib64/libkeyutils.so.1.4.debug.
Do "set use-deprecated-index-sections on" before the file is read
to use the section anyway.
Core was generated by `/home/staf-kvm-devel/autotest-devel/client/tests/kvm/qemu -S -name vm1 -nodefau'.
Program terminated with signal 8, Arithmetic exception.
#0 0x00007faa644a7ff4 in ide_set_sector (s=0x7faa67549c18, sector_num=-1) at hw/ide/core.c:488
488 cyl = sector_num / (s->heads * s->sectors);
(gdb) bt
#0 0x00007faa644a7ff4 in ide_set_sector (s=0x7faa67549c18, sector_num=-1) at hw/ide/core.c:488
#1 0x00007faa644a965b in ide_exec_cmd (bus=<optimized out>, val=<optimized out>) at hw/ide/core.c:1266
#2 0x00007faa645e6f13 in memory_region_iorange_write (iorange=<optimized out>, offset=375, width=1, data=<optimized out>)
at /usr/src/debug/qemu-1.4.0/memory.c:430
#3 0x00007faa645e3c22 in kvm_handle_io (count=1, size=1, direction=1, data=<optimized out>, port=375) at /usr/src/debug/qemu-1.4.0/kvm-all.c:1426
#4 kvm_cpu_exec (env=env@entry=0x7faa674db160) at /usr/src/debug/qemu-1.4.0/kvm-all.c:1581
#5 0x00007faa6458d871 in qemu_kvm_cpu_thread_fn (arg=0x7faa674db160) at /usr/src/debug/qemu-1.4.0/cpus.c:759
#6 0x00007faa627ffd15 in start_thread (arg=0x7faa55b16700) at pthread_create.c:308
#7 0x00007faa5ef2f46d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:114
(gdb) 2
> #3 0x00007fb3c20f3c22 in kvm_handle_io (count=1, size=1, direction=1,
> data=<optimized out>, port=45064) at /usr/src/debug/qemu-1.4.0/kvm-all.c:1426
port=45064 (0xb008). That isn't qxl but the piix-pm, please open a new bug.
> Program terminated with signal 8, Arithmetic exception.
> #0 0x00007faa644a7ff4 in ide_set_sector (s=0x7faa67549c18, sector_num=-1)
> at hw/ide/core.c:488
> 488 cyl = sector_num / (s->heads * s->sectors);
> (gdb) bt
That isn't qxl too, please open a new bug for it.
(In reply to comment #20) > > #3 0x00007fb3c20f3c22 in kvm_handle_io (count=1, size=1, direction=1, > > data=<optimized out>, port=45064) at /usr/src/debug/qemu-1.4.0/kvm-all.c:1426 > > port=45064 (0xb008). That isn't qxl but the piix-pm, please open a new bug. Bug 947691 - piix-pm triggers assert during iofuzz test (In reply to comment #21) > > Program terminated with signal 8, Arithmetic exception. > > #0 0x00007faa644a7ff4 in ide_set_sector (s=0x7faa67549c18, sector_num=-1) > > at hw/ide/core.c:488 > > 488 cyl = sector_num / (s->heads * s->sectors); > > (gdb) bt > > That isn't qxl too, please open a new bug for it. Bug 947694 - ide triggers assert during iofuzz test Try reproduce this bug, but autotest still running after 5 days, so kill process, will change the config file and re-test it after back. Thanks, Mazhang. Test this bug with new qemu-kvm and kernel package.
Host:
qemu-img-1.5.3-46.el7.x86_64
qemu-kvm-common-1.5.3-46.el7.x86_64
qemu-kvm-1.5.3-46.el7.x86_64
qemu-kvm-debuginfo-1.5.3-46.el7.x86_64
ipxe-roms-qemu-20130517-3.gitc4bce43.el7.noarch
qemu-kvm-tools-1.5.3-46.el7.x86_64
kernel-3.10.0-86.el7.x86_64
Guest:
kernel-3.10.0-48.el7.x86_64
Steps:
KVM iofuzz test:
1) Log into a guest
2) Enumerate all IO port ranges through /proc/ioports
3) On each port of the range:
* Read it
* Write 0 to it
* Write a random value to a random port on a random order
Result:
First time test, got bz1046890 ,VM has quit while write a random value to 49160, as 1046890#c2 mentioned not a bug.
Second time test, VM has quit while write a random value to 43328.
Autotest log:
02/19 04:34:05 INFO | aexpect:0907| [qemu output] qemu: virtio_ioport_write: unexpected address 0x6 value 0xdf
02/19 04:38:39 INFO | aexpect:0907| [qemu output] qemu: virtio_ioport_write: unexpected address 0x1 value 0x66
02/19 04:57:45 INFO | aexpect:0907| [qemu output] qemu: virtio_ioport_write: unexpected address 0x2 value 0xa7
02/19 05:48:24 INFO | aexpect:0907| [qemu output] qemu: virtio_ioport_write: unexpected address 0x13 value 0xb4
02/19 06:15:46 INFO | aexpect:0907| [qemu output] qemu: virtio_ioport_write: unexpected address 0x6 value 0xf7
02/19 06:20:17 INFO | aexpect:0907| [qemu output] qemu: virtio_ioport_write: unexpected address 0x0 value 0xf8
02/19 06:20:45 INFO | aexpect:0907| [qemu output] qemu: virtio_ioport_write: unexpected address 0x1 value 0x6c
02/19 06:23:45 INFO | aexpect:0907| [qemu output] qemu: virtio_ioport_write: unexpected address 0x1 value 0x1b
02/19 06:37:47 INFO | aexpect:0907| [qemu output] qemu: virtio_ioport_write: unexpected address 0x0 value 0x30
02/19 06:51:35 INFO | aexpect:0907| [qemu output] qemu: virtio_ioport_write: unexpected address 0x13 value 0x2b
02/19 07:33:59 INFO | aexpect:0907| [qemu output] qemu: virtio_ioport_write: unexpected address 0x13 value 0x1
02/19 09:02:18 INFO | aexpect:0907| [qemu output] qemu: Guest moved used index from 6140 to 0
02/19 09:02:18 INFO | aexpect:0907| [qemu output] (Process terminated with status 1)
02/19 09:02:33 ERROR| virt:0155| Test failed: TestFail: VM has quit abnormally during write: [43328, 46]
02/19 09:02:34 INFO |env_proces:0251| Video creation failed for vm virt-tests-vm1: gstreamer-python library was not found
02/19 09:02:34 ERROR| test:0414| Exception escaping from test:
Traceback (most recent call last):
File "/root/staf-kvm-devel/autotest-devel/client/shared/test.py", line 411, in _exec
_call_test_function(self.execute, *p_args, **p_dargs)
File "/root/staf-kvm-devel/autotest-devel/client/shared/test.py", line 823, in _call_test_function
return func(*args, **dargs)
File "/root/staf-kvm-devel/autotest-devel/client/shared/test.py", line 291, in execute
postprocess_profiled_run, args, dargs)
File "/root/staf-kvm-devel/autotest-devel/client/shared/test.py", line 209, in _call_run_once
*args, **dargs)
File "/root/staf-kvm-devel/autotest-devel/client/shared/test.py", line 313, in run_once_profiling
self.run_once(*args, **dargs)
File "/root/staf-kvm-devel/autotest-devel/client/tests/virt/virt.py", line 139, in run_once
run_func(self, params, env)
File "/root/staf-kvm-devel/autotest-devel/client/tests/virt/tests/iofuzz.py", line 132, in run_iofuzz
fuzz(session, inst)
File "/root/staf-kvm-devel/autotest-devel/client/tests/virt/tests/iofuzz.py", line 88, in fuzz
"%s: %s" % (op, operand))
TestFail: VM has quit abnormally during write: [43328, 46]
Created attachment 864949 [details]
autotest log
Met this on
qemu-kvm-1.5.3-47.el7.x86_64
kernel-3.10.0-88.el7.x86_64
(gdb)
#0 qxl_set_mode (d=d@entry=0x7f6c558eb470, modenr=modenr@entry=247, loadvm=loadvm@entry=0) at /usr/src/debug/qemu-1.5.3/hw/display/qxl.c:1431
#1 0x00007f6c5380c95d in ioport_write (opaque=0x7f6c558eb470, addr=6, val=247, size=1) at /usr/src/debug/qemu-1.5.3/hw/display/qxl.c:1593
#2 0x00007f6c53838993 in access_with_adjusted_size (addr=addr@entry=6, value=value@entry=0x7f6c4525ab88, size=1, access_size_min=<optimized out>,
access_size_max=<optimized out>, access=access@entry=0x7f6c53838eb0 <memory_region_write_accessor>, opaque=opaque@entry=0x7f6c558fcd08)
at /usr/src/debug/qemu-1.5.3/memory.c:365
#3 0x00007f6c53839bcf in memory_region_iorange_write (iorange=<optimized out>, offset=6, width=1, data=247) at /usr/src/debug/qemu-1.5.3/memory.c:440
#4 0x00007f6c53837a52 in kvm_handle_io (count=1, size=1, direction=1, data=<optimized out>, port=49158) at /usr/src/debug/qemu-1.5.3/kvm-all.c:1519
#5 kvm_cpu_exec (env=env@entry=0x7f6c55882bf0) at /usr/src/debug/qemu-1.5.3/kvm-all.c:1671
#6 0x00007f6c537ec1c5 in qemu_kvm_cpu_thread_fn (arg=0x7f6c55882bf0) at /usr/src/debug/qemu-1.5.3/cpus.c:793
#7 0x00007f6c51647df3 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f6c4e35339d in clone () from /lib64/libc.so.6
Retest this bug on qemu-kvm-rhev-1.5.3-47.el7.x86_64, result see 947694#c13 . upstream commit 9c70434f825fd0d2e89d1aa0f872159378d0aab3 now. backport posted. Fix included in qemu-kvm-1.5.3-51.el7 Created attachment 872542 [details]
autotest debug info
Update qemu-kvm package, iofuzz without virtio devices, vm kernel crash, debug info see comment#35 . qemu-img-1.5.3-52.el7.x86_64 qemu-kvm-common-1.5.3-52.el7.x86_64 qemu-kvm-1.5.3-52.el7.x86_64 qemu-kvm-debuginfo-1.5.3-52.el7.x86_64 ipxe-roms-qemu-20130517-3.gitc4bce43.el7.noarch qemu-kvm-tools-1.5.3-52.el7.x86_64 VM kernel crash during iofuzz is fine. This time iofuzz test (in comment#36) not hit qxl problem. This request was resolved in Red Hat Enterprise Linux 7.0. Contact your manager or support representative in case you have further questions about the request. |
Description of problem: qemu-kvm core dumps during iofuzz test Version-Release number of selected component (if applicable): qemu-kvm-0.12.1.2-2.209.el6.x86_64 How reproducible: 1 / 5 Steps to Reproduce: KVM iofuzz test: 1) Log into a guest 2) Enumerate all IO port ranges through /proc/ioports 3) On each port of the range: * Read it * Write 0 to it * Write a random value to a random port on a random order cmd line: /home/staf-kvm-devel/autotest-devel/client/tests/kvm/qemu -name 'vm1' -chardev socket,id=qmp_monitor_id_qmpmonitor1,path=/tmp/monitor-qmpmonitor1-20111105-012707-cwg7,server,nowait -mon chardev=qmp_monitor_id_qmpmonitor1,mode=control -chardev socket,id=serial_id_20111105-012707-cwg7,path=/tmp/serial-20111105-012707-cwg7,server,nowait \ \ -device isa-serial,chardev=serial_id_20111105-012707-cwg7 -drive file='RHEL-Server-6.1-64-virtio.qcow2',index=0,if=none,id=drive-virtio-disk1,media=disk,cache=none,format=qcow2,aio=native \ -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk1,id=virtio-disk1 \ -device virtio-net-pci,netdev=idQxQNpv,mac=9a:f8:52:c1:72:27,id=ndev00idQxQNpv,bus=pci.0,addr=0x3 \ -netdev tap,id=idQxQNpv,vhost=on,fd=21 \ \ -m 2048 -smp 2,cores=1,threads=1,sockets=2 -cpu cpu64-rhel6,+sse2,+x2apic \ -spice port=8000,disable-ticketing -vga qxl -rtc base=utc,clock=host,driftfix=slew \ -boot order=cdn,once=c,menu=off -no-kvm-pit-reinjection -M rhel6.2.0 \ -device intel-hda -device hda-duplex -global qxl.debug=1 \ -global qxl.output=1 -usb -device usb-tablet -enable-kvm Actual results: qemu-kvm core dumps Expected results: qemu-kvm works fine Additional info: gdb output: Program terminated with signal 11, Segmentation fault. #0 bdrv_read (bs=0x0, sector_num=878539882841, buf=0x35c1800 "\377\377\377\377", nb_sectors=1) at block.c:958 958 BlockDriver *drv = bs->drv; (gdb) #0 bdrv_read (bs=0x0, sector_num=878539882841, buf=0x35c1800 "\377\377\377\377", nb_sectors=1) at block.c:958 #1 0x0000000000439a41 in ide_sector_read (s=0x2b6f8c0) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/ide/core.c:386 #2 0x000000000042ca73 in kvm_handle_io (env=0x2b72b20) at /usr/src/debug/qemu-kvm-0.12.1.2/kvm-all.c:574 #3 kvm_run (env=0x2b72b20) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1036 #4 0x000000000042cc59 in kvm_cpu_exec (env=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1730 #5 0x000000000042da9e in kvm_main_loop_cpu (_env=0x2b72b20) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1991 #6 ap_main_loop (_env=0x2b72b20) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2041 #7 0x00000030148077f1 in start_thread (arg=0x7f690bd82700) at pthread_create.c:301 #8 0x00000030140e570d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115