Red Hat Bugzilla – Bug 751937
qxl triggers assert during iofuzz test
Last modified: 2015-04-28 13:17:00 EDT
Description of problem: qemu-kvm core dumps during iofuzz test Version-Release number of selected component (if applicable): qemu-kvm-0.12.1.2-2.209.el6.x86_64 How reproducible: 1 / 5 Steps to Reproduce: KVM iofuzz test: 1) Log into a guest 2) Enumerate all IO port ranges through /proc/ioports 3) On each port of the range: * Read it * Write 0 to it * Write a random value to a random port on a random order cmd line: /home/staf-kvm-devel/autotest-devel/client/tests/kvm/qemu -name 'vm1' -chardev socket,id=qmp_monitor_id_qmpmonitor1,path=/tmp/monitor-qmpmonitor1-20111105-012707-cwg7,server,nowait -mon chardev=qmp_monitor_id_qmpmonitor1,mode=control -chardev socket,id=serial_id_20111105-012707-cwg7,path=/tmp/serial-20111105-012707-cwg7,server,nowait \ \ -device isa-serial,chardev=serial_id_20111105-012707-cwg7 -drive file='RHEL-Server-6.1-64-virtio.qcow2',index=0,if=none,id=drive-virtio-disk1,media=disk,cache=none,format=qcow2,aio=native \ -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk1,id=virtio-disk1 \ -device virtio-net-pci,netdev=idQxQNpv,mac=9a:f8:52:c1:72:27,id=ndev00idQxQNpv,bus=pci.0,addr=0x3 \ -netdev tap,id=idQxQNpv,vhost=on,fd=21 \ \ -m 2048 -smp 2,cores=1,threads=1,sockets=2 -cpu cpu64-rhel6,+sse2,+x2apic \ -spice port=8000,disable-ticketing -vga qxl -rtc base=utc,clock=host,driftfix=slew \ -boot order=cdn,once=c,menu=off -no-kvm-pit-reinjection -M rhel6.2.0 \ -device intel-hda -device hda-duplex -global qxl.debug=1 \ -global qxl.output=1 -usb -device usb-tablet -enable-kvm Actual results: qemu-kvm core dumps Expected results: qemu-kvm works fine Additional info: gdb output: Program terminated with signal 11, Segmentation fault. #0 bdrv_read (bs=0x0, sector_num=878539882841, buf=0x35c1800 "\377\377\377\377", nb_sectors=1) at block.c:958 958 BlockDriver *drv = bs->drv; (gdb) #0 bdrv_read (bs=0x0, sector_num=878539882841, buf=0x35c1800 "\377\377\377\377", nb_sectors=1) at block.c:958 #1 0x0000000000439a41 in ide_sector_read (s=0x2b6f8c0) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/ide/core.c:386 #2 0x000000000042ca73 in kvm_handle_io (env=0x2b72b20) at /usr/src/debug/qemu-kvm-0.12.1.2/kvm-all.c:574 #3 kvm_run (env=0x2b72b20) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1036 #4 0x000000000042cc59 in kvm_cpu_exec (env=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1730 #5 0x000000000042da9e in kvm_main_loop_cpu (_env=0x2b72b20) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1991 #6 ap_main_loop (_env=0x2b72b20) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2041 #7 0x00000030148077f1 in start_thread (arg=0x7f690bd82700) at pthread_create.c:301 #8 0x00000030140e570d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
Created attachment 532435 [details] gdb bt full
Please retest, qxl upstream got a bunch of robustness patches for this over time and RHEL-7 should be in pretty good shape.
(In reply to comment #15) > Please retest, qxl upstream got a bunch of robustness patches for this over > time and RHEL-7 should be in pretty good shape. Hi Gerd, could you pls tell which version ? is the spice-server-0.12.2-1.el7.x86_64 qemu-img-1.4.0-1.el7.x86_64 and use the RHEL-7(compose 0306.0, latest one) contain the fix ? or you mean clone the git://qemu.org and compile ? Thx
latest rhel-7 compose is fine, qemu 1.4 has the fixes.
(In reply to comment #17) > latest rhel-7 compose is fine, qemu 1.4 has the fixes. Hi, 1.4 still fails: no knowing whether same BZ. (gdb) bt #0 0x0000000000000000 in ?? () #1 0x00007fb3c20f5942 in access_with_adjusted_size (addr=addr@entry=0, value=value@entry=0x7fb3a7ffeae8, size=1, access_size_min=<optimized out>, access_size_max=<optimized out>, access=access@entry=0x7fb3c20f5f60 <memory_region_write_accessor>, opaque=opaque@entry=0x7fb3c3426128) at /usr/src/debug/qemu-1.4.0/memory.c:364 #2 0x00007fb3c20f6fb7 in memory_region_iorange_write (iorange=<optimized out>, offset=0, width=1, data=0) at /usr/src/debug/qemu-1.4.0/memory.c:439 #3 0x00007fb3c20f3c22 in kvm_handle_io (count=1, size=1, direction=1, data=<optimized out>, port=45064) at /usr/src/debug/qemu-1.4.0/kvm-all.c:1426 #4 kvm_cpu_exec (env=env@entry=0x7fb3c33988d0) at /usr/src/debug/qemu-1.4.0/kvm-all.c:1581 #5 0x00007fb3c209d871 in qemu_kvm_cpu_thread_fn (arg=0x7fb3c33988d0) at /usr/src/debug/qemu-1.4.0/cpus.c:759 #6 0x00007fb3c030fd15 in start_thread (arg=0x7fb3a7fff700) at pthread_create.c:308 #7 0x00007fb3bca3f46d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:114 (gdb) Host: ipxe-bootimgs-20120328-2.gitaac9718.el7.noarch qemu-kvm-1.4.0-1.el7.x86_64 spice-server-0.12.2-1.el7.x86_64 3.8.0-0.40.el7.x86_64
yet another core dump, on same host. Using host libthread_db library "/lib64/libthread_db.so.1". warning: Skipping deprecated .gdb_index section in /usr/lib/debug/lib64/libkeyutils.so.1.4.debug. Do "set use-deprecated-index-sections on" before the file is read to use the section anyway. Core was generated by `/home/staf-kvm-devel/autotest-devel/client/tests/kvm/qemu -S -name vm1 -nodefau'. Program terminated with signal 8, Arithmetic exception. #0 0x00007faa644a7ff4 in ide_set_sector (s=0x7faa67549c18, sector_num=-1) at hw/ide/core.c:488 488 cyl = sector_num / (s->heads * s->sectors); (gdb) bt #0 0x00007faa644a7ff4 in ide_set_sector (s=0x7faa67549c18, sector_num=-1) at hw/ide/core.c:488 #1 0x00007faa644a965b in ide_exec_cmd (bus=<optimized out>, val=<optimized out>) at hw/ide/core.c:1266 #2 0x00007faa645e6f13 in memory_region_iorange_write (iorange=<optimized out>, offset=375, width=1, data=<optimized out>) at /usr/src/debug/qemu-1.4.0/memory.c:430 #3 0x00007faa645e3c22 in kvm_handle_io (count=1, size=1, direction=1, data=<optimized out>, port=375) at /usr/src/debug/qemu-1.4.0/kvm-all.c:1426 #4 kvm_cpu_exec (env=env@entry=0x7faa674db160) at /usr/src/debug/qemu-1.4.0/kvm-all.c:1581 #5 0x00007faa6458d871 in qemu_kvm_cpu_thread_fn (arg=0x7faa674db160) at /usr/src/debug/qemu-1.4.0/cpus.c:759 #6 0x00007faa627ffd15 in start_thread (arg=0x7faa55b16700) at pthread_create.c:308 #7 0x00007faa5ef2f46d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:114 (gdb) 2
> #3 0x00007fb3c20f3c22 in kvm_handle_io (count=1, size=1, direction=1, > data=<optimized out>, port=45064) at /usr/src/debug/qemu-1.4.0/kvm-all.c:1426 port=45064 (0xb008). That isn't qxl but the piix-pm, please open a new bug.
> Program terminated with signal 8, Arithmetic exception. > #0 0x00007faa644a7ff4 in ide_set_sector (s=0x7faa67549c18, sector_num=-1) > at hw/ide/core.c:488 > 488 cyl = sector_num / (s->heads * s->sectors); > (gdb) bt That isn't qxl too, please open a new bug for it.
(In reply to comment #20) > > #3 0x00007fb3c20f3c22 in kvm_handle_io (count=1, size=1, direction=1, > > data=<optimized out>, port=45064) at /usr/src/debug/qemu-1.4.0/kvm-all.c:1426 > > port=45064 (0xb008). That isn't qxl but the piix-pm, please open a new bug. Bug 947691 - piix-pm triggers assert during iofuzz test
(In reply to comment #21) > > Program terminated with signal 8, Arithmetic exception. > > #0 0x00007faa644a7ff4 in ide_set_sector (s=0x7faa67549c18, sector_num=-1) > > at hw/ide/core.c:488 > > 488 cyl = sector_num / (s->heads * s->sectors); > > (gdb) bt > > That isn't qxl too, please open a new bug for it. Bug 947694 - ide triggers assert during iofuzz test
Try reproduce this bug, but autotest still running after 5 days, so kill process, will change the config file and re-test it after back. Thanks, Mazhang.
Test this bug with new qemu-kvm and kernel package. Host: qemu-img-1.5.3-46.el7.x86_64 qemu-kvm-common-1.5.3-46.el7.x86_64 qemu-kvm-1.5.3-46.el7.x86_64 qemu-kvm-debuginfo-1.5.3-46.el7.x86_64 ipxe-roms-qemu-20130517-3.gitc4bce43.el7.noarch qemu-kvm-tools-1.5.3-46.el7.x86_64 kernel-3.10.0-86.el7.x86_64 Guest: kernel-3.10.0-48.el7.x86_64 Steps: KVM iofuzz test: 1) Log into a guest 2) Enumerate all IO port ranges through /proc/ioports 3) On each port of the range: * Read it * Write 0 to it * Write a random value to a random port on a random order Result: First time test, got bz1046890 ,VM has quit while write a random value to 49160, as 1046890#c2 mentioned not a bug. Second time test, VM has quit while write a random value to 43328. Autotest log: 02/19 04:34:05 INFO | aexpect:0907| [qemu output] qemu: virtio_ioport_write: unexpected address 0x6 value 0xdf 02/19 04:38:39 INFO | aexpect:0907| [qemu output] qemu: virtio_ioport_write: unexpected address 0x1 value 0x66 02/19 04:57:45 INFO | aexpect:0907| [qemu output] qemu: virtio_ioport_write: unexpected address 0x2 value 0xa7 02/19 05:48:24 INFO | aexpect:0907| [qemu output] qemu: virtio_ioport_write: unexpected address 0x13 value 0xb4 02/19 06:15:46 INFO | aexpect:0907| [qemu output] qemu: virtio_ioport_write: unexpected address 0x6 value 0xf7 02/19 06:20:17 INFO | aexpect:0907| [qemu output] qemu: virtio_ioport_write: unexpected address 0x0 value 0xf8 02/19 06:20:45 INFO | aexpect:0907| [qemu output] qemu: virtio_ioport_write: unexpected address 0x1 value 0x6c 02/19 06:23:45 INFO | aexpect:0907| [qemu output] qemu: virtio_ioport_write: unexpected address 0x1 value 0x1b 02/19 06:37:47 INFO | aexpect:0907| [qemu output] qemu: virtio_ioport_write: unexpected address 0x0 value 0x30 02/19 06:51:35 INFO | aexpect:0907| [qemu output] qemu: virtio_ioport_write: unexpected address 0x13 value 0x2b 02/19 07:33:59 INFO | aexpect:0907| [qemu output] qemu: virtio_ioport_write: unexpected address 0x13 value 0x1 02/19 09:02:18 INFO | aexpect:0907| [qemu output] qemu: Guest moved used index from 6140 to 0 02/19 09:02:18 INFO | aexpect:0907| [qemu output] (Process terminated with status 1) 02/19 09:02:33 ERROR| virt:0155| Test failed: TestFail: VM has quit abnormally during write: [43328, 46] 02/19 09:02:34 INFO |env_proces:0251| Video creation failed for vm virt-tests-vm1: gstreamer-python library was not found 02/19 09:02:34 ERROR| test:0414| Exception escaping from test: Traceback (most recent call last): File "/root/staf-kvm-devel/autotest-devel/client/shared/test.py", line 411, in _exec _call_test_function(self.execute, *p_args, **p_dargs) File "/root/staf-kvm-devel/autotest-devel/client/shared/test.py", line 823, in _call_test_function return func(*args, **dargs) File "/root/staf-kvm-devel/autotest-devel/client/shared/test.py", line 291, in execute postprocess_profiled_run, args, dargs) File "/root/staf-kvm-devel/autotest-devel/client/shared/test.py", line 209, in _call_run_once *args, **dargs) File "/root/staf-kvm-devel/autotest-devel/client/shared/test.py", line 313, in run_once_profiling self.run_once(*args, **dargs) File "/root/staf-kvm-devel/autotest-devel/client/tests/virt/virt.py", line 139, in run_once run_func(self, params, env) File "/root/staf-kvm-devel/autotest-devel/client/tests/virt/tests/iofuzz.py", line 132, in run_iofuzz fuzz(session, inst) File "/root/staf-kvm-devel/autotest-devel/client/tests/virt/tests/iofuzz.py", line 88, in fuzz "%s: %s" % (op, operand)) TestFail: VM has quit abnormally during write: [43328, 46]
Created attachment 864949 [details] autotest log
Met this on qemu-kvm-1.5.3-47.el7.x86_64 kernel-3.10.0-88.el7.x86_64 (gdb) #0 qxl_set_mode (d=d@entry=0x7f6c558eb470, modenr=modenr@entry=247, loadvm=loadvm@entry=0) at /usr/src/debug/qemu-1.5.3/hw/display/qxl.c:1431 #1 0x00007f6c5380c95d in ioport_write (opaque=0x7f6c558eb470, addr=6, val=247, size=1) at /usr/src/debug/qemu-1.5.3/hw/display/qxl.c:1593 #2 0x00007f6c53838993 in access_with_adjusted_size (addr=addr@entry=6, value=value@entry=0x7f6c4525ab88, size=1, access_size_min=<optimized out>, access_size_max=<optimized out>, access=access@entry=0x7f6c53838eb0 <memory_region_write_accessor>, opaque=opaque@entry=0x7f6c558fcd08) at /usr/src/debug/qemu-1.5.3/memory.c:365 #3 0x00007f6c53839bcf in memory_region_iorange_write (iorange=<optimized out>, offset=6, width=1, data=247) at /usr/src/debug/qemu-1.5.3/memory.c:440 #4 0x00007f6c53837a52 in kvm_handle_io (count=1, size=1, direction=1, data=<optimized out>, port=49158) at /usr/src/debug/qemu-1.5.3/kvm-all.c:1519 #5 kvm_cpu_exec (env=env@entry=0x7f6c55882bf0) at /usr/src/debug/qemu-1.5.3/kvm-all.c:1671 #6 0x00007f6c537ec1c5 in qemu_kvm_cpu_thread_fn (arg=0x7f6c55882bf0) at /usr/src/debug/qemu-1.5.3/cpus.c:793 #7 0x00007f6c51647df3 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f6c4e35339d in clone () from /lib64/libc.so.6
http://patchwork.ozlabs.org/patch/321829/
Retest this bug on qemu-kvm-rhev-1.5.3-47.el7.x86_64, result see 947694#c13 .
upstream commit 9c70434f825fd0d2e89d1aa0f872159378d0aab3 now.
backport posted.
Fix included in qemu-kvm-1.5.3-51.el7
Created attachment 872542 [details] autotest debug info
Update qemu-kvm package, iofuzz without virtio devices, vm kernel crash, debug info see comment#35 . qemu-img-1.5.3-52.el7.x86_64 qemu-kvm-common-1.5.3-52.el7.x86_64 qemu-kvm-1.5.3-52.el7.x86_64 qemu-kvm-debuginfo-1.5.3-52.el7.x86_64 ipxe-roms-qemu-20130517-3.gitc4bce43.el7.noarch qemu-kvm-tools-1.5.3-52.el7.x86_64
VM kernel crash during iofuzz is fine.
This time iofuzz test (in comment#36) not hit qxl problem.
This request was resolved in Red Hat Enterprise Linux 7.0. Contact your manager or support representative in case you have further questions about the request.