RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 751937 - qxl triggers assert during iofuzz test
Summary: qxl triggers assert during iofuzz test
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm
Version: 7.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: 7.0
Assignee: Gerd Hoffmann
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2011-11-08 05:17 UTC by Xiaoqing Wei
Modified: 2015-04-28 17:17 UTC (History)
18 users (show)

Fixed In Version: qemu-kvm-1.5.3-51.el7
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-06-13 10:26:38 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
gdb bt full (13.37 KB, text/plain)
2011-11-09 01:58 UTC, Xiaoqing Wei 		no flags Details
autotest log (37.15 KB, text/plain)
2014-02-19 03:36 UTC, mazhang 		no flags Details
autotest debug info (13.47 KB, text/plain)
2014-03-10 02:26 UTC, mazhang 		no flags Details
View All

Description Xiaoqing Wei 2011-11-08 05:17:33 UTC 
Description of problem:
qemu-kvm core dumps during iofuzz test

Version-Release number of selected component (if applicable):
qemu-kvm-0.12.1.2-2.209.el6.x86_64

How reproducible:
1 / 5

Steps to Reproduce:


    KVM iofuzz test:
    1) Log into a guest
    2) Enumerate all IO port ranges through /proc/ioports
    3) On each port of the range:
        * Read it
        * Write 0 to it
        * Write a random value to a random port on a random order

cmd line:
/home/staf-kvm-devel/autotest-devel/client/tests/kvm/qemu -name 'vm1' -chardev socket,id=qmp_monitor_id_qmpmonitor1,path=/tmp/monitor-qmpmonitor1-20111105-012707-cwg7,server,nowait -mon chardev=qmp_monitor_id_qmpmonitor1,mode=control -chardev socket,id=serial_id_20111105-012707-cwg7,path=/tmp/serial-20111105-012707-cwg7,server,nowait \
\
-device isa-serial,chardev=serial_id_20111105-012707-cwg7 -drive file='RHEL-Server-6.1-64-virtio.qcow2',index=0,if=none,id=drive-virtio-disk1,media=disk,cache=none,format=qcow2,aio=native \
-device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk1,id=virtio-disk1 \
-device virtio-net-pci,netdev=idQxQNpv,mac=9a:f8:52:c1:72:27,id=ndev00idQxQNpv,bus=pci.0,addr=0x3 \
-netdev tap,id=idQxQNpv,vhost=on,fd=21 \
\
-m 2048 -smp 2,cores=1,threads=1,sockets=2 -cpu cpu64-rhel6,+sse2,+x2apic \
-spice port=8000,disable-ticketing -vga qxl -rtc base=utc,clock=host,driftfix=slew \
-boot order=cdn,once=c,menu=off -no-kvm-pit-reinjection -M rhel6.2.0 \
-device intel-hda -device hda-duplex -global qxl.debug=1 \
-global qxl.output=1 -usb -device usb-tablet -enable-kvm


Actual results:
qemu-kvm core dumps

Expected results:

qemu-kvm works fine
Additional info:
gdb output:

Program terminated with signal 11, Segmentation fault.
#0  bdrv_read (bs=0x0, sector_num=878539882841, buf=0x35c1800 "\377\377\377\377", nb_sectors=1) at block.c:958
958  
BlockDriver *drv = bs->drv;

(gdb) #0  bdrv_read (bs=0x0, sector_num=878539882841, buf=0x35c1800 "\377\377\377\377", nb_sectors=1) at block.c:958
#1  0x0000000000439a41 in ide_sector_read (s=0x2b6f8c0) at /usr/src/debug/qemu-kvm-0.12.1.2/hw/ide/core.c:386
#2  0x000000000042ca73 in kvm_handle_io (env=0x2b72b20) at /usr/src/debug/qemu-kvm-0.12.1.2/kvm-all.c:574
#3  kvm_run (env=0x2b72b20) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1036
#4  0x000000000042cc59 in kvm_cpu_exec (env=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1730
#5  0x000000000042da9e in kvm_main_loop_cpu (_env=0x2b72b20) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1991
#6  ap_main_loop (_env=0x2b72b20) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2041
#7  0x00000030148077f1 in start_thread (arg=0x7f690bd82700) at pthread_create.c:301
#8  0x00000030140e570d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
Comment 3 Xiaoqing Wei 2011-11-09 01:58:33 UTC 
Created attachment 532435 [details]
gdb bt full
Comment 15 Gerd Hoffmann 2013-03-20 14:45:29 UTC 
Please retest, qxl upstream got a bunch of robustness patches for this over time and RHEL-7 should be in pretty good shape.
Comment 16 Xiaoqing Wei 2013-03-21 03:13:27 UTC 
(In reply to comment #15)
> Please retest, qxl upstream got a bunch of robustness patches for this over
> time and RHEL-7 should be in pretty good shape.

Hi Gerd,

could you pls tell which version ?

is the 
spice-server-0.12.2-1.el7.x86_64
qemu-img-1.4.0-1.el7.x86_64
and use the RHEL-7(compose 0306.0, latest one)
contain the fix ?

or you mean clone the git://qemu.org and compile ?

Thx
Comment 17 Gerd Hoffmann 2013-03-21 07:36:10 UTC 
latest rhel-7 compose is fine, qemu 1.4 has the fixes.
Comment 18 Xiaoqing Wei 2013-03-29 04:58:30 UTC 
(In reply to comment #17)
> latest rhel-7 compose is fine, qemu 1.4 has the fixes.

Hi, 1.4 still fails: no knowing whether same BZ.

(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x00007fb3c20f5942 in access_with_adjusted_size (addr=addr@entry=0, value=value@entry=0x7fb3a7ffeae8, size=1, access_size_min=<optimized out>, 
    access_size_max=<optimized out>, access=access@entry=0x7fb3c20f5f60 <memory_region_write_accessor>, opaque=opaque@entry=0x7fb3c3426128)
    at /usr/src/debug/qemu-1.4.0/memory.c:364
#2  0x00007fb3c20f6fb7 in memory_region_iorange_write (iorange=<optimized out>, offset=0, width=1, data=0) at /usr/src/debug/qemu-1.4.0/memory.c:439
#3  0x00007fb3c20f3c22 in kvm_handle_io (count=1, size=1, direction=1, data=<optimized out>, port=45064) at /usr/src/debug/qemu-1.4.0/kvm-all.c:1426
#4  kvm_cpu_exec (env=env@entry=0x7fb3c33988d0) at /usr/src/debug/qemu-1.4.0/kvm-all.c:1581
#5  0x00007fb3c209d871 in qemu_kvm_cpu_thread_fn (arg=0x7fb3c33988d0) at /usr/src/debug/qemu-1.4.0/cpus.c:759
#6  0x00007fb3c030fd15 in start_thread (arg=0x7fb3a7fff700) at pthread_create.c:308
#7  0x00007fb3bca3f46d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:114
(gdb) 

Host: 
ipxe-bootimgs-20120328-2.gitaac9718.el7.noarch
qemu-kvm-1.4.0-1.el7.x86_64
spice-server-0.12.2-1.el7.x86_64
3.8.0-0.40.el7.x86_64
Comment 19 Xiaoqing Wei 2013-04-01 02:16:07 UTC 
yet another core dump, on same host.

Using host libthread_db library "/lib64/libthread_db.so.1".

warning: Skipping deprecated .gdb_index section in /usr/lib/debug/lib64/libkeyutils.so.1.4.debug.
Do "set use-deprecated-index-sections on" before the file is read
to use the section anyway.
Core was generated by `/home/staf-kvm-devel/autotest-devel/client/tests/kvm/qemu -S -name vm1 -nodefau'.
Program terminated with signal 8, Arithmetic exception.
#0  0x00007faa644a7ff4 in ide_set_sector (s=0x7faa67549c18, sector_num=-1) at hw/ide/core.c:488
488	        cyl = sector_num / (s->heads * s->sectors);
(gdb) bt
#0  0x00007faa644a7ff4 in ide_set_sector (s=0x7faa67549c18, sector_num=-1) at hw/ide/core.c:488
#1  0x00007faa644a965b in ide_exec_cmd (bus=<optimized out>, val=<optimized out>) at hw/ide/core.c:1266
#2  0x00007faa645e6f13 in memory_region_iorange_write (iorange=<optimized out>, offset=375, width=1, data=<optimized out>)
    at /usr/src/debug/qemu-1.4.0/memory.c:430
#3  0x00007faa645e3c22 in kvm_handle_io (count=1, size=1, direction=1, data=<optimized out>, port=375) at /usr/src/debug/qemu-1.4.0/kvm-all.c:1426
#4  kvm_cpu_exec (env=env@entry=0x7faa674db160) at /usr/src/debug/qemu-1.4.0/kvm-all.c:1581
#5  0x00007faa6458d871 in qemu_kvm_cpu_thread_fn (arg=0x7faa674db160) at /usr/src/debug/qemu-1.4.0/cpus.c:759
#6  0x00007faa627ffd15 in start_thread (arg=0x7faa55b16700) at pthread_create.c:308
#7  0x00007faa5ef2f46d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:114
(gdb) 2
Comment 20 Gerd Hoffmann 2013-04-02 12:07:29 UTC 
> #3  0x00007fb3c20f3c22 in kvm_handle_io (count=1, size=1, direction=1,
> data=<optimized out>, port=45064) at /usr/src/debug/qemu-1.4.0/kvm-all.c:1426

port=45064 (0xb008).  That isn't qxl but the piix-pm, please open a new bug.
Comment 21 Gerd Hoffmann 2013-04-02 12:09:17 UTC 
> Program terminated with signal 8, Arithmetic exception.
> #0  0x00007faa644a7ff4 in ide_set_sector (s=0x7faa67549c18, sector_num=-1)
> at hw/ide/core.c:488
> 488	        cyl = sector_num / (s->heads * s->sectors);
> (gdb) bt

That isn't qxl too, please open a new bug for it.
Comment 22 Xiaoqing Wei 2013-04-03 06:14:59 UTC 
(In reply to comment #20)
> > #3  0x00007fb3c20f3c22 in kvm_handle_io (count=1, size=1, direction=1,
> > data=<optimized out>, port=45064) at /usr/src/debug/qemu-1.4.0/kvm-all.c:1426
> 
> port=45064 (0xb008).  That isn't qxl but the piix-pm, please open a new bug.

Bug 947691 - piix-pm triggers assert during iofuzz test
Comment 23 Xiaoqing Wei 2013-04-03 06:15:24 UTC 
(In reply to comment #21)
> > Program terminated with signal 8, Arithmetic exception.
> > #0  0x00007faa644a7ff4 in ide_set_sector (s=0x7faa67549c18, sector_num=-1)
> > at hw/ide/core.c:488
> > 488	        cyl = sector_num / (s->heads * s->sectors);
> > (gdb) bt
> 
> That isn't qxl too, please open a new bug for it.

Bug 947694 - ide triggers assert during iofuzz test
Comment 26 mazhang 2014-01-26 05:36:38 UTC 
Try reproduce this bug, but autotest still running after 5 days, so kill process, will change the config file and re-test it after back.

Thanks,
Mazhang.
Comment 27 mazhang 2014-02-19 03:35:10 UTC 
Test this bug with new qemu-kvm and kernel package.

Host:
qemu-img-1.5.3-46.el7.x86_64
qemu-kvm-common-1.5.3-46.el7.x86_64
qemu-kvm-1.5.3-46.el7.x86_64
qemu-kvm-debuginfo-1.5.3-46.el7.x86_64
ipxe-roms-qemu-20130517-3.gitc4bce43.el7.noarch
qemu-kvm-tools-1.5.3-46.el7.x86_64
kernel-3.10.0-86.el7.x86_64

Guest:
kernel-3.10.0-48.el7.x86_64

Steps:
    KVM iofuzz test:
    1) Log into a guest
    2) Enumerate all IO port ranges through /proc/ioports
    3) On each port of the range:
        * Read it
        * Write 0 to it
        * Write a random value to a random port on a random order

Result:
First time test, got bz1046890 ,VM has quit while write a random value to 49160, as 1046890#c2 mentioned not a bug.

Second time test, VM has quit while write a random value to 43328.
Autotest log:
02/19 04:34:05 INFO |   aexpect:0907| [qemu output] qemu: virtio_ioport_write: unexpected address 0x6 value 0xdf
02/19 04:38:39 INFO |   aexpect:0907| [qemu output] qemu: virtio_ioport_write: unexpected address 0x1 value 0x66
02/19 04:57:45 INFO |   aexpect:0907| [qemu output] qemu: virtio_ioport_write: unexpected address 0x2 value 0xa7
02/19 05:48:24 INFO |   aexpect:0907| [qemu output] qemu: virtio_ioport_write: unexpected address 0x13 value 0xb4
02/19 06:15:46 INFO |   aexpect:0907| [qemu output] qemu: virtio_ioport_write: unexpected address 0x6 value 0xf7
02/19 06:20:17 INFO |   aexpect:0907| [qemu output] qemu: virtio_ioport_write: unexpected address 0x0 value 0xf8
02/19 06:20:45 INFO |   aexpect:0907| [qemu output] qemu: virtio_ioport_write: unexpected address 0x1 value 0x6c
02/19 06:23:45 INFO |   aexpect:0907| [qemu output] qemu: virtio_ioport_write: unexpected address 0x1 value 0x1b
02/19 06:37:47 INFO |   aexpect:0907| [qemu output] qemu: virtio_ioport_write: unexpected address 0x0 value 0x30
02/19 06:51:35 INFO |   aexpect:0907| [qemu output] qemu: virtio_ioport_write: unexpected address 0x13 value 0x2b
02/19 07:33:59 INFO |   aexpect:0907| [qemu output] qemu: virtio_ioport_write: unexpected address 0x13 value 0x1
02/19 09:02:18 INFO |   aexpect:0907| [qemu output] qemu: Guest moved used index from 6140 to 0
02/19 09:02:18 INFO |   aexpect:0907| [qemu output] (Process terminated with status 1)
02/19 09:02:33 ERROR|      virt:0155| Test failed: TestFail: VM has quit abnormally during write: [43328, 46]
02/19 09:02:34 INFO |env_proces:0251| Video creation failed for vm virt-tests-vm1: gstreamer-python library was not found
02/19 09:02:34 ERROR|      test:0414| Exception escaping from test:
Traceback (most recent call last):
  File "/root/staf-kvm-devel/autotest-devel/client/shared/test.py", line 411, in _exec
    _call_test_function(self.execute, *p_args, **p_dargs)
  File "/root/staf-kvm-devel/autotest-devel/client/shared/test.py", line 823, in _call_test_function
    return func(*args, **dargs)
  File "/root/staf-kvm-devel/autotest-devel/client/shared/test.py", line 291, in execute
    postprocess_profiled_run, args, dargs)
  File "/root/staf-kvm-devel/autotest-devel/client/shared/test.py", line 209, in _call_run_once
    *args, **dargs)
  File "/root/staf-kvm-devel/autotest-devel/client/shared/test.py", line 313, in run_once_profiling
    self.run_once(*args, **dargs)
  File "/root/staf-kvm-devel/autotest-devel/client/tests/virt/virt.py", line 139, in run_once
    run_func(self, params, env)
  File "/root/staf-kvm-devel/autotest-devel/client/tests/virt/tests/iofuzz.py", line 132, in run_iofuzz
    fuzz(session, inst)
  File "/root/staf-kvm-devel/autotest-devel/client/tests/virt/tests/iofuzz.py", line 88, in fuzz
    "%s: %s" % (op, operand))
TestFail: VM has quit abnormally during write: [43328, 46]
Comment 28 mazhang 2014-02-19 03:36:10 UTC 
Created attachment 864949 [details]
autotest log
Comment 29 xhan 2014-02-19 10:05:28 UTC 
Met this on 
qemu-kvm-1.5.3-47.el7.x86_64
kernel-3.10.0-88.el7.x86_64

(gdb) 
#0  qxl_set_mode (d=d@entry=0x7f6c558eb470, modenr=modenr@entry=247, loadvm=loadvm@entry=0) at /usr/src/debug/qemu-1.5.3/hw/display/qxl.c:1431
#1  0x00007f6c5380c95d in ioport_write (opaque=0x7f6c558eb470, addr=6, val=247, size=1) at /usr/src/debug/qemu-1.5.3/hw/display/qxl.c:1593
#2  0x00007f6c53838993 in access_with_adjusted_size (addr=addr@entry=6, value=value@entry=0x7f6c4525ab88, size=1, access_size_min=<optimized out>, 
    access_size_max=<optimized out>, access=access@entry=0x7f6c53838eb0 <memory_region_write_accessor>, opaque=opaque@entry=0x7f6c558fcd08)
    at /usr/src/debug/qemu-1.5.3/memory.c:365
#3  0x00007f6c53839bcf in memory_region_iorange_write (iorange=<optimized out>, offset=6, width=1, data=247) at /usr/src/debug/qemu-1.5.3/memory.c:440
#4  0x00007f6c53837a52 in kvm_handle_io (count=1, size=1, direction=1, data=<optimized out>, port=49158) at /usr/src/debug/qemu-1.5.3/kvm-all.c:1519
#5  kvm_cpu_exec (env=env@entry=0x7f6c55882bf0) at /usr/src/debug/qemu-1.5.3/kvm-all.c:1671
#6  0x00007f6c537ec1c5 in qemu_kvm_cpu_thread_fn (arg=0x7f6c55882bf0) at /usr/src/debug/qemu-1.5.3/cpus.c:793
#7  0x00007f6c51647df3 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f6c4e35339d in clone () from /lib64/libc.so.6
Comment 30 Gerd Hoffmann 2014-02-19 10:46:24 UTC 
http://patchwork.ozlabs.org/patch/321829/
Comment 31 mazhang 2014-02-26 07:07:49 UTC 
Retest this bug on qemu-kvm-rhev-1.5.3-47.el7.x86_64, result see 947694#c13 .
Comment 32 Gerd Hoffmann 2014-02-26 11:33:20 UTC 
upstream commit 9c70434f825fd0d2e89d1aa0f872159378d0aab3 now.
Comment 33 Gerd Hoffmann 2014-02-26 11:54:55 UTC 
backport posted.
Comment 34 Miroslav Rezanina 2014-03-05 10:32:39 UTC 
Fix included in qemu-kvm-1.5.3-51.el7
Comment 35 mazhang 2014-03-10 02:26:25 UTC 
Created attachment 872542 [details]
autotest debug info
Comment 36 mazhang 2014-03-10 02:27:47 UTC 
Update qemu-kvm package, iofuzz without virtio devices, vm kernel crash, debug info see comment#35 .

qemu-img-1.5.3-52.el7.x86_64
qemu-kvm-common-1.5.3-52.el7.x86_64
qemu-kvm-1.5.3-52.el7.x86_64
qemu-kvm-debuginfo-1.5.3-52.el7.x86_64
ipxe-roms-qemu-20130517-3.gitc4bce43.el7.noarch
qemu-kvm-tools-1.5.3-52.el7.x86_64
Comment 37 Paolo Bonzini 2014-03-10 12:46:18 UTC 
VM kernel crash during iofuzz is fine.
Comment 38 mazhang 2014-03-11 02:33:09 UTC 
This time iofuzz test (in comment#36) not hit qxl problem.
Comment 41 Ludek Smid 2014-06-13 10:26:38 UTC 
This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.
Note You need to log in before you can comment on or make changes to this bug.