Bug 988768

Summary: QEMU core dumped when throttling windows guest system disk with bps_rd/iops_rd/bps/iops
Product: Red Hat Enterprise Linux 6 Reporter: Sibiao Luo <sluo>
Component: qemu-kvmAssignee: Virtualization Maintenance <virt-maint>
Status: CLOSED DUPLICATE QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 6.5CC: acathrow, bsarathy, chayang, famz, flang, juzhang, michen, mkenneth, qzhang, stefanha, virt-maint, xfu
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-07-31 09:37:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Sibiao Luo 2013-07-26 10:28:29 UTC
Description of problem:
QEMU core dumped with floating point exception when throttling system disk as bps_rd/iops_rd=1024000. If i remove it from cli, it can boot up successfully.
BTW, this issue very similar to bug 988658, but bt log not the same, so separate it to different issue.

Version-Release number of selected component (if applicable):
host info:
RHEL6.5-20130721.n.0
2.6.32-400.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.379.el6.x86_64
seabios-0.6.1.2-28.el6.x86_64
guest info:
win8.1 32bit guest
virtio-win-prewhql-0.1-65

How reproducible:
almost every time

Steps to Reproduce:
1.boot guest with throttling system disk as bps_rd/iops_rd=1024000.
e.g:...-device virtserialport,chardev=channel2,name=com.redhat.rhevm.vdsm.2,bus=virtio-serial0.0,id=port2,nr=2 -drive file=/dev/vg/system-disk.raw,if=none,id=drive-system-disk,format=raw,cache=none,aio=native,werror=stop,rerror=stop,serial="QEMU-DISK1",bps_rd=1024000 -device virtio-scsi-pci,bus=pci.0,addr=0x4,id=scsi0 -device scsi-hd,drive=drive-system-disk,id=system-disk,bootindex=1
2.
3.

Actual results:
after step 1, qemu will core dumped and fail to boot up guest.
(qemu) Floating point exception (core dumped)

Core was generated by `/usr/libexec/qemu-kvm -S -M rhel6.5.0 -cpu SandyBridge -enable-kvm -m 4096 -smp'.
Program terminated with signal 8, Arithmetic exception.
#0  0x00007f344315c6ce in bdrv_exceed_bps_limits (bs=0x7f3445337010, is_write=false, nb_sectors=1)
    at /usr/src/debug/qemu-kvm-0.12.1.2/block.c:3621
3621	    bytes_limit = bps_limit * slice_time;

(gdb) bt
#0  0x00007f344315c6ce in bdrv_exceed_bps_limits (bs=0x7f3445337010, is_write=false, nb_sectors=1)
    at /usr/src/debug/qemu-kvm-0.12.1.2/block.c:3621
#1  bdrv_exceed_io_limits (bs=0x7f3445337010, is_write=false, nb_sectors=1)
    at /usr/src/debug/qemu-kvm-0.12.1.2/block.c:3732
#2  bdrv_io_limits_intercept (bs=0x7f3445337010, is_write=false, nb_sectors=1)
    at /usr/src/debug/qemu-kvm-0.12.1.2/block.c:181
#3  0x00007f344315ccbd in bdrv_co_do_readv (bs=0x7f3445337010, sector_num=0, nb_sectors=1, qiov=0x7f3434000ab8, 
    flags=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/block.c:2136
#4  0x00007f344315cfe3 in bdrv_co_do_rw (opaque=0x7f3434000b00) at /usr/src/debug/qemu-kvm-0.12.1.2/block.c:3880
#5  0x00007f344316333b in coroutine_trampoline (i0=<value optimized out>, i1=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/coroutine-ucontext.c:129
#6  0x00007f3440a6ab70 in ?? () from /lib64/libc.so.6
#7  0x00007fff6d01b8e0 in ?? ()
#8  0x0000000000000000 in ?? ()
(gdb) bt full
#0  0x00007f344315c6ce in bdrv_exceed_bps_limits (bs=0x7f3445337010, is_write=false, nb_sectors=1)
    at /usr/src/debug/qemu-kvm-0.12.1.2/block.c:3621
        bps_limit = 1024000
        bytes_limit = <value optimized out>
        wait_time = <value optimized out>
        bytes_base = <value optimized out>
#1  bdrv_exceed_io_limits (bs=0x7f3445337010, is_write=false, nb_sectors=1)
    at /usr/src/debug/qemu-kvm-0.12.1.2/block.c:3732
        max_wait = <value optimized out>
        iops_wait = 0
        elapsed_time = 0
        bps_ret = <value optimized out>
        now = <value optimized out>
        bps_wait = 0
#2  bdrv_io_limits_intercept (bs=0x7f3445337010, is_write=false, nb_sectors=1)
    at /usr/src/debug/qemu-kvm-0.12.1.2/block.c:181
        wait_time = <value optimized out>
#3  0x00007f344315ccbd in bdrv_co_do_readv (bs=0x7f3445337010, sector_num=0, nb_sectors=1, qiov=0x7f3434000ab8, 
    flags=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/block.c:2136
        drv = 0x7f344361cc60
        req = {bs = 0x7f3400000000, sector_num = 139862255804176, nb_sectors = 1160998928, is_write = 52, list = {
            le_next = 0x28, le_prev = 0x28}, co = 0x7f3440db5e80, wait_queue = {entries = {tqh_first = 0x7fff6d01c078, 
              tqh_last = 0x7f3445339c00}}}
        ret = <value optimized out>
#4  0x00007f344315cfe3 in bdrv_co_do_rw (opaque=0x7f3434000b00) at /usr/src/debug/qemu-kvm-0.12.1.2/block.c:3880
        acb = 0x7f3434000b00
        bs = <value optimized out>
#5  0x00007f344316333b in coroutine_trampoline (i0=<value optimized out>, i1=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/coroutine-ucontext.c:129
        self = 0x7f34453b80c0
        co = 0x7f34453b80c0
#6  0x00007f3440a6ab70 in ?? () from /lib64/libc.so.6
No symbol table info available.
#7  0x00007fff6d01b8e0 in ?? ()
No symbol table info available.
#8  0x0000000000000000 in ?? ()
No symbol table info available.
(gdb) 

Expected results:
it should boot up successfully without any problem.

Additional info:
# /usr/libexec/qemu-kvm -S -M rhel6.5.0 -cpu SandyBridge -enable-kvm -m 4096 -smp 4,sockets=2,cores=2,threads=1 -no-kvm-pit-reinjection -name sluo -uuid 43425b70-86e5-4664-bf2c-3b76699a8aec -rtc base=localtime,clock=host,driftfix=slew -device virtio-serial-pci,id=virtio-serial0,max_ports=16,vectors=0,bus=pci.0,addr=0x3 -chardev socket,id=channel1,path=/tmp/helloworld1,server,nowait -device virtserialport,chardev=channel1,name=com.redhat.rhevm.vdsm.1,bus=virtio-serial0.0,id=port1,nr=1 -chardev socket,id=channel2,path=/tmp/helloworld2,server,nowait -device virtserialport,chardev=channel2,name=com.redhat.rhevm.vdsm.2,bus=virtio-serial0.0,id=port2,nr=2 -drive file=/dev/vg/system-disk.raw,if=none,id=drive-system-disk,format=raw,cache=none,aio=native,werror=stop,rerror=stop,serial="QEMU-DISK1",bps_rd=1024000 -device virtio-scsi-pci,bus=pci.0,addr=0x4,id=scsi0 -device scsi-hd,drive=drive-system-disk,id=system-disk,bootindex=1 -device virtio-balloon-pci,id=ballooning,bus=pci.0,addr=0x5 -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -netdev tap,id=hostnet0,vhost=off,script=/etc/qemu-ifup -device virtio-net-pci,netdev=hostnet0,id=virtio-net-pci0,mac=2C:41:38:B6:32:21,bus=pci.0,addr=0x6,bootindex=2 -drive file=/dev/vg/my-data-disk.raw,if=none,id=drive-data-disk,format=raw,media=disk,cache=none,aio=native,werror=stop,rerror=stop,serial="QEMU-DISK2",bps_rd=1024000 -device virtio-scsi-pci,bus=pci.0,addr=0x7,id=scsi1 -device scsi-hd,drive=drive-data-disk,id=data-disk,bootindex=3,bus=scsi1.0 -k en-us -boot menu=on -vnc :1 -spice port=5931,disable-ticketing -qmp tcp:0:4444,server,nowait -monitor stdio

Comment 1 Sibiao Luo 2013-07-26 10:30:10 UTC
My host cpu info:
# lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                8
On-line CPU(s) list:   0-7
Thread(s) per core:    2
Core(s) per socket:    4
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 42
Stepping:              7
CPU MHz:               1600.000
BogoMIPS:              6784.20
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              8192K
NUMA node0 CPU(s):     0-7


processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 42
model name	: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz
stepping	: 7
cpu MHz		: 1600.000
cache size	: 8192 KB
physical id	: 0
siblings	: 8
core id		: 3
cpu cores	: 4
apicid		: 7
initial apicid	: 7
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm ida arat epb xsaveopt pln pts dts tpr_shadow vnmi flexpriority ept vpid
bogomips	: 6784.20
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management:

Comment 2 Sibiao Luo 2013-07-29 02:30:46 UTC
If only specify the bps=1024000 or iops=1024000 for windows guest, it also hit the same issue.

e.g:...-drive file=/dev/vg/system-disk.raw,if=none,id=drive-system-disk,format=raw,cache=none,aio=native,werror=stop,rerror=stop,serial="QEMU-DISK1",bps=1024000 -device virtio-scsi-pci,bus=pci.0,addr=0x4,id=scsi0 -device scsi-hd,drive=drive-system-disk,id=system-disk,bootindex=1
(qemu) Floating point exception (core dumped)
# gdb /usr/libexec/qemu-kvm /var/core.10154-0-0-8-1375064573-dhcp-11-229.nay.redhat.com-qemu-kvm
(gdb) bt
#0  0x00007f2cc7a40436 in bdrv_exceed_bps_limits (bs=0x7f2cc8e59010, is_write=false, nb_sectors=1)
    at /usr/src/debug/qemu-kvm-0.12.1.2/block.c:3621
#1  bdrv_exceed_io_limits (bs=0x7f2cc8e59010, is_write=false, nb_sectors=1)
    at /usr/src/debug/qemu-kvm-0.12.1.2/block.c:3732
#2  bdrv_io_limits_intercept (bs=0x7f2cc8e59010, is_write=false, nb_sectors=1)
    at /usr/src/debug/qemu-kvm-0.12.1.2/block.c:181
#3  0x00007f2cc7a40cbd in bdrv_co_do_readv (bs=0x7f2cc8e59010, sector_num=0, nb_sectors=1, qiov=0x7f2cb8000ab8, 
    flags=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/block.c:2136
#4  0x00007f2cc7a40fe3 in bdrv_co_do_rw (opaque=0x7f2cb8000b00) at /usr/src/debug/qemu-kvm-0.12.1.2/block.c:3880
#5  0x00007f2cc7a4733b in coroutine_trampoline (i0=<value optimized out>, i1=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/coroutine-ucontext.c:129
#6  0x00007f2cc534eb70 in ?? () from /lib64/libc.so.6
#7  0x00007fffcb47ee10 in ?? ()
#8  0x0000000000000000 in ?? ()
(gdb) 

e.g:...-drive file=/dev/vg/system-disk.raw,if=none,id=drive-system-disk,format=raw,cache=none,aio=native,werror=stop,rerror=stop,serial="QEMU-DISK1",iops=1024000 -device virtio-scsi-pci,bus=pci.0,addr=0x4,id=scsi0 -device scsi-hd,drive=drive-system-disk,id=system-disk,bootindex=1
(gdb) bt
#0  0x00007f8a3fe9150b in bdrv_exceed_iops_limits (bs=0x7f8a41efa010, is_write=false, nb_sectors=1)
    at /usr/src/debug/qemu-kvm-0.12.1.2/block.c:3683
#1  bdrv_exceed_io_limits (bs=0x7f8a41efa010, is_write=false, nb_sectors=1)
    at /usr/src/debug/qemu-kvm-0.12.1.2/block.c:3734
#2  bdrv_io_limits_intercept (bs=0x7f8a41efa010, is_write=false, nb_sectors=1)
    at /usr/src/debug/qemu-kvm-0.12.1.2/block.c:181
#3  0x00007f8a3fe91cbd in bdrv_co_do_readv (bs=0x7f8a41efa010, sector_num=0, nb_sectors=1, qiov=0x7f8a30000ab8, 
    flags=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/block.c:2136
#4  0x00007f8a3fe91fe3 in bdrv_co_do_rw (opaque=0x7f8a30000b00) at /usr/src/debug/qemu-kvm-0.12.1.2/block.c:3880
#5  0x00007f8a3fe9833b in coroutine_trampoline (i0=<value optimized out>, i1=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/coroutine-ucontext.c:129
#6  0x00007f8a3d79fb70 in ?? () from /lib64/libc.so.6
#7  0x00007fff2f0bb0c0 in ?? ()
#8  0x0000000000000000 in ?? ()
(gdb)

Comment 3 Sibiao Luo 2013-07-29 03:03:43 UTC
1.Tried it with rhel6.5 guest that did not hit this issue.

2.Tried the the same image on rhel7 host that did not hit this issue.

Base on above two points, this issue probability not virtio-win driver issue and only trigger with windows guest.

Best Regards,
sluo.

Comment 5 Fam Zheng 2013-07-31 09:37:28 UTC
Looked into the problem, this is indeed the same thing with 988658, for more information please see:

https://bugzilla.redhat.com/show_bug.cgi?id=988658#c7

*** This bug has been marked as a duplicate of bug 988658 ***