Bug 1089632
| Summary: | guest will call trace and reboot automatically when parallel I/O to the virtio-scsi disks with multi-queue enabled and vcpu overcommit | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Sibiao Luo <sluo> | ||||
| Component: | qemu-kvm | Assignee: | Fam Zheng <famz> | ||||
| Status: | CLOSED WORKSFORME | QA Contact: | Virtualization Bugs <virt-bugs> | ||||
| Severity: | medium | Docs Contact: | |||||
| Priority: | medium | ||||||
| Version: | 7.0 | CC: | hhuang, juzhang, knoel, pbonzini, rbalakri, sluo, virt-maint | ||||
| Target Milestone: | rc | ||||||
| Target Release: | --- | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2014-11-13 09:23:17 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
KERNEL: /usr/lib/debug/lib/modules/3.10.0-121.el7.x86_64/vmlinux
DUMPFILE: /var/crash/127.0.0.1-2014.04.21-12:55:42/vmcore [PARTIAL DUMP]
CPUS: 4
DATE: Mon Apr 21 04:54:58 2014
UPTIME: 00:18:46
LOAD AVERAGE: 6.64, 2.95, 1.39
TASKS: 252
NODENAME: dhcp-66-83-239.nay.redhat.com
RELEASE: 3.10.0-121.el7.x86_64
VERSION: #1 SMP Tue Apr 8 10:48:19 EDT 2014
MACHINE: x86_64 (2700 Mhz)
MEMORY: 4 GB
PANIC: "kernel BUG at block/blk-core.c:2294!"
PID: 2964
COMMAND: "dd"
TASK: ffff8800aec038e0 [THREAD_INFO: ffff8800a491a000]
CPU: 2
STATE: TASK_RUNNING (PANIC)
crash> bt
PID: 2964 TASK: ffff8800aec038e0 CPU: 2 COMMAND: "dd"
#0 [ffff88013fd03970] machine_kexec at ffffffff81041181
#1 [ffff88013fd039c8] crash_kexec at ffffffff810cf0e2
#2 [ffff88013fd03a98] oops_end at ffffffff815f4c48
#3 [ffff88013fd03ac0] die at ffffffff8101632b
#4 [ffff88013fd03af0] do_trap at ffffffff815f4320
#5 [ffff88013fd03b40] do_invalid_op at ffffffff81013134
#6 [ffff88013fd03bf0] invalid_op at ffffffff815fdf5e
[exception RIP: blk_start_request+69]
RIP: ffffffff81293c05 RSP: ffff88013fd03ca8 RFLAGS: 00010002
RAX: 0000000000000001 RBX: ffff8800ac844c00 RCX: 0000000000000001
RDX: 00000000000da112 RSI: 00000000bda12f68 RDI: ffff88013ff86080
RBP: ffff88013fd03cb0 R8: 00000000000da112 R9: ffff88013fd1bbd0
R10: ffffffff813dd4cb R11: ffffea0000d8c580 R12: ffff880036260000
R13: ffff88013941c000 R14: ffff8800ac844c00 R15: ffff88013941c000
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#7 [ffff88013fd03ca0] blk_start_request at ffffffff81293bd2
#8 [ffff88013fd03cb8] scsi_request_fn at ffffffff813e6df8
#9 [ffff88013fd03d18] __blk_run_queue at ffffffff81290823
#10 [ffff88013fd03d30] blk_run_queue at ffffffff81290896
#11 [ffff88013fd03d50] scsi_run_queue at ffffffff813e6955
#12 [ffff88013fd03db8] scsi_next_command at ffffffff813e7627
#13 [ffff88013fd03de0] scsi_io_completion at ffffffff813e7905
#14 [ffff88013fd03e48] scsi_finish_command at ffffffff813dcd43
#15 [ffff88013fd03e78] scsi_softirq_done at ffffffff813e759f
#16 [ffff88013fd03ea8] blk_done_softirq at ffffffff81299b80
#17 [ffff88013fd03ee8] __do_softirq at ffffffff81067047
#18 [ffff88013fd03f58] call_softirq at ffffffff815fe15c
#19 [ffff88013fd03f70] do_softirq at ffffffff81014d25
#20 [ffff88013fd03f88] irq_exit at ffffffff810673e5
#21 [ffff88013fd03fa0] smp_call_function_single_interrupt at ffffffff81038a85
#22 [ffff88013fd03fb0] call_function_single_interrupt at ffffffff815fd91d
--- <IRQ stack> ---
#23 [ffff8800a491b848] call_function_single_interrupt at ffffffff815fd91d
[exception RIP: __slab_alloc+1032]
RIP: ffffffff815e90ac RSP: ffff8800a491b8f0 RFLAGS: 00000246
RAX: ffff8800773fe100 RBX: 0000000000000000 RCX: ffff88013fd17360
RDX: ffff8800773ff000 RSI: 0000000000000000 RDI: 0000000000000246
RBP: ffff8800a491b9b8 R8: ffffea0001dcff80 R9: 0000000000000000
R10: 0000000000000013 R11: ffffffffffffffdc R12: ffff8800a491b8a8
R13: ffff8800aec038e0 R14: ffff8800aec038e0 R15: 0000000000000000
ORIG_RAX: ffffffffffffff04 CS: 0010 SS: 0018
#24 [ffff8800a491b9c0] kmem_cache_alloc at ffffffff81194eb3
#25 [ffff8800a491ba00] mempool_alloc_slab at ffffffff811442f5
#26 [ffff8800a491ba10] mempool_alloc at ffffffff81144439
#27 [ffff8800a491ba90] bio_alloc_bioset at ffffffff811e74ee
#28 [ffff8800a491bae0] _submit_bh at ffffffff811e2427
#29 [ffff8800a491bb10] block_read_full_page at ffffffff811e4f77
#30 [ffff8800a491bbe8] blkdev_readpage at ffffffff811e8ee8
#31 [ffff8800a491bbf8] __do_page_cache_readahead at ffffffff8114df3a
#32 [ffff8800a491bcb8] ondemand_readahead at ffffffff8114e0d6
#33 [ffff8800a491bd00] page_cache_async_readahead at ffffffff8114e270
#34 [ffff8800a491bd38] generic_file_aio_read at ffffffff81143210
#35 [ffff8800a491be08] blkdev_aio_read at ffffffff811e938c
#36 [ffff8800a491be30] do_sync_read at ffffffff811af1dd
#37 [ffff8800a491bf08] vfs_read at ffffffff811af8bc
#38 [ffff8800a491bf38] sys_read at ffffffff811b03e8
#39 [ffff8800a491bf80] system_call_fastpath at ffffffff815fc819
RIP: 00007f1f446d9c10 RSP: 00007fffada2b8b8 RFLAGS: 00000202
RAX: 0000000000000000 RBX: ffffffff815fc819 RCX: 00007f1f446d9c70
RDX: 0000000000100000 RSI: 00007f1f44abc000 RDI: 0000000000000000
RBP: 0000000000100000 R8: 00000000ffffffff R9: 0000000000000000
R10: 00007fffada2b670 R11: 0000000000000246 R12: 00007fffada2d80d
R13: 0000000000000000 R14: 00007f1f44abc000 R15: 000000000060e640
ORIG_RAX: 0000000000000000 CS: 0033 SS: 002b
crash>
My qemu-kvm command line: # /usr/libexec/qemu-kvm -M pc -cpu host -enable-kvm -m 4096 -smp 4,sockets=2,cores=2,threads=1 -no-kvm-pit-reinjection -usb -device usb-tablet,id=input0 -name sluo_test -uuid 990ea161-6b67-47b2-b803-19fb01d30d30 -rtc base=localtime,clock=host,driftfix=slew -device virtio-serial-pci,id=virtio-serial0,max_ports=16,vectors=0,bus=pci.0,addr=0x3 -chardev socket,id=channel1,path=/tmp/helloworld1,server,nowait -device virtserialport,chardev=channel1,name=com.redhat.rhevm.vdsm,bus=virtio-serial0.0,id=port1 -chardev socket,id=channel2,path=/tmp/helloworld2,server,nowait -device virtserialport,chardev=channel2,name=com.redhat.rhevm.vdsm,bus=virtio-serial0.0,id=port2 -drive file=gluster://10.66.83.171/sluo_volume/RHEL-7.0-20140409.0_Server_x86_64.qcow2,if=none,id=drive-system-disk,format=qcow2,cache=none,aio=native,werror=stop,rerror=stop -device virtio-scsi-pci,bus=pci.0,addr=0x4,id=scsi0 -device scsi-hd,drive=drive-system-disk,id=system-disk,bus=scsi0.0,bootindex=1 -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup,queues=4 -device virtio-net-pci,netdev=hostnet0,id=virtio-net-pci0,mac=00:01:02:B6:40:21,bus=pci.0,addr=0x5,vectors=9,mq=on -device virtio-balloon-pci,id=ballooning,bus=pci.0,addr=0x6 -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -serial unix:/tmp/ttyS0,server,nowait -k en-us -boot menu=on -qmp tcp:0:4444,server,nowait -spice disable-ticketing,port=5931 -monitor stdio -device virtio-scsi-pci,id=scsi1,bus=pci.0,addr=0x7,num_queues=4,ioeventfd=on -drive file=gluster://10.66.83.171/sluo_volume/data-disk1.qcow2,if=none,id=drive-data-disk1,cache=none,format=qcow2,aio=native,werror=stop,rerror=stop -device scsi-hd,ver=luo1,wwn=0x5000c50015ea71a1,serial=balabala1,drive=drive-data-disk1,bus=scsi1.0,id=data-disk1 -drive file=gluster://10.66.83.171/sluo_volume/data-disk2.qcow2,if=none,id=drive-data-disk2,cache=none,format=qcow2,aio=native,werror=stop,rerror=stop -device scsi-hd,ver=luo2,wwn=0x5000c50015ea71a2,serial=balabala2,drive=drive-data-disk2,bus=scsi1.0,id=data-disk2 -drive file=gluster://10.66.83.171/sluo_volume/data-disk3.qcow2,if=none,id=drive-data-disk3,cache=none,format=qcow2,aio=native,werror=stop,rerror=stop -device scsi-hd,ver=luo3,wwn=0x5000c50015ea71a3,serial=balabala3,drive=drive-data-disk3,bus=scsi1.0,id=data-disk3 -drive file=gluster://10.66.83.171/sluo_volume/data-disk4.qcow2,if=none,id=drive-data-disk4,cache=none,format=qcow2,aio=native,werror=stop,rerror=stop -device scsi-hd,ver=luo4,wwn=0x5000c50015ea71a4,serial=balabala4,drive=drive-data-disk4,bus=scsi1.0,id=data-disk4 My host CPU info: # lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 2 On-line CPU(s) list: 0,1 Thread(s) per core: 1 Core(s) per socket: 2 Socket(s): 1 NUMA node(s): 1 Vendor ID: AuthenticAMD CPU family: 15 Model: 107 Model name: AMD Athlon(tm) 64 X2 Dual Core Processor 5200+ Stepping: 2 CPU MHz: 2700.000 BogoMIPS: 5411.30 Virtualization: AMD-V L1d cache: 64K L1i cache: 64K L2 cache: 512K NUMA node0 CPU(s): 0,1 Created attachment 888034 [details]
guest-vmcore-dmesg.txt
Same testing with my SandyBridge intel host which did not hit such issue. host info: # uname -r && rpm -q qemu-kvm-rhev 3.10.0-121.el7.x86_64 qemu-kvm-rhev-1.5.3-60.el7ev.x86_64 guest info: 3.10.0-121.el7.x86_64 # lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 2 Core(s) per socket: 4 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 42 Model name: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz Stepping: 7 CPU MHz: 2014.367 BogoMIPS: 6784.88 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 8192K NUMA node0 CPU(s): 0-7 guest ]# dd if=/dev/sdb of=/dev/null bs=1M & dd if=/dev/sdc of=/dev/null bs=1M & dd if=/dev/sdd of=/dev/null bs=1M & dd if=/dev/sde of=/dev/null bs=1M & [1] 2828 [2] 2829 [3] 2830 [4] 2831 guest ]# 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 45.0015 s, 239 MB/s 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 45.3035 s, 237 MB/s 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 45.3793 s, 237 MB/s 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 45.4058 s, 236 MB/s [1] Done dd if=/dev/sdb of=/dev/null bs=1M [2] Done dd if=/dev/sdc of=/dev/null bs=1M [3]- Done dd if=/dev/sdd of=/dev/null bs=1M [4]+ Done dd if=/dev/sde of=/dev/null bs=1M Best Regards, sluo Hi Sluo, Can you have a testing according to comment6 and update our testing result? Best Regards, Junyi My testing as following: +-------------------+-----------------------+-----------------------+ | Host info | host CPU / guest vCPU | Results | +-------------------+-----------------------+-----------------------+ | AMD Opteron_G2 | CPU / vCPU = 2 / 2 | call trace and reboot | | +-----------------------+-----------------------+ | Server | CPU / vCPU = 2 / 4 | call trace and reboot | +-------------------+-----------------------+-----------------------+ | AMD Opteron_G3 | CPU / vCPU = 4 / 4 | work well | | +-----------------------+-----------------------+ | Desktop | CPU / vCPU = 4 / 8 | call trace and reboot | +-------------------+-----------------------+-----------------------+ | Intel SandyBridge | CPU / vCPU = 8 / 8 | work well | | +-----------------------+-----------------------+ | Server | CPU / vCPU = 8 / 16 | call trace and reboot | +-------------------+-----------------------+-----------------------+ Best Regards, sluo |
Description of problem: Boot up a KVM guest appending 4 virtio-scsi data disk with multi-queue enabled, then do parallel dd (I/O) to the 4 multi-queue virtio-scsi data disk, then guest will call trace and reboot automatically. Version-Release number of selected component (if applicable): host info: 3.10.0-121.el7.x86_64 qemu-kvm-rhev-1.5.3-60.el7ev.x86_64 seabios-1.7.2.2-12.el7.x86_64 guest info: 3.10.0-121.el7.x86_64 How reproducible: 100% Steps to Reproduce: 1.Boot up a KVM guest appending 4 virtio-scsi data disk with multi-queue enabled. e.g:# /usr/libexec/qemu-kvm -M pc -cpu host -enable-kvm -m 4096 -smp 4,sockets=2,cores=2,threads=1 -no-kvm-pit-reinjection...-device virtio-scsi-pci,id=scsi1,bus=pci.0,addr=0x7,num_queues=4 -drive file=gluster://10.66.83.171/sluo_volume/data-disk1.qcow2,if=none,id=drive-data-disk1,cache=none,format=qcow2,aio=native,werror=stop,rerror=stop -device scsi-hd,ver=luo1,wwn=0x5000c50015ea71a1,serial=balabala1,drive=drive-data-disk1,bus=scsi1.0,id=data-disk1 -drive file=gluster://10.66.83.171/sluo_volume/data-disk2.qcow2,if=none,id=drive-data-disk2,cache=none,format=qcow2,aio=native,werror=stop,rerror=stop -device scsi-hd,ver=luo2,wwn=0x5000c50015ea71a2,serial=balabala2,drive=drive-data-disk2,bus=scsi1.0,id=data-disk2 -drive file=gluster://10.66.83.171/sluo_volume/data-disk3.qcow2,if=none,id=drive-data-disk3,cache=none,format=qcow2,aio=native,werror=stop,rerror=stop -device scsi-hd,ver=luo3,wwn=0x5000c50015ea71a3,serial=balabala3,drive=drive-data-disk3,bus=scsi1.0,id=data-disk3 -drive file=gluster://10.66.83.171/sluo_volume/data-disk4.qcow2,if=none,id=drive-data-disk4,cache=none,format=qcow2,aio=native,werror=stop,rerror=stop -device scsi-hd,ver=luo4,wwn=0x5000c50015ea71a4,serial=balabala4,drive=drive-data-disk4,bus=scsi1.0,id=data-disk4 2.do parallel dd (I/O) to the 4 multi-queue virtio-scsi data disk. # dd if=/dev/sdb of=/dev/null bs=1M & dd if=/dev/sdc of=/dev/null bs=1M & dd if=/dev/sdd of=/dev/null bs=1M & dd if=/dev/sde of=/dev/null bs=1M & Actual results: during step 2, guest will call trace and reboot automatically, I will attach the vmcore-dmesg.txt later. # nc -U /tmp/ttyS0 [ 1040.891827] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen [ 1041.231820] ata2.00: cmd a0/00:00:00:08:00/00:00:00:00:00/a0 tag 0 pio 16392 in [ 1041.231820] res 40/00:02:00:08:00/00:00:00:00:00/a0 Emask 0x4 (timeout) [ 1044.357966] ata2.00: status: { DRDY } [ 1126.367923] ------------[ cut here ]------------ [ 1126.367923] kernel BUG at block/blk-core.c:2294! [ 1126.367923] invalid opcode: 0000 [#1] SMP [ 1126.367923] Modules linked in: ip6t_rpfilter ip6t_REJECT ipt_REJECT xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw iptable_filter ip_tables sg pcspkr i2c_piix4 virtio_console serio_raw virtio_balloon mperf nfsd auth_rpcgss nfs_acl lockd sunrpc uinput xfs libcrc32c sr_mod cdrom sd_mod ata_generic crc_t10dif pata_acpi crct10dif_common cirrus syscopyarea sysfillrect sysimgblt drm_kms_helper virtio_scsi virtio_net ttm ata_piix virtio_pci drm virtio_ring libata floppy i2c_core virtio dm_mirror dm_region_hash dm_log dm_mod [ 1126.367923] CPU: 2 PID: 2964 Comm: dd Not tainted 3.10.0-121.el7.x86_64 #1 [ 1126.367923] Hardware name: Red Hat KVM, BIOS Bochs 01/01/2011 [ 1126.367923] task: ffff8800aec038e0 ti: ffff8800a491a000 task.ti: ffff8800a491a000 [ 1126.367923] RIP: 0010:[<ffffffff81293c05>] [<ffffffff81293c05>] blk_start_request+0x45/0x50 [ 1126.367923] RSP: 0018:ffff88013fd03ca8 EFLAGS: 00010002 [ 1126.367923] RAX: 0000000000000001 RBX: ffff8800ac844c00 RCX: 0000000000000001 [ 1126.367923] RDX: 00000000000da112 RSI: 00000000bda12f68 RDI: ffff88013ff86080 [ 1126.367923] RBP: ffff88013fd03cb0 R08: 00000000000da112 R09: ffff88013fd1bbd0 [ 1126.367923] R10: ffffffff813dd4cb R11: ffffea0000d8c580 R12: ffff880036260000 [ 1126.367923] R13: ffff88013941c000 R14: ffff8800ac844c00 R15: ffff88013941c000 [ 1126.367923] FS: 00007f1f44bbe740(0000) GS:ffff88013fd00000(0000) knlGS:0000000000000000 [ 1126.367923] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1126.367923] CR2: 0000000003066002 CR3: 00000000360f0000 CR4: 00000000000006e0 [ 1126.367923] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1126.367923] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 1126.367923] Stack: [ 1126.367923] ffff880036b84800 ffff88013fd03d10 ffffffff813e6df8 000000000000000b [ 1126.367923] ffffffff8199a790 ffffffff8199a760 ffff880036b84948 ffff880036b84848 [ 1126.367923] ffff880036260000 0000000000000246 ffff88013fd03d70 0000000000000246 [ 1126.367923] Call Trace: [ 1126.367923] <IRQ> [ 1126.367923] [<ffffffff813e6df8>] scsi_request_fn+0x128/0x5a0 [ 1126.367923] [<ffffffff81290823>] __blk_run_queue+0x33/0x40 [ 1126.367923] [<ffffffff81290896>] blk_run_queue+0x26/0x40 [ 1126.367923] [<ffffffff813e6955>] scsi_run_queue+0x135/0x2b0 [ 1126.367923] [<ffffffff813e7627>] scsi_next_command+0x37/0x50 [ 1126.367923] [<ffffffff813e7905>] scsi_io_completion+0x265/0x650 [ 1126.367923] [<ffffffff813dcd43>] scsi_finish_command+0xb3/0x110 [ 1126.367923] [<ffffffff813e759f>] scsi_softirq_done+0x12f/0x160 [ 1126.367923] [<ffffffff81299b80>] blk_done_softirq+0x90/0xc0 [ 1126.367923] [<ffffffff81067047>] __do_softirq+0xf7/0x290 [ 1126.367923] [<ffffffff815fe15c>] call_softirq+0x1c/0x30 [ 1126.367923] [<ffffffff81014d25>] do_softirq+0x55/0x90 [ 1126.367923] [<ffffffff810673e5>] irq_exit+0x115/0x120 [ 1126.367923] [<ffffffff81038a85>] smp_call_function_single_interrupt+0x35/0x40 [ 1126.367923] [<ffffffff815fd91d>] call_function_single_interrupt+0x6d/0x80 [ 1126.367923] <EOI> [ 1126.367923] [<ffffffff815e90ac>] ? __slab_alloc+0x408/0x482 [ 1126.367923] [<ffffffff811442f5>] ? mempool_alloc_slab+0x15/0x20 [ 1126.367923] [<ffffffff81194eb3>] kmem_cache_alloc+0x193/0x1d0 [ 1126.367923] [<ffffffff81293a3b>] ? blk_attempt_plug_merge+0xdb/0xf0 [ 1126.367923] [<ffffffff811442f5>] mempool_alloc_slab+0x15/0x20 [ 1126.367923] [<ffffffff81144439>] mempool_alloc+0x69/0x170 [ 1126.367923] [<ffffffff81194eda>] ? kmem_cache_alloc+0x1ba/0x1d0 [ 1126.367923] [<ffffffff811e1f01>] ? alloc_buffer_head+0x21/0x70 [ 1126.367923] [<ffffffff811e74ee>] bio_alloc_bioset+0x1be/0x2e0 [ 1126.367923] [<ffffffff811e2427>] _submit_bh+0x77/0x220 [ 1126.367923] [<ffffffff811e4f77>] block_read_full_page+0x217/0x350 [ 1126.367923] [<ffffffff811e87b0>] ? I_BDEV+0x10/0x10 [ 1126.367923] [<ffffffff811e8ee8>] blkdev_readpage+0x18/0x20 [ 1126.367923] [<ffffffff8114df3a>] __do_page_cache_readahead+0x23a/0x250 [ 1126.367923] [<ffffffff8114e0d6>] ondemand_readahead+0x126/0x240 [ 1126.367923] [<ffffffff812c4f80>] ? copy_user_generic_string+0x30/0x40 [ 1126.367923] [<ffffffff8114e270>] page_cache_async_readahead+0x80/0xa0 [ 1126.367923] [<ffffffff8114136e>] ? __find_get_page+0x1e/0xa0 [ 1126.367923] [<ffffffff81143210>] generic_file_aio_read+0x540/0x750 [ 1126.367923] [<ffffffff811e938c>] blkdev_aio_read+0x4c/0x70 [ 1126.367923] [<ffffffff815f7934>] ? __do_page_fault+0x204/0x540 [ 1126.367923] [<ffffffff811af1dd>] do_sync_read+0x8d/0xd0 [ 1126.367923] [<ffffffff811af8bc>] vfs_read+0x9c/0x170 [ 1126.367923] [<ffffffff811b03e8>] SyS_read+0x58/0xb0 [ 1126.367923] [<ffffffff815fc819>] system_call_fastpath+0x16/0x1b [ 1126.367923] Code: 00 48 8b 83 78 01 00 00 48 85 c0 75 13 48 8b 43 58 a8 01 75 16 48 89 df e8 c9 64 00 00 5b 5d c3 8b 50 64 89 90 3c 01 00 00 eb e2 <0f> 0b 66 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 41 [ 1126.367923] RIP [<ffffffff81293c05>] blk_start_request+0x45/0x50 [ 1126.367923] RSP <ffff88013fd03ca8> [ 1.362017] Failed to access perfctr msr (MSR c0010001 is ffffffffffffffff) systemd-fsck[272]: fsck: error 2 (No such file or directory) while executing fsck.ext2 for /dev/disk/by-uuid/5ace7911-65d8-4cdd-a2ad-61d3e687895f kdump: dump target is /dev/mapper/rhel_dhcp--66--83--239-root kdump: saving to /sysroot//var/crash/127.0.0.1-2014.04.21-12:55:42/ kdump: saving vmcore-dmesg.txt kdump: saving vmcore-dmesg.txt complete kdump: saving vmcore Copying data : [100.0 %] \ kdump: saving vmcore complete Rebooting. [ 28.430993] Restarting system. ... Expected results: It should no any guest call trace when do parallel dd (I/O) to the 4 multi-queue virtio-scsi data disks. Additional info: