642243 – guest kernel panic when transfering file from host to guest during migration

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 642243 - guest kernel panic when transfering file from host to guest during migration

Summary: guest kernel panic when transfering file from host to guest during migration

Keywords:
Status:	CLOSED DUPLICATE of bug 647367
Alias:	None
Product:	Red Hat Enterprise Linux 6
Classification:	Red Hat
Component:	qemu-kvm
Sub Component:
Version:	6.0
Hardware:	All
OS:	Linux
Priority:	high
Severity:	high
Target Milestone:	beta
Target Release:	6.1
Assignee:	Michael S. Tsirkin
QA Contact:	Virtualization Bugs
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	Rhel6KvmTier1 658437
TreeView+	depends on / blocked

Reported:	2010-10-12 12:46 UTC by Amos Kong
Modified:	2015-05-25 00:06 UTC (History)
CC List:	10 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Clones:	658437 (view as bug list)
Environment:
Last Closed:	2011-01-31 22:23:34 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Description Amos Kong 2010-10-12 12:46:39 UTC

Description of problem:
When I test jasonwang's autotest patch(http://patchwork.test.kernel.org/patch/2582/), guest kernel panic when transfering file from host to guest during migration. It can be reproduced manually, this bug only exists when using virtio nic.

Version-Release number of selected component (if applicable):
# uname -a
Linux t173 2.6.32-72.el6.x86_64 #1 SMP Mon Sep 13 13:29:07 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux
# rpm -qa |grep qemu
qemu-kvm-tools-0.12.1.2-2.113.el6_0.1.x86_64
gpxe-roms-qemu-0.9.7-6.3.el6.noarch
qemu-img-0.12.1.2-2.113.el6_0.1.x86_64
qemu-kvm-debuginfo-0.12.1.2-2.113.el6_0.1.x86_64
qemu-kvm-0.12.1.2-2.113.el6_0.1.x86_64


How reproducible:
always

Steps to Reproduce:
1. boot up src and dest guests (using virtio nic)
2. transfer file from host to guest
host) # scp a.qcow2 $guest_ip:~
3. do internal migration
src qemu) migrate -d tcp:0:4444

Actual results:
guest kernel panic

Expected results:
migration and transfer completed.

Additional info:

1. commandline:
# ps aux |grep qemu
root     12884 15.6 56.8 2595932 2158236 pts/2 Sl+  20:32   1:25 /home/devel/autotest-devel/client/tests/kvm/qemu -name vm1 -serial stdio -drive file=/home/devel/autotest-devel/client/tests/kvm/images/RHEL-Server-6.0-64-virtio.qcow2,index=0,if=none,id=drive-virtio-disk1,media=disk,cache=none,snapshot=on,boot=on,format=qcow2,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk1,id=virtio-disk1 -device virtio-net-pci,netdev=idOCJBNn,id=ndev00idOCJBNn,mac=02:A9:7C:6C:1a:90,bus=pci.0,addr=0x3 -netdev tap,id=idOCJBNn,ifname=virtio_0_8000,script=/home/devel/autotest-devel/client/tests/kvm/scripts/qemu-ifup-switch,downscript=no -m 2048 -smp 2 -cpu cpu64-rhel6,+sse2,+x2apic -vnc :0 -spice port=8000,disable-ticketing -vga qxl -rtc base=utc,clock=host,driftfix=none -M rhel6.0.0 -usbdevice tablet -no-kvm-pit-reinjection -enable-kvm
root     12929 82.5 31.9 2574588 1212016 pts/9 Sl+  20:32   7:26 /home/devel/autotest-devel/client/tests/kvm/qemu -name vm2 -serial stdio -drive file=/home/devel/autotest-devel/client/tests/kvm/images/RHEL-Server-6.0-64-virtio.qcow2,index=0,if=none,id=drive-virtio-disk1,media=disk,cache=none,snapshot=on,boot=on,format=qcow2,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk1,id=virtio-disk1 -device virtio-net-pci,netdev=idOCJBNn,id=ndev00idOCJBNn,mac=02:A9:7C:6C:1a:90,bus=pci.0,addr=0x3 -netdev tap,id=idOCJBNn,ifname=virtio_0_8001,script=/home/devel/autotest-devel/client/tests/kvm/scripts/qemu-ifup-switch,downscript=no -m 2048 -smp 2 -cpu cpu64-rhel6,+sse2,+x2apic -vnc :1 -spice port=8001,disable-ticketing -vga qxl -rtc base=utc,clock=host,driftfix=none -M rhel6.0.0 -usbdevice tablet -no-kvm-pit-reinjection -enable-kvm -incoming tcp:0:4444

2. panic msg: 

------------[ cut here ]------------
WARNING: at lib/list_debug.c:26 __list_add+0x6d/0xa0() (Not tainted)
Hardware name: KVM
list_add corruption. next->prev should be prev (ffff88000001a720), but was ffffea0001af2710. (next=ffffea0001aede38).
Modules linked in: virtio_balloon ipv6 dm_mirror dm_region_hash dm_log ppdev parport_pchandle_dev_input: detach
handle_dev_input: attach
create_cairo_context: using cairo canvas
 parport virtio_net i2c_piix4 i2c_core sg ext4 mbcache jbd2 sr_mod cdrom virtio_blk pata_acpi ata_generic ata_piix virtio_pci virtio_ring virtio dm_mod [last unloaded: speedstep_lib]
Pid: 303, comm: kdmflush Not tainted 2.6.32-70.el6.x86_64 #1
Call Trace:
 <IRQ>  [<ffffffff8106b857>] warn_slowpath_common+0x87/0xc0
 [<ffffffff8106b946>] warn_slowpath_fmt+0x46/0x50
 [<ffffffff81268acd>] __list_add+0x6d/0xa0
 [<ffffffff8111dd24>] free_pcppages_bulk+0x104/0x390
 [<ffffffff8111eb88>] free_hot_cold_page+0x1b8/0x220
 [<ffffffff8102ea69>] ? native_smp_send_reschedule+0x49/0x60
 [<ffffffff8111eccf>] free_hot_page+0x2f/0x60
 [<ffffffff8111ed60>] __free_pages+0x60/0xa0
 [<ffffffff8111ede9>] free_pages+0x49/0x50
 [<ffffffff811558cb>] kmem_freepages+0xeb/0x130
 [<ffffffff81157a23>] slab_destroy+0x33/0x90
 [<ffffffff81157ae0>] free_block+0x60/0x180
 [<ffffffff81157988>] kmem_cache_free+0x248/0x2b0
 [<ffffffff8110e617>] mempool_free_slab+0x17/0x20
 [<ffffffff8110e6d5>] mempool_free+0x95/0xa0
 [<ffffffff811a2084>] bio_free+0x64/0x70
 [<ffffffff811a20a5>] bio_fs_destructor+0x15/0x20
 [<ffffffff811a0e4b>] bio_put+0x2b/0x40
 [<ffffffff8119cb17>] end_bio_bh_io_sync+0x37/0x60
 [<ffffffff811a0d3d>] bio_endio+0x1d/0x40
 [<ffffffffa0001544>] dec_pending+0xe4/0x1e0 [dm_mod]
 [<ffffffffa00018df>] clone_endio+0x9f/0xd0 [dm_mod]
 [<ffffffff811a0d3d>] bio_endio+0x1d/0x40
 [<ffffffff8123f7fb>] req_bio_endio+0xab/0x110
 [<ffffffff8124083f>] blk_update_request+0xff/0x440
 [<ffffffff81240ba7>] blk_update_bidi_request+0x27/0x80
 [<ffffffff812418fe>] __blk_end_request_all+0x2e/0x60
 [<ffffffffa004c125>] blk_done+0x35/0xe0 [virtio_blk]
 [<ffffffffa002519c>] vring_interrupt+0x3c/0xd0 [virtio_ring]
 [<ffffffff810d8740>] handle_IRQ_event+0x60/0x170
 [<ffffffff810353d7>] ? native_apic_msr_write+0x37/0x40
 [<ffffffff810dae26>] handle_edge_irq+0xc6/0x160
 [<ffffffff81015fb9>] handle_irq+0x49/0xa0
 [<ffffffff814cf90c>] do_IRQ+0x6c/0xf0
 [<ffffffff81013ad3>] ret_from_intr+0x0/0x11
 [<ffffffff81073b8b>] ? __do_softirq+0x6b/0x1e0
 [<ffffffff81095a50>] ? hrtimer_interrupt+0x140/0x250
 [<ffffffff810142cc>] ? call_softirq+0x1c/0x30
 [<ffffffff81015f35>] ? do_softirq+0x65/0xa0
 [<ffffffff810739d5>] ? irq_exit+0x85/0x90
 [<ffffffff814cfa01>] ? smp_apic_timer_interrupt+0x71/0x9c
 [<ffffffff81013c93>] ? apic_timer_interrupt+0x13/0x20
 <EOI>  [<ffffffff81242a14>] ? __make_request+0x164/0x4c0
 [<ffffffff81240db2>] ? generic_make_request+0x1b2/0x4f0
 [<ffffffff811a1e7b>] ? bio_alloc_bioset+0x5b/0xf0
 [<ffffffffa00016ed>] ? __map_bio+0xad/0x130 [dm_mod]
 [<ffffffffa000341c>] ? __split_and_process_bio+0x46c/0x630 [dm_mod]
 [<ffffffffa00039e1>] ? dm_wq_work+0x161/0x200 [dm_mod]
handle_dev_input: detach
handle_dev_input: attach
create_cairo_context: using cairo canvas
 [<ffffffffa0003880>] ? dm_wq_work+0x0/0x200 [dm_mod]
 [<ffffffff8108c610>] ? worker_thread+0x170/0x2a0
 [<ffffffff81091ca0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff8108c4a0>] ? worker_thread+0x0/0x2a0
 [<ffffffff81091936>] ? kthread+0x96/0xa0
 [<ffffffff810141ca>] ? child_rip+0xa/0x20
 [<ffffffff810918a0>] ? kthread+0x0/0xa0
 [<ffffffff810141c0>] ? child_rip+0x0/0x20
---[ end trace d6b4961a23d7c2ee ]---
------------[ cut here ]------------
WARNING: at lib/list_debug.c:48 list_del+0x6e/0xa0() (Tainted: G        W  ---------------- )
Hardware name: KVM
list_del corruption. prev->next should be ffffea0001aa87d8, but was ffffea0001ae85a0
Modules linked in: virtio_balloon ipv6 dm_mirror dm_region_hash dm_log ppdev parport_pc parport virtio_net i2c_piix4 i2c_core sg ext4 mbcache jbd2 sr_mod cdrom virtio_blk pata_acpi ata_generic ata_piix virtio_pci virtio_ring virtio dm_mod [last unloaded: speedstep_lib]
Pid: 303, comm: kdmflush Tainted: G        W  ----------------  2.6.32-70.el6.x86_64 #1
Call Trace:
 <IRQ>  [<ffffffff8106b857>] warn_slowpath_common+0x87/0xc0
 [<ffffffff8106b946>] warn_slowpath_fmt+0x46/0x50
 [<ffffffff81268a2e>] list_del+0x6e/0xa0
 [<ffffffff8111de3e>] free_pcppages_bulk+0x21e/0x390
 [<ffffffff8111eb88>] free_hot_cold_page+0x1b8/0x220
 [<ffffffff8102ea69>] ? native_smp_send_reschedule+0x49/0x60
 [<ffffffff8111eccf>] free_hot_page+0x2f/0x60
 [<ffffffff8111ed60>] __free_pages+0x60/0xa0
 [<ffffffff8111ede9>] free_pages+0x49/0x50
 [<ffffffff811558cb>] kmem_freepages+0xeb/0x130
 [<ffffffff81157a23>] slab_destroy+0x33/0x90
 [<ffffffff81157ae0>] free_block+0x60/0x180
 [<ffffffff81157988>] kmem_cache_free+0x248/0x2b0
 [<ffffffff8110e617>] mempool_free_slab+0x17/0x20
 [<ffffffff8110e6d5>] mempool_free+0x95/0xa0
 [<ffffffff811a2084>] bio_free+0x64/0x70
 [<ffffffff811a20a5>] bio_fs_destructor+0x15/0x20
 [<ffffffff811a0e4b>] bio_put+0x2b/0x40
 [<ffffffff8119cb17>] end_bio_bh_io_sync+0x37/0x60
 [<ffffffff811a0d3d>] bio_endio+0x1d/0x40
 [<ffffffffa0001544>] dec_pending+0xe4/0x1e0 [dm_mod]
 [<ffffffffa00018df>] clone_endio+0x9f/0xd0 [dm_mod]
 [<ffffffff811a0d3d>] bio_endio+0x1d/0x40
 [<ffffffff8123f7fb>] req_bio_endio+0xab/0x110
 [<ffffffff8124083f>] blk_update_request+0xff/0x440
 [<ffffffff81240ba7>] blk_update_bidi_request+0x27/0x80
 [<ffffffff812418fe>] __blk_end_request_all+0x2e/0x60
 [<ffffffffa004c125>] blk_done+0x35/0xe0 [virtio_blk]
 [<ffffffffa002519c>] vring_interrupt+0x3c/0xd0 [virtio_ring]
 [<ffffffff810d8740>] handle_IRQ_event+0x60/0x170
 [<ffffffff810353d7>] ? native_apic_msr_write+0x37/0x40
 [<ffffffff810dae26>] handle_edge_irq+0xc6/0x160
 [<ffffffff81015fb9>] handle_irq+0x49/0xa0
 [<ffffffff814cf90c>] do_IRQ+0x6c/0xf0
 [<ffffffff81013ad3>] ret_from_intr+0x0/0x11
 [<ffffffff81073b8b>] ? __do_softirq+0x6b/0x1e0
 [<ffffffff81095a50>] ? hrtimer_interrupt+0x140/0x250
 [<ffffffff810142cc>] ? call_softirq+0x1c/0x30
 [<ffffffff81015f35>] ? do_softirq+0x65/0xa0
 [<ffffffff810739d5>] ? irq_exit+0x85/0x90
 [<ffffffff814cfa01>] ? smp_apic_timer_interrupt+0x71/0x9c
 [<ffffffff81013c93>] ? apic_timer_interrupt+0x13/0x20
 <EOI>  [<ffffffff81242a14>] ? __make_request+0x164/0x4c0
 [<ffffffff81240db2>] ? generic_make_request+0x1b2/0x4f0
 [<ffffffff811a1e7b>] ? bio_alloc_bioset+0x5b/0xf0
 [<ffffffffa00016ed>] ? __map_bio+0xad/0x130 [dm_mod]
 [<ffffffffa000341c>] ? __split_and_process_bio+0x46c/0x630 [dm_mod]
 [<ffffffffa00039e1>] ? dm_wq_work+0x161/0x200 [dm_mod]
 [<ffffffffa0003880>] ? dm_wq_work+0x0/0x200 [dm_mod]
 [<ffffffff8108c610>] ? worker_thread+0x170/0x2a0
 [<ffffffff81091ca0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff8108c4a0>] ? worker_thread+0x0/0x2a0
 [<ffffffff81091936>] ? kthread+0x96/0xa0
 [<ffffffff810141ca>] ? child_rip+0xa/0x20
 [<ffffffff810918a0>] ? kthread+0x0/0xa0
 [<ffffffff810141c0>] ? child_rip+0x0/0x20
---[ end trace d6b4961a23d7c2ef ]---
------------[ cut here ]------------
WARNING: at lib/list_debug.c:48 list_del+0x6e/0xa0() (Tainted: G        W  ---------------- )
Hardware name: KVM
list_del corruption. prev->next should be ffffea0001aa8768, but was ffffea0000c0e1b8
Modules linked in: virtio_balloon ipv6 dm_mirror dm_region_hash dm_log ppdev parport_pc parport virtio_net i2c_piix4 i2c_core sg ext4 mbcache jbd2 sr_mod cdrom virtio_blk pata_acpi ata_generic ata_piix virtio_pci virtio_ring virtio dm_mod [last unloaded: speedstep_lib]
Pid: 303, comm: kdmflush Tainted: G        W  ----------------  2.6.32-70.el6.x86_64 #1
Call Trace:
 <IRQ>  [<ffffffff8106b857>] warn_slowpath_common+0x87/0xc0
 [<ffffffff8106b946>] warn_slowpath_fmt+0x46/0x50
 [<ffffffff81268a2e>] list_del+0x6e/0xa0
 [<ffffffff8111de3e>] free_pcppages_bulk+0x21e/0x390
 [<ffffffff8111eb88>] free_hot_cold_page+0x1b8/0x220
 [<ffffffff8102ea69>] ? native_smp_send_reschedule+0x49/0x60
 [<ffffffff8111eccf>] free_hot_page+0x2f/0x60
 [<ffffffff8111ed60>] __free_pages+0x60/0xa0
 [<ffffffff8111ede9>] free_pages+0x49/0x50
 [<ffffffff811558cb>] kmem_freepages+0xeb/0x130
 [<ffffffff81157a23>] slab_destroy+0x33/0x90
 [<ffffffff81157ae0>] free_block+0x60/0x180
 [<ffffffff81157988>] kmem_cache_free+0x248/0x2b0
 [<ffffffff8110e617>] mempool_free_slab+0x17/0x20
 [<ffffffff8110e6d5>] mempool_free+0x95/0xa0
 [<ffffffff811a2084>] bio_free+0x64/0x70
 [<ffffffff811a20a5>] bio_fs_destructor+0x15/0x20
 [<ffffffff811a0e4b>] bio_put+0x2b/0x40
 [<ffffffff8119cb17>] end_bio_bh_io_sync+0x37/0x60
 [<ffffffff811a0d3d>] bio_endio+0x1d/0x40
 [<ffffffffa0001544>] dec_pending+0xe4/0x1e0 [dm_mod]
 [<ffffffffa00018df>] clone_endio+0x9f/0xd0 [dm_mod]
 [<ffffffff811a0d3d>] bio_endio+0x1d/0x40
 [<ffffffff8123f7fb>] req_bio_endio+0xab/0x110
 [<ffffffff8124083f>] blk_update_request+0xff/0x440
 [<ffffffff81240ba7>] blk_update_bidi_request+0x27/0x80
 [<ffffffff812418fe>] __blk_end_request_all+0x2e/0x60
 [<ffffffffa004c125>] blk_done+0x35/0xe0 [virtio_blk]
 [<ffffffffa002519c>] vring_interrupt+0x3c/0xd0 [virtio_ring]
 [<ffffffff810d8740>] handle_IRQ_event+0x60/0x170
 [<ffffffff810353d7>] ? native_apic_msr_write+0x37/0x40
 [<ffffffff810dae26>] handle_edge_irq+0xc6/0x160
 [<ffffffff81015fb9>] handle_irq+0x49/0xa0
 [<ffffffff814cf90c>] do_IRQ+0x6c/0xf0
 [<ffffffff81013ad3>] ret_from_intr+0x0/0x11
 [<ffffffff81073b8b>] ? __do_softirq+0x6b/0x1e0
 [<ffffffff81095a50>] ? hrtimer_interrupt+0x140/0x250
 [<ffffffff810142cc>] ? call_softirq+0x1c/0x30
 [<ffffffff81015f35>] ? do_softirq+0x65/0xa0
 [<ffffffff810739d5>] ? irq_exit+0x85/0x90
 [<ffffffff814cfa01>] ? smp_apic_timer_interrupt+0x71/0x9c
 [<ffffffff81013c93>] ? apic_timer_interrupt+0x13/0x20
 <EOI>  [<ffffffff81242a14>] ? __make_request+0x164/0x4c0
 [<ffffffff81240db2>] ? generic_make_request+0x1b2/0x4f0
 [<ffffffff811a1e7b>] ? bio_alloc_bioset+0x5b/0xf0
 [<ffffffffa00016ed>] ? __map_bio+0xad/0x130 [dm_mod]
 [<ffffffffa000341c>] ? __split_and_process_bio+0x46c/0x630 [dm_mod]
 [<ffffffffa00039e1>] ? dm_wq_work+0x161/0x200 [dm_mod]
 [<ffffffffa0003880>] ? dm_wq_work+0x0/0x200 [dm_mod]
 [<ffffffff8108c610>] ? worker_thread+0x170/0x2a0
 [<ffffffff81091ca0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff8108c4a0>] ? worker_thread+0x0/0x2a0
 [<ffffffff81091936>] ? kthread+0x96/0xa0
 [<ffffffff810141ca>] ? child_rip+0xa/0x20
 [<ffffffff810918a0>] ? kthread+0x0/0xa0
 [<ffffffff810141c0>] ? child_rip+0x0/0x20
---[ end trace d6b4961a23d7c2f0 ]---
------------[ cut here ]------------
WARNING: at lib/list_debug.c:26 __list_add+0x6d/0xa0() (Tainted: G        W  ---------------- )
Hardware name: KVM
list_add corruption. next->prev should be prev (ffff88000001a7d0), but was ffff88007b13c000. (next=ffffea0001aec548).
Modules linked in: virtio_balloon ipv6 dm_mirror dm_region_hash dm_log ppdev parport_pc parport virtio_net i2c_piix4 i2c_core sg ext4 mbcache jbd2 sr_mod cdrom virtio_blk pata_acpi ata_generic ata_piix virtio_pci virtio_ring virtio dm_mod [last unloaded: speedstep_lib]
Pid: 303, comm: kdmflush Tainted: G        W  ----------------  2.6.32-70.el6.x86_64 #1
Call Trace:
 <IRQ>  [<ffffffff8106b857>] warn_slowpath_common+0x87/0xc0
 [<ffffffff8106b946>] warn_slowpath_fmt+0x46/0x50
 [<ffffffff81268acd>] __list_add+0x6d/0xa0
 [<ffffffff8111dd24>] free_pcppages_bulk+0x104/0x390
 [<ffffffff8111eb88>] free_hot_cold_page+0x1b8/0x220
 [<ffffffff8102ea69>] ? native_smp_send_reschedule+0x49/0x60
 [<ffffffff8111eccf>] free_hot_page+0x2f/0x60
 [<ffffffff8111ed60>] __free_pages+0x60/0xa0
 [<ffffffff8111ede9>] free_pages+0x49/0x50
 [<ffffffff811558cb>] kmem_freepages+0xeb/0x130
 [<ffffffff81157a23>] slab_destroy+0x33/0x90
 [<ffffffff81157ae0>] free_block+0x60/0x180
 [<ffffffff81157988>] kmem_cache_free+0x248/0x2b0
 [<ffffffff8110e617>] mempool_free_slab+0x17/0x20
 [<ffffffff8110e6d5>] mempool_free+0x95/0xa0
 [<ffffffff811a2084>] bio_free+0x64/0x70
 [<ffffffff811a20a5>] bio_fs_destructor+0x15/0x20
 [<ffffffff811a0e4b>] bio_put+0x2b/0x40
 [<ffffffff8119cb17>] end_bio_bh_io_sync+0x37/0x60
 [<ffffffff811a0d3d>] bio_endio+0x1d/0x40
 [<ffffffffa0001544>] dec_pending+0xe4/0x1e0 [dm_mod]
 [<ffffffffa00018df>] clone_endio+0x9f/0xd0 [dm_mod]
 [<ffffffff811a0d3d>] bio_endio+0x1d/0x40
 [<ffffffff8123f7fb>] req_bio_endio+0xab/0x110
 [<ffffffff8124083f>] blk_update_request+0xff/0x440
 [<ffffffff81240ba7>] blk_update_bidi_request+0x27/0x80
 [<ffffffff812418fe>] __blk_end_request_all+0x2e/0x60
 [<ffffffffa004c125>] blk_done+0x35/0xe0 [virtio_blk]
 [<ffffffffa002519c>] vring_interrupt+0x3c/0xd0 [virtio_ring]
 [<ffffffff810d8740>] handle_IRQ_event+0x60/0x170
 [<ffffffff810353d7>] ? native_apic_msr_write+0x37/0x40
 [<ffffffff810dae26>] handle_edge_irq+0xc6/0x160
 [<ffffffff81015fb9>] handle_irq+0x49/0xa0
 [<ffffffff814cf90c>] do_IRQ+0x6c/0xf0
 [<ffffffff81013ad3>] ret_from_intr+0x0/0x11
 [<ffffffff81073b8b>] ? __do_softirq+0x6b/0x1e0
 [<ffffffff81095a50>] ? hrtimer_interrupt+0x140/0x250
 [<ffffffff810142cc>] ? call_softirq+0x1c/0x30
 [<ffffffff81015f35>] ? do_softirq+0x65/0xa0
 [<ffffffff810739d5>] ? irq_exit+0x85/0x90
 [<ffffffff814cfa01>] ? smp_apic_timer_interrupt+0x71/0x9c
 [<ffffffff81013c93>] ? apic_timer_interrupt+0x13/0x20
 <EOI>  [<ffffffff81242a14>] ? __make_request+0x164/0x4c0
 [<ffffffff81240db2>] ? generic_make_request+0x1b2/0x4f0
 [<ffffffff811a1e7b>] ? bio_alloc_bioset+0x5b/0xf0
 [<ffffffffa00016ed>] ? __map_bio+0xad/0x130 [dm_mod]
 [<ffffffffa000341c>] ? __split_and_process_bio+0x46c/0x630 [dm_mod]
 [<ffffffffa00039e1>] ? dm_wq_work+0x161/0x200 [dm_mod]
 [<ffffffffa0003880>] ? dm_wq_work+0x0/0x200 [dm_mod]
 [<ffffffff8108c610>] ? worker_thread+0x170/0x2a0
 [<ffffffff81091ca0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff8108c4a0>] ? worker_thread+0x0/0x2a0
 [<ffffffff81091936>] ? kthread+0x96/0xa0
 [<ffffffff810141ca>] ? child_rip+0xa/0x20
 [<ffffffff810918a0>] ? kthread+0x0/0xa0
 [<ffffffff810141c0>] ? child_rip+0x0/0x20
---[ end trace d6b4961a23d7c2f1 ]---
------------[ cut here ]------------
WARNING: at lib/list_debug.c:26 __list_add+0x6d/0xa0() (Tainted: G        W  ---------------- )
Hardware name: KVM
list_add corruption. next->prev should be prev (ffff88000001a828), but was ffffea0000c1f5c8. (next=ffffea0001abb628).
Modules linked in: virtio_balloon ipv6 dm_mirror dm_region_hash dm_log ppdev parport_pc parport virtio_net i2c_piix4 i2c_core sg ext4 mbcache jbd2 sr_mod cdrom virtio_blk pata_acpi ata_generic ata_piix virtio_pci virtio_ring virtio dm_mod [last unloaded: speedstep_lib]
Pid: 303, comm: kdmflush Tainted: G        W  ----------------  2.6.32-70.el6.x86_64 #1
Call Trace:
 <IRQ>  [<ffffffff8106b857>] warn_slowpath_common+0x87/0xc0
 [<ffffffff8106b946>] warn_slowpath_fmt+0x46/0x50
 [<ffffffff81268acd>] __list_add+0x6d/0xa0
 [<ffffffff8111dd24>] free_pcppages_bulk+0x104/0x390
 [<ffffffff8111eb88>] free_hot_cold_page+0x1b8/0x220
 [<ffffffff8102ea69>] ? native_smp_send_reschedule+0x49/0x60
 [<ffffffff8111eccf>] free_hot_page+0x2f/0x60
 [<ffffffff8111ed60>] __free_pages+0x60/0xa0
 [<ffffffff8111ede9>] free_pages+0x49/0x50
 [<ffffffff811558cb>] kmem_freepages+0xeb/0x130
 [<ffffffff81157a23>] slab_destroy+0x33/0x90
 [<ffffffff81157ae0>] free_block+0x60/0x180
 [<ffffffff81157988>] kmem_cache_free+0x248/0x2b0
 [<ffffffff8110e617>] mempool_free_slab+0x17/0x20
 [<ffffffff8110e6d5>] mempool_free+0x95/0xa0
 [<ffffffff811a2084>] bio_free+0x64/0x70
 [<ffffffff811a20a5>] bio_fs_destructor+0x15/0x20
 [<ffffffff811a0e4b>] bio_put+0x2b/0x40
 [<ffffffff8119cb17>] end_bio_bh_io_sync+0x37/0x60
 [<ffffffff811a0d3d>] bio_endio+0x1d/0x40
 [<ffffffffa0001544>] dec_pending+0xe4/0x1e0 [dm_mod]
 [<ffffffffa00018df>] clone_endio+0x9f/0xd0 [dm_mod]
 [<ffffffff811a0d3d>] bio_endio+0x1d/0x40
 [<ffffffff8123f7fb>] req_bio_endio+0xab/0x110
 [<ffffffff8124083f>] blk_update_request+0xff/0x440
 [<ffffffff81240ba7>] blk_update_bidi_request+0x27/0x80
 [<ffffffff812418fe>] __blk_end_request_all+0x2e/0x60
 [<ffffffffa004c125>] blk_done+0x35/0xe0 [virtio_blk]
 [<ffffffffa002519c>] vring_interrupt+0x3c/0xd0 [virtio_ring]
 [<ffffffff810d8740>] handle_IRQ_event+0x60/0x170
 [<ffffffff810353d7>] ? native_apic_msr_write+0x37/0x40
 [<ffffffff810dae26>] handle_edge_irq+0xc6/0x160
 [<ffffffff81015fb9>] handle_irq+0x49/0xa0
 [<ffffffff814cf90c>] do_IRQ+0x6c/0xf0
 [<ffffffff81013ad3>] ret_from_intr+0x0/0x11
 [<ffffffff81073b8b>] ? __do_softirq+0x6b/0x1e0
 [<ffffffff81095a50>] ? hrtimer_interrupt+0x140/0x250
 [<ffffffff810142cc>] ? call_softirq+0x1c/0x30
 [<ffffffff81015f35>] ? do_softirq+0x65/0xa0
 [<ffffffff810739d5>] ? irq_exit+0x85/0x90
 [<ffffffff814cfa01>] ? smp_apic_timer_interrupt+0x71/0x9c
 [<ffffffff81013c93>] ? apic_timer_interrupt+0x13/0x20
 <EOI>  [<ffffffff81242a14>] ? __make_request+0x164/0x4c0
 [<ffffffff81240db2>] ? generic_make_request+0x1b2/0x4f0
 [<ffffffff811a1e7b>] ? bio_alloc_bioset+0x5b/0xf0
 [<ffffffffa00016ed>] ? __map_bio+0xad/0x130 [dm_mod]
 [<ffffffffa000341c>] ? __split_and_process_bio+0x46c/0x630 [dm_mod]
 [<ffffffffa00039e1>] ? dm_wq_work+0x161/0x200 [dm_mod]
 [<ffffffffa0003880>] ? dm_wq_work+0x0/0x200 [dm_mod]
 [<ffffffff8108c610>] ? worker_thread+0x170/0x2a0
 [<ffffffff81091ca0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff8108c4a0>] ? worker_thread+0x0/0x2a0
 [<ffffffff81091936>] ? kthread+0x96/0xa0
 [<ffffffff810141ca>] ? child_rip+0xa/0x20
 [<ffffffff810918a0>] ? kthread+0x0/0xa0
 [<ffffffff810141c0>] ? child_rip+0x0/0x20
---[ end trace d6b4961a23d7c2f2 ]---
------------[ cut here ]------------
WARNING: at lib/list_debug.c:48 list_del+0x6e/0xa0() (Tainted: G        W  ---------------- )
Hardware name: KVM
list_del corruption. prev->next should be ffffea0001ae85a0, but was ffffea0001aa8848
Modules linked in: virtio_balloon ipv6 dm_mirror dm_region_hash dm_log ppdev parport_pc parport virtio_net i2c_piix4 i2c_core sg ext4 mbcache jbd2 sr_mod cdrom virtio_blk pata_acpi ata_generic ata_piix virtio_pci virtio_ring virtio dm_mod [last unloaded: speedstep_lib]
Pid: 303, comm: kdmflush Tainted: G        W  ----------------  2.6.32-70.el6.x86_64 #1
Call Trace:
 <IRQ>  [<ffffffff8106b857>] warn_slowpath_common+0x87/0xc0
 [<ffffffff8106b946>] warn_slowpath_fmt+0x46/0x50
 [<ffffffff81268a2e>] list_del+0x6e/0xa0
 [<ffffffff8111de3e>] free_pcppages_bulk+0x21e/0x390
 [<ffffffff8111eb88>] free_hot_cold_page+0x1b8/0x220
 [<ffffffff8102ea69>] ? native_smp_send_reschedule+0x49/0x60
 [<ffffffff8111eccf>] free_hot_page+0x2f/0x60
 [<ffffffff8111ed60>] __free_pages+0x60/0xa0
 [<ffffffff8111ede9>] free_pages+0x49/0x50
 [<ffffffff811558cb>] kmem_freepages+0xeb/0x130
 [<ffffffff81157a23>] slab_destroy+0x33/0x90
 [<ffffffff81157ae0>] free_block+0x60/0x180
 [<ffffffff81157988>] kmem_cache_free+0x248/0x2b0
 [<ffffffff8110e617>] mempool_free_slab+0x17/0x20
 [<ffffffff8110e6d5>] mempool_free+0x95/0xa0
 [<ffffffff811a2084>] bio_free+0x64/0x70
 [<ffffffff811a20a5>] bio_fs_destructor+0x15/0x20
 [<ffffffff811a0e4b>] bio_put+0x2b/0x40
 [<ffffffff8119cb17>] end_bio_bh_io_sync+0x37/0x60
 [<ffffffff811a0d3d>] bio_endio+0x1d/0x40
 [<ffffffffa0001544>] dec_pending+0xe4/0x1e0 [dm_mod]
 [<ffffffffa00018df>] clone_endio+0x9f/0xd0 [dm_mod]
 [<ffffffff811a0d3d>] bio_endio+0x1d/0x40
 [<ffffffff8123f7fb>] req_bio_endio+0xab/0x110
 [<ffffffff8124083f>] blk_update_request+0xff/0x440
 [<ffffffff81240ba7>] blk_update_bidi_request+0x27/0x80
 [<ffffffff812418fe>] __blk_end_request_all+0x2e/0x60
 [<ffffffffa004c125>] blk_done+0x35/0xe0 [virtio_blk]
 [<ffffffffa002519c>] vring_interrupt+0x3c/0xd0 [virtio_ring]
 [<ffffffff810d8740>] handle_IRQ_event+0x60/0x170
 [<ffffffff810353d7>] ? native_apic_msr_write+0x37/0x40
 [<ffffffff810dae26>] handle_edge_irq+0xc6/0x160
 [<ffffffff81015fb9>] handle_irq+0x49/0xa0
 [<ffffffff814cf90c>] do_IRQ+0x6c/0xf0
 [<ffffffff81013ad3>] ret_from_intr+0x0/0x11
 [<ffffffff81073b8b>] ? __do_softirq+0x6b/0x1e0
 [<ffffffff81095a50>] ? hrtimer_interrupt+0x140/0x250
 [<ffffffff810142cc>] ? call_softirq+0x1c/0x30
 [<ffffffff81015f35>] ? do_softirq+0x65/0xa0
 [<ffffffff810739d5>] ? irq_exit+0x85/0x90
 [<ffffffff814cfa01>] ? smp_apic_timer_interrupt+0x71/0x9c
 [<ffffffff81013c93>] ? apic_timer_interrupt+0x13/0x20
 <EOI>  [<ffffffff81242a14>] ? __make_request+0x164/0x4c0
 [<ffffffff81240db2>] ? generic_make_request+0x1b2/0x4f0
 [<ffffffff811a1e7b>] ? bio_alloc_bioset+0x5b/0xf0
 [<ffffffffa00016ed>] ? __map_bio+0xad/0x130 [dm_mod]
 [<ffffffffa000341c>] ? __split_and_process_bio+0x46c/0x630 [dm_mod]
 [<ffffffffa00039e1>] ? dm_wq_work+0x161/0x200 [dm_mod]
 [<ffffffffa0003880>] ? dm_wq_work+0x0/0x200 [dm_mod]
 [<ffffffff8108c610>] ? worker_thread+0x170/0x2a0
 [<ffffffff81091ca0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff8108c4a0>] ? worker_thread+0x0/0x2a0
 [<ffffffff81091936>] ? kthread+0x96/0xa0
 [<ffffffff810141ca>] ? child_rip+0xa/0x20
 [<ffffffff810918a0>] ? kthread+0x0/0xa0
 [<ffffffff810141c0>] ? child_rip+0x0/0x20
---[ end trace d6b4961a23d7c2f3 ]---
------------[ cut here ]------------
WARNING: at lib/list_debug.c:48 list_del+0x6e/0xa0() (Tainted: G        W  ---------------- )
Hardware name: KVM
list_del corruption. prev->next should be ffffea0001aa8848, but was ffffea0001aede38
Modules linked in: virtio_balloon ipv6 dm_mirror dm_region_hash dm_log ppdev parport_pc parport virtio_net i2c_piix4 i2c_core sg ext4 mbcache jbd2 sr_mod cdrom virtio_blk pata_acpi ata_generic ata_piix virtio_pci virtio_ring virtio dm_mod [last unloaded: speedstep_lib]
Pid: 303, comm: kdmflush Tainted: G        W  ----------------  2.6.32-70.el6.x86_64 #1
Call Trace:
 <IRQ>  [<ffffffff8106b857>] warn_slowpath_common+0x87/0xc0
 [<ffffffff8106b946>] warn_slowpath_fmt+0x46/0x50
 [<ffffffff81268a2e>] list_del+0x6e/0xa0
 [<ffffffff8111de3e>] free_pcppages_bulk+0x21e/0x390
 [<ffffffff8111eb88>] free_hot_cold_page+0x1b8/0x220
 [<ffffffff8102ea69>] ? native_smp_send_reschedule+0x49/0x60
 [<ffffffff8111eccf>] free_hot_page+0x2f/0x60
 [<ffffffff8111ed60>] __free_pages+0x60/0xa0
 [<ffffffff8111ede9>] free_pages+0x49/0x50
 [<ffffffff811558cb>] kmem_freepages+0xeb/0x130
 [<ffffffff81157a23>] slab_destroy+0x33/0x90
 [<ffffffff81157ae0>] free_block+0x60/0x180
 [<ffffffff81157988>] kmem_cache_free+0x248/0x2b0
 [<ffffffff8110e617>] mempool_free_slab+0x17/0x20
 [<ffffffff8110e6d5>] mempool_free+0x95/0xa0
 [<ffffffff811a2084>] bio_free+0x64/0x70
 [<ffffffff811a20a5>] bio_fs_destructor+0x15/0x20
 [<ffffffff811a0e4b>] bio_put+0x2b/0x40
 [<ffffffff8119cb17>] end_bio_bh_io_sync+0x37/0x60
 [<ffffffff811a0d3d>] bio_endio+0x1d/0x40
 [<ffffffffa0001544>] dec_pending+0xe4/0x1e0 [dm_mod]
 [<ffffffffa00018df>] clone_endio+0x9f/0xd0 [dm_mod]
 [<ffffffff811a0d3d>] bio_endio+0x1d/0x40
 [<ffffffff8123f7fb>] req_bio_endio+0xab/0x110
 [<ffffffff8124083f>] blk_update_request+0xff/0x440
 [<ffffffff81240ba7>] blk_update_bidi_request+0x27/0x80
 [<ffffffff812418fe>] __blk_end_request_all+0x2e/0x60
 [<ffffffffa004c125>] blk_done+0x35/0xe0 [virtio_blk]
 [<ffffffffa002519c>] vring_interrupt+0x3c/0xd0 [virtio_ring]
 [<ffffffff810d8740>] handle_IRQ_event+0x60/0x170
 [<ffffffff810353d7>] ? native_apic_msr_write+0x37/0x40
 [<ffffffff810dae26>] handle_edge_irq+0xc6/0x160
 [<ffffffff81015fb9>] handle_irq+0x49/0xa0
 [<ffffffff814cf90c>] do_IRQ+0x6c/0xf0
 [<ffffffff81013ad3>] ret_from_intr+0x0/0x11
 [<ffffffff81073b8b>] ? __do_softirq+0x6b/0x1e0
 [<ffffffff81095a50>] ? hrtimer_interrupt+0x140/0x250
 [<ffffffff810142cc>] ? call_softirq+0x1c/0x30
 [<ffffffff81015f35>] ? do_softirq+0x65/0xa0
 [<ffffffff810739d5>] ? irq_exit+0x85/0x90
 [<ffffffff814cfa01>] ? smp_apic_timer_interrupt+0x71/0x9c
 [<ffffffff81013c93>] ? apic_timer_interrupt+0x13/0x20
 <EOI>  [<ffffffff81242a14>] ? __make_request+0x164/0x4c0
 [<ffffffff81240db2>] ? generic_make_request+0x1b2/0x4f0
 [<ffffffff811a1e7b>] ? bio_alloc_bioset+0x5b/0xf0
 [<ffffffffa00016ed>] ? __map_bio+0xad/0x130 [dm_mod]
 [<ffffffffa000341c>] ? __split_and_process_bio+0x46c/0x630 [dm_mod]
 [<ffffffffa00039e1>] ? dm_wq_work+0x161/0x200 [dm_mod]
 [<ffffffffa0003880>] ? dm_wq_work+0x0/0x200 [dm_mod]
 [<ffffffff8108c610>] ? worker_thread+0x170/0x2a0
 [<ffffffff81091ca0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff8108c4a0>] ? worker_thread+0x0/0x2a0
 [<ffffffff81091936>] ? kthread+0x96/0xa0
 [<ffffffff810141ca>] ? child_rip+0xa/0x20
 [<ffffffff810918a0>] ? kthread+0x0/0xa0
 [<ffffffff810141c0>] ? child_rip+0x0/0x20
---[ end trace d6b4961a23d7c2f4 ]---
------------[ cut here ]------------
WARNING: at lib/list_debug.c:48 list_del+0x6e/0xa0() (Tainted: G        W  ---------------- )
Hardware name: KVM
list_del corruption. prev->next should be ffffea0001aa85a8, but was ffffea0001ae8568
Modules linked in: virtio_balloon ipv6 dm_mirror dm_region_hash dm_log ppdev parport_pc parport virtio_net i2c_piix4 i2c_core sg ext4 mbcache jbd2 sr_mod cdrom virtio_blk pata_acpi ata_generic ata_piix virtio_pci virtio_ring virtio dm_mod [last unloaded: speedstep_lib]
Pid: 303, comm: kdmflush Tainted: G        W  ----------------  2.6.32-70.el6.x86_64 #1
Call Trace:
 <IRQ>  [<ffffffff8106b857>] warn_slowpath_common+0x87/0xc0
 [<ffffffff8106b946>] warn_slowpath_fmt+0x46/0x50
 [<ffffffff81268a2e>] list_del+0x6e/0xa0
 [<ffffffff8111de3e>] free_pcppages_bulk+0x21e/0x390
 [<ffffffff8111eb88>] free_hot_cold_page+0x1b8/0x220
 [<ffffffff8102ea69>] ? native_smp_send_reschedule+0x49/0x60
 [<ffffffff8111eccf>] free_hot_page+0x2f/0x60
 [<ffffffff8111ed60>] __free_pages+0x60/0xa0
 [<ffffffff8111ede9>] free_pages+0x49/0x50
 [<ffffffff811558cb>] kmem_freepages+0xeb/0x130
 [<ffffffff81157a23>] slab_destroy+0x33/0x90
 [<ffffffff81157ae0>] free_block+0x60/0x180
 [<ffffffff81157988>] kmem_cache_free+0x248/0x2b0
 [<ffffffff8110e617>] mempool_free_slab+0x17/0x20
 [<ffffffff8110e6d5>] mempool_free+0x95/0xa0
 [<ffffffff811a2084>] bio_free+0x64/0x70
 [<ffffffff811a20a5>] bio_fs_destructor+0x15/0x20
 [<ffffffff811a0e4b>] bio_put+0x2b/0x40
 [<ffffffff8119cb17>] end_bio_bh_io_sync+0x37/0x60
 [<ffffffff811a0d3d>] bio_endio+0x1d/0x40
 [<ffffffffa0001544>] dec_pending+0xe4/0x1e0 [dm_mod]
 [<ffffffffa00018df>] clone_endio+0x9f/0xd0 [dm_mod]
 [<ffffffff811a0d3d>] bio_endio+0x1d/0x40
 [<ffffffff8123f7fb>] req_bio_endio+0xab/0x110
 [<ffffffff8124083f>] blk_update_request+0xff/0x440
 [<ffffffff81240ba7>] blk_update_bidi_request+0x27/0x80
 [<ffffffff812418fe>] __blk_end_request_all+0x2e/0x60
 [<ffffffffa004c125>] blk_done+0x35/0xe0 [virtio_blk]
 [<ffffffffa002519c>] vring_interrupt+0x3c/0xd0 [virtio_ring]
 [<ffffffff810d8740>] handle_IRQ_event+0x60/0x170
 [<ffffffff810353d7>] ? native_apic_msr_write+0x37/0x40
 [<ffffffff810dae26>] handle_edge_irq+0xc6/0x160
 [<ffffffff81015fb9>] handle_irq+0x49/0xa0
 [<ffffffff814cf90c>] do_IRQ+0x6c/0xf0
 [<ffffffff81013ad3>] ret_from_intr+0x0/0x11
 [<ffffffff81073b8b>] ? __do_softirq+0x6b/0x1e0
 [<ffffffff81095a50>] ? hrtimer_interrupt+0x140/0x250
 [<ffffffff810142cc>] ? call_softirq+0x1c/0x30
 [<ffffffff81015f35>] ? do_softirq+0x65/0xa0
 [<ffffffff810739d5>] ? irq_exit+0x85/0x90
 [<ffffffff814cfa01>] ? smp_apic_timer_interrupt+0x71/0x9c
 [<ffffffff81013c93>] ? apic_timer_interrupt+0x13/0x20
 <EOI>  [<ffffffff81242a14>] ? __make_request+0x164/0x4c0
 [<ffffffff81240db2>] ? generic_make_request+0x1b2/0x4f0
 [<ffffffff811a1e7b>] ? bio_alloc_bioset+0x5b/0xf0
 [<ffffffffa00016ed>] ? __map_bio+0xad/0x130 [dm_mod]
 [<ffffffffa000341c>] ? __split_and_process_bio+0x46c/0x630 [dm_mod]
 [<ffffffffa00039e1>] ? dm_wq_work+0x161/0x200 [dm_mod]
 [<ffffffffa0003880>] ? dm_wq_work+0x0/0x200 [dm_mod]
 [<ffffffff8108c610>] ? worker_thread+0x170/0x2a0
 [<ffffffff81091ca0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff8108c4a0>] ? worker_thread+0x0/0x2a0
 [<ffffffff81091936>] ? kthread+0x96/0xa0
 [<ffffffff810141ca>] ? child_rip+0xa/0x20
 [<ffffffff810918a0>] ? kthread+0x0/0xa0
 [<ffffffff810141c0>] ? child_rip+0x0/0x20
---[ end trace d6b4961a23d7c2f5 ]---
------------[ cut here ]------------
WARNING: at lib/list_debug.c:26 __list_add+0x6d/0xa0() (Tainted: G        W  ---------------- )
Hardware name: KVM
list_add corruption. next->prev should be prev (ffff88000001a880), but was ffffea0001b17428. (next=ffffea0001ac76a8).
Modules linked in: virtio_balloon ipv6 dm_mirror dm_region_hash dm_log ppdev parport_pc parport virtio_net i2c_piix4 i2c_core sg ext4 mbcache jbd2 sr_mod cdrom virtio_blk pata_acpi ata_generic ata_piix virtio_pci virtio_ring virtio dm_mod [last unloaded: speedstep_lib]
Pid: 303, comm: kdmflush Tainted: G        W  ----------------  2.6.32-70.el6.x86_64 #1
Call Trace:
 <IRQ>  [<ffffffff8106b857>] warn_slowpath_common+0x87/0xc0
 [<ffffffff8106b946>] warn_slowpath_fmt+0x46/0x50
 [<ffffffff81268acd>] __list_add+0x6d/0xa0
 [<ffffffff8111dd24>] free_pcppages_bulk+0x104/0x390
 [<ffffffff8111eb88>] free_hot_cold_page+0x1b8/0x220
 [<ffffffff8102ea69>] ? native_smp_send_reschedule+0x49/0x60
 [<ffffffff8111eccf>] free_hot_page+0x2f/0x60
 [<ffffffff8111ed60>] __free_pages+0x60/0xa0
 [<ffffffff8111ede9>] free_pages+0x49/0x50
 [<ffffffff811558cb>] kmem_freepages+0xeb/0x130
 [<ffffffff81157a23>] slab_destroy+0x33/0x90
 [<ffffffff81157ae0>] free_block+0x60/0x180
 [<ffffffff81157988>] kmem_cache_free+0x248/0x2b0
 [<ffffffff8110e617>] mempool_free_slab+0x17/0x20
 [<ffffffff8110e6d5>] mempool_free+0x95/0xa0
 [<ffffffff811a2084>] bio_free+0x64/0x70
 [<ffffffff811a20a5>] bio_fs_destructor+0x15/0x20
 [<ffffffff811a0e4b>] bio_put+0x2b/0x40
 [<ffffffff8119cb17>] end_bio_bh_io_sync+0x37/0x60
 [<ffffffff811a0d3d>] bio_endio+0x1d/0x40
 [<ffffffffa0001544>] dec_pending+0xe4/0x1e0 [dm_mod]
 [<ffffffffa00018df>] clone_endio+0x9f/0xd0 [dm_mod]
 [<ffffffff811a0d3d>] bio_endio+0x1d/0x40
 [<ffffffff8123f7fb>] req_bio_endio+0xab/0x110
 [<ffffffff8124083f>] blk_update_request+0xff/0x440
 [<ffffffff81240ba7>] blk_update_bidi_request+0x27/0x80
 [<ffffffff812418fe>] __blk_end_request_all+0x2e/0x60
 [<ffffffffa004c125>] blk_done+0x35/0xe0 [virtio_blk]
 [<ffffffffa002519c>] vring_interrupt+0x3c/0xd0 [virtio_ring]
 [<ffffffff810d8740>] handle_IRQ_event+0x60/0x170
 [<ffffffff810353d7>] ? native_apic_msr_write+0x37/0x40
 [<ffffffff810dae26>] handle_edge_irq+0xc6/0x160
 [<ffffffff81015fb9>] handle_irq+0x49/0xa0
 [<ffffffff814cf90c>] do_IRQ+0x6c/0xf0
 [<ffffffff81013ad3>] ret_from_intr+0x0/0x11
 [<ffffffff81073b8b>] ? __do_softirq+0x6b/0x1e0
 [<ffffffff81095a50>] ? hrtimer_interrupt+0x140/0x250
 [<ffffffff810142cc>] ? call_softirq+0x1c/0x30
 [<ffffffff81015f35>] ? do_softirq+0x65/0xa0
 [<ffffffff810739d5>] ? irq_exit+0x85/0x90
 [<ffffffff814cfa01>] ? smp_apic_timer_interrupt+0x71/0x9c
 [<ffffffff81013c93>] ? apic_timer_interrupt+0x13/0x20
 <EOI>  [<ffffffff81242a14>] ? __make_request+0x164/0x4c0
 [<ffffffff81240db2>] ? generic_make_request+0x1b2/0x4f0
 [<ffffffff811a1e7b>] ? bio_alloc_bioset+0x5b/0xf0
 [<ffffffffa00016ed>] ? __map_bio+0xad/0x130 [dm_mod]
 [<ffffffffa000341c>] ? __split_and_process_bio+0x46c/0x630 [dm_mod]
 [<ffffffffa00039e1>] ? dm_wq_work+0x161/0x200 [dm_mod]
 [<ffffffffa0003880>] ? dm_wq_work+0x0/0x200 [dm_mod]
 [<ffffffff8108c610>] ? worker_thread+0x170/0x2a0
 [<ffffffff81091ca0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff8108c4a0>] ? worker_thread+0x0/0x2a0
 [<ffffffff81091936>] ? kthread+0x96/0xa0
 [<ffffffff810141ca>] ? child_rip+0xa/0x20
 [<ffffffff810918a0>] ? kthread+0x0/0xa0
 [<ffffffff810141c0>] ? child_rip+0x0/0x20
---[ end trace d6b4961a23d7c2f6 ]---
BUG: unable to handle kernel paging request at ffffea000dfa72f0
IP: [<ffffffff81157b63>] free_block+0xe3/0x180
PGD 20a2067 PUD 20a3067 PMD 0 
Oops: 0002 [#1] SMP 
last sysfs file: /sys/devices/virtio-pci/virtio1/block/vda/dev
CPU 0 
Modules linked in: virtio_balloon ipv6 dm_mirror dm_region_hash dm_log ppdev parport_pc parport virtio_net i2c_piix4 i2c_core sg ext4 mbcache jbd2 sr_mod cdrom virtio_blk pata_acpi ata_generic ata_piix virtio_pci virtio_ring virtio dm_mod [last unloaded: speedstep_lib]

Modules linked in: virtio_balloon ipv6 dm_mirror dm_region_hash dm_log ppdev parport_pc parport virtio_net i2c_piix4 i2c_core sg ext4 mbcache jbd2 sr_mod cdrom virtio_blk pata_acpi ata_generic ata_piix virtio_pci virtio_ring virtio dm_mod [last unloaded: speedstep_lib]
Pid: 303, comm: kdmflush Tainted: G        W  ----------------  2.6.32-70.el6.x86_64 #1 KVM
RIP: 0010:[<ffffffff81157b63>]  [<ffffffff81157b63>] free_block+0xe3/0x180
RSP: 0018:ffff880001e03ab0  EFLAGS: 00010002
RAX: ffffea0001abaac8 RBX: ffff8800377825c0 RCX: 000000000313b1fe
RDX: dead000000100100 RSI: 0000000000000000 RDI: ffffea0001abaac8
RBP: ffff880001e03b00 R08: ffffea0001abaac8 R09: 0000000000000000
R10: 00000000ffffffff R11: 000000000000003d R12: 000000000000003c
R13: ffff880037626908 R14: 000000000000001e R15: ffffea0000000000
FS:  0000000000000000(0000) GS:ffff880001e00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: ffffea000dfa72f0 CR3: 000000007a279000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kdmflush (pid: 303, threadinfo ffff880079636000, task ffff88007c8020c0)
Stack:
 ffff8800375c20c0 ffffea0001abaac8 000000000000100c ffff88007b13cfb0
<0> ffffea0000cc3d68 ffff880037626800 0000000000000082 ffff88007b470ba0
<0> ffff8800377825c0 ffff880037626818 ffff880001e03b70 ffffffff81157988
Call Trace:
 <IRQ> 
 [<ffffffff81157988>] kmem_cache_free+0x248/0x2b0
 [<ffffffff811a2084>] ? bio_free+0x64/0x70
 [<ffffffff8110e617>] mempool_free_slab+0x17/0x20
 [<ffffffff8110e6d5>] mempool_free+0x95/0xa0
 [<ffffffffa0001520>] dec_pending+0xc0/0x1e0 [dm_mod]
 [<ffffffffa00018df>] clone_endio+0x9f/0xd0 [dm_mod]
 [<ffffffff811a0d3d>] bio_endio+0x1d/0x40
 [<ffffffff8123f7fb>] req_bio_endio+0xab/0x110
 [<ffffffff8124083f>] blk_update_request+0xff/0x440
 [<ffffffff81240ba7>] blk_update_bidi_request+0x27/0x80
 [<ffffffff812418fe>] __blk_end_request_all+0x2e/0x60
 [<ffffffffa004c125>] blk_done+0x35/0xe0 [virtio_blk]
 [<ffffffffa002519c>] vring_interrupt+0x3c/0xd0 [virtio_ring]
 [<ffffffff810d8740>] handle_IRQ_event+0x60/0x170
 [<ffffffff810353d7>] ? native_apic_msr_write+0x37/0x40
 [<ffffffff810dae26>] handle_edge_irq+0xc6/0x160
 [<ffffffff81015fb9>] handle_irq+0x49/0xa0
 [<ffffffff814cf90c>] do_IRQ+0x6c/0xf0
 [<ffffffff81013ad3>] ret_from_intr+0x0/0x11
 [<ffffffff81073b8b>] ? __do_softirq+0x6b/0x1e0
 [<ffffffff81095a50>] ? hrtimer_interrupt+0x140/0x250
 [<ffffffff810142cc>] ? call_softirq+0x1c/0x30
 [<ffffffff81015f35>] ? do_softirq+0x65/0xa0
 [<ffffffff810739d5>] ? irq_exit+0x85/0x90
 [<ffffffff814cfa01>] ? smp_apic_timer_interrupt+0x71/0x9c
 [<ffffffff81013c93>] ? apic_timer_interrupt+0x13/0x20
 <EOI> 
 [<ffffffff81242a14>] ? __make_request+0x164/0x4c0
 [<ffffffff81240db2>] ? generic_make_request+0x1b2/0x4f0
 [<ffffffff811a1e7b>] ? bio_alloc_bioset+0x5b/0xf0
 [<ffffffffa00016ed>] ? __map_bio+0xad/0x130 [dm_mod]
 [<ffffffffa000341c>] ? __split_and_process_bio+0x46c/0x630 [dm_mod]
 [<ffffffffa00039e1>] ? dm_wq_work+0x161/0x200 [dm_mod]
 [<ffffffffa0003880>] ? dm_wq_work+0x0/0x200 [dm_mod]
 [<ffffffff8108c610>] ? worker_thread+0x170/0x2a0
 [<ffffffff81091ca0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff8108c4a0>] ? worker_thread+0x0/0x2a0
 [<ffffffff81091936>] ? kthread+0x96/0xa0
 [<ffffffff810141ca>] ? child_rip+0xa/0x20
 [<ffffffff810918a0>] ? kthread+0x0/0xa0
 [<ffffffff810141c0>] ? child_rip+0x0/0x20
Code: 48 89 c7 48 89 45 b8 48 89 55 b0 e8 78 0e 11 00 48 8b 45 b8 8b b3 10 80 00 00 8b 4d c8 2b 48 18 48 0f af ce 8b 70 24 48 c1 e9 20 <89> 74 88 30 83 68 20 01 89 48 24 48 8b 55 b0 48 8b 4a 30 48 83 
RIP  [<ffffffff81157b63>] free_block+0xe3/0x180
 RSP <ffff880001e03ab0>
CR2: ffffea000dfa72f0
---[ end trace d6b4961a23d7c2f7 ]---
Kernel panic - not syncing: Fatal exception in interrupt
Pid: 303, comm: kdmflush Tainted: G      D W  ----------------  2.6.32-70.el6.x86_64 #1
Call Trace:
 <IRQ>  [<ffffffff814c7b23>] panic+0x78/0x137
 [<ffffffff814cbc02>] oops_end+0xf2/0x100
 [<ffffffff8104651b>] no_context+0xfb/0x260
 [<ffffffff810467a5>] __bad_area_nosemaphore+0x125/0x1e0
 [<ffffffff81046873>] bad_area_nosemaphore+0x13/0x20
 [<ffffffff814cd658>] do_page_fault+0x2a8/0x3a0
 [<ffffffff8111ede9>] ? free_pages+0x49/0x50
 [<ffffffff814caf45>] page_fault+0x25/0x30
 [<ffffffff81157b63>] ? free_block+0xe3/0x180
 [<ffffffff81157988>] kmem_cache_free+0x248/0x2b0
 [<ffffffff811a2084>] ? bio_free+0x64/0x70
 [<ffffffff8110e617>] mempool_free_slab+0x17/0x20
 [<ffffffff8110e6d5>] mempool_free+0x95/0xa0
 [<ffffffffa0001520>] dec_pending+0xc0/0x1e0 [dm_mod]
 [<ffffffffa00018df>] clone_endio+0x9f/0xd0 [dm_mod]
 [<ffffffff811a0d3d>] bio_endio+0x1d/0x40
 [<ffffffff8123f7fb>] req_bio_endio+0xab/0x110
 [<ffffffff8124083f>] blk_update_request+0xff/0x440
 [<ffffffff81240ba7>] blk_update_bidi_request+0x27/0x80
 [<ffffffff812418fe>] __blk_end_request_all+0x2e/0x60
 [<ffffffffa004c125>] blk_done+0x35/0xe0 [virtio_blk]
 [<ffffffffa002519c>] vring_interrupt+0x3c/0xd0 [virtio_ring]
 [<ffffffff810d8740>] handle_IRQ_event+0x60/0x170
 [<ffffffff810353d7>] ? native_apic_msr_write+0x37/0x40
 [<ffffffff810dae26>] handle_edge_irq+0xc6/0x160
 [<ffffffff81015fb9>] handle_irq+0x49/0xa0
 [<ffffffff814cf90c>] do_IRQ+0x6c/0xf0
 [<ffffffff81013ad3>] ret_from_intr+0x0/0x11
 [<ffffffff81073b8b>] ? __do_softirq+0x6b/0x1e0
 [<ffffffff81095a50>] ? hrtimer_interrupt+0x140/0x250
 [<ffffffff810142cc>] ? call_softirq+0x1c/0x30
 [<ffffffff81015f35>] ? do_softirq+0x65/0xa0
 [<ffffffff810739d5>] ? irq_exit+0x85/0x90
 [<ffffffff814cfa01>] ? smp_apic_timer_interrupt+0x71/0x9c
 [<ffffffff81013c93>] ? apic_timer_interrupt+0x13/0x20
 <EOI>  [<ffffffff81242a14>] ? __make_request+0x164/0x4c0
 [<ffffffff81240db2>] ? generic_make_request+0x1b2/0x4f0
 [<ffffffff811a1e7b>] ? bio_alloc_bioset+0x5b/0xf0
 [<ffffffffa00016ed>] ? __map_bio+0xad/0x130 [dm_mod]
 [<ffffffffa000341c>] ? __split_and_process_bio+0x46c/0x630 [dm_mod]
 [<ffffffffa00039e1>] ? dm_wq_work+0x161/0x200 [dm_mod]
 [<ffffffffa0003880>] ? dm_wq_work+0x0/0x200 [dm_mod]
 [<ffffffff8108c610>] ? worker_thread+0x170/0x2a0
 [<ffffffff81091ca0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff8108c4a0>] ? worker_thread+0x0/0x2a0
 [<ffffffff81091936>] ? kthread+0x96/0xa0
 [<ffffffff810141ca>] ? child_rip+0xa/0x20
 [<ffffffff810918a0>] ? kthread+0x0/0xa0
 [<ffffffff810141c0>] ? child_rip+0x0/0x20

Comment 1 Amit Shah 2010-10-13 12:06:42 UTC

Looks like it's related to virtio-block:

BUG: unable to handle kernel paging request at ffffea000dfa72f0
IP: [<ffffffff81157b63>] free_block+0xe3/0x180
PGD 20a2067 PUD 20a3067 PMD 0 
Oops: 0002 [#1] SMP 
last sysfs file: /sys/devices/virtio-pci/virtio1/block/vda/dev
CPU 0 
Modules linked in: virtio_balloon ipv6 dm_mirror dm_region_hash dm_log ppdev
parport_pc parport virtio_net i2c_piix4 i2c_core sg ext4 mbcache jbd2 sr_mod
cdrom virtio_blk pata_acpi ata_generic ata_piix virtio_pci virtio_ring virtio
dm_mod [last unloaded: speedstep_lib]

Modules linked in: virtio_balloon ipv6 dm_mirror dm_region_hash dm_log ppdev
parport_pc parport virtio_net i2c_piix4 i2c_core sg ext4 mbcache jbd2 sr_mod
cdrom virtio_blk pata_acpi ata_generic ata_piix virtio_pci virtio_ring virtio
dm_mod [last unloaded: speedstep_lib]
Pid: 303, comm: kdmflush Tainted: G        W  ---------------- 
2.6.32-70.el6.x86_64 #1 KVM
RIP: 0010:[<ffffffff81157b63>]  [<ffffffff81157b63>] free_block+0xe3/0x180
RSP: 0018:ffff880001e03ab0  EFLAGS: 00010002
RAX: ffffea0001abaac8 RBX: ffff8800377825c0 RCX: 000000000313b1fe
RDX: dead000000100100 RSI: 0000000000000000 RDI: ffffea0001abaac8
RBP: ffff880001e03b00 R08: ffffea0001abaac8 R09: 0000000000000000
R10: 00000000ffffffff R11: 000000000000003d R12: 000000000000003c
R13: ffff880037626908 R14: 000000000000001e R15: ffffea0000000000
FS:  0000000000000000(0000) GS:ffff880001e00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: ffffea000dfa72f0 CR3: 000000007a279000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kdmflush (pid: 303, threadinfo ffff880079636000, task ffff88007c8020c0)
Stack:
 ffff8800375c20c0 ffffea0001abaac8 000000000000100c ffff88007b13cfb0
<0> ffffea0000cc3d68 ffff880037626800 0000000000000082 ffff88007b470ba0
<0> ffff8800377825c0 ffff880037626818 ffff880001e03b70 ffffffff81157988
Call Trace:
 <IRQ> 
 [<ffffffff81157988>] kmem_cache_free+0x248/0x2b0
 [<ffffffff811a2084>] ? bio_free+0x64/0x70
 [<ffffffff8110e617>] mempool_free_slab+0x17/0x20
 [<ffffffff8110e6d5>] mempool_free+0x95/0xa0
 [<ffffffffa0001520>] dec_pending+0xc0/0x1e0 [dm_mod]
 [<ffffffffa00018df>] clone_endio+0x9f/0xd0 [dm_mod]
 [<ffffffff811a0d3d>] bio_endio+0x1d/0x40
 [<ffffffff8123f7fb>] req_bio_endio+0xab/0x110
 [<ffffffff8124083f>] blk_update_request+0xff/0x440
 [<ffffffff81240ba7>] blk_update_bidi_request+0x27/0x80
 [<ffffffff812418fe>] __blk_end_request_all+0x2e/0x60
 [<ffffffffa004c125>] blk_done+0x35/0xe0 [virtio_blk]
 [<ffffffffa002519c>] vring_interrupt+0x3c/0xd0 [virtio_ring]

Comment 2 Amos Kong 2010-10-15 05:25:40 UTC

(In reply to comment #1)
> Looks like it's related to virtio-block:

I can also reproduce this bug with ide/e1000 configuration.

host kernel:2.6.32-71.3.1.el6_0.x86_64
qemu-kvm-0.12.1.2-2.113.el6_0.1.x86_64


Panic msg: 
---------------------------------------------------------

general protection fault: 0000 [#1] SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:03.0/irq
CPU 1
Modules linked in: ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log ppdev parport_pc parport e1000 i2c_piix4 i2c_core sg ext4 mbcache jbd2 sd_mod crc_t10dif sr_mod cdrom pata_acpi ata_generic ata_piix dm_mod [last unloaded: speedstep_lib]

Modules linked in: ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log ppdev parport_pc parport e1000 i2c_piix4 i2c_core sg ext4 mbcache jbd2 sd_mod crc_t10dif sr_mod cdrom pata_acpi ata_generic ata_piix dm_mod [last unloaded: speedstep_lib]
Pid: 1438, comm: sshd Not tainted 2.6.32-71.el6.x86_64 #1 KVM
RIP: 0010:[<ffffffff81268a77>]  [<ffffffff81268a77>] __list_add+0x17/0xa0
RSP: 0018:ffff88007a161a88  EFLAGS: 00010086
RAX: ffffea0001aa9f88 RBX: ffffea0001aa9fb0 RCX: ffffea0001aa9fb0
RDX: dead000000100100 RSI: ffffea0001aa9fb0 RDI: ffffea0001aa9fb0
RBP: ffff88007a161aa8 R08: ffffea0001aa9fb0 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000001000 R12: 0000000000000000
R13: ffff8800000126c0 R14: 0000000000000001 R15: ffffea0001aa9fb0
FS:  00007f498c65f7c0(0000) GS:ffff880001f00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fa756381000 CR3: 0000000037f6a000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process sshd (pid: 1438, threadinfo ffff88007a160000, task ffff880037be6a70)
Stack:
 0000000000000000 000000000000001f 0000000000000000 ffff8800000126c0
<0> ffff88007a161bd8 ffffffff8111d186 ffffea0001aa9fb0 ffffea0001aa9f88
<0> 0000000100000001 ffff88007ccbcec0 000000000000000f 00000040ffffffff
Call Trace:
 [<ffffffff8111d186>] get_page_from_freelist+0x5c6/0x820
 [<ffffffff8111e1c6>] __alloc_pages_nodemask+0xf6/0x810
 [<ffffffff811502a7>] alloc_pages_current+0x87/0xd0
 [<ffffffff8117636c>] pipe_write+0x36c/0x650
 [<ffffffff8116c51a>] do_sync_write+0xfa/0x140
 [<ffffffff81091ca0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff8120bf0f>] ? selinux_file_permission+0xbf/0x150
 [<ffffffff811ff3b6>] ? security_file_permission+0x16/0x20
 [<ffffffff8116c818>] vfs_write+0xb8/0x1a0
 [<ffffffff810830a1>] ? sigprocmask+0x71/0x110
 [<ffffffff8116d251>] sys_write+0x51/0x90
 [<ffffffff81013172>] system_call_fastpath+0x16/0x1b
Code: ff 48 8b 03 eb 92 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 48 83 ec 20 48 89 5d e8 4c 89 65 f0 48 89 fb 4c 89 6d f8 <4c> 8b 42 08 49 89 f5 49 89 d4 49 39 f0 75 27 4d 8b 45 00 4d 39
RIP  [<ffffffff81268a77>] __list_add+0x17/0xa0
 RSP <ffff88007a161a88>
---[ end trace 9cd7f43d04294bd4 ]---
Kernel panic - not syncing: Fatal exception
Pid: 1438, comm: sshd Tainted: G      D    ----------------  2.6.32-71.el6.x86_64 #1
Call Trace:
 [<ffffffff814c7b23>] panic+0x78/0x137
 [<ffffffff814cbbf4>] oops_end+0xe4/0x100
 [<ffffffff8101733b>] die+0x5b/0x90
 [<ffffffff814cb742>] do_general_protection+0x152/0x160
 [<ffffffff814caf15>] general_protection+0x25/0x30
 [<ffffffff81268a77>] ? __list_add+0x17/0xa0
 [<ffffffff8111d186>] get_page_from_freelist+0x5c6/0x820
 [<ffffffff8111e1c6>] __alloc_pages_nodemask+0xf6/0x810
 [<ffffffff811502a7>] alloc_pages_current+0x87/0xd0
 [<ffffffff8117636c>] pipe_write+0x36c/0x650
 [<ffffffff8116c51a>] do_sync_write+0xfa/0x140
 [<ffffffff81091ca0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff8120bf0f>] ? selinux_file_permission+0xbf/0x150
 [<ffffffff811ff3b6>] ? security_file_permission+0x16/0x20
 [<ffffffff8116c818>] vfs_write+0xb8/0x1a0
 [<ffffffff810830a1>] ? sigprocmask+0x71/0x110
 [<ffffffff8116d251>] sys_write+0x51/0x90
 [<ffffffff81013172>] system_call_fastpath+0x16/0x1b

Comment 3 chellwig@redhat.com 2010-10-15 21:06:15 UTC

This looks like random memory allocator freelist corruption.  I don't think the path that we reach it with has anything to do with the root cause.

Does it happen with different hardware and different configurations?

Comment 4 Amos Kong 2010-10-16 05:21:37 UTC

(In reply to comment #3)
> This looks like random memory allocator freelist corruption.  I don't think the
> path that we reach it with has anything to do with the root cause.
> 
> Does it happen with different hardware and different configurations?

Yes, it can be reproduced in different hosts, and different configuration.

Comment 5 chellwig@redhat.com 2010-10-17 01:09:55 UTC

Which makes it pretty clear it's not related to virtio.  Any chance you could check if there is any older guest kernel and/or qemu that does not show these issues?

Comment 7 Amos Kong 2010-10-18 03:34:36 UTC

I can reproduce this bug with those version combination.

guest kernel    qemu-kvm
------------    --------
2.6.32-76       0.12.1.2-2.113.el6_0.3
2.6.32-70       0.12.1.2-2.113.el6_0.3
2.6.32-66       0.12.1.2-2.113.el6_0.3
2.6.32-70       0.12.1.2-2.104.el6
2.6.32-70       0.12.1.2-2.97.el6

host kernel: 2.6.32-71.3.1

Comment 11 Michael S. Tsirkin 2011-01-03 10:07:17 UTC

Could be a duplicate of:
https://bugzilla.redhat.com/show_bug.cgi?id=647367

Can you check with host 2.6.32-83.el6 please?

Comment 12 Michael S. Tsirkin 2011-01-10 16:10:27 UTC

Is this fixed? Can we close?

Comment 13 Mike Cao 2011-01-11 02:02:30 UTC

(In reply to comment #12)
> Is this fixed? Can we close?

Hi,mst
I am working on it ,will check whether this issue is dup of bug #647367

Comment 14 Amos Kong 2011-01-13 07:55:54 UTC

(In reply to comment #12)
> Is this fixed? Can we close?

I've verified bug 658437.
https://bugzilla.redhat.com/show_bug.cgi?id=658437#c6

qemu-kvm-0.12.1.2-2.129.el6.x86_64
host kernel: kernel-2.6.32-94.el6

So can we duplicate this bug with 658437 ?

Comment 15 Amos Kong 2011-01-13 08:09:34 UTC

more detail: 

vhost: on
kernel-2.6.32-71.el6: host crash of bz #623915
kernel-2.6.32-94.el6: not reproduced

vhost: off
kernel-2.6.32-71.el6: reproduced
kernel-2.6.32-94.el6: not reproduced

Comment 16 Michael S. Tsirkin 2011-01-13 08:44:37 UTC

I think this is not a duplicate of 658437
as that deals with vhost-net issue only.

this one happens without vhost, so
I think this one is same as https://bugzilla.redhat.com/show_bug.cgi?id=647367

Comment 17 Michael S. Tsirkin 2011-01-31 22:23:34 UTC


*** This bug has been marked as a duplicate of bug 647367 ***

Note You need to log in before you can comment on or make changes to this bug.