Bug 619560

Summary: KVM VMs problems, disks not bootable, network cards not seen, hanging
Product: [Fedora] Fedora Reporter: Joshua Rosen <bjrosen>
Component: qemuAssignee: Justin M. Forbes <jforbes>
Status: CLOSED WONTFIX QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: high    
Version: 13CC: amit.shah, a.r.welter, berrange, clalance, dwmw2, ehabkost, extras-orphan, gcosta, itamar, jaswinder, jforbes, karlcz, knoel, markmc, notting, ondrejj, quintela, rh_bugzilla, scottt.tw, skothari, virt-maint
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-06-29 12:47:49 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
/var/log/messages file, Contains QEMU errors
none
libvirt log file
none
/var/log/messages
none
/var/log/libvirt/log
none
Bad .config file, used to build a 2.6.35.2 kernel that exhibits the VM problem
none
Good .config file, used to build a 2.6.35.2 kernel that doesn't have the VM problem none

Description Joshua Rosen 2010-07-29 20:12:04 UTC
Description of problem: I've started to experience a series of problems with KVM in the last day or two (it's now July 29). I'm running CentOS 5.5 and XP VMs on two Core2 systems. The problem that I'm seeing are,
1) Disks are unbootable. I've been able to fix this by deleting and then re-adding the virtual disks.
2) Virtual network cards not see.
3) Programs on VMs are hanging. Usually I can ssh into the VM for a while but eventually it the VM stops responding.
4) Can't access the virtual console. This happen on a system where I reverted back to the 124 kernel from the 147 kernel.


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:

Comment 1 Joshua Rosen 2010-07-30 16:44:23 UTC
Created attachment 435601 [details]
/var/log/messages file, Contains QEMU errors

Comment 2 Joshua Rosen 2010-07-31 14:53:45 UTC
I put F12 on the box with the CentOS5.5 VM and ran an overnight Verilog simulation on the VM. On F12 the VM behaved normally. The same VM on F13 consistently crashed when running this load. This probably eliminates a hardware problem. 

The system has multiple root partitions with a shared /home partition. I can reboot it into F13 if you want me to instrument something.

Comment 3 Joshua Rosen 2010-08-04 17:07:14 UTC
I just updated my iCore7 system and I'm getting the same problem. After the system was updated I was unable to boot my VMs because the virtual disks weren't seen. Deleting and reattaching the virtual disks fixed that problem.

After running simulations on the CentOS5.5 VM the VM seems to have hung.

I found this in /var/log/messages

Aug  3 20:54:31 localhost libvirtd: 20:54:31.622: warning : qemudParsePCIDeviceStrs:1411 : Unexpected exit status '1', qemu probably failed
Aug  3 20:54:31 localhost libvirtd: 20:54:31.758: error : qemudDomainGetVcpus:5801 : Requested operation is not valid: cannot list vcpu pinning for an inactive domain
Aug  3 20:54:31 localhost libvirtd: 20:54:31.760: error : qemudDomainGetVcpus:5801 : Requested operation is not valid: cannot list vcpu pinning for an inactive domain
Aug  3 20:54:31 localhost libvirtd: 20:54:31.761: error : qemudDomainGetVcpus:5801 : Requested operation is not valid: cannot list vcpu pinning for an inactive domain
Aug  3 20:54:31 localhost libvirtd: 20:54:31.763: error : qemudDomainGetVcpus:5801 : Requested operation is not valid: cannot list vcpu pinning for an inactive domain
Aug  3 20:54:31 localhost libvirtd: 20:54:31.764: error : qemudDomainGetVcpus:5801 : Requested operation is not valid: cannot list vcpu pinning for an inactive domain
Aug  3 20:54:31 localhost libvirtd: 20:54:31.766: error : qemudDomainGetVcpus:5801 : Requested operation is not valid: cannot list vcpu pinning for an inactive domain
Aug  3 20:54:31 localhost libvirtd: 20:54:31.767: error : qemudDomainGetVcpus:5801 : Requested operation is not valid: cannot list vcpu pinning for an inactive domain
Aug  3 20:54:31 localhost libvirtd: 20:54:31.768: error : qemudDomainGetVcpus:5801 : Requested operation is not valid: cannot list vcpu pinning for an inactive domain
Aug  3 20:54:31 localhost libvirtd: 20:54:31.770: error : qemudDomainGetVcpus:5801 : Requested operation is not valid: cannot list vcpu pinning for an inactive domain
Aug  3 20:54:31 localhost libvirtd: 20:54:31.771: error : qemudDomainGetVcpus:5801 : Requested operation is not valid: cannot list vcpu pinning for an inactive domain
Aug  3 20:54:31 localhost libvirtd: 20:54:31.773: error : qemudDomainGetVcpus:5801 : Requested operation is not valid: cannot list vcpu pinning for an inactive domain
Aug  3 20:54:48 localhost libvirtd: 20:54:48.189: warning : qemudParsePCIDeviceStrs:1411 : Unexpected exit status '1', qemu probably failed

Comment 4 Joshua Rosen 2010-08-05 13:09:56 UTC
This does appear to be a kernel problem that's fixed in later kernels. I built and installed a 2.6.35 kernel on my iCore7 system. So far so good, I have four simulation streams that have been running for the past 10 hours, the VM is still functioning properly.

Comment 5 Brian Lane 2010-08-05 23:03:38 UTC
Running f13 with  2.6.33.6-147.2.4.fc13.x86_64 (upgraded yesterday) and I'm seeing similar problems.

Comment 6 Patrick 2010-08-10 16:14:34 UTC
Just would like to point to bz619798 which is also about VMs hanging so you may want to monitor that too. The issues I'm seeing are similar to Joshua's in #3.

Comment 7 Joshua Rosen 2010-08-10 16:28:28 UTC
I have 2.6.35 on two machines now, an iCore7 and a Core2. I've been running heavily loaded CentOS 5.5 VMs on both systems for several days without a problem. Both systems had hanging VMs with the stock Fedora 13 kernel.

Comment 8 Patrick 2010-08-10 16:44:31 UTC
Thanks for the feedback Joshua. When you talk about 2.6.35 do I understand correctly that you mean a plain vanilla 2.6.35 kernel from kernel.org? If you used a plain vanilla 2.6.35 kernel would it be possible for you to test the latest F14 2.6.35 kernel from koji.fedoraproject.org ? It might help the developers if this is a plain vanilla kernel bug or something caused by the Fedora special sauce added to the kernel. Link to koji kernel packages: http://koji.fedoraproject.org/koji/packageinfo?packageID=8
You may need to upgrade linux-firmware to a newer version which can also be found on koji.

Comment 9 Joshua Rosen 2010-08-10 17:10:36 UTC
I see a 2.6.36.rc kernel in Rawhide and a 2.6.34 kernel in updates testing, I don't see a 2.6.35 kernel anywhere. I could give each one of them a try. My 2.6.35 is a vanilla kernel.org kernel. I configured it by coping the F13 .config file into the build directory, ran gnome-config and set the processor type to be Core2 or Later instead of the generic x86-64 and then built it.

Which Fedora kernel would you like me to try first?

Comment 10 Patrick 2010-08-10 17:18:21 UTC
I tried the 2.6.34 Fedora kernel and experienced the hangs. The rawhide one might be a bit too experimental but I'm not a kernel developer. Would be nice if any of the developers could give us some feedback what to test (hint). How about this one: kernel-2.6.35.1-5.rc1.fc14. You can download it here:
http://koji.fedoraproject.org/koji/buildinfo?buildID=189140

Comment 11 Joshua Rosen 2010-08-10 17:54:48 UTC
I've installed this one which is a 2.6.35.0, I'll reboot tonight and give it a try. 

http://koji.fedoraproject.org/koji/buildinfo?buildID=187655

Comment 12 Joshua Rosen 2010-08-12 12:06:00 UTC
The Koji 2.6.35.0 kernel seems to be working fine, give it a try to see if it fixes your problem.

Comment 13 Justin M. Forbes 2010-08-12 15:32:54 UTC
Can you please include:

qemu version: 
seabios version: 
qemu command line used (if using virt-manager/libvirt this will be in /var/logs/libvirt/qemu/<guestname>.log )

We should be able to start tracking down the issues with that information.

Comment 14 Joshua Rosen 2010-08-12 16:17:54 UTC
Created attachment 438475 [details]
libvirt log file

This is the latest log file. I'm currently running with a 2.6.35.0 kernel which works. I won't be able to revert back to the busted 2.6.13 kernel until Monday.

Linux reagan 2.6.35-0.fc14.x86_64 #1 SMP Mon Aug 2 01:07:47 UTC 2010 x86_64 x86_64 x86_64 GNU/Linux

Comment 15 Joshua Rosen 2010-08-12 16:19:39 UTC
QEMU PC emulator version 0.12.3 (qemu-kvm-0.12.3), Copyright (c) 2003-2008 Fabrice Bellard

It seems curious that the copyright date is 2008 on the F13 version of QEMU, is this correct?

Comment 16 Joshua Rosen 2010-08-12 16:27:03 UTC
seabios is not installed.

Comment 17 Joshua Rosen 2010-08-12 16:34:28 UTC
I've installed seabios, when I reboot on Monday with the F13 kernel I'll see if this makes a difference.

Comment 18 Joshua Rosen 2010-08-13 12:34:05 UTC
Created attachment 438671 [details]
/var/log/messages

This is from a 4th system, a Core2 laptop running the stock F13 kernels. I can't do a Force Off of this VM and I can't access it.

Linux hornet 2.6.33.6-147.2.4.fc13.x86_64 #1 SMP Fri Jul 23 17:14:44 UTC 2010 x86_64 x86_64 x86_64 GNU/Linux

QEMU PC emulator version 0.12.3 (qemu-kvm-0.12.3), Copyright (c) 2003-2008 Fabrice Bellard

Comment 19 Joshua Rosen 2010-08-13 12:35:18 UTC
Created attachment 438672 [details]
/var/log/libvirt/log

This is the libvirt log file for the Core2 laptop system

Comment 20 Joshua Rosen 2010-08-14 21:30:57 UTC
Created attachment 438803 [details]
Bad .config file, used to build a 2.6.35.2 kernel that exhibits the VM problem

I think I stumbled onto something. I built a 2.6.35.2 kernel for my Core2 laptop. Since it's a laptop system I selected the low latency preemptible kernel option instead of leaving it at the Voluntary Kernel option. The resulting kernel resulted in hanging VMs. In fact the problem seems to be worse. I then built another 2.6.35.2 kernel using the .config file that I used to build the 2.6.35 kernel that I reported on earlier. The second 2.6.35.2 kernel works fine.

I'm attaching the good and bad .config files.

Comment 21 Joshua Rosen 2010-08-14 21:32:46 UTC
Created attachment 438804 [details]
Good .config file, used to build a 2.6.35.2 kernel that doesn't have the VM problem

This .config file builds a kernel which doesn't exhibit the hanging VM problem.

Comment 22 Need Real Name 2010-08-18 17:24:27 UTC
I am also seeing KVM guest hangs with up to date CentOS 5.5 guests on up to date Fedora 13, on an Intel Xeon E5530 based host.

After trying every tweak found via the web (changing virtio to other emulated disk and NIC types, turning off APIC in VM machine settings, trying the centos-plus kernel instead of the regular guest kernel) it seems changing to 1 VCPU has made the guest more stable.

It has now been running for over 24 hours, where it would freeze up in strange ways after only an hour or two of light development use. (The CentOS guest would not emit unusual log messages, but some system calls seemed to block, blocking new logins, blocking disk flush/sync from apps, and blocking 'top' while 'ps' still worked. The only recourse was forced power off of the guest.)

Meanwhile, a Windows XP guest with 2 VCPUs seems to run without issue on the same host.

Comment 23 Patrick 2010-08-18 17:30:59 UTC
I have been running the 2.6.35.1-5.rc1.fc14.x86_64 kernel on F13 for about two weeks now without any issue running CentOS 5.5 x86_64 guests with 2 VCPUs and 2GB mem.

Comment 24 Joshua Rosen 2010-08-18 17:45:19 UTC
Has anyone else tried building a kernel with the low latency option to see if it breaks?

Comment 25 Joshua Rosen 2010-08-19 15:43:39 UTC
I just experienced the problem with the Linux reagan 2.6.35-0.fc14.x86_64 kernel. I've disabled ksm and ksmtuned and I'm rebooting. I'll let you know if this helps.

Comment 26 Joshua Rosen 2010-08-29 16:43:53 UTC
My CentOS 5.5 VM has crashed twice on a 2.6.35.2 kernel. I've introduced a new factor, I've added a 128G virtual disk, mounted as IDE. I'm running my Verilog simulations on the virtual disk instead of on NFS mounted directories as I was doing before. After several hours of simulations the VM crashes. This has happened twice. I don't see any errors in the host machine's logs however in the VM's /var/log/messages I see the following,


Aug  29 03:24:52 localhost kernel: BUG: soft lockup - CPU#1 stuck for 18s! [swapper:0]
Aug  29 03:24:52 localhost kernel: CPU 1:
Aug  29 03:24:52 localhost kernel: Modules linked in: ipv6 xfrm_nalgo crypto_api autofs4 hidp nfs fscache nfs_acl rfcomm l2cap bluetooth lockd sunrpc dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport floppy snd_ens1370 gameport snd_rawmidi snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm i2c_piix4 snd_timer snd_ak4531_codec serio_raw virtio_balloon snd soundcore 8139too virtio_pci virtio_ring virtio i2c_core snd_page_alloc 8139cp mii pcspkr dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod ata_piix libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Aug  29 03:24:52 localhost kernel: Pid: 0, comm: swapper Not tainted 2.6.18-194.11.1.el5 #1
Aug  29 03:24:52 localhost kernel: RIP: 0010:[<ffffffff8004f4db>]  [<ffffffff8004f4db>] hrtimer_run_queues+0xb4/0x197
Aug  29 03:24:52 localhost kernel: RSP: 0018:ffff810108637ed0  EFLAGS: 00000282
Aug  29 03:24:52 localhost kernel: RAX: 0000000225c17d03 RBX: 0000000000000001 RCX: 000000002aae83f7
Aug  29 03:24:52 localhost kernel: RDX: ffffffffb6743cec RSI: 111fb9681838d2b0 RDI: 7fffffffffffffff
Aug  29 03:24:52 localhost kernel: RBP: ffff810108637e50 R08: 0000000001786599 R09: ffff81010862fe48
Aug  29 03:24:52 localhost kernel: R10: 0000000000000001 R11: ffff8101ba523c20 R12: ffffffff8005dc8e
Aug  29 03:24:52 localhost kernel: R13: 0000000000000046 R14: ffffffff80078203 R15: ffff810108637e50
Aug  29 03:24:52 localhost kernel: FS:  0000000000000000(0000) GS:ffff8101085f57c0(0000) knlGS:0000000000000000
Aug  29 03:24:52 localhost kernel: CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
Aug  29 03:25:05 localhost kernel: CR2: 0000000000b23370 CR3: 00000001b9ecb000 CR4: 00000000000006e0
Aug  29 03:43:06 localhost kernel: 
Aug  29 03:43:07 localhost kernel: Call Trace:
Aug  29 03:43:08 localhost kernel:  <IRQ>  [<ffffffff8009793e>] run_timer_softirq+0x27/0x241
Aug  29 03:43:08 localhost kernel:  [<ffffffff800123b4>] __do_softirq+0x89/0x133
Aug  29 03:43:09 localhost kernel:  [<ffffffff8005e2fc>] call_softirq+0x1c/0x28
Aug  29 03:43:09 localhost kernel:  [<ffffffff8006cb8e>] do_softirq+0x2c/0x85
Aug  29 03:43:09 localhost kernel:  [<ffffffff8006b346>] default_idle+0x0/0x50
Aug  29 03:43:09 localhost kernel:  [<ffffffff8005dc8e>] apic_timer_interrupt+0x66/0x6c
Aug  29 03:43:09 localhost kernel:  <EOI>  [<ffffffff8006b36f>] default_idle+0x29/0x50
Aug  29 03:43:10 localhost kernel:  [<ffffffff800491ab>] cpu_idle+0x95/0xb8
Aug  29 03:43:10 localhost kernel:  [<ffffffff8007796f>] start_secondary+0x498/0x4a7
Aug  29 03:43:10 localhost kernel: 
Aug  29 03:43:10 localhost kernel: BUG: soft lockup - CPU#6 stuck for 17s! [swapper:0]
Aug  29 03:43:10 localhost kernel:  DriveReady Error }
Aug  29 03:43:11 localhost kernel: hdb: task_out_intr: error=0x04 { DriveStatusError }
Aug  29 03:43:11 localhost kernel: ide: failed opcode was: unknown
Aug  29 03:54:03 localhost kernel: hdb: task_out_intr: status=0x41 { DriveReady Error }
Aug  29 03:54:04 localhost kernel: hdb: task_out_intr: error=0x04 { DriveStatusError }
Aug  29 03:54:04 localhost kernel: ide: failed opcode was: unknown
Aug  29 03:54:05 localhost kernel: hdb: task_out_intr: status=0x41 { DriveReady Error }
Aug  29 03:54:05 localhost kernel: hdb: task_out_intr: error=0x04 { DriveStatusError }
Aug  29 03:54:06 localhost kernel: ide: failed opcode was: unknown
Aug  29 03:58:40 localhost kernel: hdb: task_out_intr: status=0x41 { DriveReady Error }
Aug  29 03:58:40 localhost kernel: hdb: task_out_intr: error=0x04 { DriveStatusError }
Aug  29 03:58:40 localhost kernel: ide: failed opcode was: unknown
Aug  29 03:58:41 localhost kernel: hdb: task_out_intr: status=0x41 { DriveReady Error }
Aug  29 03:58:41 localhost kernel: hdb: task_out_intr: error=0x04 { DriveStatusError }
Aug  29 03:58:41 localhost kernel: ide: failed opcode was: unknown
Aug  29 03:58:41 localhost kernel: hdb: task_out_intr: status=0x41 { DriveReady Error }
Aug  29 03:58:42 localhost kernel: hdb: task_out_intr: error=0x04 { DriveStatusError }
Aug  29 03:58:42 localhost kernel: ide: failed opcode was: unknown
Aug  29 03:58:42 localhost kernel: hdb: task_out_intr: status=0x41 { DriveReady Error }
Aug  29 03:58:42 localhost kernel: hdb: task_out_intr: error=0x04 { DriveStatusError }
Aug  29 03:58:42 localhost kernel: ide: failed opcode was: unknown
Aug  29 03:58:42 localhost kernel: hdb: task_out_intr: status=0x41 { DriveReady Error }
Aug  29 03:58:43 localhost kernel: hdb: task_out_intr: error=0x04 { DriveStatusError }
Aug  29 03:58:43 localhost kernel: ide: failed opcode was: unknown
Aug  29 03:58:43 localhost kernel: hdb: task_out_intr: status=0x41 { DriveReady Error }
Aug  29 03:58:43 localhost kernel: hdb: task_out_intr: error=0x04 { DriveStatusError }
Aug  29 03:58:43 localhost kernel: ide: failed opcode was: unknown
Aug  29 03:58:44 localhost kernel: hdb: task_out_intr: status=0x41 { DriveReady Error }
Aug  29 03:58:44 localhost kernel: hdb: task_out_intr: error=0x04 { DriveStatusError }
Aug  29 03:58:44 localhost kernel: ide: failed opcode was: unknown
Aug  29 03:58:44 localhost kernel: hdb: task_out_intr: status=0x41 { DriveReady Error }
Aug  29 03:58:45 localhost kernel: hdb: task_out_intr: error=0x04 { DriveStatusError }
Aug  29 03:58:45 localhost kernel: ide: failed opcode was: unknown
Aug  29 03:58:45 localhost kernel: hdb: task_out_intr: status=0x41 { DriveReady Error }
Aug  29 03:58:45 localhost kernel: hdb: task_out_intr: error=0x04 { DriveStatusError }
Aug  29 03:58:46 localhost kernel: ide: failed opcode was: unknown
Aug  29 03:58:46 localhost kernel: hdb: task_out_intr: status=0x41 { DriveReady Error }
Aug  29 03:58:46 localhost kernel: hdb: task_out_intr: error=0x04 { DriveStatusError }
Aug  29 03:58:46 localhost kernel: ide: failed opcode was: unknown

Comment 27 Joshua Rosen 2010-08-29 22:17:13 UTC
I tried mounting the virtual disk as virtio instead of IDE, the VM crashed as before,

Aug 29 00:13:29 localhost kernel: Pid: 0, comm: swapper Not tainted 2.6.18-194.11.1.el5 #1
Aug 29 00:13:29 localhost kernel: RIP: 0010:[<ffffffff881eb5cb>]  [<ffffffff881eb5cb>] :8139cp:cp_interrupt+0x2c/0x362
Aug 29 00:13:29 localhost kernel: RSP: 0018:ffff8101087afd18  EFLAGS: 00010286
Aug 29 00:13:29 localhost kernel: RAX: 0000000000000001 RBX: ffff81025c1ba500 RCX: 0000000000000001
Aug 29 00:13:29 localhost kernel: RDX: ffffc20000028200 RSI: ffff81025c1ba000 RDI: 000000000000000b
Aug 29 00:13:29 localhost kernel: RBP: ffff8101087afc90 R08: ffffffff8032aad8 R09: 0000000000001000
Aug 29 00:13:29 localhost kernel: R10: ffff810240d821b8 R11: ffffffff800c6944 R12: ffffffff8005dc8e
Aug 29 00:13:29 localhost kernel: R13: 0000000000000080 R14: ffffffff80078203 R15: ffff8101087afc90
Aug 29 00:13:29 localhost kernel: FS:  0000000000000000(0000) GS:ffff810108753340(0000) knlGS:0000000000000000
Aug 29 00:13:29 localhost kernel: CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
Aug 29 00:13:29 localhost kernel: CR2: 00000000087d8d60 CR3: 0000000247563000 CR4: 00000000000006e0
Aug 29 00:13:29 localhost kernel: 
Aug 29 00:13:29 localhost kernel: Call Trace:
Aug 29 00:13:29 localhost kernel:  <IRQ>  [<ffffffff881a3082>] :virtio_ring:vring_interrupt+0x26/0x2f
Aug 29 00:13:30 localhost kernel:  [<ffffffff881ae107>] :virtio_pci:vp_interrupt+0xa9/0xb8
Aug 29 00:13:30 localhost kernel:  [<ffffffff80010bab>] handle_IRQ_event+0x51/0xa6
Aug 29 00:13:30 localhost kernel:  [<ffffffff800bae28>] __do_IRQ+0xa4/0x103
Aug 29 00:13:30 localhost kernel:  [<ffffffff8006ca11>] do_IRQ+0xe7/0xf5
Aug 29 00:13:30 localhost kernel:  [<ffffffff8005d615>] ret_from_intr+0x0/0xa
Aug 29 00:13:31 localhost kernel:  [<ffffffff881ed280>] :8139cp:cp_rx_poll+0x535/0x58c
Aug 29 00:13:31 localhost kernel:  [<ffffffff8000c88c>] net_rx_action+0xac/0x1e0
Aug 29 00:13:31 localhost kernel:  [<ffffffff800123b4>] __do_softirq+0x89/0x133
Aug 29 00:13:31 localhost kernel:  [<ffffffff8005e2fc>] call_softirq+0x1c/0x28
Aug 29 00:13:31 localhost kernel:  [<ffffffff8006cb8e>] do_softirq+0x2c/0x85
Aug 29 00:13:32 localhost kernel:  [<ffffffff8006ca16>] do_IRQ+0xec/0xf5
Aug 29 00:13:32 localhost kernel:  [<ffffffff8006b346>] default_idle+0x0/0x50
Aug 29 00:13:32 localhost kernel:  [<ffffffff8005d615>] ret_from_intr+0x0/0xa
Aug 29 00:13:32 localhost kernel:  <EOI>  [<ffffffff8006b36f>] default_idle+0x29/0x50
Aug 29 00:13:32 localhost kernel:  [<ffffffff800491ab>] cpu_idle+0x95/0xb8
Aug 29 00:13:32 localhost kernel:  [<ffffffff8007796f>] start_secondary+0x498/0x4a7
Aug 29 00:13:33 localhost kernel: 
Aug 29 01:13:01 localhost kernel: INFO: task ncvlog:11720 blocked for more than 120 seconds.
Aug 29 01:13:01 localhost kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 29 01:13:01 localhost kernel: ncvlog        D ffff81000103f1a0     0 11720  11715                     (NOTLB)
Aug 29 01:13:01 localhost kernel:  ffff810117de1d88 0000000000000082 ffff810117de1d98 ffffffff80062ff8
Aug 29 01:13:01 localhost kernel:  0000000000001000 0000000000000008 ffff810170c497a0 ffff810108764100
Aug 29 01:13:01 localhost kernel:  000003dd5444933d 00000000000007c1 ffff810170c49988 0000000700084cf3
Aug 29 01:13:01 localhost kernel: Call Trace:
Aug 29 01:13:01 localhost kernel:  [<ffffffff80062ff8>] thread_return+0x62/0xfe
Aug 29 01:13:01 localhost kernel:  [<ffffffff88036d8a>] :jbd:log_wait_commit+0xa3/0xf5
Aug 29 01:13:01 localhost kernel:  [<ffffffff800a09d8>] autoremove_wake_function+0x0/0x2e
Aug 29 01:13:01 localhost kernel:  [<ffffffff80097cc5>] process_timeout+0x0/0x5
Aug 29 01:13:01 localhost kernel:  [<ffffffff8803178a>] :jbd:journal_stop+0x1cf/0x1ff
Aug 29 01:13:01 localhost kernel:  [<ffffffff8002fbdf>] __writeback_single_inode+0x1e9/0x328
Aug 29 01:13:02 localhost kernel:  [<ffffffff800f333b>] sync_inode+0x24/0x33
Aug 29 01:13:02 localhost kernel:  [<ffffffff8804c36d>] :ext3:ext3_sync_file+0xc9/0xdc
Aug 29 01:13:03 localhost kernel:  [<ffffffff80050127>] do_fsync+0x52/0xa4
Aug 29 01:13:03 localhost kernel:  [<ffffffff800e0f84>] __do_fsync+0x23/0x36
Aug 29 01:13:03 localhost kernel:  [<ffffffff8006149d>] sysenter_do_call+0x1e/0x76
Aug 29 01:13:03 localhost kernel: 
Aug 29 01:13:04 localhost kernel: INFO: task ncvlog:11721 blocked for more than 120 seconds.
Aug 29 01:13:04 localhost kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 29 01:13:04 localhost kernel: ncvlog        D ffff810001004420     0 11721  11713                     (NOTLB)
Aug 29 01:13:05 localhost kernel:  ffff810117debd88 0000000000000082 ffff810117debd98 ffffffff80062ff8
Aug 29 01:13:05 localhost kernel:  0000000000001000 0000000000000008 ffff81025e44f080 ffffffff80308b60
Aug 29 01:13:05 localhost kernel:  000003dd4b81a2b6 0000000000000729 ffff81025e44f268 000000000007bf51
Aug 29 01:13:06 localhost kernel: Call Trace:
Aug 29 01:13:06 localhost kernel:  [<ffffffff80062ff8>] thread_return+0x62/0xfe
Aug 29 01:13:07 localhost kernel:  [<ffffffff88036d8a>] :jbd:log_wait_commit+0xa3/0xf5
Aug 29 01:13:07 localhost kernel:  [<ffffffff800a09d8>] autoremove_wake_function+0x0/0x2e
Aug 29 01:13:07 localhost kernel:  [<ffffffff8803178a>] :jbd:journal_stop+0x1cf/0x1ff
Aug 29 01:13:07 localhost kernel:  [<ffffffff8002fbdf>] __writeback_single_inode+0x1e9/0x328
Aug 29 01:13:07 localhost kernel:  [<ffffffff800f333b>] sync_inode+0x24/0x33
Aug 29 01:13:08 localhost kernel:  [<ffffffff8804c36d>] :ext3:ext3_sync_file+0xc9/0xdc
Aug 29 01:13:08 localhost kernel:  [<ffffffff80050127>] do_fsync+0x52/0xa4
Aug 29 01:13:08 localhost kernel:  [<ffffffff800e0f84>] __do_fsync+0x23/0x36
Aug 29 01:13:08 localhost kernel:  [<ffffffff8006149d>] sysenter_do_call+0x1e/0x76
Aug 29 01:13:08 localhost kernel: 
Aug 29 01:18:55 localhost kernel: BUG: soft lockup - CPU#0 stuck for 10s! [ncsim:18812]
Aug 29 01:18:55 localhost kernel: CPU 0:
Aug 29 01:18:55 localhost kernel: Modules linked in: ipv6 xfrm_nalgo crypto_api autofs4 hidp nfs fscache nfs_acl rfcomm l2cap bluetooth lockd sunrpc dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport floppy snd_ens1370 gameport snd_rawmidi snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_ak4531_codec snd 8139too i2c_piix4 soundcore virtio_blk 8139cp virtio_balloon i2c_core snd_page_alloc serio_raw virtio_pci virtio_ring virtio mii pcspkr dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod ata_piix libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Aug 29 01:18:55 localhost kernel: Pid: 18812, comm: ncsim Not tainted 2.6.18-194.11.1.el5 #1
Aug 29 01:18:55 localhost kernel: RIP: 0010:[<ffffffff881ecd88>]  [<ffffffff881ecd88>] :8139cp:cp_rx_poll+0x3d/0x58c
Aug 29 01:18:55 localhost kernel: RSP: 0018:ffffffff80446ea0  EFLAGS: 00010202
Aug 29 01:18:55 localhost kernel: RAX: ffffc20000016100 RBX: ffff81025ca1e500 RCX: ffff81025ca1e6b8
Aug 29 01:18:55 localhost kernel: RDX: 0000000000000003 RSI: ffff81025fe0b680 RDI: 0000000000000000
Aug 29 01:18:55 localhost kernel: RBP: ffffffff80446e20 R08: ffff810259625d00 R09: 0000000000000010
Aug 29 01:18:55 localhost kernel: R10: ffff81010872e738 R11: 0000000000000000 R12: ffffffff8005dc8e
Aug 29 01:18:55 localhost kernel: R13: ffff81024cb19880 R14: ffffffff80078203 R15: ffffffff80446e20
Aug 29 01:18:55 localhost kernel: FS:  0000000000000000(0000) GS:ffffffff803ca000(0063) knlGS:00000000f7f9a6c0
Aug 29 01:18:55 localhost kernel: CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
Aug 29 01:18:58 localhost kernel: CR2: 00007fff5bcc3ec8 CR3: 000000012cb9f000 CR4: 00000000000006e0
Aug 29 01:18:58 localhost kernel: 
Aug 29 01:18:59 localhost kernel: Call Trace:
Aug 29 01:19:01 localhost kernel:  <IRQ>  [<ffffffff8000c88c>] net_rx_action+0xac/0x1e0
Aug 29 01:19:01 localhost kernel:  [<ffffffff800123b4>] __do_softirq+0x89/0x133
Aug 29 01:19:02 localhost kernel:  [<ffffffff8005e2fc>] call_softirq+0x1c/0x28
Aug 29 01:19:03 localhost kernel:  [<ffffffff8006cb8e>] do_softirq+0x2c/0x85
Aug 29 01:19:04 localhost kernel:  [<ffffffff8005dc8e>] apic_timer_interrupt+0x66/0x6c
Aug 29 01:19:04 localhost kernel:  <EOI>  [<ffffffff8014e118>] cfq_may_queue+0x0/0x112
Aug 29 01:19:04 localhost kernel:  [<ffffffff8804e080>] :ext3:ext3_journal_dirty_data+0x0/0x34
Aug 29 01:19:05 localhost kernel:  [<ffffffff800133bb>] get_request+0x186/0x36f
Aug 29 01:19:05 localhost kernel:  [<ffffffff80013265>] get_request+0x30/0x36f
Aug 29 01:19:05 localhost kernel:  [<ffffffff80028b6f>] get_request_wait+0x2a/0x130
Aug 29 01:19:05 localhost kernel:  [<ffffffff8000c041>] __make_request+0x350/0x414
Aug 29 01:19:06 localhost kernel:  [<ffffffff8000b3e7>] __find_get_block+0x15c/0x16c
Aug 29 01:19:06 localhost kernel:  [<ffffffff8001c1c1>] generic_make_request+0x211/0x228
Aug 29 01:19:06 localhost kernel:  [<ffffffff80019b79>] __getblk+0x25/0x236
Aug 29 01:19:06 localhost kernel:  [<ffffffff800332f4>] submit_bio+0xe4/0xeb
Aug 29 01:19:06 localhost kernel:  [<ffffffff8001a90b>] submit_bh+0xf1/0x111
Aug 29 01:19:06 localhost kernel:  [<ffffffff8003a8d4>] sync_dirty_buffer+0x7a/0xcb
Aug 29 01:19:06 localhost kernel:  [<ffffffff88032ba8>] :jbd:journal_dirty_data+0x100/0x205
Aug 29 01:19:06 localhost kernel:  [<ffffffff8804e08f>] :ext3:ext3_journal_dirty_data+0xf/0x34
Aug 29 01:19:06 localhost kernel:  [<ffffffff8804d4b8>] :ext3:walk_page_buffers+0x65/0x8b
Aug 29 01:19:06 localhost kernel:  [<ffffffff8804e080>] :ext3:ext3_journal_dirty_data+0x0/0x34
Aug 29 01:19:07 localhost kernel:  [<ffffffff8804ff3d>] :ext3:ext3_ordered_write_end+0x7c/0x116
Aug 29 01:19:07 localhost kernel:  [<ffffffff8000fd6c>] generic_file_buffered_write+0x1cc/0x675
Aug 29 01:19:07 localhost kernel:  [<ffffffff880317ae>] :jbd:journal_stop+0x1f3/0x1ff
Aug 29 01:19:07 localhost kernel:  [<ffffffff8001660f>] __generic_file_aio_write_nolock+0x369/0x3b6
Aug 29 01:19:07 localhost kernel:  [<ffffffff8000ae3c>] ia32_setup_sigcontext+0x141/0x1ef
Aug 29 01:19:07 localhost kernel:  [<ffffffff800217b2>] generic_file_aio_write+0x65/0xc1
Aug 29 01:19:07 localhost kernel:  [<ffffffff8804c1b6>] :ext3:ext3_file_write+0x16/0x91
Aug 29 01:19:07 localhost kernel:  [<ffffffff80018234>] do_sync_write+0xc7/0x104
Aug 29 01:19:07 localhost kernel:  [<ffffffff800a09d8>] autoremove_wake_function+0x0/0x2e
Aug 29 01:19:08 localhost kernel:  [<ffffffff800624b6>] __sched_text_start+0xf6/0xbd6
Aug 29 01:19:08 localhost kernel:  [<ffffffff80033713>] do_setitimer+0x171/0x55a
Aug 29 01:19:08 localhost kernel:  [<ffffffff80016a17>] vfs_write+0xce/0x174
Aug 29 01:19:08 localhost kernel:  [<ffffffff800172e4>] sys_write+0x45/0x6e
Aug 29 01:19:08 localhost kernel:  [<ffffffff8006149d>] sysenter_do_call+0x1e/0x76
Aug 29 01:19:08 localhost kernel: 
Aug 29 01:27:02 localhost kernel: BUG: soft lockup - CPU#7 stuck for 10s! [ncvlog:19815]
Aug 29 01:27:02 localhost kernel: CPU 7:
Aug 29 01:27:02 localhost kernel: Modules linked in: ipv6 xfrm_nalgo crypto_api autofs4 hidp nfs fscache nfs_acl rfcomm l2cap bluetooth lockd sunrpc dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport floppy snd_ens1370 gameport snd_rawmidi snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_ak4531_codec snd 8139too i2c_piix4 soundcore virtio_blk 8139cp virtio_balloon i2c_core snd_page_alloc serio_raw virtio_pci virtio_ring virtio mii pcspkr dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod ata_piix libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Aug 29 01:27:02 localhost kernel: Pid: 19815, comm: ncvlog Not tainted 2.6.18-194.11.1.el5 #1
Aug 29 01:27:02 localhost kernel: RIP: 0010:[<ffffffff881eb687>]  [<ffffffff881eb687>] :8139cp:cp_interrupt+0xe8/0x362
Aug 29 01:27:02 localhost kernel: RSP: 0018:ffff8101087afd38  EFLAGS: 00010246
Aug 29 01:27:02 localhost kernel: RAX: ffffc200000280ac RBX: ffff81025c1ba500 RCX: 0000000000000001
Aug 29 01:27:02 localhost kernel: RDX: ffffc20000028200 RSI: ffff81025c1ba000 RDI: ffff81025c1ba000
Aug 29 01:27:02 localhost kernel: RBP: ffff8101087afcb0 R08: ffffffff8032aad8 R09: 0000000000001000
Aug 29 01:27:02 localhost kernel: R10: ffff81016b9b5178 R11: ffffffff800c6944 R12: ffffffff8005dc8e
Aug 29 01:27:02 localhost kernel: R13: 0000000000000080 R14: ffffffff80078203 R15: ffff8101087afcb0
Aug 29 01:27:02 localhost kernel: FS:  0000000000000000(0000) GS:ffff810108753340(0063) knlGS:00000000f7f6c6c0
Aug 29 01:27:02 localhost kernel: CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
Aug 29 01:27:02 localhost kernel: CR2: 00007fff25944000 CR3: 00000001802bd000 CR4: 00000000000006e0
Aug 29 01:27:03 localhost kernel: 
Aug 29 01:27:03 localhost kernel: Call Trace:
Aug 29 01:27:04 localhost kernel:  <IRQ>  [<ffffffff881eb62c>] :8139cp:cp_interrupt+0x8d/0x362
Aug 29 01:27:04 localhost kernel:  [<ffffffff881ae107>] :virtio_pci:vp_interrupt+0xa9/0xb8
Aug 29 01:27:04 localhost kernel:  [<ffffffff80010bab>] handle_IRQ_event+0x51/0xa6
Aug 29 01:27:04 localhost kernel:  [<ffffffff800bae28>] __do_IRQ+0xa4/0x103
Aug 29 01:27:05 localhost kernel:  [<ffffffff8006ca11>] do_IRQ+0xe7/0xf5
Aug 29 01:27:05 localhost kernel:  [<ffffffff8005d615>] ret_from_intr+0x0/0xa
Aug 29 01:27:05 localhost kernel:  [<ffffffff881ed280>] :8139cp:cp_rx_poll+0x535/0x58c
Aug 29 01:27:05 localhost kernel:  [<ffffffff8001549c>] sync_buffer+0x0/0x3f
Aug 29 01:27:06 localhost kernel:  [<ffffffff8000c88c>] net_rx_action+0xac/0x1e0
Aug 29 01:27:06 localhost kernel:  [<ffffffff800123b4>] __do_softirq+0x89/0x133
Aug 29 01:27:06 localhost kernel:  [<ffffffff8005e2fc>] call_softirq+0x1c/0x28
Aug 29 01:27:06 localhost kernel:  [<ffffffff8006cb8e>] do_softirq+0x2c/0x85
Aug 29 01:27:07 localhost kernel:  [<ffffffff8005dc8e>] apic_timer_interrupt+0x66/0x6c
Aug 29 01:27:07 localhost kernel:  <EOI>  [<ffffffff881ae53d>] :virtio_pci:vp_notify+0x0/0x1c
Aug 29 01:27:07 localhost kernel:  [<ffffffff8005a7db>] generic_unplug_device+0x31/0x32
Aug 29 01:27:07 localhost kernel:  [<ffffffff800154d2>] sync_buffer+0x36/0x3f
Aug 29 01:27:08 localhost kernel:  [<ffffffff80063a16>] __wait_on_bit+0x40/0x6e
Aug 29 01:27:08 localhost kernel:  [<ffffffff8001549c>] sync_buffer+0x0/0x3f
Aug 29 01:27:08 localhost kernel:  [<ffffffff80063ab0>] out_of_line_wait_on_bit+0x6c/0x78
Aug 29 01:27:08 localhost kernel:  [<ffffffff800a0a06>] wake_bit_function+0x0/0x23
Aug 29 01:27:09 localhost kernel:  [<ffffffff8003a8f0>] sync_dirty_buffer+0x96/0xcb
Aug 29 01:27:09 localhost kernel:  [<ffffffff88032ba8>] :jbd:journal_dirty_data+0x100/0x205
Aug 29 01:27:09 localhost kernel:  [<ffffffff8804e08f>] :ext3:ext3_journal_dirty_data+0xf/0x34
Aug 29 01:27:09 localhost kernel:  [<ffffffff8804d4b8>] :ext3:walk_page_buffers+0x65/0x8b
Aug 29 01:27:09 localhost kernel:  [<ffffffff8804e080>] :ext3:ext3_journal_dirty_data+0x0/0x34
Aug 29 01:27:10 localhost kernel:  [<ffffffff8804ff3d>] :ext3:ext3_ordered_write_end+0x7c/0x116
Aug 29 01:27:10 localhost kernel:  [<ffffffff8000fd6c>] generic_file_buffered_write+0x1cc/0x675
Aug 29 01:27:10 localhost kernel:  [<ffffffff8805c1a2>] :ext3:ext3_get_acl+0x63/0x310
Aug 29 01:27:10 localhost kernel:  [<ffffffff8001660f>] __generic_file_aio_write_nolock+0x369/0x3b6
Aug 29 01:27:10 localhost kernel:  [<ffffffff8000a73e>] __link_path_walk+0xf32/0xf5b
Aug 29 01:27:11 localhost kernel:  [<ffffffff800217b2>] generic_file_aio_write+0x65/0xc1
Aug 29 01:27:11 localhost kernel:  [<ffffffff8804c1b6>] :ext3:ext3_file_write+0x16/0x91
Aug 29 01:27:11 localhost kernel:  [<ffffffff80018234>] do_sync_write+0xc7/0x104
Aug 29 01:27:11 localhost kernel:  [<ffffffff80066b88>] do_page_fault+0x4fe/0x874
Aug 29 01:27:12 localhost kernel:  [<ffffffff800a09d8>] autoremove_wake_function+0x0/0x2e
Aug 29 01:27:12 localhost kernel:  [<ffffffff800274b2>] do_filp_open+0x2a/0x38
Aug 29 01:27:12 localhost kernel:  [<ffffffff80016a17>] vfs_write+0xce/0x174
Aug 29 01:27:12 localhost kernel:  [<ffffffff800172e4>] sys_write+0x45/0x6e
Aug 29 01:27:12 localhost kernel:  [<ffffffff8006149d>] sysenter_do_call+0x1e/0x76
Aug 29 01:27:12 localhost kernel: 
Aug 29 01:30:45 localhost kernel: BUG: soft lockup - CPU#7 stuck for 10s! [swapper:0]
Aug 29 01:30:45 localhost kernel: CPU 7:
Aug 29 01:30:45 localhost kernel: Modules linked in: ipv6 xfrm_nalgo crypto_api autofs4 hidp nfs fscache nfs_acl rfcomm l2cap bluetooth lockd sunrpc dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport floppy snd_ens1370 gameport snd_rawmidi snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_ak4531_codec snd 8139too i2c_piix4 soundcore virtio_blk 8139cp virtio_balloon i2c_core snd_page_alloc serio_raw virtio_pci virtio_ring virtio mii pcspkr dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod ata_piix libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Aug 29 01:30:45 localhost kernel: Pid: 0, comm: swapper Not tainted 2.6.18-194.11.1.el5 #1
Aug 29 01:30:45 localhost kernel: RIP: 0010:[<ffffffff80064b50>]  [<ffffffff80064b50>] _spin_unlock_irqrestore+0x8/0x9
Aug 29 01:30:45 localhost kernel: RSP: 0018:ffff8101087af870  EFLAGS: 00000246
Aug 29 01:30:45 localhost kernel: RAX: ffff81025c8a92a8 RBX: ffff81025c8a9000 RCX: 0000000000000001
Aug 29 01:30:45 localhost kernel: RDX: ffff81025c8a92a8 RSI: 0000000000000246 RDI: ffff81025c8a92a0
Aug 29 01:30:45 localhost kernel: RBP: ffff8101087af7f0 R08: 0000000000000000 R09: ffff81025b18a000
Aug 29 01:30:45 localhost kernel: R10: 00000000000000c0 R11: ffffffff881ae53d R12: ffffffff8005dc8e
Aug 29 01:30:45 localhost kernel: R13: ffff81025c8a9290 R14: ffffffff80078203 R15: ffff8101087af7f0
Aug 29 01:30:45 localhost kernel: FS:  0000000000000000(0000) GS:ffff810108753340(0000) knlGS:0000000000000000
Aug 29 01:30:45 localhost kernel: CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
Aug 29 01:30:46 localhost kernel: CR2: 0000003d3ba9a490 CR3: 0000000000201000 CR4: 00000000000006e0
Aug 29 01:30:46 localhost kernel: 
Aug 29 01:30:46 localhost kernel: Call Trace:
Aug 29 01:30:46 localhost kernel:  <IRQ>  [<ffffffff881ae107>] :virtio_pci:vp_interrupt+0xa9/0xb8
Aug 29 01:30:47 localhost kernel:  [<ffffffff80010bab>] handle_IRQ_event+0x51/0xa6
Aug 29 01:30:47 localhost kernel:  [<ffffffff800bae28>] __do_IRQ+0xa4/0x103
Aug 29 01:30:47 localhost kernel:  [<ffffffff8006ca11>] do_IRQ+0xe7/0xf5
Aug 29 01:30:47 localhost kernel:  [<ffffffff8005d615>] ret_from_intr+0x0/0xa
Aug 29 01:30:47 localhost kernel:  [<ffffffff881ed7c6>] :8139cp:cp_start_xmit+0x4ef/0x511
Aug 29 01:30:48 localhost kernel:  [<ffffffff8022f296>] dev_hard_start_xmit+0x1b7/0x28a
Aug 29 01:30:48 localhost kernel:  [<ffffffff8023f5f8>] __qdisc_run+0x136/0x1f9
Aug 29 01:30:48 localhost kernel:  [<ffffffff8002f872>] dev_queue_xmit+0x150/0x271
Aug 29 01:30:48 localhost kernel:  [<ffffffff80031f5a>] ip_output+0x29a/0x2dd
Aug 29 01:30:48 localhost kernel:  [<ffffffff8003456f>] ip_queue_xmit+0x434/0x48e
Aug 29 01:30:49 localhost kernel:  [<ffffffff88423476>] :sunrpc:__rpc_do_wake_up_task+0x15a/0x180
Aug 29 01:30:49 localhost kernel:  [<ffffffff80046e4c>] try_to_wake_up+0x472/0x484
Aug 29 01:30:49 localhost kernel:  [<ffffffff88423a3b>] :sunrpc:rpc_wake_up_task+0x3e/0x55
Aug 29 01:30:49 localhost kernel:  [<ffffffff80021faa>] tcp_transmit_skb+0x62f/0x667
Aug 29 01:30:49 localhost kernel:  [<ffffffff80035ea3>] tcp_read_sock+0x189/0x19c
Aug 29 01:30:50 localhost kernel:  [<ffffffff88421c69>] :sunrpc:xs_tcp_data_recv+0x0/0x4f9
Aug 29 01:30:50 localhost kernel:  [<ffffffff88422629>] :sunrpc:xs_tcp_data_ready+0x60/0x6d
Aug 29 01:30:50 localhost kernel:  [<ffffffff8001beda>] tcp_rcv_established+0x84f/0x925
Aug 29 01:30:53 localhost kernel:  [<ffffffff8003b494>] tcp_v4_do_rcv+0x2a/0x2fa
Aug 29 01:30:53 localhost kernel:  [<ffffffff800272ae>] tcp_v4_rcv+0x9e0/0xa49
Aug 29 01:30:53 localhost kernel:  [<ffffffff80079352>] end_level_ioapic_vector+0x9/0x16
Aug 29 01:30:53 localhost kernel:  [<ffffffff800bae6f>] __do_IRQ+0xeb/0x103
Aug 29 01:30:54 localhost kernel:  [<ffffffff80034766>] ip_local_deliver+0x19d/0x263
Aug 29 01:30:54 localhost kernel:  [<ffffffff800358d4>] ip_rcv+0x539/0x57c
Aug 29 01:30:54 localhost kernel:  [<ffffffff80020a3b>] netif_receive_skb+0x470/0x49f
Aug 29 01:30:54 localhost kernel:  [<ffffffff881ed187>] :8139cp:cp_rx_poll+0x43c/0x58c
Aug 29 01:30:54 localhost kernel:  [<ffffffff8000c88c>] net_rx_action+0xac/0x1e0
Aug 29 01:30:55 localhost kernel:  [<ffffffff800123b4>] __do_softirq+0x89/0x133
Aug 29 01:30:55 localhost kernel:  [<ffffffff8005e2fc>] call_softirq+0x1c/0x28
Aug 29 01:30:55 localhost kernel:  [<ffffffff8006cb8e>] do_softirq+0x2c/0x85
Aug 29 01:30:55 localhost kernel:  [<ffffffff8006ca16>] do_IRQ+0xec/0xf5
Aug 29 01:30:55 localhost kernel:  [<ffffffff8006b346>] default_idle+0x0/0x50
Aug 29 01:30:55 localhost kernel:  [<ffffffff8005d615>] ret_from_intr+0x0/0xa
Aug 29 01:30:56 localhost kernel:  <EOI>  [<ffffffff8006b36f>] default_idle+0x29/0x50
Aug 29 01:30:56 localhost kernel:  [<ffffffff800491ab>] cpu_idle+0x95/0xb8
Aug 29 01:30:56 localhost kernel:  [<ffffffff8007796f>] start_secondary+0x498/0x4a7
Aug 29 01:30:56 localhost kernel: 
Aug 29 01:34:26 localhost kernel: BUG: soft lockup - CPU#7 stuck for 10s! [swapper:0]
Aug 29 01:34:26 localhost kernel: CPU 7:
Aug 29 01:34:26 localhost kernel: Modules linked in: ipv6 xfrm_nalgo crypto_api autofs4 hidp nfs fscache nfs_acl rfcomm l2cap bluetooth lockd sunrpc dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport floppy snd_ens1370 gameport snd_rawmidi snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_ak4531_codec snd 8139too i2c_piix4 soundcore virtio_blk 8139cp virtio_balloon i2c_core snd_page_alloc serio_raw virtio_pci virtio_ring virtio mii pcspkr dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod ata_piix libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Aug 29 01:34:26 localhost kernel: Pid: 0, comm: swapper Not tainted 2.6.18-194.11.1.el5 #1
Aug 29 01:34:26 localhost kernel: RIP: 0010:[<ffffffff88023adf>]  [<ffffffff88023adf>] :uhci_hcd:uhci_irq+0x24/0x15c
Aug 29 01:34:26 localhost kernel: RSP: 0000:ffff8101087af868  EFLAGS: 00000206
Aug 29 01:34:26 localhost kernel: RAX: ffffffff88020020 RBX: ffff81025eb9a150 RCX: 000000000000000b
Aug 29 01:34:26 localhost kernel: RDX: 000000000000c022 RSI: ffff8101087af928 RDI: ffff81025eb9a000
Aug 29 01:34:26 localhost kernel: RBP: ffff8101087af7e0 R08: ffff810256953ce0 R09: 0000000000000000
Aug 29 01:34:26 localhost kernel: R10: 00000000f14f961d R11: 0000000000000004 R12: ffffffff8005dc8e
Aug 29 01:34:26 localhost kernel: R13: ffff81025eb9a000 R14: ffffffff80078203 R15: ffff8101087af7e0
Aug 29 01:34:26 localhost kernel: FS:  0000000000000000(0000) GS:ffff810108753340(0000) knlGS:0000000000000000
Aug 29 01:34:26 localhost kernel: CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
Aug 29 01:34:30 localhost kernel: CR2: 000000000000f022 CR3: 0000000000201000 CR4: 00000000000006e0
Aug 29 01:34:31 localhost kernel: 
Aug 29 01:34:31 localhost kernel: Call Trace:
Aug 29 01:34:31 localhost kernel:  <IRQ>  [<ffffffff801f74dc>] usb_hcd_irq+0x27/0x55
Aug 29 01:34:31 localhost kernel:  [<ffffffff80010bab>] handle_IRQ_event+0x51/0xa6
Aug 29 01:34:31 localhost kernel:  [<ffffffff800bae28>] __do_IRQ+0xa4/0x103
Aug 29 01:34:31 localhost kernel:  [<ffffffff8006ca11>] do_IRQ+0xe7/0xf5
Aug 29 01:34:31 localhost kernel:  [<ffffffff8005d615>] ret_from_intr+0x0/0xa
Aug 29 01:34:32 localhost kernel:  [<ffffffff881ed7c6>] :8139cp:cp_start_xmit+0x4ef/0x511
Aug 29 01:34:32 localhost kernel:  [<ffffffff8022f296>] dev_hard_start_xmit+0x1b7/0x28a
Aug 29 01:34:32 localhost kernel:  [<ffffffff8023f5f8>] __qdisc_run+0x136/0x1f9
Aug 29 01:34:32 localhost kernel:  [<ffffffff8002f872>] dev_queue_xmit+0x150/0x271
Aug 29 01:34:32 localhost kernel:  [<ffffffff80031f5a>] ip_output+0x29a/0x2dd
Aug 29 01:34:32 localhost kernel:  [<ffffffff8003456f>] ip_queue_xmit+0x434/0x48e
Aug 29 01:34:32 localhost kernel:  [<ffffffff88423476>] :sunrpc:__rpc_do_wake_up_task+0x15a/0x180
Aug 29 01:34:33 localhost kernel:  [<ffffffff88423a3b>] :sunrpc:rpc_wake_up_task+0x3e/0x55
Aug 29 01:34:33 localhost kernel:  [<ffffffff80021faa>] tcp_transmit_skb+0x62f/0x667
Aug 29 01:34:33 localhost kernel:  [<ffffffff80035ea3>] tcp_read_sock+0x189/0x19c
Aug 29 01:34:33 localhost kernel:  [<ffffffff88421c69>] :sunrpc:xs_tcp_data_recv+0x0/0x4f9
Aug 29 01:34:33 localhost kernel:  [<ffffffff88422629>] :sunrpc:xs_tcp_data_ready+0x60/0x6d
Aug 29 01:34:33 localhost kernel:  [<ffffffff80055358>] sk_reset_timer+0xf/0x19
Aug 29 01:34:33 localhost kernel:  [<ffffffff8001bcc5>] tcp_rcv_established+0x63a/0x925
Aug 29 01:34:34 localhost kernel:  [<ffffffff8003b494>] tcp_v4_do_rcv+0x2a/0x2fa
Aug 29 01:34:34 localhost kernel:  [<ffffffff800272ae>] tcp_v4_rcv+0x9e0/0xa49
Aug 29 01:34:34 localhost kernel:  [<ffffffff80079352>] end_level_ioapic_vector+0x9/0x16
Aug 29 01:34:34 localhost kernel:  [<ffffffff800bae6f>] __do_IRQ+0xeb/0x103
Aug 29 01:34:34 localhost kernel:  [<ffffffff80034766>] ip_local_deliver+0x19d/0x263
Aug 29 01:34:34 localhost kernel:  [<ffffffff800358d4>] ip_rcv+0x539/0x57c
Aug 29 01:34:35 localhost kernel:  [<ffffffff80020a3b>] netif_receive_skb+0x470/0x49f
Aug 29 01:34:35 localhost kernel:  [<ffffffff881ed187>] :8139cp:cp_rx_poll+0x43c/0x58c
Aug 29 01:34:35 localhost kernel:  [<ffffffff8000c88c>] net_rx_action+0xac/0x1e0
Aug 29 01:34:35 localhost kernel:  [<ffffffff800123b4>] __do_softirq+0x89/0x133
Aug 29 01:34:35 localhost kernel:  [<ffffffff8005e2fc>] call_softirq+0x1c/0x28
Aug 29 01:34:35 localhost kernel:  [<ffffffff8006cb8e>] do_softirq+0x2c/0x85
Aug 29 01:34:35 localhost kernel:  [<ffffffff8006ca16>] do_IRQ+0xec/0xf5
Aug 29 01:34:36 localhost kernel:  [<ffffffff8006b346>] default_idle+0x0/0x50
Aug 29 01:34:36 localhost kernel:  [<ffffffff8005d615>] ret_from_intr+0x0/0xa
Aug 29 01:34:36 localhost kernel:  <EOI>  [<ffffffff8006b36f>] default_idle+0x29/0x50
Aug 29 01:34:36 localhost kernel:  [<ffffffff800491ab>] cpu_idle+0x95/0xb8
Aug 29 01:34:37 localhost kernel:  [<ffffffff8007796f>] start_secondary+0x498/0x4a7
Aug 29 01:34:37 localhost kernel: 
Aug 29 02:19:41 localhost kernel: BUG: soft lockup - CPU#7 stuck for 10s! [swapper:0]
Aug 29 02:19:41 localhost kernel: CPU 7:
Aug 29 02:19:41 localhost kernel: Modules linked in: ipv6 xfrm_nalgo crypto_api autofs4 hidp nfs fscache nfs_acl rfcomm l2cap bluetooth lockd sunrpc dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport floppy snd_ens1370 gameport snd_rawmidi snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_ak4531_codec snd 8139too i2c_piix4 soundcore virtio_blk 8139cp virtio_balloon i2c_core snd_page_alloc serio_raw virtio_pci virtio_ring virtio mii pcspkr dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod ata_piix libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Aug 29 02:19:41 localhost kernel: Pid: 0, comm: swapper Not tainted 2.6.18-194.11.1.el5 #1
Aug 29 02:19:41 localhost kernel: RIP: 0010:[<ffffffff80154837>]  [<ffffffff80154837>] ioread8+0x25/0x2f
Aug 29 02:19:41 localhost kernel: RSP: 0018:ffff8101087afd90  EFLAGS: 00000246
Aug 29 02:19:41 localhost kernel: RAX: 0000000000010001 RBX: ffff81025c8a9000 RCX: 000000000000000b
Aug 29 02:19:41 localhost kernel: RDX: 000000000000c513 RSI: ffff81025c8a9000 RDI: 000000000001c513
Aug 29 02:19:41 localhost kernel: RBP: ffff8101087afd10 R08: ffffffff80363008 R09: ffff8101087afe48
Aug 29 02:19:41 localhost kernel: R10: ffff8101087afd90 R11: ffffffff80363008 R12: ffffffff8005dc8e
Aug 29 02:19:41 localhost kernel: R13: 0000000000000000 R14: ffffffff80078203 R15: ffff8101087afd10
Aug 29 02:19:41 localhost kernel: FS:  0000000000000000(0000) GS:ffff810108753340(0000) knlGS:0000000000000000
Aug 29 02:19:41 localhost kernel: CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
Aug 29 02:19:41 localhost kernel: CR2: 0000003d3b6055d0 CR3: 0000000252db2000 CR4: 00000000000006e0
Aug 29 02:19:41 localhost kernel: 
Aug 29 02:19:42 localhost kernel: Call Trace:
Aug 29 02:19:42 localhost kernel:  <IRQ>  [<ffffffff881ae085>] :virtio_pci:vp_interrupt+0x27/0xb8
Aug 29 02:19:43 localhost kernel:  [<ffffffff80010bab>] handle_IRQ_event+0x51/0xa6
Aug 29 02:19:43 localhost kernel:  [<ffffffff800bae28>] __do_IRQ+0xa4/0x103
Aug 29 02:19:43 localhost kernel:  [<ffffffff80251b1f>] inet_twdr_hangman+0x0/0xa5
Aug 29 02:19:43 localhost kernel:  [<ffffffff8006ca11>] do_IRQ+0xe7/0xf5
Aug 29 02:19:44 localhost kernel:  [<ffffffff80046e4c>] try_to_wake_up+0x472/0x484
Aug 29 02:19:44 localhost kernel:  [<ffffffff8005d615>] ret_from_intr+0x0/0xa
Aug 29 02:19:44 localhost kernel:  [<ffffffff80097a79>] run_timer_softirq+0x162/0x241
Aug 29 02:19:44 localhost kernel:  [<ffffffff800123b4>] __do_softirq+0x89/0x133
Aug 29 02:19:44 localhost kernel:  [<ffffffff8005e2fc>] call_softirq+0x1c/0x28
Aug 29 02:19:45 localhost kernel:  [<ffffffff8006cb8e>] do_softirq+0x2c/0x85
Aug 29 02:19:45 localhost kernel:  [<ffffffff8006ca16>] do_IRQ+0xec/0xf5
Aug 29 02:19:45 localhost kernel:  [<ffffffff8006b346>] default_idle+0x0/0x50
Aug 29 02:19:45 localhost kernel:  [<ffffffff8005d615>] ret_from_intr+0x0/0xa
Aug 29 02:19:45 localhost kernel:  <EOI>  [<ffffffff8006b36f>] default_idle+0x29/0x50
Aug 29 02:19:45 localhost kernel:  [<ffffffff800491ab>] cpu_idle+0x95/0xb8
Aug 29 02:19:45 localhost kernel:  [<ffffffff8007796f>] start_secondary+0x498/0x4a7
Aug 29 02:19:46 localhost kernel: 
Aug 29 02:21:25 localhost kernel: BUG: soft lockup - CPU#7 stuck for 10s! [events/7:33]
Aug 29 02:21:25 localhost kernel: CPU 7:
Aug 29 02:21:25 localhost kernel: Modules linked in: ipv6 xfrm_nalgo crypto_api autofs4 hidp nfs fscache nfs_acl rfcomm l2cap bluetooth lockd sunrpc dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport floppy snd_ens1370 gameport snd_rawmidi snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_ak4531_codec snd 8139too i2c_piix4 soundcore virtio_blk 8139cp virtio_balloon i2c_core snd_page_alloc serio_raw virtio_pci virtio_ring virtio mii pcspkr dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod ata_piix libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Aug 29 02:21:25 localhost kernel: Pid: 33, comm: events/7 Not tainted 2.6.18-194.11.1.el5 #1
Aug 29 02:21:25 localhost kernel: RIP: 0010:[<ffffffff80154837>]  [<ffffffff80154837>] ioread8+0x25/0x2f
Aug 29 02:21:25 localhost kernel: RSP: 0018:ffff8101087afd20  EFLAGS: 00000246
Aug 29 02:21:25 localhost kernel: RAX: 0000000000010001 RBX: ffff81025c8a9000 RCX: 000000000000000b
Aug 29 02:21:25 localhost kernel: RDX: 000000000000c513 RSI: ffff81025c8a9000 RDI: 000000000001c513
Aug 29 02:21:25 localhost kernel: RBP: ffff8101087afca0 R08: ffff810108547b88 R09: ffff8101087afdd8
Aug 29 02:21:25 localhost kernel: R10: 000000000003e380 R11: 0000000000000074 R12: ffffffff8005dc8e
Aug 29 02:21:25 localhost kernel: R13: 0000000000000000 R14: ffffffff80078203 R15: ffff8101087afca0
Aug 29 02:21:25 localhost kernel: FS:  0000000000000000(0000) GS:ffff810108753340(0000) knlGS:0000000000000000
Aug 29 02:21:25 localhost kernel: CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
Aug 29 02:21:26 localhost kernel: CR2: 0000000008db0000 CR3: 00000002532c0000 CR4: 00000000000006e0
Aug 29 02:21:26 localhost kernel: 
Aug 29 02:21:26 localhost kernel: Call Trace:
Aug 29 02:21:27 localhost kernel:  <IRQ>  [<ffffffff881ae085>] :virtio_pci:vp_interrupt+0x27/0xb8
Aug 29 02:21:27 localhost kernel:  [<ffffffff80010bab>] handle_IRQ_event+0x51/0xa6
Aug 29 02:21:27 localhost kernel:  [<ffffffff800bae28>] __do_IRQ+0xa4/0x103
Aug 29 02:21:27 localhost kernel:  [<ffffffff8006ca11>] do_IRQ+0xe7/0xf5
Aug 29 02:21:28 localhost kernel:  [<ffffffff8005d615>] ret_from_intr+0x0/0xa
Aug 29 02:21:28 localhost kernel:  [<ffffffff881ed280>] :8139cp:cp_rx_poll+0x535/0x58c
Aug 29 02:21:28 localhost kernel:  [<ffffffff8000c88c>] net_rx_action+0xac/0x1e0
Aug 29 02:21:29 localhost kernel:  [<ffffffff800123b4>] __do_softirq+0x89/0x133
Aug 29 02:21:29 localhost kernel:  [<ffffffff8005e2fc>] call_softirq+0x1c/0x28
Aug 29 02:21:29 localhost kernel:  [<ffffffff8006cb8e>] do_softirq+0x2c/0x85
Aug 29 02:21:29 localhost kernel:  [<ffffffff8006ca16>] do_IRQ+0xec/0xf5
Aug 29 02:21:29 localhost kernel:  [<ffffffff8005d615>] ret_from_intr+0x0/0xa
Aug 29 02:21:30 localhost kernel:  <EOI>  [<ffffffff800dc069>] drain_array+0xb5/0xc0
Aug 29 02:21:30 localhost kernel:  [<ffffffff800dca7c>] cache_reap+0x0/0x217
Aug 29 02:21:30 localhost kernel:  [<ffffffff800dcb21>] cache_reap+0xa5/0x217
Aug 29 02:21:30 localhost kernel:  [<ffffffff8004d624>] run_workqueue+0x94/0xe4
Aug 29 02:21:30 localhost kernel:  [<ffffffff80049e5f>] worker_thread+0x0/0x122
Aug 29 02:21:31 localhost kernel:  [<ffffffff80049f4f>] worker_thread+0xf0/0x122
Aug 29 02:21:31 localhost kernel:  [<ffffffff8008cfa1>] default_wake_function+0x0/0xe
Aug 29 02:21:31 localhost kernel:  [<ffffffff8003287b>] kthread+0xfe/0x132
Aug 29 02:21:32 localhost kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Aug 29 02:21:32 localhost kernel:  [<ffffffff8003277d>] kthread+0x0/0x132
Aug 29 02:21:32 localhost kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Aug 29 02:21:32 localhost kernel: 
Aug 29 02:30:43 localhost kernel: BUG: soft lockup - CPU#7 stuck for 10s! [swapper:0]
Aug 29 02:30:43 localhost kernel: CPU 7:
Aug 29 02:30:43 localhost kernel: Modules linked in: ipv6 xfrm_nalgo crypto_api autofs4 hidp nfs fscache nfs_acl rfcomm l2cap bluetooth lockd sunrpc dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport floppy snd_ens1370 gameport snd_rawmidi snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_ak4531_codec snd 8139too i2c_piix4 soundcore virtio_blk 8139cp virtio_balloon i2c_core snd_page_alloc serio_raw virtio_pci virtio_ring virtio mii pcspkr dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod ata_piix libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Aug 29 02:30:43 localhost kernel: Pid: 0, comm: swapper Not tainted 2.6.18-194.11.1.el5 #1
Aug 29 02:30:43 localhost kernel: RIP: 0010:[<ffffffff80064b50>]  [<ffffffff80064b50>] _spin_unlock_irqrestore+0x8/0x9
Aug 29 02:30:43 localhost kernel: RSP: 0018:ffff8101087afd20  EFLAGS: 00000246
Aug 29 02:30:43 localhost kernel: RAX: ffff81025c8a92a8 RBX: ffff81025c8a9000 RCX: 0000000000000001
Aug 29 02:30:43 localhost kernel: RDX: ffff81025c8a92a8 RSI: 0000000000000246 RDI: ffff81025c8a92a0
Aug 29 02:30:43 localhost kernel: RBP: ffff8101087afca0 R08: ffffffff8032aad8 R09: 0000000000001000
Aug 29 02:30:43 localhost kernel: R10: ffff810253ab6a38 R11: ffffffff800c6944 R12: ffffffff8005dc8e
Aug 29 02:30:43 localhost kernel: R13: ffff81025c8a9290 R14: ffffffff80078203 R15: ffff8101087afca0
Aug 29 02:30:43 localhost kernel: FS:  0000000000000000(0000) GS:ffff810108753340(0000) knlGS:0000000000000000
Aug 29 02:30:43 localhost kernel: CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
Aug 29 02:30:43 localhost kernel: CR2: 0000003d3ba9a490 CR3: 00000001515ef000 CR4: 00000000000006e0
Aug 29 02:30:44 localhost kernel: 
Aug 29 02:30:44 localhost kernel: Call Trace:
Aug 29 02:30:44 localhost kernel:  <IRQ>  [<ffffffff881ae107>] :virtio_pci:vp_interrupt+0xa9/0xb8
Aug 29 02:30:44 localhost kernel:  [<ffffffff80010bab>] handle_IRQ_event+0x51/0xa6
Aug 29 02:30:45 localhost kernel:  [<ffffffff800bae28>] __do_IRQ+0xa4/0x103
Aug 29 02:30:45 localhost kernel:  [<ffffffff8006ca11>] do_IRQ+0xe7/0xf5
Aug 29 02:30:45 localhost kernel:  [<ffffffff8005d615>] ret_from_intr+0x0/0xa
Aug 29 02:30:45 localhost kernel:  [<ffffffff881ed280>] :8139cp:cp_rx_poll+0x535/0x58c
Aug 29 02:30:45 localhost kernel:  [<ffffffff8000c88c>] net_rx_action+0xac/0x1e0
Aug 29 02:30:45 localhost kernel:  [<ffffffff800123b4>] __do_softirq+0x89/0x133
Aug 29 02:30:45 localhost kernel:  [<ffffffff8005e2fc>] call_softirq+0x1c/0x28
Aug 29 02:30:45 localhost kernel:  [<ffffffff8006cb8e>] do_softirq+0x2c/0x85
Aug 29 02:30:46 localhost kernel:  [<ffffffff8006ca16>] do_IRQ+0xec/0xf5
Aug 29 02:30:46 localhost kernel:  [<ffffffff8006b346>] default_idle+0x0/0x50
Aug 29 02:30:46 localhost kernel:  [<ffffffff8005d615>] ret_from_intr+0x0/0xa
Aug 29 02:30:46 localhost kernel:  <EOI>  [<ffffffff8014e118>] cfq_may_queue+0x0/0x112
Aug 29 02:30:46 localhost kernel:  [<ffffffff8006b36f>] default_idle+0x29/0x50
Aug 29 02:30:46 localhost kernel:  [<ffffffff800491ab>] cpu_idle+0x95/0xb8
Aug 29 02:30:47 localhost kernel:  [<ffffffff8007796f>] start_secondary+0x498/0x4a7
Aug 29 02:30:47 localhost kernel: 
Aug 29 03:09:27 localhost kernel: BUG: soft lockup - CPU#7 stuck for 10s! [ncvlog:28284]
Aug 29 03:09:27 localhost kernel: CPU 7:
Aug 29 03:09:27 localhost kernel: Modules linked in: ipv6 xfrm_nalgo crypto_api autofs4 hidp nfs fscache nfs_acl rfcomm l2cap bluetooth lockd sunrpc dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport floppy snd_ens1370 gameport snd_rawmidi snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_ak4531_codec snd 8139too i2c_piix4 soundcore virtio_blk 8139cp virtio_balloon i2c_core snd_page_alloc serio_raw virtio_pci virtio_ring virtio mii pcspkr dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod ata_piix libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Aug 29 03:09:27 localhost kernel: Pid: 28284, comm: ncvlog Not tainted 2.6.18-194.11.1.el5 #1
Aug 29 03:09:27 localhost kernel: RIP: 0010:[<ffffffff88023adf>]  [<ffffffff88023adf>] :uhci_hcd:uhci_irq+0x24/0x15c
Aug 29 03:09:27 localhost kernel: RSP: 0018:ffff8101087afdb8  EFLAGS: 00000206
Aug 29 03:09:27 localhost kernel: RAX: ffffffff88020020 RBX: ffff81025eb9a150 RCX: 000000000000000b
Aug 29 03:09:27 localhost kernel: RDX: 000000000000c022 RSI: ffff8101087afe78 RDI: ffff81025eb9a000
Aug 29 03:09:27 localhost kernel: RBP: ffff8101087afd30 R08: 0000000000000040 R09: 000000000000006c
Aug 29 03:09:27 localhost kernel: R10: 000000000003e380 R11: ffff8101087afee0 R12: ffffffff8005dc8e
Aug 29 03:09:27 localhost kernel: R13: ffff81025eb9a000 R14: ffffffff80078203 R15: ffff8101087afd30
Aug 29 03:09:27 localhost kernel: FS:  0000000000000000(0000) GS:ffff810108753340(0063) knlGS:00000000f7f2e6c0
Aug 29 03:09:27 localhost kernel: CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
Aug 29 03:09:27 localhost kernel: CR2: 0000000008636f78 CR3: 0000000212f89000 CR4: 00000000000006e0
Aug 29 03:09:27 localhost kernel: 
Aug 29 03:09:28 localhost kernel: Call Trace:
Aug 29 03:09:28 localhost kernel:  <IRQ>  [<ffffffff801f74dc>] usb_hcd_irq+0x27/0x55
Aug 29 03:09:28 localhost kernel:  [<ffffffff80010bab>] handle_IRQ_event+0x51/0xa6
Aug 29 03:09:28 localhost kernel:  [<ffffffff800bae28>] __do_IRQ+0xa4/0x103
Aug 29 03:09:29 localhost kernel:  [<ffffffff80258bf4>] tcp_delack_timer+0x0/0x1ed
Aug 29 03:09:29 localhost kernel:  [<ffffffff8006ca11>] do_IRQ+0xe7/0xf5
Aug 29 03:09:29 localhost kernel:  [<ffffffff881ed280>] :8139cp:cp_rx_poll+0x535/0x58c
Aug 29 03:09:29 localhost kernel:  [<ffffffff8005d615>] ret_from_intr+0x0/0xa
Aug 29 03:09:30 localhost kernel:  [<ffffffff80097b4d>] run_timer_softirq+0x236/0x241
Aug 29 03:09:30 localhost kernel:  [<ffffffff800123b4>] __do_softirq+0x89/0x133
Aug 29 03:09:30 localhost kernel:  [<ffffffff8005e2fc>] call_softirq+0x1c/0x28
Aug 29 03:09:31 localhost kernel:  [<ffffffff8006cb8e>] do_softirq+0x2c/0x85
Aug 29 03:09:31 localhost kernel:  [<ffffffff8005dc8e>] apic_timer_interrupt+0x66/0x6c
Aug 29 03:09:31 localhost kernel:  <EOI>  [<ffffffff881ae53d>] :virtio_pci:vp_notify+0x0/0x1c
Aug 29 03:09:31 localhost kernel:  [<ffffffff8000c0f4>] __make_request+0x403/0x414
Aug 29 03:09:32 localhost kernel:  [<ffffffff8000c0b6>] __make_request+0x3c5/0x414
Aug 29 03:09:32 localhost kernel:  [<ffffffff8000fdab>] generic_file_buffered_write+0x20b/0x675
Aug 29 03:09:32 localhost kernel:  [<ffffffff8001c1c1>] generic_make_request+0x211/0x228
Aug 29 03:09:32 localhost kernel:  [<ffffffff800332f4>] submit_bio+0xe4/0xeb
Aug 29 03:09:32 localhost kernel:  [<ffffffff8001a90b>] submit_bh+0xf1/0x111
Aug 29 03:09:33 localhost kernel:  [<ffffffff8001c3ce>] __block_write_full_page+0x1f6/0x301
Aug 29 03:09:33 localhost kernel:  [<ffffffff8804eceb>] :ext3:ext3_get_block+0x0/0xf7
Aug 29 03:09:33 localhost kernel:  [<ffffffff8805056a>] :ext3:ext3_ordered_writepage+0xf1/0x198
Aug 29 03:09:33 localhost kernel:  [<ffffffff8001cf41>] mpage_writepages+0x1bf/0x37d
Aug 29 03:09:33 localhost kernel:  [<ffffffff88050479>] :ext3:ext3_ordered_writepage+0x0/0x198
Aug 29 03:09:34 localhost kernel:  [<ffffffff8005abe0>] do_writepages+0x29/0x2f
Aug 29 03:09:34 localhost kernel:  [<ffffffff8004f9fb>] __filemap_fdatawrite_range+0x50/0x5b
Aug 29 03:09:34 localhost kernel:  [<ffffffff80050104>] do_fsync+0x2f/0xa4
Aug 29 03:09:34 localhost kernel:  [<ffffffff800e0f84>] __do_fsync+0x23/0x36
Aug 29 03:09:34 localhost kernel:  [<ffffffff8006149d>] sysenter_do_call+0x1e/0x76
Aug 29 03:09:35 localhost kernel: 
Aug 29 03:33:14 localhost kernel: BUG: soft lockup - CPU#7 stuck for 10s! [swapper:0]
Aug 29 03:33:14 localhost kernel: CPU 7:
Aug 29 03:33:14 localhost kernel: Modules linked in: ipv6 xfrm_nalgo crypto_api autofs4 hidp nfs fscache nfs_acl rfcomm l2cap bluetooth lockd sunrpc dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport floppy snd_ens1370 gameport snd_rawmidi snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_ak4531_codec snd 8139too i2c_piix4 soundcore virtio_blk 8139cp virtio_balloon i2c_core snd_page_alloc serio_raw virtio_pci virtio_ring virtio mii pcspkr dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod ata_piix libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Aug 29 03:33:14 localhost kernel: Pid: 0, comm: swapper Not tainted 2.6.18-194.11.1.el5 #1
Aug 29 03:33:14 localhost kernel: RIP: 0010:[<ffffffff88023adf>]  [<ffffffff88023adf>] :uhci_hcd:uhci_irq+0x24/0x15c
Aug 29 03:33:14 localhost kernel: RSP: 0018:ffff8101087afa78  EFLAGS: 00000206
Aug 29 03:33:14 localhost kernel: RAX: ffffffff88020020 RBX: ffff81025eb9a150 RCX: 000000000000000b
Aug 29 03:33:14 localhost kernel: RDX: 000000000000c022 RSI: ffff8101087afb38 RDI: ffff81025eb9a000
Aug 29 03:33:14 localhost kernel: RBP: ffff8101087af9f0 R08: ffff81010853bac8 R09: 000000000000006c
Aug 29 03:33:14 localhost kernel: R10: ffff8101087af930 R11: 000000000000037f R12: ffffffff8005dc8e
Aug 29 03:33:14 localhost kernel: R13: ffff81025eb9a000 R14: ffffffff80078203 R15: ffff8101087af9f0
Aug 29 03:33:14 localhost kernel: FS:  0000000000000000(0000) GS:ffff810108753340(0000) knlGS:0000000000000000
Aug 29 03:33:14 localhost kernel: CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
Aug 29 03:33:15 localhost kernel: CR2: 0000003d3d6aa700 CR3: 00000001f2432000 CR4: 00000000000006e0
Aug 29 03:33:15 localhost kernel: 
Aug 29 03:33:16 localhost kernel: Call Trace:
Aug 29 03:33:16 localhost kernel:  <IRQ>  [<ffffffff801f74dc>] usb_hcd_irq+0x27/0x55
Aug 29 03:33:16 localhost kernel:  [<ffffffff80010bab>] handle_IRQ_event+0x51/0xa6
Aug 29 03:33:17 localhost kernel:  [<ffffffff800bae28>] __do_IRQ+0xa4/0x103
Aug 29 03:33:17 localhost kernel:  [<ffffffff8006ca11>] do_IRQ+0xe7/0xf5
Aug 29 03:33:19 localhost kernel:  [<ffffffff88423476>] :sunrpc:__rpc_do_wake_up_task+0x15a/0x180
Aug 29 03:33:19 localhost kernel:  [<ffffffff8005d615>] ret_from_intr+0x0/0xa
Aug 29 03:33:19 localhost kernel:  [<ffffffff80229cde>] skb_release_head_state+0x15/0x108
Aug 29 03:33:19 localhost kernel:  [<ffffffff80028b1c>] __kfree_skb+0x9/0x1a
Aug 29 03:33:19 localhost kernel:  [<ffffffff80035e23>] tcp_read_sock+0x109/0x19c
Aug 29 03:33:20 localhost kernel:  [<ffffffff88421c69>] :sunrpc:xs_tcp_data_recv+0x0/0x4f9
Aug 29 03:33:20 localhost kernel:  [<ffffffff88422629>] :sunrpc:xs_tcp_data_ready+0x60/0x6d
Aug 29 03:33:20 localhost kernel:  [<ffffffff8001bcc5>] tcp_rcv_established+0x63a/0x925
Aug 29 03:33:20 localhost kernel:  [<ffffffff8005dc8e>] apic_timer_interrupt+0x66/0x6c
Aug 29 03:33:21 localhost kernel:  [<ffffffff8003b494>] tcp_v4_do_rcv+0x2a/0x2fa
Aug 29 03:33:21 localhost kernel:  [<ffffffff80064b50>] _spin_unlock_irqrestore+0x8/0x9
Aug 29 03:33:21 localhost kernel:  [<ffffffff800272ae>] tcp_v4_rcv+0x9e0/0xa49
Aug 29 03:33:22 localhost kernel:  [<ffffffff80079352>] end_level_ioapic_vector+0x9/0x16
Aug 29 03:33:22 localhost kernel:  [<ffffffff800bae6f>] __do_IRQ+0xeb/0x103
Aug 29 03:33:22 localhost kernel:  [<ffffffff80034766>] ip_local_deliver+0x19d/0x263
Aug 29 03:33:23 localhost kernel:  [<ffffffff800358d4>] ip_rcv+0x539/0x57c
Aug 29 03:33:23 localhost kernel:  [<ffffffff80020a3b>] netif_receive_skb+0x470/0x49f
Aug 29 03:33:23 localhost kernel:  [<ffffffff881ed187>] :8139cp:cp_rx_poll+0x43c/0x58c
Aug 29 03:33:24 localhost kernel:  [<ffffffff8000c88c>] net_rx_action+0xac/0x1e0
Aug 29 03:33:24 localhost kernel:  [<ffffffff800123b4>] __do_softirq+0x89/0x133
Aug 29 03:33:25 localhost kernel:  [<ffffffff8005e2fc>] call_softirq+0x1c/0x28
Aug 29 03:33:25 localhost kernel:  [<ffffffff8006cb8e>] do_softirq+0x2c/0x85
Aug 29 03:33:25 localhost kernel:  [<ffffffff8006ca16>] do_IRQ+0xec/0xf5
Aug 29 03:33:26 localhost kernel:  [<ffffffff8006b346>] default_idle+0x0/0x50
Aug 29 03:33:26 localhost kernel:  [<ffffffff8005d615>] ret_from_intr+0x0/0xa
Aug 29 03:33:26 localhost kernel:  <EOI>  [<ffffffff8005a7aa>] generic_unplug_device+0x0/0x32
Aug 29 03:33:27 localhost kernel:  [<ffffffff8006b36f>] default_idle+0x29/0x50
Aug 29 03:33:27 localhost kernel:  [<ffffffff800491ab>] cpu_idle+0x95/0xb8
Aug 29 03:33:28 localhost kernel:  [<ffffffff8007796f>] start_secondary+0x498/0x4a7
Aug 29 03:33:28 localhost kernel: 
Aug 29 03:34:14 localhost kernel: BUG: soft lockup - CPU#7 stuck for 10s! [swapper:0]
Aug 29 03:34:14 localhost kernel: CPU 7:
Aug 29 03:34:14 localhost kernel: Modules linked in: ipv6 xfrm_nalgo crypto_api autofs4 hidp nfs fscache nfs_acl rfcomm l2cap bluetooth lockd sunrpc dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport floppy snd_ens1370 gameport snd_rawmidi snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_ak4531_codec snd 8139too i2c_piix4 soundcore virtio_blk 8139cp virtio_balloon i2c_core snd_page_alloc serio_raw virtio_pci virtio_ring virtio mii pcspkr dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod ata_piix libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Aug 29 03:34:14 localhost kernel: Pid: 0, comm: swapper Not tainted 2.6.18-194.11.1.el5 #1
Aug 29 03:34:14 localhost kernel: RIP: 0010:[<ffffffff88023adf>]  [<ffffffff88023adf>] :uhci_hcd:uhci_irq+0x24/0x15c
Aug 29 03:34:14 localhost kernel: RSP: 0018:ffff8101087af868  EFLAGS: 00000206
Aug 29 03:34:14 localhost kernel: RAX: ffffffff88020020 RBX: ffff81025eb9a150 RCX: 000000000000000b
Aug 29 03:34:14 localhost kernel: RDX: 000000000000c022 RSI: ffff8101087af928 RDI: ffff81025eb9a000
Aug 29 03:34:14 localhost kernel: RBP: ffff8101087af7e0 R08: ffff810256953ce0 R09: 0000000000000000
Aug 29 03:34:14 localhost kernel: R10: 000000001166b020 R11: 0000000000000004 R12: ffffffff8005dc8e
Aug 29 03:34:14 localhost kernel: R13: ffff81025eb9a000 R14: ffffffff80078203 R15: ffff8101087af7e0
Aug 29 03:34:14 localhost kernel: FS:  0000000000000000(0000) GS:ffff810108753340(0000) knlGS:0000000000000000
Aug 29 03:34:14 localhost kernel: CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
Aug 29 03:34:15 localhost kernel: CR2: 0000000000bd9ac0 CR3: 0000000000201000 CR4: 00000000000006e0
Aug 29 03:34:16 localhost kernel: 
Aug 29 03:34:17 localhost kernel: Call Trace:
Aug 29 03:34:17 localhost kernel:  <IRQ>  [<ffffffff801f74dc>] usb_hcd_irq+0x27/0x55
Aug 29 03:34:17 localhost kernel:  [<ffffffff80010bab>] handle_IRQ_event+0x51/0xa6
Aug 29 03:34:18 localhost kernel:  [<ffffffff800bae28>] __do_IRQ+0xa4/0x103
Aug 29 03:34:18 localhost kernel:  [<ffffffff8006ca11>] do_IRQ+0xe7/0xf5
Aug 29 03:34:18 localhost kernel:  [<ffffffff8005d615>] ret_from_intr+0x0/0xa
Aug 29 03:34:18 localhost kernel:  [<ffffffff881ed7c6>] :8139cp:cp_start_xmit+0x4ef/0x511
Aug 29 03:34:18 localhost kernel:  [<ffffffff8022f296>] dev_hard_start_xmit+0x1b7/0x28a
Aug 29 03:34:19 localhost kernel:  [<ffffffff8023f5f8>] __qdisc_run+0x136/0x1f9
Aug 29 03:34:19 localhost kernel:  [<ffffffff8002f872>] dev_queue_xmit+0x150/0x271
Aug 29 03:34:19 localhost kernel:  [<ffffffff80031f5a>] ip_output+0x29a/0x2dd
Aug 29 03:34:19 localhost kernel:  [<ffffffff8003456f>] ip_queue_xmit+0x434/0x48e
Aug 29 03:34:19 localhost kernel:  [<ffffffff88423476>] :sunrpc:__rpc_do_wake_up_task+0x15a/0x180
Aug 29 03:34:20 localhost kernel:  [<ffffffff80046e4c>] try_to_wake_up+0x472/0x484
Aug 29 03:34:20 localhost kernel:  [<ffffffff88423a3b>] :sunrpc:rpc_wake_up_task+0x3e/0x55
Aug 29 03:34:20 localhost kernel:  [<ffffffff80021faa>] tcp_transmit_skb+0x62f/0x667
Aug 29 03:34:20 localhost kernel:  [<ffffffff80035ea3>] tcp_read_sock+0x189/0x19c
Aug 29 03:34:20 localhost kernel:  [<ffffffff88421c69>] :sunrpc:xs_tcp_data_recv+0x0/0x4f9
Aug 29 03:34:21 localhost kernel:  [<ffffffff88422629>] :sunrpc:xs_tcp_data_ready+0x60/0x6d
Aug 29 03:34:21 localhost kernel:  [<ffffffff8001beda>] tcp_rcv_established+0x84f/0x925
Aug 29 03:34:21 localhost kernel:  [<ffffffff8003b494>] tcp_v4_do_rcv+0x2a/0x2fa
Aug 29 03:34:21 localhost kernel:  [<ffffffff881eb5cb>] :8139cp:cp_interrupt+0x2c/0x362
Aug 29 03:34:21 localhost kernel:  [<ffffffff800272ae>] tcp_v4_rcv+0x9e0/0xa49
Aug 29 03:34:22 localhost kernel:  [<ffffffff80034766>] ip_local_deliver+0x19d/0x263
Aug 29 03:34:22 localhost kernel:  [<ffffffff800358d4>] ip_rcv+0x539/0x57c
Aug 29 03:34:22 localhost kernel:  [<ffffffff80020a3b>] netif_receive_skb+0x470/0x49f
Aug 29 03:34:22 localhost kernel:  [<ffffffff881ed187>] :8139cp:cp_rx_poll+0x43c/0x58c
Aug 29 03:34:23 localhost kernel:  [<ffffffff8000c88c>] net_rx_action+0xac/0x1e0
Aug 29 03:34:23 localhost kernel:  [<ffffffff800123b4>] __do_softirq+0x89/0x133
Aug 29 03:34:23 localhost kernel:  [<ffffffff8005e2fc>] call_softirq+0x1c/0x28
Aug 29 03:34:23 localhost kernel:  [<ffffffff8006cb8e>] do_softirq+0x2c/0x85
Aug 29 03:34:23 localhost kernel:  [<ffffffff8006ca16>] do_IRQ+0xec/0xf5
Aug 29 03:34:24 localhost kernel:  [<ffffffff8006b346>] default_idle+0x0/0x50
Aug 29 03:34:24 localhost kernel:  [<ffffffff8005d615>] ret_from_intr+0x0/0xa
Aug 29 03:34:24 localhost kernel:  <EOI>  [<ffffffff8006b36f>] default_idle+0x29/0x50
Aug 29 03:34:24 localhost kernel:  [<ffffffff800491ab>] cpu_idle+0x95/0xb8
Aug 29 03:34:24 localhost kernel:  [<ffffffff8007796f>] start_secondary+0x498/0x4a7
Aug 29 03:34:25 localhost kernel:

Comment 28 Robert Dale 2010-11-04 19:18:01 UTC
Still seeing this with 2.6.34.7-61.fc13.x86_64

Comment 29 Bug Zapper 2011-06-01 12:32:11 UTC
This message is a reminder that Fedora 13 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 13.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '13'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 13's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 13 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Comment 30 Bug Zapper 2011-06-29 12:47:49 UTC
Fedora 13 changed to end-of-life (EOL) status on 2011-06-25. Fedora 13 is 
no longer maintained, which means that it will not receive any further 
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of 
Fedora please feel free to reopen this bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.