Bug 619560
Summary: | KVM VMs problems, disks not bootable, network cards not seen, hanging | ||
---|---|---|---|
Product: | [Fedora] Fedora | Reporter: | Joshua Rosen <bjrosen> |
Component: | qemu | Assignee: | Justin M. Forbes <jforbes> |
Status: | CLOSED WONTFIX | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
Severity: | high | Docs Contact: | |
Priority: | high | ||
Version: | 13 | CC: | amit.shah, a.r.welter, berrange, clalance, dwmw2, ehabkost, extras-orphan, gcosta, itamar, jaswinder, jforbes, karlcz, knoel, markmc, notting, ondrejj, quintela, rh_bugzilla, scottt.tw, skothari, virt-maint |
Target Milestone: | --- | ||
Target Release: | --- | ||
Hardware: | All | ||
OS: | Linux | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | Bug Fix | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2011-06-29 12:47:49 UTC | Type: | --- |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Attachments: |
Description
Joshua Rosen
2010-07-29 20:12:04 UTC
Created attachment 435601 [details]
/var/log/messages file, Contains QEMU errors
I put F12 on the box with the CentOS5.5 VM and ran an overnight Verilog simulation on the VM. On F12 the VM behaved normally. The same VM on F13 consistently crashed when running this load. This probably eliminates a hardware problem. The system has multiple root partitions with a shared /home partition. I can reboot it into F13 if you want me to instrument something. I just updated my iCore7 system and I'm getting the same problem. After the system was updated I was unable to boot my VMs because the virtual disks weren't seen. Deleting and reattaching the virtual disks fixed that problem. After running simulations on the CentOS5.5 VM the VM seems to have hung. I found this in /var/log/messages Aug 3 20:54:31 localhost libvirtd: 20:54:31.622: warning : qemudParsePCIDeviceStrs:1411 : Unexpected exit status '1', qemu probably failed Aug 3 20:54:31 localhost libvirtd: 20:54:31.758: error : qemudDomainGetVcpus:5801 : Requested operation is not valid: cannot list vcpu pinning for an inactive domain Aug 3 20:54:31 localhost libvirtd: 20:54:31.760: error : qemudDomainGetVcpus:5801 : Requested operation is not valid: cannot list vcpu pinning for an inactive domain Aug 3 20:54:31 localhost libvirtd: 20:54:31.761: error : qemudDomainGetVcpus:5801 : Requested operation is not valid: cannot list vcpu pinning for an inactive domain Aug 3 20:54:31 localhost libvirtd: 20:54:31.763: error : qemudDomainGetVcpus:5801 : Requested operation is not valid: cannot list vcpu pinning for an inactive domain Aug 3 20:54:31 localhost libvirtd: 20:54:31.764: error : qemudDomainGetVcpus:5801 : Requested operation is not valid: cannot list vcpu pinning for an inactive domain Aug 3 20:54:31 localhost libvirtd: 20:54:31.766: error : qemudDomainGetVcpus:5801 : Requested operation is not valid: cannot list vcpu pinning for an inactive domain Aug 3 20:54:31 localhost libvirtd: 20:54:31.767: error : qemudDomainGetVcpus:5801 : Requested operation is not valid: cannot list vcpu pinning for an inactive domain Aug 3 20:54:31 localhost libvirtd: 20:54:31.768: error : qemudDomainGetVcpus:5801 : Requested operation is not valid: cannot list vcpu pinning for an inactive domain Aug 3 20:54:31 localhost libvirtd: 20:54:31.770: error : qemudDomainGetVcpus:5801 : Requested operation is not valid: cannot list vcpu pinning for an inactive domain Aug 3 20:54:31 localhost libvirtd: 20:54:31.771: error : qemudDomainGetVcpus:5801 : Requested operation is not valid: cannot list vcpu pinning for an inactive domain Aug 3 20:54:31 localhost libvirtd: 20:54:31.773: error : qemudDomainGetVcpus:5801 : Requested operation is not valid: cannot list vcpu pinning for an inactive domain Aug 3 20:54:48 localhost libvirtd: 20:54:48.189: warning : qemudParsePCIDeviceStrs:1411 : Unexpected exit status '1', qemu probably failed This does appear to be a kernel problem that's fixed in later kernels. I built and installed a 2.6.35 kernel on my iCore7 system. So far so good, I have four simulation streams that have been running for the past 10 hours, the VM is still functioning properly. Running f13 with 2.6.33.6-147.2.4.fc13.x86_64 (upgraded yesterday) and I'm seeing similar problems. Just would like to point to bz619798 which is also about VMs hanging so you may want to monitor that too. The issues I'm seeing are similar to Joshua's in #3. I have 2.6.35 on two machines now, an iCore7 and a Core2. I've been running heavily loaded CentOS 5.5 VMs on both systems for several days without a problem. Both systems had hanging VMs with the stock Fedora 13 kernel. Thanks for the feedback Joshua. When you talk about 2.6.35 do I understand correctly that you mean a plain vanilla 2.6.35 kernel from kernel.org? If you used a plain vanilla 2.6.35 kernel would it be possible for you to test the latest F14 2.6.35 kernel from koji.fedoraproject.org ? It might help the developers if this is a plain vanilla kernel bug or something caused by the Fedora special sauce added to the kernel. Link to koji kernel packages: http://koji.fedoraproject.org/koji/packageinfo?packageID=8 You may need to upgrade linux-firmware to a newer version which can also be found on koji. I see a 2.6.36.rc kernel in Rawhide and a 2.6.34 kernel in updates testing, I don't see a 2.6.35 kernel anywhere. I could give each one of them a try. My 2.6.35 is a vanilla kernel.org kernel. I configured it by coping the F13 .config file into the build directory, ran gnome-config and set the processor type to be Core2 or Later instead of the generic x86-64 and then built it. Which Fedora kernel would you like me to try first? I tried the 2.6.34 Fedora kernel and experienced the hangs. The rawhide one might be a bit too experimental but I'm not a kernel developer. Would be nice if any of the developers could give us some feedback what to test (hint). How about this one: kernel-2.6.35.1-5.rc1.fc14. You can download it here: http://koji.fedoraproject.org/koji/buildinfo?buildID=189140 I've installed this one which is a 2.6.35.0, I'll reboot tonight and give it a try. http://koji.fedoraproject.org/koji/buildinfo?buildID=187655 The Koji 2.6.35.0 kernel seems to be working fine, give it a try to see if it fixes your problem. Can you please include: qemu version: seabios version: qemu command line used (if using virt-manager/libvirt this will be in /var/logs/libvirt/qemu/<guestname>.log ) We should be able to start tracking down the issues with that information. Created attachment 438475 [details]
libvirt log file
This is the latest log file. I'm currently running with a 2.6.35.0 kernel which works. I won't be able to revert back to the busted 2.6.13 kernel until Monday.
Linux reagan 2.6.35-0.fc14.x86_64 #1 SMP Mon Aug 2 01:07:47 UTC 2010 x86_64 x86_64 x86_64 GNU/Linux
QEMU PC emulator version 0.12.3 (qemu-kvm-0.12.3), Copyright (c) 2003-2008 Fabrice Bellard It seems curious that the copyright date is 2008 on the F13 version of QEMU, is this correct? seabios is not installed. I've installed seabios, when I reboot on Monday with the F13 kernel I'll see if this makes a difference. Created attachment 438671 [details]
/var/log/messages
This is from a 4th system, a Core2 laptop running the stock F13 kernels. I can't do a Force Off of this VM and I can't access it.
Linux hornet 2.6.33.6-147.2.4.fc13.x86_64 #1 SMP Fri Jul 23 17:14:44 UTC 2010 x86_64 x86_64 x86_64 GNU/Linux
QEMU PC emulator version 0.12.3 (qemu-kvm-0.12.3), Copyright (c) 2003-2008 Fabrice Bellard
Created attachment 438672 [details]
/var/log/libvirt/log
This is the libvirt log file for the Core2 laptop system
Created attachment 438803 [details]
Bad .config file, used to build a 2.6.35.2 kernel that exhibits the VM problem
I think I stumbled onto something. I built a 2.6.35.2 kernel for my Core2 laptop. Since it's a laptop system I selected the low latency preemptible kernel option instead of leaving it at the Voluntary Kernel option. The resulting kernel resulted in hanging VMs. In fact the problem seems to be worse. I then built another 2.6.35.2 kernel using the .config file that I used to build the 2.6.35 kernel that I reported on earlier. The second 2.6.35.2 kernel works fine.
I'm attaching the good and bad .config files.
Created attachment 438804 [details]
Good .config file, used to build a 2.6.35.2 kernel that doesn't have the VM problem
This .config file builds a kernel which doesn't exhibit the hanging VM problem.
I am also seeing KVM guest hangs with up to date CentOS 5.5 guests on up to date Fedora 13, on an Intel Xeon E5530 based host. After trying every tweak found via the web (changing virtio to other emulated disk and NIC types, turning off APIC in VM machine settings, trying the centos-plus kernel instead of the regular guest kernel) it seems changing to 1 VCPU has made the guest more stable. It has now been running for over 24 hours, where it would freeze up in strange ways after only an hour or two of light development use. (The CentOS guest would not emit unusual log messages, but some system calls seemed to block, blocking new logins, blocking disk flush/sync from apps, and blocking 'top' while 'ps' still worked. The only recourse was forced power off of the guest.) Meanwhile, a Windows XP guest with 2 VCPUs seems to run without issue on the same host. I have been running the 2.6.35.1-5.rc1.fc14.x86_64 kernel on F13 for about two weeks now without any issue running CentOS 5.5 x86_64 guests with 2 VCPUs and 2GB mem. Has anyone else tried building a kernel with the low latency option to see if it breaks? I just experienced the problem with the Linux reagan 2.6.35-0.fc14.x86_64 kernel. I've disabled ksm and ksmtuned and I'm rebooting. I'll let you know if this helps. My CentOS 5.5 VM has crashed twice on a 2.6.35.2 kernel. I've introduced a new factor, I've added a 128G virtual disk, mounted as IDE. I'm running my Verilog simulations on the virtual disk instead of on NFS mounted directories as I was doing before. After several hours of simulations the VM crashes. This has happened twice. I don't see any errors in the host machine's logs however in the VM's /var/log/messages I see the following, Aug 29 03:24:52 localhost kernel: BUG: soft lockup - CPU#1 stuck for 18s! [swapper:0] Aug 29 03:24:52 localhost kernel: CPU 1: Aug 29 03:24:52 localhost kernel: Modules linked in: ipv6 xfrm_nalgo crypto_api autofs4 hidp nfs fscache nfs_acl rfcomm l2cap bluetooth lockd sunrpc dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport floppy snd_ens1370 gameport snd_rawmidi snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm i2c_piix4 snd_timer snd_ak4531_codec serio_raw virtio_balloon snd soundcore 8139too virtio_pci virtio_ring virtio i2c_core snd_page_alloc 8139cp mii pcspkr dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod ata_piix libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd Aug 29 03:24:52 localhost kernel: Pid: 0, comm: swapper Not tainted 2.6.18-194.11.1.el5 #1 Aug 29 03:24:52 localhost kernel: RIP: 0010:[<ffffffff8004f4db>] [<ffffffff8004f4db>] hrtimer_run_queues+0xb4/0x197 Aug 29 03:24:52 localhost kernel: RSP: 0018:ffff810108637ed0 EFLAGS: 00000282 Aug 29 03:24:52 localhost kernel: RAX: 0000000225c17d03 RBX: 0000000000000001 RCX: 000000002aae83f7 Aug 29 03:24:52 localhost kernel: RDX: ffffffffb6743cec RSI: 111fb9681838d2b0 RDI: 7fffffffffffffff Aug 29 03:24:52 localhost kernel: RBP: ffff810108637e50 R08: 0000000001786599 R09: ffff81010862fe48 Aug 29 03:24:52 localhost kernel: R10: 0000000000000001 R11: ffff8101ba523c20 R12: ffffffff8005dc8e Aug 29 03:24:52 localhost kernel: R13: 0000000000000046 R14: ffffffff80078203 R15: ffff810108637e50 Aug 29 03:24:52 localhost kernel: FS: 0000000000000000(0000) GS:ffff8101085f57c0(0000) knlGS:0000000000000000 Aug 29 03:24:52 localhost kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 29 03:25:05 localhost kernel: CR2: 0000000000b23370 CR3: 00000001b9ecb000 CR4: 00000000000006e0 Aug 29 03:43:06 localhost kernel: Aug 29 03:43:07 localhost kernel: Call Trace: Aug 29 03:43:08 localhost kernel: <IRQ> [<ffffffff8009793e>] run_timer_softirq+0x27/0x241 Aug 29 03:43:08 localhost kernel: [<ffffffff800123b4>] __do_softirq+0x89/0x133 Aug 29 03:43:09 localhost kernel: [<ffffffff8005e2fc>] call_softirq+0x1c/0x28 Aug 29 03:43:09 localhost kernel: [<ffffffff8006cb8e>] do_softirq+0x2c/0x85 Aug 29 03:43:09 localhost kernel: [<ffffffff8006b346>] default_idle+0x0/0x50 Aug 29 03:43:09 localhost kernel: [<ffffffff8005dc8e>] apic_timer_interrupt+0x66/0x6c Aug 29 03:43:09 localhost kernel: <EOI> [<ffffffff8006b36f>] default_idle+0x29/0x50 Aug 29 03:43:10 localhost kernel: [<ffffffff800491ab>] cpu_idle+0x95/0xb8 Aug 29 03:43:10 localhost kernel: [<ffffffff8007796f>] start_secondary+0x498/0x4a7 Aug 29 03:43:10 localhost kernel: Aug 29 03:43:10 localhost kernel: BUG: soft lockup - CPU#6 stuck for 17s! [swapper:0] Aug 29 03:43:10 localhost kernel: DriveReady Error } Aug 29 03:43:11 localhost kernel: hdb: task_out_intr: error=0x04 { DriveStatusError } Aug 29 03:43:11 localhost kernel: ide: failed opcode was: unknown Aug 29 03:54:03 localhost kernel: hdb: task_out_intr: status=0x41 { DriveReady Error } Aug 29 03:54:04 localhost kernel: hdb: task_out_intr: error=0x04 { DriveStatusError } Aug 29 03:54:04 localhost kernel: ide: failed opcode was: unknown Aug 29 03:54:05 localhost kernel: hdb: task_out_intr: status=0x41 { DriveReady Error } Aug 29 03:54:05 localhost kernel: hdb: task_out_intr: error=0x04 { DriveStatusError } Aug 29 03:54:06 localhost kernel: ide: failed opcode was: unknown Aug 29 03:58:40 localhost kernel: hdb: task_out_intr: status=0x41 { DriveReady Error } Aug 29 03:58:40 localhost kernel: hdb: task_out_intr: error=0x04 { DriveStatusError } Aug 29 03:58:40 localhost kernel: ide: failed opcode was: unknown Aug 29 03:58:41 localhost kernel: hdb: task_out_intr: status=0x41 { DriveReady Error } Aug 29 03:58:41 localhost kernel: hdb: task_out_intr: error=0x04 { DriveStatusError } Aug 29 03:58:41 localhost kernel: ide: failed opcode was: unknown Aug 29 03:58:41 localhost kernel: hdb: task_out_intr: status=0x41 { DriveReady Error } Aug 29 03:58:42 localhost kernel: hdb: task_out_intr: error=0x04 { DriveStatusError } Aug 29 03:58:42 localhost kernel: ide: failed opcode was: unknown Aug 29 03:58:42 localhost kernel: hdb: task_out_intr: status=0x41 { DriveReady Error } Aug 29 03:58:42 localhost kernel: hdb: task_out_intr: error=0x04 { DriveStatusError } Aug 29 03:58:42 localhost kernel: ide: failed opcode was: unknown Aug 29 03:58:42 localhost kernel: hdb: task_out_intr: status=0x41 { DriveReady Error } Aug 29 03:58:43 localhost kernel: hdb: task_out_intr: error=0x04 { DriveStatusError } Aug 29 03:58:43 localhost kernel: ide: failed opcode was: unknown Aug 29 03:58:43 localhost kernel: hdb: task_out_intr: status=0x41 { DriveReady Error } Aug 29 03:58:43 localhost kernel: hdb: task_out_intr: error=0x04 { DriveStatusError } Aug 29 03:58:43 localhost kernel: ide: failed opcode was: unknown Aug 29 03:58:44 localhost kernel: hdb: task_out_intr: status=0x41 { DriveReady Error } Aug 29 03:58:44 localhost kernel: hdb: task_out_intr: error=0x04 { DriveStatusError } Aug 29 03:58:44 localhost kernel: ide: failed opcode was: unknown Aug 29 03:58:44 localhost kernel: hdb: task_out_intr: status=0x41 { DriveReady Error } Aug 29 03:58:45 localhost kernel: hdb: task_out_intr: error=0x04 { DriveStatusError } Aug 29 03:58:45 localhost kernel: ide: failed opcode was: unknown Aug 29 03:58:45 localhost kernel: hdb: task_out_intr: status=0x41 { DriveReady Error } Aug 29 03:58:45 localhost kernel: hdb: task_out_intr: error=0x04 { DriveStatusError } Aug 29 03:58:46 localhost kernel: ide: failed opcode was: unknown Aug 29 03:58:46 localhost kernel: hdb: task_out_intr: status=0x41 { DriveReady Error } Aug 29 03:58:46 localhost kernel: hdb: task_out_intr: error=0x04 { DriveStatusError } Aug 29 03:58:46 localhost kernel: ide: failed opcode was: unknown I tried mounting the virtual disk as virtio instead of IDE, the VM crashed as before, Aug 29 00:13:29 localhost kernel: Pid: 0, comm: swapper Not tainted 2.6.18-194.11.1.el5 #1 Aug 29 00:13:29 localhost kernel: RIP: 0010:[<ffffffff881eb5cb>] [<ffffffff881eb5cb>] :8139cp:cp_interrupt+0x2c/0x362 Aug 29 00:13:29 localhost kernel: RSP: 0018:ffff8101087afd18 EFLAGS: 00010286 Aug 29 00:13:29 localhost kernel: RAX: 0000000000000001 RBX: ffff81025c1ba500 RCX: 0000000000000001 Aug 29 00:13:29 localhost kernel: RDX: ffffc20000028200 RSI: ffff81025c1ba000 RDI: 000000000000000b Aug 29 00:13:29 localhost kernel: RBP: ffff8101087afc90 R08: ffffffff8032aad8 R09: 0000000000001000 Aug 29 00:13:29 localhost kernel: R10: ffff810240d821b8 R11: ffffffff800c6944 R12: ffffffff8005dc8e Aug 29 00:13:29 localhost kernel: R13: 0000000000000080 R14: ffffffff80078203 R15: ffff8101087afc90 Aug 29 00:13:29 localhost kernel: FS: 0000000000000000(0000) GS:ffff810108753340(0000) knlGS:0000000000000000 Aug 29 00:13:29 localhost kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 29 00:13:29 localhost kernel: CR2: 00000000087d8d60 CR3: 0000000247563000 CR4: 00000000000006e0 Aug 29 00:13:29 localhost kernel: Aug 29 00:13:29 localhost kernel: Call Trace: Aug 29 00:13:29 localhost kernel: <IRQ> [<ffffffff881a3082>] :virtio_ring:vring_interrupt+0x26/0x2f Aug 29 00:13:30 localhost kernel: [<ffffffff881ae107>] :virtio_pci:vp_interrupt+0xa9/0xb8 Aug 29 00:13:30 localhost kernel: [<ffffffff80010bab>] handle_IRQ_event+0x51/0xa6 Aug 29 00:13:30 localhost kernel: [<ffffffff800bae28>] __do_IRQ+0xa4/0x103 Aug 29 00:13:30 localhost kernel: [<ffffffff8006ca11>] do_IRQ+0xe7/0xf5 Aug 29 00:13:30 localhost kernel: [<ffffffff8005d615>] ret_from_intr+0x0/0xa Aug 29 00:13:31 localhost kernel: [<ffffffff881ed280>] :8139cp:cp_rx_poll+0x535/0x58c Aug 29 00:13:31 localhost kernel: [<ffffffff8000c88c>] net_rx_action+0xac/0x1e0 Aug 29 00:13:31 localhost kernel: [<ffffffff800123b4>] __do_softirq+0x89/0x133 Aug 29 00:13:31 localhost kernel: [<ffffffff8005e2fc>] call_softirq+0x1c/0x28 Aug 29 00:13:31 localhost kernel: [<ffffffff8006cb8e>] do_softirq+0x2c/0x85 Aug 29 00:13:32 localhost kernel: [<ffffffff8006ca16>] do_IRQ+0xec/0xf5 Aug 29 00:13:32 localhost kernel: [<ffffffff8006b346>] default_idle+0x0/0x50 Aug 29 00:13:32 localhost kernel: [<ffffffff8005d615>] ret_from_intr+0x0/0xa Aug 29 00:13:32 localhost kernel: <EOI> [<ffffffff8006b36f>] default_idle+0x29/0x50 Aug 29 00:13:32 localhost kernel: [<ffffffff800491ab>] cpu_idle+0x95/0xb8 Aug 29 00:13:32 localhost kernel: [<ffffffff8007796f>] start_secondary+0x498/0x4a7 Aug 29 00:13:33 localhost kernel: Aug 29 01:13:01 localhost kernel: INFO: task ncvlog:11720 blocked for more than 120 seconds. Aug 29 01:13:01 localhost kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 29 01:13:01 localhost kernel: ncvlog D ffff81000103f1a0 0 11720 11715 (NOTLB) Aug 29 01:13:01 localhost kernel: ffff810117de1d88 0000000000000082 ffff810117de1d98 ffffffff80062ff8 Aug 29 01:13:01 localhost kernel: 0000000000001000 0000000000000008 ffff810170c497a0 ffff810108764100 Aug 29 01:13:01 localhost kernel: 000003dd5444933d 00000000000007c1 ffff810170c49988 0000000700084cf3 Aug 29 01:13:01 localhost kernel: Call Trace: Aug 29 01:13:01 localhost kernel: [<ffffffff80062ff8>] thread_return+0x62/0xfe Aug 29 01:13:01 localhost kernel: [<ffffffff88036d8a>] :jbd:log_wait_commit+0xa3/0xf5 Aug 29 01:13:01 localhost kernel: [<ffffffff800a09d8>] autoremove_wake_function+0x0/0x2e Aug 29 01:13:01 localhost kernel: [<ffffffff80097cc5>] process_timeout+0x0/0x5 Aug 29 01:13:01 localhost kernel: [<ffffffff8803178a>] :jbd:journal_stop+0x1cf/0x1ff Aug 29 01:13:01 localhost kernel: [<ffffffff8002fbdf>] __writeback_single_inode+0x1e9/0x328 Aug 29 01:13:02 localhost kernel: [<ffffffff800f333b>] sync_inode+0x24/0x33 Aug 29 01:13:02 localhost kernel: [<ffffffff8804c36d>] :ext3:ext3_sync_file+0xc9/0xdc Aug 29 01:13:03 localhost kernel: [<ffffffff80050127>] do_fsync+0x52/0xa4 Aug 29 01:13:03 localhost kernel: [<ffffffff800e0f84>] __do_fsync+0x23/0x36 Aug 29 01:13:03 localhost kernel: [<ffffffff8006149d>] sysenter_do_call+0x1e/0x76 Aug 29 01:13:03 localhost kernel: Aug 29 01:13:04 localhost kernel: INFO: task ncvlog:11721 blocked for more than 120 seconds. Aug 29 01:13:04 localhost kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 29 01:13:04 localhost kernel: ncvlog D ffff810001004420 0 11721 11713 (NOTLB) Aug 29 01:13:05 localhost kernel: ffff810117debd88 0000000000000082 ffff810117debd98 ffffffff80062ff8 Aug 29 01:13:05 localhost kernel: 0000000000001000 0000000000000008 ffff81025e44f080 ffffffff80308b60 Aug 29 01:13:05 localhost kernel: 000003dd4b81a2b6 0000000000000729 ffff81025e44f268 000000000007bf51 Aug 29 01:13:06 localhost kernel: Call Trace: Aug 29 01:13:06 localhost kernel: [<ffffffff80062ff8>] thread_return+0x62/0xfe Aug 29 01:13:07 localhost kernel: [<ffffffff88036d8a>] :jbd:log_wait_commit+0xa3/0xf5 Aug 29 01:13:07 localhost kernel: [<ffffffff800a09d8>] autoremove_wake_function+0x0/0x2e Aug 29 01:13:07 localhost kernel: [<ffffffff8803178a>] :jbd:journal_stop+0x1cf/0x1ff Aug 29 01:13:07 localhost kernel: [<ffffffff8002fbdf>] __writeback_single_inode+0x1e9/0x328 Aug 29 01:13:07 localhost kernel: [<ffffffff800f333b>] sync_inode+0x24/0x33 Aug 29 01:13:08 localhost kernel: [<ffffffff8804c36d>] :ext3:ext3_sync_file+0xc9/0xdc Aug 29 01:13:08 localhost kernel: [<ffffffff80050127>] do_fsync+0x52/0xa4 Aug 29 01:13:08 localhost kernel: [<ffffffff800e0f84>] __do_fsync+0x23/0x36 Aug 29 01:13:08 localhost kernel: [<ffffffff8006149d>] sysenter_do_call+0x1e/0x76 Aug 29 01:13:08 localhost kernel: Aug 29 01:18:55 localhost kernel: BUG: soft lockup - CPU#0 stuck for 10s! [ncsim:18812] Aug 29 01:18:55 localhost kernel: CPU 0: Aug 29 01:18:55 localhost kernel: Modules linked in: ipv6 xfrm_nalgo crypto_api autofs4 hidp nfs fscache nfs_acl rfcomm l2cap bluetooth lockd sunrpc dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport floppy snd_ens1370 gameport snd_rawmidi snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_ak4531_codec snd 8139too i2c_piix4 soundcore virtio_blk 8139cp virtio_balloon i2c_core snd_page_alloc serio_raw virtio_pci virtio_ring virtio mii pcspkr dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod ata_piix libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd Aug 29 01:18:55 localhost kernel: Pid: 18812, comm: ncsim Not tainted 2.6.18-194.11.1.el5 #1 Aug 29 01:18:55 localhost kernel: RIP: 0010:[<ffffffff881ecd88>] [<ffffffff881ecd88>] :8139cp:cp_rx_poll+0x3d/0x58c Aug 29 01:18:55 localhost kernel: RSP: 0018:ffffffff80446ea0 EFLAGS: 00010202 Aug 29 01:18:55 localhost kernel: RAX: ffffc20000016100 RBX: ffff81025ca1e500 RCX: ffff81025ca1e6b8 Aug 29 01:18:55 localhost kernel: RDX: 0000000000000003 RSI: ffff81025fe0b680 RDI: 0000000000000000 Aug 29 01:18:55 localhost kernel: RBP: ffffffff80446e20 R08: ffff810259625d00 R09: 0000000000000010 Aug 29 01:18:55 localhost kernel: R10: ffff81010872e738 R11: 0000000000000000 R12: ffffffff8005dc8e Aug 29 01:18:55 localhost kernel: R13: ffff81024cb19880 R14: ffffffff80078203 R15: ffffffff80446e20 Aug 29 01:18:55 localhost kernel: FS: 0000000000000000(0000) GS:ffffffff803ca000(0063) knlGS:00000000f7f9a6c0 Aug 29 01:18:55 localhost kernel: CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b Aug 29 01:18:58 localhost kernel: CR2: 00007fff5bcc3ec8 CR3: 000000012cb9f000 CR4: 00000000000006e0 Aug 29 01:18:58 localhost kernel: Aug 29 01:18:59 localhost kernel: Call Trace: Aug 29 01:19:01 localhost kernel: <IRQ> [<ffffffff8000c88c>] net_rx_action+0xac/0x1e0 Aug 29 01:19:01 localhost kernel: [<ffffffff800123b4>] __do_softirq+0x89/0x133 Aug 29 01:19:02 localhost kernel: [<ffffffff8005e2fc>] call_softirq+0x1c/0x28 Aug 29 01:19:03 localhost kernel: [<ffffffff8006cb8e>] do_softirq+0x2c/0x85 Aug 29 01:19:04 localhost kernel: [<ffffffff8005dc8e>] apic_timer_interrupt+0x66/0x6c Aug 29 01:19:04 localhost kernel: <EOI> [<ffffffff8014e118>] cfq_may_queue+0x0/0x112 Aug 29 01:19:04 localhost kernel: [<ffffffff8804e080>] :ext3:ext3_journal_dirty_data+0x0/0x34 Aug 29 01:19:05 localhost kernel: [<ffffffff800133bb>] get_request+0x186/0x36f Aug 29 01:19:05 localhost kernel: [<ffffffff80013265>] get_request+0x30/0x36f Aug 29 01:19:05 localhost kernel: [<ffffffff80028b6f>] get_request_wait+0x2a/0x130 Aug 29 01:19:05 localhost kernel: [<ffffffff8000c041>] __make_request+0x350/0x414 Aug 29 01:19:06 localhost kernel: [<ffffffff8000b3e7>] __find_get_block+0x15c/0x16c Aug 29 01:19:06 localhost kernel: [<ffffffff8001c1c1>] generic_make_request+0x211/0x228 Aug 29 01:19:06 localhost kernel: [<ffffffff80019b79>] __getblk+0x25/0x236 Aug 29 01:19:06 localhost kernel: [<ffffffff800332f4>] submit_bio+0xe4/0xeb Aug 29 01:19:06 localhost kernel: [<ffffffff8001a90b>] submit_bh+0xf1/0x111 Aug 29 01:19:06 localhost kernel: [<ffffffff8003a8d4>] sync_dirty_buffer+0x7a/0xcb Aug 29 01:19:06 localhost kernel: [<ffffffff88032ba8>] :jbd:journal_dirty_data+0x100/0x205 Aug 29 01:19:06 localhost kernel: [<ffffffff8804e08f>] :ext3:ext3_journal_dirty_data+0xf/0x34 Aug 29 01:19:06 localhost kernel: [<ffffffff8804d4b8>] :ext3:walk_page_buffers+0x65/0x8b Aug 29 01:19:06 localhost kernel: [<ffffffff8804e080>] :ext3:ext3_journal_dirty_data+0x0/0x34 Aug 29 01:19:07 localhost kernel: [<ffffffff8804ff3d>] :ext3:ext3_ordered_write_end+0x7c/0x116 Aug 29 01:19:07 localhost kernel: [<ffffffff8000fd6c>] generic_file_buffered_write+0x1cc/0x675 Aug 29 01:19:07 localhost kernel: [<ffffffff880317ae>] :jbd:journal_stop+0x1f3/0x1ff Aug 29 01:19:07 localhost kernel: [<ffffffff8001660f>] __generic_file_aio_write_nolock+0x369/0x3b6 Aug 29 01:19:07 localhost kernel: [<ffffffff8000ae3c>] ia32_setup_sigcontext+0x141/0x1ef Aug 29 01:19:07 localhost kernel: [<ffffffff800217b2>] generic_file_aio_write+0x65/0xc1 Aug 29 01:19:07 localhost kernel: [<ffffffff8804c1b6>] :ext3:ext3_file_write+0x16/0x91 Aug 29 01:19:07 localhost kernel: [<ffffffff80018234>] do_sync_write+0xc7/0x104 Aug 29 01:19:07 localhost kernel: [<ffffffff800a09d8>] autoremove_wake_function+0x0/0x2e Aug 29 01:19:08 localhost kernel: [<ffffffff800624b6>] __sched_text_start+0xf6/0xbd6 Aug 29 01:19:08 localhost kernel: [<ffffffff80033713>] do_setitimer+0x171/0x55a Aug 29 01:19:08 localhost kernel: [<ffffffff80016a17>] vfs_write+0xce/0x174 Aug 29 01:19:08 localhost kernel: [<ffffffff800172e4>] sys_write+0x45/0x6e Aug 29 01:19:08 localhost kernel: [<ffffffff8006149d>] sysenter_do_call+0x1e/0x76 Aug 29 01:19:08 localhost kernel: Aug 29 01:27:02 localhost kernel: BUG: soft lockup - CPU#7 stuck for 10s! [ncvlog:19815] Aug 29 01:27:02 localhost kernel: CPU 7: Aug 29 01:27:02 localhost kernel: Modules linked in: ipv6 xfrm_nalgo crypto_api autofs4 hidp nfs fscache nfs_acl rfcomm l2cap bluetooth lockd sunrpc dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport floppy snd_ens1370 gameport snd_rawmidi snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_ak4531_codec snd 8139too i2c_piix4 soundcore virtio_blk 8139cp virtio_balloon i2c_core snd_page_alloc serio_raw virtio_pci virtio_ring virtio mii pcspkr dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod ata_piix libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd Aug 29 01:27:02 localhost kernel: Pid: 19815, comm: ncvlog Not tainted 2.6.18-194.11.1.el5 #1 Aug 29 01:27:02 localhost kernel: RIP: 0010:[<ffffffff881eb687>] [<ffffffff881eb687>] :8139cp:cp_interrupt+0xe8/0x362 Aug 29 01:27:02 localhost kernel: RSP: 0018:ffff8101087afd38 EFLAGS: 00010246 Aug 29 01:27:02 localhost kernel: RAX: ffffc200000280ac RBX: ffff81025c1ba500 RCX: 0000000000000001 Aug 29 01:27:02 localhost kernel: RDX: ffffc20000028200 RSI: ffff81025c1ba000 RDI: ffff81025c1ba000 Aug 29 01:27:02 localhost kernel: RBP: ffff8101087afcb0 R08: ffffffff8032aad8 R09: 0000000000001000 Aug 29 01:27:02 localhost kernel: R10: ffff81016b9b5178 R11: ffffffff800c6944 R12: ffffffff8005dc8e Aug 29 01:27:02 localhost kernel: R13: 0000000000000080 R14: ffffffff80078203 R15: ffff8101087afcb0 Aug 29 01:27:02 localhost kernel: FS: 0000000000000000(0000) GS:ffff810108753340(0063) knlGS:00000000f7f6c6c0 Aug 29 01:27:02 localhost kernel: CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b Aug 29 01:27:02 localhost kernel: CR2: 00007fff25944000 CR3: 00000001802bd000 CR4: 00000000000006e0 Aug 29 01:27:03 localhost kernel: Aug 29 01:27:03 localhost kernel: Call Trace: Aug 29 01:27:04 localhost kernel: <IRQ> [<ffffffff881eb62c>] :8139cp:cp_interrupt+0x8d/0x362 Aug 29 01:27:04 localhost kernel: [<ffffffff881ae107>] :virtio_pci:vp_interrupt+0xa9/0xb8 Aug 29 01:27:04 localhost kernel: [<ffffffff80010bab>] handle_IRQ_event+0x51/0xa6 Aug 29 01:27:04 localhost kernel: [<ffffffff800bae28>] __do_IRQ+0xa4/0x103 Aug 29 01:27:05 localhost kernel: [<ffffffff8006ca11>] do_IRQ+0xe7/0xf5 Aug 29 01:27:05 localhost kernel: [<ffffffff8005d615>] ret_from_intr+0x0/0xa Aug 29 01:27:05 localhost kernel: [<ffffffff881ed280>] :8139cp:cp_rx_poll+0x535/0x58c Aug 29 01:27:05 localhost kernel: [<ffffffff8001549c>] sync_buffer+0x0/0x3f Aug 29 01:27:06 localhost kernel: [<ffffffff8000c88c>] net_rx_action+0xac/0x1e0 Aug 29 01:27:06 localhost kernel: [<ffffffff800123b4>] __do_softirq+0x89/0x133 Aug 29 01:27:06 localhost kernel: [<ffffffff8005e2fc>] call_softirq+0x1c/0x28 Aug 29 01:27:06 localhost kernel: [<ffffffff8006cb8e>] do_softirq+0x2c/0x85 Aug 29 01:27:07 localhost kernel: [<ffffffff8005dc8e>] apic_timer_interrupt+0x66/0x6c Aug 29 01:27:07 localhost kernel: <EOI> [<ffffffff881ae53d>] :virtio_pci:vp_notify+0x0/0x1c Aug 29 01:27:07 localhost kernel: [<ffffffff8005a7db>] generic_unplug_device+0x31/0x32 Aug 29 01:27:07 localhost kernel: [<ffffffff800154d2>] sync_buffer+0x36/0x3f Aug 29 01:27:08 localhost kernel: [<ffffffff80063a16>] __wait_on_bit+0x40/0x6e Aug 29 01:27:08 localhost kernel: [<ffffffff8001549c>] sync_buffer+0x0/0x3f Aug 29 01:27:08 localhost kernel: [<ffffffff80063ab0>] out_of_line_wait_on_bit+0x6c/0x78 Aug 29 01:27:08 localhost kernel: [<ffffffff800a0a06>] wake_bit_function+0x0/0x23 Aug 29 01:27:09 localhost kernel: [<ffffffff8003a8f0>] sync_dirty_buffer+0x96/0xcb Aug 29 01:27:09 localhost kernel: [<ffffffff88032ba8>] :jbd:journal_dirty_data+0x100/0x205 Aug 29 01:27:09 localhost kernel: [<ffffffff8804e08f>] :ext3:ext3_journal_dirty_data+0xf/0x34 Aug 29 01:27:09 localhost kernel: [<ffffffff8804d4b8>] :ext3:walk_page_buffers+0x65/0x8b Aug 29 01:27:09 localhost kernel: [<ffffffff8804e080>] :ext3:ext3_journal_dirty_data+0x0/0x34 Aug 29 01:27:10 localhost kernel: [<ffffffff8804ff3d>] :ext3:ext3_ordered_write_end+0x7c/0x116 Aug 29 01:27:10 localhost kernel: [<ffffffff8000fd6c>] generic_file_buffered_write+0x1cc/0x675 Aug 29 01:27:10 localhost kernel: [<ffffffff8805c1a2>] :ext3:ext3_get_acl+0x63/0x310 Aug 29 01:27:10 localhost kernel: [<ffffffff8001660f>] __generic_file_aio_write_nolock+0x369/0x3b6 Aug 29 01:27:10 localhost kernel: [<ffffffff8000a73e>] __link_path_walk+0xf32/0xf5b Aug 29 01:27:11 localhost kernel: [<ffffffff800217b2>] generic_file_aio_write+0x65/0xc1 Aug 29 01:27:11 localhost kernel: [<ffffffff8804c1b6>] :ext3:ext3_file_write+0x16/0x91 Aug 29 01:27:11 localhost kernel: [<ffffffff80018234>] do_sync_write+0xc7/0x104 Aug 29 01:27:11 localhost kernel: [<ffffffff80066b88>] do_page_fault+0x4fe/0x874 Aug 29 01:27:12 localhost kernel: [<ffffffff800a09d8>] autoremove_wake_function+0x0/0x2e Aug 29 01:27:12 localhost kernel: [<ffffffff800274b2>] do_filp_open+0x2a/0x38 Aug 29 01:27:12 localhost kernel: [<ffffffff80016a17>] vfs_write+0xce/0x174 Aug 29 01:27:12 localhost kernel: [<ffffffff800172e4>] sys_write+0x45/0x6e Aug 29 01:27:12 localhost kernel: [<ffffffff8006149d>] sysenter_do_call+0x1e/0x76 Aug 29 01:27:12 localhost kernel: Aug 29 01:30:45 localhost kernel: BUG: soft lockup - CPU#7 stuck for 10s! [swapper:0] Aug 29 01:30:45 localhost kernel: CPU 7: Aug 29 01:30:45 localhost kernel: Modules linked in: ipv6 xfrm_nalgo crypto_api autofs4 hidp nfs fscache nfs_acl rfcomm l2cap bluetooth lockd sunrpc dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport floppy snd_ens1370 gameport snd_rawmidi snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_ak4531_codec snd 8139too i2c_piix4 soundcore virtio_blk 8139cp virtio_balloon i2c_core snd_page_alloc serio_raw virtio_pci virtio_ring virtio mii pcspkr dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod ata_piix libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd Aug 29 01:30:45 localhost kernel: Pid: 0, comm: swapper Not tainted 2.6.18-194.11.1.el5 #1 Aug 29 01:30:45 localhost kernel: RIP: 0010:[<ffffffff80064b50>] [<ffffffff80064b50>] _spin_unlock_irqrestore+0x8/0x9 Aug 29 01:30:45 localhost kernel: RSP: 0018:ffff8101087af870 EFLAGS: 00000246 Aug 29 01:30:45 localhost kernel: RAX: ffff81025c8a92a8 RBX: ffff81025c8a9000 RCX: 0000000000000001 Aug 29 01:30:45 localhost kernel: RDX: ffff81025c8a92a8 RSI: 0000000000000246 RDI: ffff81025c8a92a0 Aug 29 01:30:45 localhost kernel: RBP: ffff8101087af7f0 R08: 0000000000000000 R09: ffff81025b18a000 Aug 29 01:30:45 localhost kernel: R10: 00000000000000c0 R11: ffffffff881ae53d R12: ffffffff8005dc8e Aug 29 01:30:45 localhost kernel: R13: ffff81025c8a9290 R14: ffffffff80078203 R15: ffff8101087af7f0 Aug 29 01:30:45 localhost kernel: FS: 0000000000000000(0000) GS:ffff810108753340(0000) knlGS:0000000000000000 Aug 29 01:30:45 localhost kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 29 01:30:46 localhost kernel: CR2: 0000003d3ba9a490 CR3: 0000000000201000 CR4: 00000000000006e0 Aug 29 01:30:46 localhost kernel: Aug 29 01:30:46 localhost kernel: Call Trace: Aug 29 01:30:46 localhost kernel: <IRQ> [<ffffffff881ae107>] :virtio_pci:vp_interrupt+0xa9/0xb8 Aug 29 01:30:47 localhost kernel: [<ffffffff80010bab>] handle_IRQ_event+0x51/0xa6 Aug 29 01:30:47 localhost kernel: [<ffffffff800bae28>] __do_IRQ+0xa4/0x103 Aug 29 01:30:47 localhost kernel: [<ffffffff8006ca11>] do_IRQ+0xe7/0xf5 Aug 29 01:30:47 localhost kernel: [<ffffffff8005d615>] ret_from_intr+0x0/0xa Aug 29 01:30:47 localhost kernel: [<ffffffff881ed7c6>] :8139cp:cp_start_xmit+0x4ef/0x511 Aug 29 01:30:48 localhost kernel: [<ffffffff8022f296>] dev_hard_start_xmit+0x1b7/0x28a Aug 29 01:30:48 localhost kernel: [<ffffffff8023f5f8>] __qdisc_run+0x136/0x1f9 Aug 29 01:30:48 localhost kernel: [<ffffffff8002f872>] dev_queue_xmit+0x150/0x271 Aug 29 01:30:48 localhost kernel: [<ffffffff80031f5a>] ip_output+0x29a/0x2dd Aug 29 01:30:48 localhost kernel: [<ffffffff8003456f>] ip_queue_xmit+0x434/0x48e Aug 29 01:30:49 localhost kernel: [<ffffffff88423476>] :sunrpc:__rpc_do_wake_up_task+0x15a/0x180 Aug 29 01:30:49 localhost kernel: [<ffffffff80046e4c>] try_to_wake_up+0x472/0x484 Aug 29 01:30:49 localhost kernel: [<ffffffff88423a3b>] :sunrpc:rpc_wake_up_task+0x3e/0x55 Aug 29 01:30:49 localhost kernel: [<ffffffff80021faa>] tcp_transmit_skb+0x62f/0x667 Aug 29 01:30:49 localhost kernel: [<ffffffff80035ea3>] tcp_read_sock+0x189/0x19c Aug 29 01:30:50 localhost kernel: [<ffffffff88421c69>] :sunrpc:xs_tcp_data_recv+0x0/0x4f9 Aug 29 01:30:50 localhost kernel: [<ffffffff88422629>] :sunrpc:xs_tcp_data_ready+0x60/0x6d Aug 29 01:30:50 localhost kernel: [<ffffffff8001beda>] tcp_rcv_established+0x84f/0x925 Aug 29 01:30:53 localhost kernel: [<ffffffff8003b494>] tcp_v4_do_rcv+0x2a/0x2fa Aug 29 01:30:53 localhost kernel: [<ffffffff800272ae>] tcp_v4_rcv+0x9e0/0xa49 Aug 29 01:30:53 localhost kernel: [<ffffffff80079352>] end_level_ioapic_vector+0x9/0x16 Aug 29 01:30:53 localhost kernel: [<ffffffff800bae6f>] __do_IRQ+0xeb/0x103 Aug 29 01:30:54 localhost kernel: [<ffffffff80034766>] ip_local_deliver+0x19d/0x263 Aug 29 01:30:54 localhost kernel: [<ffffffff800358d4>] ip_rcv+0x539/0x57c Aug 29 01:30:54 localhost kernel: [<ffffffff80020a3b>] netif_receive_skb+0x470/0x49f Aug 29 01:30:54 localhost kernel: [<ffffffff881ed187>] :8139cp:cp_rx_poll+0x43c/0x58c Aug 29 01:30:54 localhost kernel: [<ffffffff8000c88c>] net_rx_action+0xac/0x1e0 Aug 29 01:30:55 localhost kernel: [<ffffffff800123b4>] __do_softirq+0x89/0x133 Aug 29 01:30:55 localhost kernel: [<ffffffff8005e2fc>] call_softirq+0x1c/0x28 Aug 29 01:30:55 localhost kernel: [<ffffffff8006cb8e>] do_softirq+0x2c/0x85 Aug 29 01:30:55 localhost kernel: [<ffffffff8006ca16>] do_IRQ+0xec/0xf5 Aug 29 01:30:55 localhost kernel: [<ffffffff8006b346>] default_idle+0x0/0x50 Aug 29 01:30:55 localhost kernel: [<ffffffff8005d615>] ret_from_intr+0x0/0xa Aug 29 01:30:56 localhost kernel: <EOI> [<ffffffff8006b36f>] default_idle+0x29/0x50 Aug 29 01:30:56 localhost kernel: [<ffffffff800491ab>] cpu_idle+0x95/0xb8 Aug 29 01:30:56 localhost kernel: [<ffffffff8007796f>] start_secondary+0x498/0x4a7 Aug 29 01:30:56 localhost kernel: Aug 29 01:34:26 localhost kernel: BUG: soft lockup - CPU#7 stuck for 10s! [swapper:0] Aug 29 01:34:26 localhost kernel: CPU 7: Aug 29 01:34:26 localhost kernel: Modules linked in: ipv6 xfrm_nalgo crypto_api autofs4 hidp nfs fscache nfs_acl rfcomm l2cap bluetooth lockd sunrpc dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport floppy snd_ens1370 gameport snd_rawmidi snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_ak4531_codec snd 8139too i2c_piix4 soundcore virtio_blk 8139cp virtio_balloon i2c_core snd_page_alloc serio_raw virtio_pci virtio_ring virtio mii pcspkr dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod ata_piix libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd Aug 29 01:34:26 localhost kernel: Pid: 0, comm: swapper Not tainted 2.6.18-194.11.1.el5 #1 Aug 29 01:34:26 localhost kernel: RIP: 0010:[<ffffffff88023adf>] [<ffffffff88023adf>] :uhci_hcd:uhci_irq+0x24/0x15c Aug 29 01:34:26 localhost kernel: RSP: 0000:ffff8101087af868 EFLAGS: 00000206 Aug 29 01:34:26 localhost kernel: RAX: ffffffff88020020 RBX: ffff81025eb9a150 RCX: 000000000000000b Aug 29 01:34:26 localhost kernel: RDX: 000000000000c022 RSI: ffff8101087af928 RDI: ffff81025eb9a000 Aug 29 01:34:26 localhost kernel: RBP: ffff8101087af7e0 R08: ffff810256953ce0 R09: 0000000000000000 Aug 29 01:34:26 localhost kernel: R10: 00000000f14f961d R11: 0000000000000004 R12: ffffffff8005dc8e Aug 29 01:34:26 localhost kernel: R13: ffff81025eb9a000 R14: ffffffff80078203 R15: ffff8101087af7e0 Aug 29 01:34:26 localhost kernel: FS: 0000000000000000(0000) GS:ffff810108753340(0000) knlGS:0000000000000000 Aug 29 01:34:26 localhost kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 29 01:34:30 localhost kernel: CR2: 000000000000f022 CR3: 0000000000201000 CR4: 00000000000006e0 Aug 29 01:34:31 localhost kernel: Aug 29 01:34:31 localhost kernel: Call Trace: Aug 29 01:34:31 localhost kernel: <IRQ> [<ffffffff801f74dc>] usb_hcd_irq+0x27/0x55 Aug 29 01:34:31 localhost kernel: [<ffffffff80010bab>] handle_IRQ_event+0x51/0xa6 Aug 29 01:34:31 localhost kernel: [<ffffffff800bae28>] __do_IRQ+0xa4/0x103 Aug 29 01:34:31 localhost kernel: [<ffffffff8006ca11>] do_IRQ+0xe7/0xf5 Aug 29 01:34:31 localhost kernel: [<ffffffff8005d615>] ret_from_intr+0x0/0xa Aug 29 01:34:32 localhost kernel: [<ffffffff881ed7c6>] :8139cp:cp_start_xmit+0x4ef/0x511 Aug 29 01:34:32 localhost kernel: [<ffffffff8022f296>] dev_hard_start_xmit+0x1b7/0x28a Aug 29 01:34:32 localhost kernel: [<ffffffff8023f5f8>] __qdisc_run+0x136/0x1f9 Aug 29 01:34:32 localhost kernel: [<ffffffff8002f872>] dev_queue_xmit+0x150/0x271 Aug 29 01:34:32 localhost kernel: [<ffffffff80031f5a>] ip_output+0x29a/0x2dd Aug 29 01:34:32 localhost kernel: [<ffffffff8003456f>] ip_queue_xmit+0x434/0x48e Aug 29 01:34:32 localhost kernel: [<ffffffff88423476>] :sunrpc:__rpc_do_wake_up_task+0x15a/0x180 Aug 29 01:34:33 localhost kernel: [<ffffffff88423a3b>] :sunrpc:rpc_wake_up_task+0x3e/0x55 Aug 29 01:34:33 localhost kernel: [<ffffffff80021faa>] tcp_transmit_skb+0x62f/0x667 Aug 29 01:34:33 localhost kernel: [<ffffffff80035ea3>] tcp_read_sock+0x189/0x19c Aug 29 01:34:33 localhost kernel: [<ffffffff88421c69>] :sunrpc:xs_tcp_data_recv+0x0/0x4f9 Aug 29 01:34:33 localhost kernel: [<ffffffff88422629>] :sunrpc:xs_tcp_data_ready+0x60/0x6d Aug 29 01:34:33 localhost kernel: [<ffffffff80055358>] sk_reset_timer+0xf/0x19 Aug 29 01:34:33 localhost kernel: [<ffffffff8001bcc5>] tcp_rcv_established+0x63a/0x925 Aug 29 01:34:34 localhost kernel: [<ffffffff8003b494>] tcp_v4_do_rcv+0x2a/0x2fa Aug 29 01:34:34 localhost kernel: [<ffffffff800272ae>] tcp_v4_rcv+0x9e0/0xa49 Aug 29 01:34:34 localhost kernel: [<ffffffff80079352>] end_level_ioapic_vector+0x9/0x16 Aug 29 01:34:34 localhost kernel: [<ffffffff800bae6f>] __do_IRQ+0xeb/0x103 Aug 29 01:34:34 localhost kernel: [<ffffffff80034766>] ip_local_deliver+0x19d/0x263 Aug 29 01:34:34 localhost kernel: [<ffffffff800358d4>] ip_rcv+0x539/0x57c Aug 29 01:34:35 localhost kernel: [<ffffffff80020a3b>] netif_receive_skb+0x470/0x49f Aug 29 01:34:35 localhost kernel: [<ffffffff881ed187>] :8139cp:cp_rx_poll+0x43c/0x58c Aug 29 01:34:35 localhost kernel: [<ffffffff8000c88c>] net_rx_action+0xac/0x1e0 Aug 29 01:34:35 localhost kernel: [<ffffffff800123b4>] __do_softirq+0x89/0x133 Aug 29 01:34:35 localhost kernel: [<ffffffff8005e2fc>] call_softirq+0x1c/0x28 Aug 29 01:34:35 localhost kernel: [<ffffffff8006cb8e>] do_softirq+0x2c/0x85 Aug 29 01:34:35 localhost kernel: [<ffffffff8006ca16>] do_IRQ+0xec/0xf5 Aug 29 01:34:36 localhost kernel: [<ffffffff8006b346>] default_idle+0x0/0x50 Aug 29 01:34:36 localhost kernel: [<ffffffff8005d615>] ret_from_intr+0x0/0xa Aug 29 01:34:36 localhost kernel: <EOI> [<ffffffff8006b36f>] default_idle+0x29/0x50 Aug 29 01:34:36 localhost kernel: [<ffffffff800491ab>] cpu_idle+0x95/0xb8 Aug 29 01:34:37 localhost kernel: [<ffffffff8007796f>] start_secondary+0x498/0x4a7 Aug 29 01:34:37 localhost kernel: Aug 29 02:19:41 localhost kernel: BUG: soft lockup - CPU#7 stuck for 10s! [swapper:0] Aug 29 02:19:41 localhost kernel: CPU 7: Aug 29 02:19:41 localhost kernel: Modules linked in: ipv6 xfrm_nalgo crypto_api autofs4 hidp nfs fscache nfs_acl rfcomm l2cap bluetooth lockd sunrpc dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport floppy snd_ens1370 gameport snd_rawmidi snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_ak4531_codec snd 8139too i2c_piix4 soundcore virtio_blk 8139cp virtio_balloon i2c_core snd_page_alloc serio_raw virtio_pci virtio_ring virtio mii pcspkr dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod ata_piix libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd Aug 29 02:19:41 localhost kernel: Pid: 0, comm: swapper Not tainted 2.6.18-194.11.1.el5 #1 Aug 29 02:19:41 localhost kernel: RIP: 0010:[<ffffffff80154837>] [<ffffffff80154837>] ioread8+0x25/0x2f Aug 29 02:19:41 localhost kernel: RSP: 0018:ffff8101087afd90 EFLAGS: 00000246 Aug 29 02:19:41 localhost kernel: RAX: 0000000000010001 RBX: ffff81025c8a9000 RCX: 000000000000000b Aug 29 02:19:41 localhost kernel: RDX: 000000000000c513 RSI: ffff81025c8a9000 RDI: 000000000001c513 Aug 29 02:19:41 localhost kernel: RBP: ffff8101087afd10 R08: ffffffff80363008 R09: ffff8101087afe48 Aug 29 02:19:41 localhost kernel: R10: ffff8101087afd90 R11: ffffffff80363008 R12: ffffffff8005dc8e Aug 29 02:19:41 localhost kernel: R13: 0000000000000000 R14: ffffffff80078203 R15: ffff8101087afd10 Aug 29 02:19:41 localhost kernel: FS: 0000000000000000(0000) GS:ffff810108753340(0000) knlGS:0000000000000000 Aug 29 02:19:41 localhost kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 29 02:19:41 localhost kernel: CR2: 0000003d3b6055d0 CR3: 0000000252db2000 CR4: 00000000000006e0 Aug 29 02:19:41 localhost kernel: Aug 29 02:19:42 localhost kernel: Call Trace: Aug 29 02:19:42 localhost kernel: <IRQ> [<ffffffff881ae085>] :virtio_pci:vp_interrupt+0x27/0xb8 Aug 29 02:19:43 localhost kernel: [<ffffffff80010bab>] handle_IRQ_event+0x51/0xa6 Aug 29 02:19:43 localhost kernel: [<ffffffff800bae28>] __do_IRQ+0xa4/0x103 Aug 29 02:19:43 localhost kernel: [<ffffffff80251b1f>] inet_twdr_hangman+0x0/0xa5 Aug 29 02:19:43 localhost kernel: [<ffffffff8006ca11>] do_IRQ+0xe7/0xf5 Aug 29 02:19:44 localhost kernel: [<ffffffff80046e4c>] try_to_wake_up+0x472/0x484 Aug 29 02:19:44 localhost kernel: [<ffffffff8005d615>] ret_from_intr+0x0/0xa Aug 29 02:19:44 localhost kernel: [<ffffffff80097a79>] run_timer_softirq+0x162/0x241 Aug 29 02:19:44 localhost kernel: [<ffffffff800123b4>] __do_softirq+0x89/0x133 Aug 29 02:19:44 localhost kernel: [<ffffffff8005e2fc>] call_softirq+0x1c/0x28 Aug 29 02:19:45 localhost kernel: [<ffffffff8006cb8e>] do_softirq+0x2c/0x85 Aug 29 02:19:45 localhost kernel: [<ffffffff8006ca16>] do_IRQ+0xec/0xf5 Aug 29 02:19:45 localhost kernel: [<ffffffff8006b346>] default_idle+0x0/0x50 Aug 29 02:19:45 localhost kernel: [<ffffffff8005d615>] ret_from_intr+0x0/0xa Aug 29 02:19:45 localhost kernel: <EOI> [<ffffffff8006b36f>] default_idle+0x29/0x50 Aug 29 02:19:45 localhost kernel: [<ffffffff800491ab>] cpu_idle+0x95/0xb8 Aug 29 02:19:45 localhost kernel: [<ffffffff8007796f>] start_secondary+0x498/0x4a7 Aug 29 02:19:46 localhost kernel: Aug 29 02:21:25 localhost kernel: BUG: soft lockup - CPU#7 stuck for 10s! [events/7:33] Aug 29 02:21:25 localhost kernel: CPU 7: Aug 29 02:21:25 localhost kernel: Modules linked in: ipv6 xfrm_nalgo crypto_api autofs4 hidp nfs fscache nfs_acl rfcomm l2cap bluetooth lockd sunrpc dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport floppy snd_ens1370 gameport snd_rawmidi snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_ak4531_codec snd 8139too i2c_piix4 soundcore virtio_blk 8139cp virtio_balloon i2c_core snd_page_alloc serio_raw virtio_pci virtio_ring virtio mii pcspkr dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod ata_piix libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd Aug 29 02:21:25 localhost kernel: Pid: 33, comm: events/7 Not tainted 2.6.18-194.11.1.el5 #1 Aug 29 02:21:25 localhost kernel: RIP: 0010:[<ffffffff80154837>] [<ffffffff80154837>] ioread8+0x25/0x2f Aug 29 02:21:25 localhost kernel: RSP: 0018:ffff8101087afd20 EFLAGS: 00000246 Aug 29 02:21:25 localhost kernel: RAX: 0000000000010001 RBX: ffff81025c8a9000 RCX: 000000000000000b Aug 29 02:21:25 localhost kernel: RDX: 000000000000c513 RSI: ffff81025c8a9000 RDI: 000000000001c513 Aug 29 02:21:25 localhost kernel: RBP: ffff8101087afca0 R08: ffff810108547b88 R09: ffff8101087afdd8 Aug 29 02:21:25 localhost kernel: R10: 000000000003e380 R11: 0000000000000074 R12: ffffffff8005dc8e Aug 29 02:21:25 localhost kernel: R13: 0000000000000000 R14: ffffffff80078203 R15: ffff8101087afca0 Aug 29 02:21:25 localhost kernel: FS: 0000000000000000(0000) GS:ffff810108753340(0000) knlGS:0000000000000000 Aug 29 02:21:25 localhost kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 29 02:21:26 localhost kernel: CR2: 0000000008db0000 CR3: 00000002532c0000 CR4: 00000000000006e0 Aug 29 02:21:26 localhost kernel: Aug 29 02:21:26 localhost kernel: Call Trace: Aug 29 02:21:27 localhost kernel: <IRQ> [<ffffffff881ae085>] :virtio_pci:vp_interrupt+0x27/0xb8 Aug 29 02:21:27 localhost kernel: [<ffffffff80010bab>] handle_IRQ_event+0x51/0xa6 Aug 29 02:21:27 localhost kernel: [<ffffffff800bae28>] __do_IRQ+0xa4/0x103 Aug 29 02:21:27 localhost kernel: [<ffffffff8006ca11>] do_IRQ+0xe7/0xf5 Aug 29 02:21:28 localhost kernel: [<ffffffff8005d615>] ret_from_intr+0x0/0xa Aug 29 02:21:28 localhost kernel: [<ffffffff881ed280>] :8139cp:cp_rx_poll+0x535/0x58c Aug 29 02:21:28 localhost kernel: [<ffffffff8000c88c>] net_rx_action+0xac/0x1e0 Aug 29 02:21:29 localhost kernel: [<ffffffff800123b4>] __do_softirq+0x89/0x133 Aug 29 02:21:29 localhost kernel: [<ffffffff8005e2fc>] call_softirq+0x1c/0x28 Aug 29 02:21:29 localhost kernel: [<ffffffff8006cb8e>] do_softirq+0x2c/0x85 Aug 29 02:21:29 localhost kernel: [<ffffffff8006ca16>] do_IRQ+0xec/0xf5 Aug 29 02:21:29 localhost kernel: [<ffffffff8005d615>] ret_from_intr+0x0/0xa Aug 29 02:21:30 localhost kernel: <EOI> [<ffffffff800dc069>] drain_array+0xb5/0xc0 Aug 29 02:21:30 localhost kernel: [<ffffffff800dca7c>] cache_reap+0x0/0x217 Aug 29 02:21:30 localhost kernel: [<ffffffff800dcb21>] cache_reap+0xa5/0x217 Aug 29 02:21:30 localhost kernel: [<ffffffff8004d624>] run_workqueue+0x94/0xe4 Aug 29 02:21:30 localhost kernel: [<ffffffff80049e5f>] worker_thread+0x0/0x122 Aug 29 02:21:31 localhost kernel: [<ffffffff80049f4f>] worker_thread+0xf0/0x122 Aug 29 02:21:31 localhost kernel: [<ffffffff8008cfa1>] default_wake_function+0x0/0xe Aug 29 02:21:31 localhost kernel: [<ffffffff8003287b>] kthread+0xfe/0x132 Aug 29 02:21:32 localhost kernel: [<ffffffff8005dfb1>] child_rip+0xa/0x11 Aug 29 02:21:32 localhost kernel: [<ffffffff8003277d>] kthread+0x0/0x132 Aug 29 02:21:32 localhost kernel: [<ffffffff8005dfa7>] child_rip+0x0/0x11 Aug 29 02:21:32 localhost kernel: Aug 29 02:30:43 localhost kernel: BUG: soft lockup - CPU#7 stuck for 10s! [swapper:0] Aug 29 02:30:43 localhost kernel: CPU 7: Aug 29 02:30:43 localhost kernel: Modules linked in: ipv6 xfrm_nalgo crypto_api autofs4 hidp nfs fscache nfs_acl rfcomm l2cap bluetooth lockd sunrpc dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport floppy snd_ens1370 gameport snd_rawmidi snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_ak4531_codec snd 8139too i2c_piix4 soundcore virtio_blk 8139cp virtio_balloon i2c_core snd_page_alloc serio_raw virtio_pci virtio_ring virtio mii pcspkr dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod ata_piix libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd Aug 29 02:30:43 localhost kernel: Pid: 0, comm: swapper Not tainted 2.6.18-194.11.1.el5 #1 Aug 29 02:30:43 localhost kernel: RIP: 0010:[<ffffffff80064b50>] [<ffffffff80064b50>] _spin_unlock_irqrestore+0x8/0x9 Aug 29 02:30:43 localhost kernel: RSP: 0018:ffff8101087afd20 EFLAGS: 00000246 Aug 29 02:30:43 localhost kernel: RAX: ffff81025c8a92a8 RBX: ffff81025c8a9000 RCX: 0000000000000001 Aug 29 02:30:43 localhost kernel: RDX: ffff81025c8a92a8 RSI: 0000000000000246 RDI: ffff81025c8a92a0 Aug 29 02:30:43 localhost kernel: RBP: ffff8101087afca0 R08: ffffffff8032aad8 R09: 0000000000001000 Aug 29 02:30:43 localhost kernel: R10: ffff810253ab6a38 R11: ffffffff800c6944 R12: ffffffff8005dc8e Aug 29 02:30:43 localhost kernel: R13: ffff81025c8a9290 R14: ffffffff80078203 R15: ffff8101087afca0 Aug 29 02:30:43 localhost kernel: FS: 0000000000000000(0000) GS:ffff810108753340(0000) knlGS:0000000000000000 Aug 29 02:30:43 localhost kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 29 02:30:43 localhost kernel: CR2: 0000003d3ba9a490 CR3: 00000001515ef000 CR4: 00000000000006e0 Aug 29 02:30:44 localhost kernel: Aug 29 02:30:44 localhost kernel: Call Trace: Aug 29 02:30:44 localhost kernel: <IRQ> [<ffffffff881ae107>] :virtio_pci:vp_interrupt+0xa9/0xb8 Aug 29 02:30:44 localhost kernel: [<ffffffff80010bab>] handle_IRQ_event+0x51/0xa6 Aug 29 02:30:45 localhost kernel: [<ffffffff800bae28>] __do_IRQ+0xa4/0x103 Aug 29 02:30:45 localhost kernel: [<ffffffff8006ca11>] do_IRQ+0xe7/0xf5 Aug 29 02:30:45 localhost kernel: [<ffffffff8005d615>] ret_from_intr+0x0/0xa Aug 29 02:30:45 localhost kernel: [<ffffffff881ed280>] :8139cp:cp_rx_poll+0x535/0x58c Aug 29 02:30:45 localhost kernel: [<ffffffff8000c88c>] net_rx_action+0xac/0x1e0 Aug 29 02:30:45 localhost kernel: [<ffffffff800123b4>] __do_softirq+0x89/0x133 Aug 29 02:30:45 localhost kernel: [<ffffffff8005e2fc>] call_softirq+0x1c/0x28 Aug 29 02:30:45 localhost kernel: [<ffffffff8006cb8e>] do_softirq+0x2c/0x85 Aug 29 02:30:46 localhost kernel: [<ffffffff8006ca16>] do_IRQ+0xec/0xf5 Aug 29 02:30:46 localhost kernel: [<ffffffff8006b346>] default_idle+0x0/0x50 Aug 29 02:30:46 localhost kernel: [<ffffffff8005d615>] ret_from_intr+0x0/0xa Aug 29 02:30:46 localhost kernel: <EOI> [<ffffffff8014e118>] cfq_may_queue+0x0/0x112 Aug 29 02:30:46 localhost kernel: [<ffffffff8006b36f>] default_idle+0x29/0x50 Aug 29 02:30:46 localhost kernel: [<ffffffff800491ab>] cpu_idle+0x95/0xb8 Aug 29 02:30:47 localhost kernel: [<ffffffff8007796f>] start_secondary+0x498/0x4a7 Aug 29 02:30:47 localhost kernel: Aug 29 03:09:27 localhost kernel: BUG: soft lockup - CPU#7 stuck for 10s! [ncvlog:28284] Aug 29 03:09:27 localhost kernel: CPU 7: Aug 29 03:09:27 localhost kernel: Modules linked in: ipv6 xfrm_nalgo crypto_api autofs4 hidp nfs fscache nfs_acl rfcomm l2cap bluetooth lockd sunrpc dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport floppy snd_ens1370 gameport snd_rawmidi snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_ak4531_codec snd 8139too i2c_piix4 soundcore virtio_blk 8139cp virtio_balloon i2c_core snd_page_alloc serio_raw virtio_pci virtio_ring virtio mii pcspkr dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod ata_piix libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd Aug 29 03:09:27 localhost kernel: Pid: 28284, comm: ncvlog Not tainted 2.6.18-194.11.1.el5 #1 Aug 29 03:09:27 localhost kernel: RIP: 0010:[<ffffffff88023adf>] [<ffffffff88023adf>] :uhci_hcd:uhci_irq+0x24/0x15c Aug 29 03:09:27 localhost kernel: RSP: 0018:ffff8101087afdb8 EFLAGS: 00000206 Aug 29 03:09:27 localhost kernel: RAX: ffffffff88020020 RBX: ffff81025eb9a150 RCX: 000000000000000b Aug 29 03:09:27 localhost kernel: RDX: 000000000000c022 RSI: ffff8101087afe78 RDI: ffff81025eb9a000 Aug 29 03:09:27 localhost kernel: RBP: ffff8101087afd30 R08: 0000000000000040 R09: 000000000000006c Aug 29 03:09:27 localhost kernel: R10: 000000000003e380 R11: ffff8101087afee0 R12: ffffffff8005dc8e Aug 29 03:09:27 localhost kernel: R13: ffff81025eb9a000 R14: ffffffff80078203 R15: ffff8101087afd30 Aug 29 03:09:27 localhost kernel: FS: 0000000000000000(0000) GS:ffff810108753340(0063) knlGS:00000000f7f2e6c0 Aug 29 03:09:27 localhost kernel: CS: 0010 DS: 002b ES: 002b CR0: 0000000080050033 Aug 29 03:09:27 localhost kernel: CR2: 0000000008636f78 CR3: 0000000212f89000 CR4: 00000000000006e0 Aug 29 03:09:27 localhost kernel: Aug 29 03:09:28 localhost kernel: Call Trace: Aug 29 03:09:28 localhost kernel: <IRQ> [<ffffffff801f74dc>] usb_hcd_irq+0x27/0x55 Aug 29 03:09:28 localhost kernel: [<ffffffff80010bab>] handle_IRQ_event+0x51/0xa6 Aug 29 03:09:28 localhost kernel: [<ffffffff800bae28>] __do_IRQ+0xa4/0x103 Aug 29 03:09:29 localhost kernel: [<ffffffff80258bf4>] tcp_delack_timer+0x0/0x1ed Aug 29 03:09:29 localhost kernel: [<ffffffff8006ca11>] do_IRQ+0xe7/0xf5 Aug 29 03:09:29 localhost kernel: [<ffffffff881ed280>] :8139cp:cp_rx_poll+0x535/0x58c Aug 29 03:09:29 localhost kernel: [<ffffffff8005d615>] ret_from_intr+0x0/0xa Aug 29 03:09:30 localhost kernel: [<ffffffff80097b4d>] run_timer_softirq+0x236/0x241 Aug 29 03:09:30 localhost kernel: [<ffffffff800123b4>] __do_softirq+0x89/0x133 Aug 29 03:09:30 localhost kernel: [<ffffffff8005e2fc>] call_softirq+0x1c/0x28 Aug 29 03:09:31 localhost kernel: [<ffffffff8006cb8e>] do_softirq+0x2c/0x85 Aug 29 03:09:31 localhost kernel: [<ffffffff8005dc8e>] apic_timer_interrupt+0x66/0x6c Aug 29 03:09:31 localhost kernel: <EOI> [<ffffffff881ae53d>] :virtio_pci:vp_notify+0x0/0x1c Aug 29 03:09:31 localhost kernel: [<ffffffff8000c0f4>] __make_request+0x403/0x414 Aug 29 03:09:32 localhost kernel: [<ffffffff8000c0b6>] __make_request+0x3c5/0x414 Aug 29 03:09:32 localhost kernel: [<ffffffff8000fdab>] generic_file_buffered_write+0x20b/0x675 Aug 29 03:09:32 localhost kernel: [<ffffffff8001c1c1>] generic_make_request+0x211/0x228 Aug 29 03:09:32 localhost kernel: [<ffffffff800332f4>] submit_bio+0xe4/0xeb Aug 29 03:09:32 localhost kernel: [<ffffffff8001a90b>] submit_bh+0xf1/0x111 Aug 29 03:09:33 localhost kernel: [<ffffffff8001c3ce>] __block_write_full_page+0x1f6/0x301 Aug 29 03:09:33 localhost kernel: [<ffffffff8804eceb>] :ext3:ext3_get_block+0x0/0xf7 Aug 29 03:09:33 localhost kernel: [<ffffffff8805056a>] :ext3:ext3_ordered_writepage+0xf1/0x198 Aug 29 03:09:33 localhost kernel: [<ffffffff8001cf41>] mpage_writepages+0x1bf/0x37d Aug 29 03:09:33 localhost kernel: [<ffffffff88050479>] :ext3:ext3_ordered_writepage+0x0/0x198 Aug 29 03:09:34 localhost kernel: [<ffffffff8005abe0>] do_writepages+0x29/0x2f Aug 29 03:09:34 localhost kernel: [<ffffffff8004f9fb>] __filemap_fdatawrite_range+0x50/0x5b Aug 29 03:09:34 localhost kernel: [<ffffffff80050104>] do_fsync+0x2f/0xa4 Aug 29 03:09:34 localhost kernel: [<ffffffff800e0f84>] __do_fsync+0x23/0x36 Aug 29 03:09:34 localhost kernel: [<ffffffff8006149d>] sysenter_do_call+0x1e/0x76 Aug 29 03:09:35 localhost kernel: Aug 29 03:33:14 localhost kernel: BUG: soft lockup - CPU#7 stuck for 10s! [swapper:0] Aug 29 03:33:14 localhost kernel: CPU 7: Aug 29 03:33:14 localhost kernel: Modules linked in: ipv6 xfrm_nalgo crypto_api autofs4 hidp nfs fscache nfs_acl rfcomm l2cap bluetooth lockd sunrpc dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport floppy snd_ens1370 gameport snd_rawmidi snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_ak4531_codec snd 8139too i2c_piix4 soundcore virtio_blk 8139cp virtio_balloon i2c_core snd_page_alloc serio_raw virtio_pci virtio_ring virtio mii pcspkr dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod ata_piix libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd Aug 29 03:33:14 localhost kernel: Pid: 0, comm: swapper Not tainted 2.6.18-194.11.1.el5 #1 Aug 29 03:33:14 localhost kernel: RIP: 0010:[<ffffffff88023adf>] [<ffffffff88023adf>] :uhci_hcd:uhci_irq+0x24/0x15c Aug 29 03:33:14 localhost kernel: RSP: 0018:ffff8101087afa78 EFLAGS: 00000206 Aug 29 03:33:14 localhost kernel: RAX: ffffffff88020020 RBX: ffff81025eb9a150 RCX: 000000000000000b Aug 29 03:33:14 localhost kernel: RDX: 000000000000c022 RSI: ffff8101087afb38 RDI: ffff81025eb9a000 Aug 29 03:33:14 localhost kernel: RBP: ffff8101087af9f0 R08: ffff81010853bac8 R09: 000000000000006c Aug 29 03:33:14 localhost kernel: R10: ffff8101087af930 R11: 000000000000037f R12: ffffffff8005dc8e Aug 29 03:33:14 localhost kernel: R13: ffff81025eb9a000 R14: ffffffff80078203 R15: ffff8101087af9f0 Aug 29 03:33:14 localhost kernel: FS: 0000000000000000(0000) GS:ffff810108753340(0000) knlGS:0000000000000000 Aug 29 03:33:14 localhost kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 29 03:33:15 localhost kernel: CR2: 0000003d3d6aa700 CR3: 00000001f2432000 CR4: 00000000000006e0 Aug 29 03:33:15 localhost kernel: Aug 29 03:33:16 localhost kernel: Call Trace: Aug 29 03:33:16 localhost kernel: <IRQ> [<ffffffff801f74dc>] usb_hcd_irq+0x27/0x55 Aug 29 03:33:16 localhost kernel: [<ffffffff80010bab>] handle_IRQ_event+0x51/0xa6 Aug 29 03:33:17 localhost kernel: [<ffffffff800bae28>] __do_IRQ+0xa4/0x103 Aug 29 03:33:17 localhost kernel: [<ffffffff8006ca11>] do_IRQ+0xe7/0xf5 Aug 29 03:33:19 localhost kernel: [<ffffffff88423476>] :sunrpc:__rpc_do_wake_up_task+0x15a/0x180 Aug 29 03:33:19 localhost kernel: [<ffffffff8005d615>] ret_from_intr+0x0/0xa Aug 29 03:33:19 localhost kernel: [<ffffffff80229cde>] skb_release_head_state+0x15/0x108 Aug 29 03:33:19 localhost kernel: [<ffffffff80028b1c>] __kfree_skb+0x9/0x1a Aug 29 03:33:19 localhost kernel: [<ffffffff80035e23>] tcp_read_sock+0x109/0x19c Aug 29 03:33:20 localhost kernel: [<ffffffff88421c69>] :sunrpc:xs_tcp_data_recv+0x0/0x4f9 Aug 29 03:33:20 localhost kernel: [<ffffffff88422629>] :sunrpc:xs_tcp_data_ready+0x60/0x6d Aug 29 03:33:20 localhost kernel: [<ffffffff8001bcc5>] tcp_rcv_established+0x63a/0x925 Aug 29 03:33:20 localhost kernel: [<ffffffff8005dc8e>] apic_timer_interrupt+0x66/0x6c Aug 29 03:33:21 localhost kernel: [<ffffffff8003b494>] tcp_v4_do_rcv+0x2a/0x2fa Aug 29 03:33:21 localhost kernel: [<ffffffff80064b50>] _spin_unlock_irqrestore+0x8/0x9 Aug 29 03:33:21 localhost kernel: [<ffffffff800272ae>] tcp_v4_rcv+0x9e0/0xa49 Aug 29 03:33:22 localhost kernel: [<ffffffff80079352>] end_level_ioapic_vector+0x9/0x16 Aug 29 03:33:22 localhost kernel: [<ffffffff800bae6f>] __do_IRQ+0xeb/0x103 Aug 29 03:33:22 localhost kernel: [<ffffffff80034766>] ip_local_deliver+0x19d/0x263 Aug 29 03:33:23 localhost kernel: [<ffffffff800358d4>] ip_rcv+0x539/0x57c Aug 29 03:33:23 localhost kernel: [<ffffffff80020a3b>] netif_receive_skb+0x470/0x49f Aug 29 03:33:23 localhost kernel: [<ffffffff881ed187>] :8139cp:cp_rx_poll+0x43c/0x58c Aug 29 03:33:24 localhost kernel: [<ffffffff8000c88c>] net_rx_action+0xac/0x1e0 Aug 29 03:33:24 localhost kernel: [<ffffffff800123b4>] __do_softirq+0x89/0x133 Aug 29 03:33:25 localhost kernel: [<ffffffff8005e2fc>] call_softirq+0x1c/0x28 Aug 29 03:33:25 localhost kernel: [<ffffffff8006cb8e>] do_softirq+0x2c/0x85 Aug 29 03:33:25 localhost kernel: [<ffffffff8006ca16>] do_IRQ+0xec/0xf5 Aug 29 03:33:26 localhost kernel: [<ffffffff8006b346>] default_idle+0x0/0x50 Aug 29 03:33:26 localhost kernel: [<ffffffff8005d615>] ret_from_intr+0x0/0xa Aug 29 03:33:26 localhost kernel: <EOI> [<ffffffff8005a7aa>] generic_unplug_device+0x0/0x32 Aug 29 03:33:27 localhost kernel: [<ffffffff8006b36f>] default_idle+0x29/0x50 Aug 29 03:33:27 localhost kernel: [<ffffffff800491ab>] cpu_idle+0x95/0xb8 Aug 29 03:33:28 localhost kernel: [<ffffffff8007796f>] start_secondary+0x498/0x4a7 Aug 29 03:33:28 localhost kernel: Aug 29 03:34:14 localhost kernel: BUG: soft lockup - CPU#7 stuck for 10s! [swapper:0] Aug 29 03:34:14 localhost kernel: CPU 7: Aug 29 03:34:14 localhost kernel: Modules linked in: ipv6 xfrm_nalgo crypto_api autofs4 hidp nfs fscache nfs_acl rfcomm l2cap bluetooth lockd sunrpc dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport floppy snd_ens1370 gameport snd_rawmidi snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_ak4531_codec snd 8139too i2c_piix4 soundcore virtio_blk 8139cp virtio_balloon i2c_core snd_page_alloc serio_raw virtio_pci virtio_ring virtio mii pcspkr dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod ata_piix libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd Aug 29 03:34:14 localhost kernel: Pid: 0, comm: swapper Not tainted 2.6.18-194.11.1.el5 #1 Aug 29 03:34:14 localhost kernel: RIP: 0010:[<ffffffff88023adf>] [<ffffffff88023adf>] :uhci_hcd:uhci_irq+0x24/0x15c Aug 29 03:34:14 localhost kernel: RSP: 0018:ffff8101087af868 EFLAGS: 00000206 Aug 29 03:34:14 localhost kernel: RAX: ffffffff88020020 RBX: ffff81025eb9a150 RCX: 000000000000000b Aug 29 03:34:14 localhost kernel: RDX: 000000000000c022 RSI: ffff8101087af928 RDI: ffff81025eb9a000 Aug 29 03:34:14 localhost kernel: RBP: ffff8101087af7e0 R08: ffff810256953ce0 R09: 0000000000000000 Aug 29 03:34:14 localhost kernel: R10: 000000001166b020 R11: 0000000000000004 R12: ffffffff8005dc8e Aug 29 03:34:14 localhost kernel: R13: ffff81025eb9a000 R14: ffffffff80078203 R15: ffff8101087af7e0 Aug 29 03:34:14 localhost kernel: FS: 0000000000000000(0000) GS:ffff810108753340(0000) knlGS:0000000000000000 Aug 29 03:34:14 localhost kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Aug 29 03:34:15 localhost kernel: CR2: 0000000000bd9ac0 CR3: 0000000000201000 CR4: 00000000000006e0 Aug 29 03:34:16 localhost kernel: Aug 29 03:34:17 localhost kernel: Call Trace: Aug 29 03:34:17 localhost kernel: <IRQ> [<ffffffff801f74dc>] usb_hcd_irq+0x27/0x55 Aug 29 03:34:17 localhost kernel: [<ffffffff80010bab>] handle_IRQ_event+0x51/0xa6 Aug 29 03:34:18 localhost kernel: [<ffffffff800bae28>] __do_IRQ+0xa4/0x103 Aug 29 03:34:18 localhost kernel: [<ffffffff8006ca11>] do_IRQ+0xe7/0xf5 Aug 29 03:34:18 localhost kernel: [<ffffffff8005d615>] ret_from_intr+0x0/0xa Aug 29 03:34:18 localhost kernel: [<ffffffff881ed7c6>] :8139cp:cp_start_xmit+0x4ef/0x511 Aug 29 03:34:18 localhost kernel: [<ffffffff8022f296>] dev_hard_start_xmit+0x1b7/0x28a Aug 29 03:34:19 localhost kernel: [<ffffffff8023f5f8>] __qdisc_run+0x136/0x1f9 Aug 29 03:34:19 localhost kernel: [<ffffffff8002f872>] dev_queue_xmit+0x150/0x271 Aug 29 03:34:19 localhost kernel: [<ffffffff80031f5a>] ip_output+0x29a/0x2dd Aug 29 03:34:19 localhost kernel: [<ffffffff8003456f>] ip_queue_xmit+0x434/0x48e Aug 29 03:34:19 localhost kernel: [<ffffffff88423476>] :sunrpc:__rpc_do_wake_up_task+0x15a/0x180 Aug 29 03:34:20 localhost kernel: [<ffffffff80046e4c>] try_to_wake_up+0x472/0x484 Aug 29 03:34:20 localhost kernel: [<ffffffff88423a3b>] :sunrpc:rpc_wake_up_task+0x3e/0x55 Aug 29 03:34:20 localhost kernel: [<ffffffff80021faa>] tcp_transmit_skb+0x62f/0x667 Aug 29 03:34:20 localhost kernel: [<ffffffff80035ea3>] tcp_read_sock+0x189/0x19c Aug 29 03:34:20 localhost kernel: [<ffffffff88421c69>] :sunrpc:xs_tcp_data_recv+0x0/0x4f9 Aug 29 03:34:21 localhost kernel: [<ffffffff88422629>] :sunrpc:xs_tcp_data_ready+0x60/0x6d Aug 29 03:34:21 localhost kernel: [<ffffffff8001beda>] tcp_rcv_established+0x84f/0x925 Aug 29 03:34:21 localhost kernel: [<ffffffff8003b494>] tcp_v4_do_rcv+0x2a/0x2fa Aug 29 03:34:21 localhost kernel: [<ffffffff881eb5cb>] :8139cp:cp_interrupt+0x2c/0x362 Aug 29 03:34:21 localhost kernel: [<ffffffff800272ae>] tcp_v4_rcv+0x9e0/0xa49 Aug 29 03:34:22 localhost kernel: [<ffffffff80034766>] ip_local_deliver+0x19d/0x263 Aug 29 03:34:22 localhost kernel: [<ffffffff800358d4>] ip_rcv+0x539/0x57c Aug 29 03:34:22 localhost kernel: [<ffffffff80020a3b>] netif_receive_skb+0x470/0x49f Aug 29 03:34:22 localhost kernel: [<ffffffff881ed187>] :8139cp:cp_rx_poll+0x43c/0x58c Aug 29 03:34:23 localhost kernel: [<ffffffff8000c88c>] net_rx_action+0xac/0x1e0 Aug 29 03:34:23 localhost kernel: [<ffffffff800123b4>] __do_softirq+0x89/0x133 Aug 29 03:34:23 localhost kernel: [<ffffffff8005e2fc>] call_softirq+0x1c/0x28 Aug 29 03:34:23 localhost kernel: [<ffffffff8006cb8e>] do_softirq+0x2c/0x85 Aug 29 03:34:23 localhost kernel: [<ffffffff8006ca16>] do_IRQ+0xec/0xf5 Aug 29 03:34:24 localhost kernel: [<ffffffff8006b346>] default_idle+0x0/0x50 Aug 29 03:34:24 localhost kernel: [<ffffffff8005d615>] ret_from_intr+0x0/0xa Aug 29 03:34:24 localhost kernel: <EOI> [<ffffffff8006b36f>] default_idle+0x29/0x50 Aug 29 03:34:24 localhost kernel: [<ffffffff800491ab>] cpu_idle+0x95/0xb8 Aug 29 03:34:24 localhost kernel: [<ffffffff8007796f>] start_secondary+0x498/0x4a7 Aug 29 03:34:25 localhost kernel: Still seeing this with 2.6.34.7-61.fc13.x86_64 This message is a reminder that Fedora 13 is nearing its end of life. Approximately 30 (thirty) days from now Fedora will stop maintaining and issuing updates for Fedora 13. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as WONTFIX if it remains open with a Fedora 'version' of '13'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version prior to Fedora 13's end of life. Bug Reporter: Thank you for reporting this issue and we are sorry that we may not be able to fix it before Fedora 13 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora please change the 'version' of this bug to the applicable version. If you are unable to change the version, please add a comment here and someone will do it for you. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. The process we are following is described here: http://fedoraproject.org/wiki/BugZappers/HouseKeeping Fedora 13 changed to end-of-life (EOL) status on 2011-06-25. Fedora 13 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. Thank you for reporting this bug and we are sorry it could not be fixed. |