libreport version: 2.0.6 abrt_version: 2.0.4.981 cmdline: BOOT_IMAGE=/boot/vmlinuz-3.1.1-1.fc16.x86_64 root=UUID=2c692d84-1ec2-4c2d-b04f-271650aaf24d ro rd.md=0 rd.lvm=0 rd.dm=0 quiet SYSFONT=latarcyrheb-sun16 rhgb KEYTABLE=uk rd.luks=0 LANG=en_US.UTF-8 comment: I do not know event_log: 2011-11-15-11:09:41> Smolt profile successfully saved kernel: undefined reason: BUG: soft lockup - CPU#3 stuck for 22s! [kworker/3:1:3639] time: Tue Nov 15 09:50:55 2011 backtrace: :BUG: soft lockup - CPU#3 stuck for 22s! [kworker/3:1:3639] :Modules linked in: ppdev parport_pc lp parport fuse fcoe 8021q garp stp llc libfcoe libfc scsi_transport_fc scsi_tgt ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 xt_state ip6table_filter nf_conntrack ip6_tables snd_hda_codec_realtek arc4 snd_hda_intel snd_hda_codec rtl8192ce rtl8192c_common snd_hwdep snd_seq snd_seq_device snd_pcm rtlwifi mac80211 uvcvideo cfg80211 snd_timer snd videodev media v4l2_compat_ioctl32 soundcore sparse_keymap snd_page_alloc microcode r8169 iTCO_wdt rfkill i2c_i801 mii serio_raw joydev uinput iTCO_vendor_support wmi uas ums_realtek usb_storage i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] :CPU 3 :Modules linked in: ppdev parport_pc lp parport fuse fcoe 8021q garp stp llc libfcoe libfc scsi_transport_fc scsi_tgt ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 xt_state ip6table_filter nf_conntrack ip6_tables snd_hda_codec_realtek arc4 snd_hda_intel snd_hda_codec rtl8192ce rtl8192c_common snd_hwdep snd_seq snd_seq_device snd_pcm rtlwifi mac80211 uvcvideo cfg80211 snd_timer snd videodev media v4l2_compat_ioctl32 soundcore sparse_keymap snd_page_alloc microcode r8169 iTCO_wdt rfkill i2c_i801 mii serio_raw joydev uinput iTCO_vendor_support wmi uas ums_realtek usb_storage i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] :Pid: 3639, comm: kworker/3:1 Not tainted 3.1.1-1.fc16.x86_64 #1 TOSHIBA SATELLITE C670-14P/Oneonta Falls :RIP: 0010:[<ffffffff81085dc1>] [<ffffffff81085dc1>] do_raw_spin_lock+0x1e/0x25 :RSP: 0018:ffff88012208dda0 EFLAGS: 00000293 :RAX: 000000000000471d RBX: ffff880100000001 RCX: 0000000000000000 :RDX: 000000000000471b RSI: 0000000000000004 RDI: ffff880183a61d74 :RBP: ffff88012208dda0 R08: 0000000000000000 R09: 0000000000000000 :R10: 000000000055f14b R11: ffff8801952c7fd8 R12: ffff88012208de00 :R13: 0000000000000000 R14: 0000000000000003 R15: 00000001008f226f :FS: 0000000000000000(0000) GS:ffff88019fa60000(0000) knlGS:0000000000000000 :CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b :CR2: 00000030504043c5 CR3: 0000000001a05000 CR4: 00000000000406e0 :DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 :DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 :Process kworker/3:1 (pid: 3639, threadinfo ffff88012208c000, task ffff880172885cc0) :Stack: : ffff88012208ddb0 ffffffff814b712e ffff88012208ddd0 ffffffffa020b866 : ffff880183a61d40 ffff880183a60540 ffff88012208de40 ffffffffa0204771 : 0000000000012f80 0000000583b4aa00 0000000000000000 0000000000000000 :Call Trace: : [<ffffffff814b712e>] _raw_spin_lock+0xe/0x10 : [<ffffffffa020b866>] rtl_lps_leave+0x1c/0xea [rtlwifi] : [<ffffffffa0204771>] rtl_watchdog_wq_callback+0x1ae/0x261 [rtlwifi] : [<ffffffffa02045c3>] ? rtl_watch_dog_timer_callback+0x48/0x48 [rtlwifi] : [<ffffffff8106ed84>] process_one_work+0x176/0x2a9 : [<ffffffff8106f892>] worker_thread+0xda/0x15d : [<ffffffff8106f7b8>] ? manage_workers+0x176/0x176 : [<ffffffff81072cdf>] kthread+0x84/0x8c : [<ffffffff814bfa34>] kernel_thread_helper+0x4/0x10 : [<ffffffff81072c5b>] ? kthread_worker_fn+0x148/0x148 : [<ffffffff814bfa30>] ? gs_change+0x13/0x13 :Code: 00 00 10 00 74 05 e8 ef 45 1a 00 5d c3 55 48 89 e5 66 66 66 66 90 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 07 f3 90 <0f> b7 17 eb f5 5d c3 55 48 89 e5 66 66 66 66 90 8b 07 89 c2 c1 smolt_data: : : :General :================================= :UUID: 8253d965-7289-4fed-a9e4-3da1f949ca09 :OS: Fedora release 16 (Verne) :Default run level: Unknown :Language: en_GB.utf8 :Platform: x86_64 :BogoMIPS: 4191.26 :CPU Vendor: GenuineIntel :CPU Model: Intel(R) Core(TM) i3-2310M CPU @ 2.10GHz :CPU Stepping: 7 :CPU Family: 6 :CPU Model Num: 42 :Number of CPUs: 4 :CPU Speed: 2100 :System Memory: 5883 :System Swap: 0 :Vendor: TOSHIBA :System: SATELLITE C670-14P PSC3UE-00Y009EN :Form factor: Laptop :Kernel: 3.1.1-1.fc16.x86_64 :SELinux Enabled: 1 :SELinux Policy: targeted :SELinux Enforce: Enforcing :MythTV Remote: Unknown :MythTV Role: Unknown :MythTV Theme: Unknown :MythTV Plugin: :MythTV Tuner: -1 : : :Devices :================================= :(32902:7241:4473:64630) pci, None, PCI/ISA, HM65 Express Chipset Family LPC Controller :(32902:278:4473:64624) pci, i915, VIDEO, 2nd Generation Core Processor Family Integrated Graphics Controller :(32902:7171:4473:64630) pci, ahci, STORAGE, 6 Series/C200 Series Chipset Family 6 port SATA AHCI Controller :(32902:7184:4473:64630) pci, pcieport, PCI/PCI, 6 Series/C200 Series Chipset Family PCI Express Root Port 1 :(32902:7186:4473:64630) pci, pcieport, PCI/PCI, 6 Series/C200 Series Chipset Family PCI Express Root Port 2 :(32902:7196:4473:64630) pci, pcieport, PCI/PCI, 6 Series/C200 Series Chipset Family PCI Express Root Port 7 :(32902:7200:4473:64628) pci, snd_hda_intel, MULTIMEDIA, 6 Series/C200 Series Chipset Family High Definition Audio Controller :(32902:7202:4473:64630) pci, i801_smbus, SERIAL, 6 Series/C200 Series Chipset Family SMBus Controller :(4332:33078:4473:64614) pci, r8169, ETHERNET, RTL8101E/RTL8102E PCI Express Fast Ethernet controller :(4332:33142:4332:33154) pci, rtl8192ce, NETWORK, RTL8188CE 802.11b/g/n WiFi Adapter :(32902:7213:4473:64630) pci, ehci_hcd, USB, 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 :(32902:7206:4473:64630) pci, ehci_hcd, USB, 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 :(32902:260:4473:64624) pci, agpgart-intel, HOST/PCI, 2nd Generation Core Processor Family DRAM Controller :(32902:7226:4473:64630) pci, None, SIMPLE, 6 Series/C200 Series Chipset Family MEI Controller #1 :(32902:7194:4473:64630) pci, pcieport, PCI/PCI, 6 Series/C200 Series Chipset Family PCI Express Root Port 6 : : :Filesystem Information :================================= :device mtpt type bsize frsize blocks bfree bavail file ffree favail :------------------------------------------------------------------- :/dev/sda3 / ext4 4096 4096 71815018 63022819 62293494 18243584 17975712 17975712 :
*** Bug 756140 has been marked as a duplicate of this bug. ***
Please try the following (as root): modprobe -rv rtl8192ce modprobe -v rtl8192ce ips=0 The above will disable power save in the interface, which seems to be indicated. Does it help?
(In reply to comment #2) > Please try the following (as root): > > modprobe -rv rtl8192ce > modprobe -v rtl8192ce ips=0 > > The above will disable power save in the interface, which seems to be > indicated. Does it help? Hi Larry, Many thanks for help. The first line have killed the wifi and second line just freezed without any result. I've tried yesterday: iw dev wlan0 set power_save off Seems not appear again but who knows...
I just updated a desktop/server system to Fedora 16 over the weekend and I am seeing this exact (and very annoying) problem. It uses wired ethernet for the network connection. In contrast to the above user, my system has absolutely NO wifi hardware of any sort in it. In one case, I was just editing a file (from vim running in an xterm), and it just froze solid as a rock, no response of any sort. What was odd, was I could open another xterm without any problems, and started "top" to see a kworker process taking 95-100% of one CPU. It took perhaps 2-3 minutes before the editor session became responsive again. Just this morning, I sat down in front of the system which had been up over-night but not doing anything of particular significance. I tried typing a command in an xterm window that was already open from the previous evening and it was already totally unresponsive. Once again, I was able to open another xterm without difficulty and found a kworker process running at close to 100% CPU utilization. I saw the problem with the kernel that Anaconda installed from the full DVD installer, and the latest kernel that was installed after doing a "yum update". More specifically; kernel-3.1.0-7.fc16.x86_64 kernel-3.1.1-2.fc16.x86_64 The system is a Nettop style system I built myself with an Intel motherboard and dual core Atom processor (330 if memory serves me correctly). Let me know what sort of detailed information would be helpful. Jast an FYI. What little searching I have done suggests this problem was observed with the 2.6.x kernels, mostly from the Ubuntu community.
I found a discussion of kworker threads "going bonkers" on the kernel mailing list; https://lkml.org/lkml/2011/3/30/836 I was able to collect statistics with perf as recommended in the post above. I'm not that familiar with the kernel internals, so I am not sure what to make of it, but I see lots of time spent in ACPI stuff. A trimmed version of the output from "perf report -stdio" is attached to avoid spamming the comments here. As I mentioned above, there is no wifi networking hardware in this (nettop) box. There was mention of this being associated with Intel video, which this system does have. Trimmed output from lspci; 00:02.0 VGA compatible controller: Intel Corporation 82945G/GZ Integrated Graphics Controller (rev 02) Mind you, I'm not ranting here, but this bug is a true show stopper for me. I did a fresh reboot before turning in last night. The system was doing nothing of particular significance. When I checked it this morning, it had been up just shy of 14 hours, while one kworker thread had already consumed over 6 hours of CPU time! % uptime 10:16:57 up 13:55, 4 users, load average: 2.74, 1.72, 1.33 % ps aux | grep kworker root 5 0.0 0.0 0 0 ? S Nov25 0:00 [kworker/u:0] root 11 0.0 0.0 0 0 ? S Nov25 0:00 [kworker/0:1] root 40 0.0 0.0 0 0 ? S Nov25 0:00 [kworker/u:2] root 44 44.2 0.0 0 0 ? R Nov25 369:39 [kworker/0:2] root 2309 0.0 0.0 0 0 ? S 02:52 0:00 [kworker/0:0] root 2466 0.4 0.0 0 0 ? S 08:08 0:31 [kworker/1:1] root 2522 0.0 0.0 0 0 ? S 10:06 0:00 [kworker/1:0] root 2529 0.0 0.0 0 0 ? S 10:14 0:00 [kworker/1:2]
Created attachment 536902 [details] Output of "perf report --stdio" Waited for the kworker thread to go "cuckoo for cocoa puffs" then from another xterm; perf record -ag sleep 180 perf report --stdio
John, did you try to use kernel-debug ? Perhaps it will print more detail information where the problem is. For tracking Terry problem with rtl8192ce driver I'm using bug 755154 now, and I reassign this one back to kernel-main since John problem is not wireless related.
*** Bug 756916 has been marked as a duplicate of this bug. ***
Is there significance in the 22 seconds? I have 82 soft lockups in my logs this morning, they're almost all 22 seconds with a few that are 23. The events are spaced exactly 28 seconds apart...and I do mean exactly. Dec 9 09:07:23 studio kernel: [21852.192997] BUG: soft lockup - CPU#0 stuck for 22s! [kswapd0:27] Dec 9 09:07:51 studio kernel: [21880.192997] BUG: soft lockup - CPU#0 stuck for 22s! [kswapd0:27] Dec 9 09:08:19 studio kernel: [21908.192997] BUG: soft lockup - CPU#0 stuck for 22s! [kswapd0:27] Dec 9 09:08:47 studio kernel: [21936.192997] BUG: soft lockup - CPU#0 stuck for 22s! [kswapd0:27] Dec 9 09:09:15 studio kernel: [21964.192997] BUG: soft lockup - CPU#0 stuck for 22s! [kswapd0:27] Dec 9 09:09:43 studio kernel: [21992.192997] BUG: soft lockup - CPU#0 stuck for 22s! [kswapd0:27] (3.2.0-0.rc4.git5.1.fc17.x86_64)
(In reply to comment #9) > Is there significance in the 22 seconds? I have 82 soft lockups in my logs this > morning, they're almost all 22 seconds with a few that are 23. The events are > spaced exactly 28 seconds apart...and I do mean exactly. There is a timer ticking in the kernel to detect soft lockups. It ticks at (watchdog_thresh * 2) seconds. I believe the default value of watchdog_thresh is 10 now. So every 20 seconds or so a high priority thread is supposed to write to a variable that indicates the watchdog is able to run stuff on this CPU. If it doesn't get scheduled, then you get this warning. Being highly regular is expected in the case where something is hogging the CPU.
Created attachment 545442 [details] Output of perf top on kswapd thread while in a soft-lockup loop I don't know if this is in any way useful.. currently running 3.2.0-0.rc4.git5.1.fc17.x86_64
The patch does not occur in mainline until 3.2-rc5. If you still get lockups with that version, then please post again.
John, your issue seems to be related with ACPI. Derek, in your case this seems to be i915 driver issue. Please install kernel-debug, it should print some more verbose information where the problem is.
Sorry for not following up sooner, I think I have diagnosed my problem. I did not notice it right away, but there were some entries in /var/log/messages that seem to confirm my problem is related to ACPI. Entries like the one below appear at the same time that kworker goes bonkers; Nov 21 06:32:51 octagon kernel: [41779.816616] ACPI: While loop taking a really long time. loop_count=0xfff [.... snip ....] Nov 21 06:41:46 octagon kernel: [42315.419942] ACPI: While loop taking a really long time. loop_count=0xffff00 Nov 21 06:41:46 octagon kernel: [42315.428453] ACPI Error: Method parse/execution failed [\_SB_.PCI0.LPC_.SMBR] (Node ffff88007bb44f28), AE_AML_INFINITE_LOOP (20110623/psparse-536) Nov 21 06:41:46 octagon kernel: [42315.428487] ACPI Error: Method parse/execution failed [\_SB_.PCI0.LPC_.INIT] (Node ffff88007bb44f00), AE_AML_INFINITE_LOOP (20110623/psparse-536) Nov 21 06:41:46 octagon kernel: [42315.428509] ACPI Error: Method parse/execution failed [\_GPE._L00] (Node ffff88007bb402d0), AE_AML_INFINITE_LOOP (20110623/psparse-536) Nov 21 06:41:46 octagon kernel: [42315.428538] ACPI Exception: AE_AML_INFINITE_LOOP, while evaluating GPE method [_L00] (20110623/evgpe-560) In hunting around to see if others have this problem, I discovered that many people having this problem have the same system board as I have, the Intel 945GCLF2. To make a long story short, it appears this board has an unfixed ACPI bug. Upon learning of this, I downloaded and flashed the latest BIOS from Intel, but that unfortunately did NOT fix anything, same drill. Although I'm not really 100% certain this is the issue, it seemed to be pretty likely. It was reported by Intel 945GCLF2 owners using other Linux distributions, and even by users of other operating systems such as FreeBSD, so that was good enough for me: http://www.mail-archive.com/acpi-bugzilla@lists.sourceforge.net/msg27069.html https://bugzilla.novell.com/show_bug.cgi?id=689848 http://forums.freebsd.org/showpost.php?s=c7ec091918772edc6ccac2448feadc3d&p=62755&postcount=15 As I mentioned earlier, the problem really made the system unusable. So, I just installed an Intel D525MW board (the successor product, about $115 with memory) and everything works fine now. I thought Intel had mostly eliminated problems like this with all the time and money they sunk into using Formal Methods for hardware and software verification after the infamous FDIV bug back in the 1990's. Apparently not.
Are all the issues associated with this bug satisfied by either getting a new mainboard, or by the fix to rtlwifi? If so, then the bug should be closed.
In retrospect, the problem I was having was completely different than the person who filed the original bug report (Terry Wallwork). As I noted above, my system had/has no wifi of any kind in it. I posted here for lack of a more precise place to report it. If I understood correctly (comment 7 above), Terri's problem is now being tracked by bug 755154
Yes, that bug is fixed, at least in mainline. It is my understanding that a kernel update has been issued by Fedora.
This bug was a hardware issue, and the original reporter has their issue tracked by a different bug. Closing.