Bug 755154 - [abrt] kernel: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u:1:3289]
Summary: [abrt] kernel: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u:1:3289]
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: kernel
Version: 16
Hardware: x86_64
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Stanislaw Gruszka
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: abrt_hash:5cca75ec309f85e005663b1a2e8...
: 710783 757066 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2011-11-19 06:06 UTC by vjain02
Modified: 2011-12-22 01:17 UTC (History)
12 users (show)

Fixed In Version: kernel-3.1.5-1.fc16
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2011-12-12 21:57:39 UTC
Type: ---


Attachments (Terms of Use)
0001-rtlwifi-fix-lps_lock-deadlock.patch (7.33 KB, text/plain)
2011-11-28 09:35 UTC, Stanislaw Gruszka
no flags Details

Description vjain02 2011-11-19 06:06:59 UTC
libreport version: 2.0.6
abrt_version:   2.0.4.981
cmdline:        BOOT_IMAGE=/boot/vmlinuz-3.1.0-7.fc16.x86_64 root=UUID=98d33203-8003-4ac4-b642-f857109021b6 ro rd.md=0 rd.lvm=0 rd.dm=0 KEYTABLE=us SYSFONT=latarcyrheb-sun16 rd.luks=0 LANG=en_US.UTF-8 init=/sbin/bootchartd drm.debug=0x02
kernel:         undefined
reason:         BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u:1:3289]
time:           Sat Nov 19 00:47:00 2011

backtrace:
:BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u:1:3289]
:Modules linked in: tcp_lp ppdev parport_pc lp parport fuse ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_CHECKSUM iptable_mangle tun bridge stp llc snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device rtl8192se rtlwifi snd_pcm mac80211 iTCO_wdt e1000e intel_ips i2c_i801 cfg80211 snd_timer iTCO_vendor_support snd_page_alloc uvcvideo videodev media v4l2_compat_ioctl32 thinkpad_acpi rfkill snd soundcore joydev microcode virtio_net kvm_intel kvm uinput mxm_wmi firewire_ohci firewire_core crc_itu_t sdhci_pci sdhci mmc_core wmi i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
:CPU 0 
:Modules linked in: tcp_lp ppdev parport_pc lp parport fuse ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_CHECKSUM iptable_mangle tun bridge stp llc snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device rtl8192se rtlwifi snd_pcm mac80211 iTCO_wdt e1000e intel_ips i2c_i801 cfg80211 snd_timer iTCO_vendor_support snd_page_alloc uvcvideo videodev media v4l2_compat_ioctl32 thinkpad_acpi rfkill snd soundcore joydev microcode virtio_net kvm_intel kvm uinput mxm_wmi firewire_ohci firewire_core crc_itu_t sdhci_pci sdhci mmc_core wmi i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
:Pid: 3289, comm: kworker/u:1 Not tainted 3.1.0-7.fc16.x86_64 #1 LENOVO 2516CTO/2516CTO
:RIP: 0010:[<ffffffff81085e09>]  [<ffffffff81085e09>] do_raw_spin_lock+0x1e/0x25
:RSP: 0018:ffff8800431c3da0  EFLAGS: 00000293
:RAX: 000000000000c654 RBX: 0000000000000286 RCX: 0000000000001b83
:RDX: 000000000000c652 RSI: 0000000000000286 RDI: ffff8800a2ba9d74
:RBP: ffff8800431c3da0 R08: ffff8800a2ba9408 R09: 5554849d651c2cf8
:R10: 5554849d651c2cf8 R11: ffff880043039b00 R12: ffff8800431c3d28
:R13: 0000000000000000 R14: 0000000000000000 R15: 0000000100348422
:FS:  0000000000000000(0000) GS:ffff8800b7000000(0000) knlGS:0000000000000000
:CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
:CR2: 0000000003bf4001 CR3: 0000000001a05000 CR4: 00000000000006f0
:DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
:DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
:Process kworker/u:1 (pid: 3289, threadinfo ffff8800431c2000, task ffff880066645cc0)
:Stack:
: ffff8800431c3db0 ffffffff814b5cfe ffff8800431c3dd0 ffffffffa036bbaf
: ffff8800a2ba9d40 ffff8800a2ba8540 ffff8800431c3e00 ffffffffa0368c8c
: ffff8800a2ba8540 0000000000000010 ffff8800ac6da000 ffffffffa0301710
:Call Trace:
: [<ffffffff814b5cfe>] _raw_spin_lock+0xe/0x10
: [<ffffffffa036bbaf>] rtl_swlps_rf_awake+0x54/0x6c [rtlwifi]
: [<ffffffffa0368c8c>] rtl_op_config+0x128/0x328 [rtlwifi]
: [<ffffffffa0301710>] ? ieee80211_rx_mgmt_beacon+0x4d3/0x4d3 [mac80211]
: [<ffffffffa02f7fcb>] ieee80211_hw_config+0x102/0x107 [mac80211]
: [<ffffffffa0301742>] ieee80211_dynamic_ps_disable_work+0x32/0x47 [mac80211]
: [<ffffffff8106edbc>] process_one_work+0x176/0x2a9
: [<ffffffff8106ce54>] ? need_to_create_worker+0x12/0x26
: [<ffffffff8106f8ca>] worker_thread+0xda/0x15d
: [<ffffffff8106f7f0>] ? manage_workers+0x176/0x176
: [<ffffffff81072d17>] kthread+0x84/0x8c
: [<ffffffff814be5f4>] kernel_thread_helper+0x4/0x10
: [<ffffffff81072c93>] ? kthread_worker_fn+0x148/0x148
: [<ffffffff814be5f0>] ? gs_change+0x13/0x13
:Code: 00 00 10 00 74 05 e8 77 42 1a 00 5d c3 55 48 89 e5 66 66 66 66 90 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 07 f3 90 <0f> b7 17 eb f5 5d c3 55 48 89 e5 66 66 66 66 90 8b 07 89 c2 c1 

comment:
:Plain gnome 3 session
:xchat + google-chrome + rhythmbox

event_log:
:2011-11-19-01:02:51> Smolt profile successfully saved
:2011-11-19-01:05:50> Submitting oops report to http://submit.kerneloops.org/submitoops.php
:2011-11-19-01:06:53  Kernel oops has not been sent due to Couldn't connect to server
:2011-11-19-01:06:53* (exited with 1)

smolt_data:
:
:
:General
:=================================
:UUID: 05b1b33d-6ccd-463e-a112-e2c41df79e94
:OS: Fedora release 16 (Verne)
:Default run level: Unknown
:Language: en_US.utf8
:Platform: x86_64
:BogoMIPS: 5320.11
:CPU Vendor: GenuineIntel
:CPU Model: Intel(R) Core(TM) i5 CPU       M 560  @ 2.67GHz
:CPU Stepping: 5
:CPU Family: 6
:CPU Model Num: 37
:Number of CPUs: 4
:CPU Speed: 2667
:System Memory: 2874
:System Swap: 6143
:Vendor: LENOVO
:System: 2516CTO ThinkPad T410
:Form factor: Notebook
:Kernel: 3.1.0-7.fc16.x86_64
:SELinux Enabled: 0
:SELinux Policy: targeted
:SELinux Enforce: Unknown
:MythTV Remote: Unknown
:MythTV Role: Unknown
:MythTV Theme: Unknown
:MythTV Plugin: 
:MythTV Tuner: -1
:
:
:Devices
:=================================
:(4480:59426:6058:8499) pci, sdhci-pci, BASE, MMC/SD Host Controller
:(4480:57904:6058:8500) pci, None, BASE, Memory Stick Host Controller
:(4480:59442:6058:8502) pci, firewire_ohci, FIREWIRE, FireWire Host Controller
:(32902:11362:6058:8598) pci, None, HOST/PCI, Core Processor QuickPath Architecture Generic Non-core Registers
:(32902:11521:6058:8598) pci, None, HOST/PCI, Core Processor QuickPath Architecture System Address Decoder
:(32902:68:6058:8595) pci, agpgart-intel, HOST/PCI, Core Processor DRAM Controller
:(32902:11536:6058:8598) pci, None, HOST/PCI, Core Processor QPI Link 0
:(32902:11537:6058:8598) pci, None, HOST/PCI, Core Processor QPI Physical 0
:(32902:11538:6058:8598) pci, None, HOST/PCI, Core Processor Reserved
:(32902:11539:6058:8598) pci, None, HOST/PCI, Core Processor Reserved
:(32902:15152:6058:8551) pci, i801_smbus, SERIAL, 5 Series/3400 Series Chipset SMBus Controller
:(32902:15151:6058:8552) pci, ahci, STORAGE, 5 Series/3400 Series Chipset 6 port SATA AHCI Controller
:(32902:15111:6058:8550) pci, None, PCI/ISA, Mobile 5 Series Chipset LPC Interface Controller
:(32902:15154:6058:8592) pci, intel ips, NONE, 5 Series/3400 Series Chipset Thermal Subsystem
:(32902:4330:6058:8531) pci, e1000e, ETHERNET, 82577LM Gigabit Network Connection
:(32902:9288:6058:8549) pci, None, PCI/PCI, 82801 Mobile PCI Bridge
:(32902:15164:6058:8547) pci, ehci_hcd, USB, 5 Series/3400 Series Chipset USB2 Enhanced Host Controller
:(32902:15156:6058:8547) pci, ehci_hcd, USB, 5 Series/3400 Series Chipset USB2 Enhanced Host Controller
:(32902:15204:6058:8543) pci, None, SIMPLE, 5 Series/3400 Series Chipset HECI Controller
:(4332:33138:4332:57376) pci, rtl8192se, NETWORK, RTL8191SEvB Wireless LAN Controller
:(32902:70:6058:8538) pci, i915, VIDEO, Core Processor Integrated Graphics Controller
:(32902:15176:6058:8548) pci, pcieport, PCI/PCI, 5 Series/3400 Series Chipset PCI Express Root Port 4
:(32902:15170:6058:8548) pci, pcieport, PCI/PCI, 5 Series/3400 Series Chipset PCI Express Root Port 1
:(32902:15172:6058:8548) pci, pcieport, PCI/PCI, 5 Series/3400 Series Chipset PCI Express Root Port 2
:(32902:15178:6058:8548) pci, pcieport, PCI/PCI, 5 Series/3400 Series Chipset PCI Express Root Port 5
:(32902:15190:6058:8542) pci, snd_hda_intel, MULTIMEDIA, 5 Series/3400 Series Chipset High Definition Audio
:(32902:15207:6058:8546) pci, serial, 16550_SERIAL, 5 Series/3400 Series Chipset KT Controller
:
:
:Filesystem Information
:=================================
:device mtpt type bsize frsize blocks bfree bavail file ffree favail
:-------------------------------------------------------------------
:/dev/sda8 / ext3 4096 4096 4913631 2883128 2633528 1248480 1007052 1007052
:/dev/sda7 WITHHELD ext4 4096 4096 6457055 1612419 1284419 1640160 1311231 1311231
:/dev/sda5 WITHHELD fuseblk UNKNOWN UNKNOWN UNKNOWN UNKNOWN UNKNOWN UNKNOWN UNKNOWN UNKNOWN
:

Comment 1 Josh Boyer 2011-11-22 15:12:06 UTC
This looks related or possibly a dup of bug 754075

Comment 2 Larry Finger 2011-11-22 21:08:31 UTC
I think related. This one involves rtl8192se, and the other rtl8192se; however, both use rtlwifi, and the problem may be there.

Please try the following (as root):

modprobe -rv rtl8192se
modprobe -v rtl8192se ips=0

The above will disable power save in the interface, which seems to be indicated. Does it help?

Comment 3 Larry Finger 2011-11-22 21:09:50 UTC
I think related. This one involves rtl8192se, and the other rtl8192ce; however, both use rtlwifi, and the problem may be there.

Please try the following (as root):

modprobe -rv rtl8192se
modprobe -v rtl8192se ips=0

The above will disable power save in the interface, which seems to be indicated. Does it help?

Comment 4 vjain02 2011-11-23 01:48:13 UTC
The system freeze has quite an irregular pattern, I have seen it hang 2-4 hrs after reboot. I haven't experienced a freeze till now after 4-6 hours after executing the above command. 
Will report if the freeze happens again.

Also, I have seen it happen more often when I resume from sleep than actually a fresh restart.

Comment 5 Larry Finger 2011-11-23 02:25:40 UTC
That makes sense. I never see this fault with rtl8192ce, rtl8192se, or rtl8192de, but I never put my machine to sleep - that feature does not work here.

Thanks for testing.

Comment 6 vjain02 2011-11-23 23:41:30 UTC
So, there is no support for sleep/resume. What should an end user do in this case ?

Comment 7 Larry Finger 2011-11-24 00:51:25 UTC
I did not say that. What I cannot do is test, not because of problems with rtl8192ce, but with another component. If you want sleep/resume, you need to test. I will support any problems that you find, but I cannot find them for you.

BTW, hibernate works.

Comment 8 vjain02 2011-11-24 03:19:01 UTC
OK, I will be happy to test.
I don't get messages like the one above now, but the system still freezes producing no output in /var/log/messages
What should be the next step ?

I tried the ips=0 option, I can't say exactly whether it had an affect or not. It still froze but I guess it ran longer.

Comment 9 Larry Finger 2011-11-24 03:36:29 UTC
If you do not see messages like the one above, then you have a different problem. Note, if you have to power off the system to recover, anything logged will be lost.

When you get a freeze, does the system recover in 30 seconds, or so?

If you switch to the debug console (CTRL-ALT-F10 on my openSUSE KDE system), is anything logged to the screen?

Do you see a kernel panic (keyboard lights flashing once per second)?

Comment 10 Larry Finger 2011-11-24 03:38:47 UTC
If you do not see messages like the one above, then you have a different problem. Note, if you have to power off the system to recover, anything logged will be lost.

When you get a freeze, does the system recover in 30 seconds, or so?

If you switch to the debug console (CTRL-ALT-F10 on my openSUSE KDE system), is anything logged to the screen?

Do you see a kernel panic (keyboard lights flashing once per second)?

Comment 11 vjain02 2011-11-24 03:51:45 UTC
I don't see the above messages now, I was getting them earlier.

I have to do a hard reboot all the time, the system becomes completely unresponsive. 
ctrl-alt-f2-  (fedora) doesn't work. No key combination works.
The system restart takes the normal amount of time as it does ~50 secs or so.

No, I don't see a kernel panic (flashing caps lock light).

Comment 12 vjain02 2011-11-25 06:22:33 UTC
Package: kernel
Architecture: x86_64
OS Release: Fedora release 16 (Verne)

Comment
-----
plain gnome 3 session

Comment 13 Stanislaw Gruszka 2011-11-28 09:35:17 UTC
Created attachment 537412 [details]
0001-rtlwifi-fix-lps_lock-deadlock.patch

Fix for that bug. Posted and CCed to -stable:
http://thread.gmane.org/gmane.linux.kernel.wireless.general/81542/focus=81657

Comment 14 John W. Linville 2011-11-28 18:15:27 UTC
*** Bug 757066 has been marked as a duplicate of this bug. ***

Comment 15 Stanislaw Gruszka 2011-11-29 08:49:02 UTC
Josh, could you apply patch from comment 13? Thanks.

Comment 16 Josh Boyer 2011-11-29 15:07:19 UTC
(In reply to comment #15)
> Josh, could you apply patch from comment 13? Thanks.

Yep.  I'll get it into both f15 and f16 today.

Comment 17 Josh Boyer 2011-11-29 15:37:39 UTC
Applied.  Should be in the next build.

Comment 18 Brad 2011-12-06 06:56:05 UTC
*** Bug 710783 has been marked as a duplicate of this bug. ***

Comment 19 Fedora Update System 2011-12-09 20:23:51 UTC
kernel-3.1.5-1.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/kernel-3.1.5-1.fc16

Comment 20 Brad 2011-12-09 21:58:15 UTC
(In reply to comment #19)
> kernel-3.1.5-1.fc16 has been submitted as an update for Fedora 16.
> https://admin.fedoraproject.org/updates/kernel-3.1.5-1.fc16

Stansliaw, what do you want for Christmas?

Comment 21 Fedora Update System 2011-12-10 19:34:07 UTC
Package kernel-3.1.5-1.fc16:
* should fix your issue,
* was pushed to the Fedora 16 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing kernel-3.1.5-1.fc16'
as soon as you are able to, then reboot.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2011-16840/kernel-3.1.5-1.fc16
then log in and leave karma (feedback).

Comment 22 Stefano Tabone 2011-12-10 22:46:39 UTC
(In reply to comment #21)
> Package kernel-3.1.5-1.fc16:
> * should fix your issue,
> * was pushed to the Fedora 16 testing repository,
> * should be available at your local mirror within two days.
> Update it with:
> # su -c 'yum update --enablerepo=updates-testing kernel-3.1.5-1.fc16'
> as soon as you are able to, then reboot.
> Please go to the following url:
> https://admin.fedoraproject.org/updates/FEDORA-2011-16840/kernel-3.1.5-1.fc16
> then log in and leave karma (feedback).

I've enables the test updates repository but this update doesn't come up Should I just wait a few more days?

Comment 23 Stanislaw Gruszka 2011-12-12 11:53:43 UTC
(In reply to comment #20)
> Stansliaw, what do you want for Christmas?
LOL, robot capable to fix quickly fedora bugs. Hmm, does Santa use fedora? :-)

(In reply to comment #22)
> I've enables the test updates repository but this update doesn't come up Should
> I just wait a few more days?

Probably updated did not propagate into mirrors. You can try comment out
> mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=updates-testing-f$releasever&arch=$basearch
and uncomment
baseurl=http://download.fedoraproject.org/pub/fedora/linux/updates/testing/$releasever/$basearch/
line in /etc/yum.repos.d/fedora-updates-testing.repo

Comment 24 Fedora Update System 2011-12-12 12:20:35 UTC
kernel-2.6.41.5-1.fc15 has been submitted as an update for Fedora 15.
https://admin.fedoraproject.org/updates/kernel-2.6.41.5-1.fc15

Comment 25 Fedora Update System 2011-12-12 21:57:39 UTC
kernel-3.1.5-1.fc16 has been pushed to the Fedora 16 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 26 Fedora Update System 2011-12-15 18:54:15 UTC
kernel-2.6.41.5-4.fc15 has been submitted as an update for Fedora 15.
https://admin.fedoraproject.org/updates/kernel-2.6.41.5-4.fc15

Comment 27 Fedora Update System 2011-12-22 01:17:59 UTC
kernel-2.6.41.6-1.fc15 has been submitted as an update for Fedora 15.
https://admin.fedoraproject.org/updates/kernel-2.6.41.6-1.fc15


Note You need to log in before you can comment on or make changes to this bug.