Description of problem: Started to get soft lockups like these after a few days of uptime with the latest kernel: [282777.559308] BUG: soft lockup - CPU#1 stuck for 23s! [java:897] [282777.559309] BUG: soft lockup - CPU#0 stuck for 23s! [java:931] [282777.559328] Modules linked in: nfnetlink_queue nfnetlink_log nfnetlink fuse bnep bluetooth cfg80211 rfkill snd_hda_codec_realtek snd_hda_codec_hdmi snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_page_alloc snd_timer e1000e snd ptp iTCO_wdt iTCO_vendor_support mei_me mei lpc_ich serio_raw i2c_i801 x86_pkg_temp_thermal shpchp dcdbas coretemp pps_core mfd_core kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel soundcore ghash_clmulni_intel microcode bcache i915 i2c_algo_bit drm_kms_helper drm i2c_core video [282777.559329] CPU: 0 PID: 931 Comm: java Not tainted 3.12.6-300.fc20.x86_64 #1 [282777.559330] Hardware name: Dell Inc. OptiPlex 9020/0PC5F7, BIOS A02 08/15/2013 [282777.559331] task: ffff8807f156e300 ti: ffff8807e6344000 task.ti: ffff8807e6344000 [282777.559335] RIP: 0010:[<ffffffff81669b0a>] [<ffffffff81669b0a>] _raw_spin_lock+0x2a/0x40 [282777.559336] RSP: 0018:ffff8807e6345ce8 EFLAGS: 00000a83 [282777.559336] RAX: 0000000000005548 RBX: 0000000000014540 RCX: 00000000000081f4 [282777.559337] RDX: 00000000000081f4 RSI: ffff8807f156e300 RDI: ffff8807e6345d20 [282777.559337] RBP: ffff8807e6345ce8 R08: ffff8807e6344000 R09: 0000000000000031 [282777.559338] R10: 0000000000000004 R11: 0000000000000005 R12: 0000000000014540 [282777.559338] R13: ffff8807e6345fd8 R14: ffffffff816678cc R15: ffff8807e6345c98 [282777.559339] FS: 00007f4f6d0cc700(0000) GS:ffff88081ea00000(0000) knlGS:0000000000000000 [282777.559340] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [282777.559340] CR2: 00007f178f1cf000 CR3: 00000007f14cd000 CR4: 00000000001407f0 [282777.559341] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [282777.559341] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [282777.559342] Stack: [282777.559343] ffff8807e6345e30 ffffffff810cd098 ffffffff811a019e ffff8807e6345f38 [282777.559345] ffff8807e6345d70 0000000200000001 00007f4f740b1c54 ffffffff81f55548 [282777.559347] 0000000000000064 ffff8807e6345d30 ffff8807e6345d30 ffffffff81f45550 [282777.559347] Call Trace: [282777.559351] [<ffffffff810cd098>] futex_wait+0xe8/0x290 [282777.559354] [<ffffffff811a019e>] ? lookup_page_cgroup_used+0xe/0x30 [282777.559357] [<ffffffff8108e440>] ? hrtimer_get_res+0x50/0x50 [282777.559358] [<ffffffff8108ed34>] ? hrtimer_start_range_ns+0x14/0x20 [282777.559359] [<ffffffff810cec96>] do_futex+0xe6/0xc30 [282777.559362] [<ffffffff810a18fc>] ? update_curr+0xcc/0x160 [282777.559364] [<ffffffff81011611>] ? __switch_to+0x181/0x4b0 [282777.559366] [<ffffffff810cf851>] SyS_futex+0x71/0x150 [282777.559368] [<ffffffff81672129>] system_call_fastpath+0x16/0x1b [282777.559381] Code: 00 0f 1f 44 00 00 55 48 89 e5 b8 00 00 01 00 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 75 02 5d c3 0f b7 07 89 d1 66 39 d0 74 f4 f3 90 <0f> b7 07 66 39 c8 75 f6 5d c3 66 66 66 2e 0f 1f 84 00 00 00 00 [282777.559729] Modules linked in: nfnetlink_queue nfnetlink_log nfnetlink fuse bnep bluetooth cfg80211 rfkill snd_hda_codec_realtek snd_hda_codec_hdmi snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_page_alloc snd_timer e1000e snd ptp iTCO_wdt iTCO_vendor_support mei_me mei lpc_ich serio_raw i2c_i801 x86_pkg_temp_thermal shpchp dcdbas coretemp pps_core mfd_core kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel soundcore ghash_clmulni_intel microcode bcache i915 i2c_algo_bit drm_kms_helper drm i2c_core video [282777.559742] CPU: 1 PID: 897 Comm: java Not tainted 3.12.6-300.fc20.x86_64 #1 [282777.559743] Hardware name: Dell Inc. OptiPlex 9020/0PC5F7, BIOS A02 08/15/2013 [282777.559743] task: ffff8807e301ca40 ti: ffff8807e3226000 task.ti: ffff8807e3226000 [282777.559744] RIP: 0010:[<ffffffff81669b0d>] [<ffffffff81669b0d>] _raw_spin_lock+0x2d/0x40 [282777.559746] RSP: 0018:ffff8807e3227ce8 EFLAGS: 00000a97 [282777.559746] RAX: 00000000000046c0 RBX: 0000000000014540 RCX: 00000000000081f4 [282777.559747] RDX: 00000000000081f4 RSI: ffff8807e301ca40 RDI: ffff8807e3227d20 [282777.559748] RBP: ffff8807e3227ce8 R08: ffff8807e3226000 R09: 0000000000000009 [282777.559748] R10: 0000000000000004 R11: 0000000000000005 R12: 0000000000014540 [282777.559749] R13: ffff8807e3227fd8 R14: ffffffff816678cc R15: ffff8807e3227c98 [282777.559750] FS: 00007f01e39f9700(0000) GS:ffff88081ea40000(0000) knlGS:0000000000000000 [282777.559750] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [282777.559751] CR2: 00007f41f80e0550 CR3: 00000007e63cf000 CR4: 00000000001407e0 [282777.559752] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [282777.559752] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [282777.559753] Stack: [282777.559753] ffff8807e3227e30 ffffffff810cd098 ffffffff811a019e ffff8807e3227f38 [282777.559755] ffff8807e3227d70 0000000200000001 00007f02080a9c54 ffffffff81f546c0 [282777.559756] 0000000000000064 ffff8807e3227d30 ffff8807e3227d30 ffff8807f1115d40 [282777.559757] Call Trace: [282777.559760] [<ffffffff810cd098>] futex_wait+0xe8/0x290 [282777.559762] [<ffffffff811a019e>] ? lookup_page_cgroup_used+0xe/0x30 [282777.559763] [<ffffffff8108e440>] ? hrtimer_get_res+0x50/0x50 [282777.559765] [<ffffffff8108ed34>] ? hrtimer_start_range_ns+0x14/0x20 [282777.559766] [<ffffffff810cec96>] do_futex+0xe6/0xc30 [282777.559768] [<ffffffff810a18a0>] ? update_curr+0x70/0x160 [282777.559769] [<ffffffff810cf851>] SyS_futex+0x71/0x150 [282777.559770] [<ffffffff81672129>] system_call_fastpath+0x16/0x1b [282777.559771] Code: 44 00 00 55 48 89 e5 b8 00 00 01 00 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 75 02 5d c3 0f b7 07 89 d1 66 39 d0 74 f4 f3 90 0f b7 07 <66> 39 c8 75 f6 5d c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f [282777.571313] BUG: soft lockup - CPU#2 stuck for 23s! [java:3350] [282777.571520] Modules linked in: nfnetlink_queue nfnetlink_log nfnetlink fuse bnep bluetooth cfg80211 rfkill snd_hda_codec_realtek snd_hda_codec_hdmi snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_page_alloc snd_timer e1000e snd ptp iTCO_wdt iTCO_vendor_support mei_me mei lpc_ich serio_raw i2c_i801 x86_pkg_temp_thermal shpchp dcdbas coretemp pps_core mfd_core kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel soundcore ghash_clmulni_intel microcode bcache i915 i2c_algo_bit drm_kms_helper drm i2c_core video [282777.571532] CPU: 2 PID: 3350 Comm: java Not tainted 3.12.6-300.fc20.x86_64 #1 [282777.571533] Hardware name: Dell Inc. OptiPlex 9020/0PC5F7, BIOS A02 08/15/2013 [282777.571534] task: ffff8807a4314a40 ti: ffff8807a3272000 task.ti: ffff8807a3272000 [282777.571535] RIP: 0010:[<ffffffff81669b0d>] [<ffffffff81669b0d>] _raw_spin_lock+0x2d/0x40 [282777.571537] RSP: 0018:ffff8807a3273ce8 EFLAGS: 00000a83 [282777.571538] RAX: 0000000000004e28 RBX: 0000000000014540 RCX: 00000000000081f4 [282777.571538] RDX: 00000000000081f4 RSI: ffff8807a4314a40 RDI: ffff8807a3273d20 [282777.571539] RBP: ffff8807a3273ce8 R08: ffff8807a3272000 R09: 0000000000000031 [282777.571540] R10: 0000000000000004 R11: 0000000000000005 R12: 0000000000014540 [282777.571540] R13: ffff8807a3273fd8 R14: ffffffff816678cc R15: ffff8807a3273c98 [282777.571541] FS: 00007fb92d3f4700(0000) GS:ffff88081ea80000(0000) knlGS:0000000000000000 [282777.571542] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [282777.571543] CR2: 00007f4238ad6480 CR3: 00000007f05fb000 CR4: 00000000001407e0 [282777.571544] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [282777.571544] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [282777.571545] Stack: [282777.571545] ffff8807a3273e30 ffffffff810cd098 ffffffff811a019e ffff8807a3273f38 [282777.571547] ffff8807a3273d70 0000000200000001 00007fb9380a9a54 ffffffff81f54e28 [282777.571548] 0000000000000064 ffff8807a3273d30 ffff8807a3273d30 ffff880037111d40 [282777.571550] Call Trace: [282777.571552] [<ffffffff810cd098>] futex_wait+0xe8/0x290 [282777.571554] [<ffffffff811a019e>] ? lookup_page_cgroup_used+0xe/0x30 [282777.571555] [<ffffffff8108e440>] ? hrtimer_get_res+0x50/0x50 [282777.571556] [<ffffffff8108ed34>] ? hrtimer_start_range_ns+0x14/0x20 [282777.571558] [<ffffffff810cec96>] do_futex+0xe6/0xc30 [282777.571559] [<ffffffff810a18fc>] ? update_curr+0xcc/0x160 [282777.571561] [<ffffffff81011611>] ? __switch_to+0x181/0x4b0 [282777.571563] [<ffffffff810cf851>] SyS_futex+0x71/0x150 [282777.571564] [<ffffffff81672129>] system_call_fastpath+0x16/0x1b [282777.571565] Code: 44 00 00 55 48 89 e5 b8 00 00 01 00 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 75 02 5d c3 0f b7 07 89 d1 66 39 d0 74 f4 f3 90 0f b7 07 <66> 39 c8 75 f6 5d c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f [282777.595324] BUG: soft lockup - CPU#4 stuck for 23s! [java:1203] [282777.595674] Modules linked in: nfnetlink_queue nfnetlink_log nfnetlink fuse bnep bluetooth cfg80211 rfkill snd_hda_codec_realtek snd_hda_codec_hdmi snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_page_alloc snd_timer e1000e snd ptp iTCO_wdt iTCO_vendor_support mei_me mei lpc_ich serio_raw i2c_i801 x86_pkg_temp_thermal shpchp dcdbas coretemp pps_core mfd_core kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel soundcore ghash_clmulni_intel microcode bcache i915 i2c_algo_bit drm_kms_helper drm i2c_core video [282777.595699] CPU: 4 PID: 1203 Comm: java Not tainted 3.12.6-300.fc20.x86_64 #1 [282777.595700] Hardware name: Dell Inc. OptiPlex 9020/0PC5F7, BIOS A02 08/15/2013 [282777.595701] task: ffff8807e33ea940 ti: ffff8807ef640000 task.ti: ffff8807ef640000 [282777.595702] RIP: 0010:[<ffffffff81669b0d>] [<ffffffff81669b0d>] _raw_spin_lock+0x2d/0x40 [282777.595707] RSP: 0018:ffff8807ef641ce8 EFLAGS: 00000a97 [282777.595708] RAX: 0000000000005da0 RBX: 0000000000014540 RCX: 00000000000081f4 [282777.595709] RDX: 00000000000081f4 RSI: ffff8807e33ea940 RDI: ffff8807ef641d20 [282777.595710] RBP: ffff8807ef641ce8 R08: ffff8807ef640000 R09: 0000000000000031 [282777.595711] R10: 0000000000000004 R11: 0000000000000005 R12: 0000000000014540 [282777.595712] R13: ffff8807ef641fd8 R14: ffffffff816678cc R15: ffff8807ef641c98 [282777.595713] FS: 00007f420eded700(0000) GS:ffff88081eb00000(0000) knlGS:0000000000000000 [282777.595714] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [282777.595716] CR2: 00007f4f39dcd7b8 CR3: 00000007f1b94000 CR4: 00000000001407e0 [282777.595717] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [282777.595718] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [282777.595719] Stack: [282777.595720] ffff8807ef641e30 ffffffff810cd098 ffffffff811a019e ffff8807ef641f38 [282777.595722] ffff8807ef641d70 0000000200000001 00007f42300a9d54 ffffffff81f55da0 [282777.595724] 0000000000000064 ffff8807ef641d30 ffff8807ef641d30 ffffffff81f45da8 [282777.595726] Call Trace: [282777.595731] [<ffffffff810cd098>] futex_wait+0xe8/0x290 [282777.595734] [<ffffffff811a019e>] ? lookup_page_cgroup_used+0xe/0x30 [282777.595737] [<ffffffff8108e440>] ? hrtimer_get_res+0x50/0x50 [282777.595739] [<ffffffff8108ed34>] ? hrtimer_start_range_ns+0x14/0x20 [282777.595741] [<ffffffff810cec96>] do_futex+0xe6/0xc30 [282777.595744] [<ffffffff810a18fc>] ? update_curr+0xcc/0x160 [282777.595747] [<ffffffff81011611>] ? __switch_to+0x181/0x4b0 [282777.595748] [<ffffffff810cf851>] SyS_futex+0x71/0x150 [282777.595751] [<ffffffff81672129>] system_call_fastpath+0x16/0x1b [282777.595752] Code: 44 00 00 55 48 89 e5 b8 00 00 01 00 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 75 02 5d c3 0f b7 07 89 d1 66 39 d0 74 f4 f3 90 0f b7 07 <66> 39 c8 75 f6 5d c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f CPU is: $ cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 60 model name : Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz stepping : 3 microcode : 0x12 cpu MHz : 3699.890 cache size : 8192 KB physical id : 0 siblings : 8 core id : 0 cpu cores : 4 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm bogomips : 6784.72 clflush size : 64 cache_alignment : 64 address sizes : 39 bits physical, 48 bits virtual power management: Version-Release number of selected component (if applicable): kernel is 3.12.6-300.fc20.x86_64 I was able to run top for short time, then my remote access via SSH stopped working: top - 19:00:42 up 3 days, 6:38, 6 users, load average: 47.83, 46.42, 42.64 Tasks: 227 total, 16 running, 211 sleeping, 0 stopped, 0 zombie %Cpu0 : 0.0 us,100.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu1 : 0.0 us,100.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu2 : 0.0 us, 99.7 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.3 hi, 0.0 si, 0.0 st %Cpu3 : 0.0 us, 0.0 sy, 0.0 ni, 0.0 id, 99.7 wa, 0.0 hi, 0.3 si, 0.0 st %Cpu4 : 0.0 us,100.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu5 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu6 : 0.0 us, 0.0 sy, 0.0 ni, 0.0 id,100.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu7 : 0.0 us, 0.0 sy, 0.0 ni, 0.0 id,100.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 32856528 total, 29409304 used, 3447224 free, 289276 buffers KiB Swap: 16777212 total, 0 used, 16777212 free, 25098972 cached System is still answering pings but I'm unable to log in. Can access the system locally only after the weekend.
Created attachment 848337 [details] Full dmesg output
In the full dmesg output this may be interesting: [282818.373615] INFO: rcu_sched self-detected stall on CPU [282818.373616] INFO: rcu_sched self-detected stall on CPU [282818.373617] INFO: rcu_sched self-detected stall on CPU [282818.373617] INFO: rcu_sched self-detected stall on CPU [282818.373618] { [282818.373618] { [282818.373620] { [282818.373620] 4 [282818.373621] 1 [282818.373621] 2 [282818.373622] } [282818.373622] } [282818.373623] (t=2400039 jiffies g=288243 c=288242 q=200719) [282818.373623] (t=2400039 jiffies g=288243 c=288242 q=200719) [282818.373624] } (t=2400039 jiffies g=288243 c=288242 q=200719) [282818.373624] sending NMI to all CPUs: [282818.373626] NMI backtrace for cpu 4 [282818.373627] CPU: 4 PID: 1203 Comm: java Not tainted 3.12.6-300.fc20.x86_64 #1 [282818.373628] Hardware name: Dell Inc. OptiPlex 9020/0PC5F7, BIOS A02 08/15/2013 [282818.373628] task: ffff8807e33ea940 ti: ffff8807ef640000 task.ti: ffff8807ef640000 [282818.373633] RIP: 0010:[<ffffffff81312474>] [<ffffffff81312474>] __bitmap_andnot+0x24/0x50 [282818.373633] RSP: 0018:ffff88081eb03d78 EFLAGS: 00000016 [282818.373634] RAX: 0000000000000000 RBX: 00000000000000ff RCX: 0000000000000004 [282818.373634] RDX: ffff88081ea0df80 RSI: ffff88081eb0df00 RDI: ffff88081eb0df00 [282818.373635] RBP: ffff88081eb03d78 R08: 0000000000000000 R09: 0000000000000010 [282818.373636] R10: 0000000000013f5c R11: 0000000000040000 R12: ffff88081eb0df00 [282818.373636] R13: 000000000000e000 R14: ffff88081ea0df80 R15: 0000000000080000 [282818.373637] FS: 00007f420eded700(0000) GS:ffff88081eb00000(0000) knlGS:0000000000000000 [282818.373638] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [282818.373638] CR2: 00007f4f39dcd7b8 CR3: 00000007f1b94000 CR4: 00000000001407e0 [282818.373639] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [282818.373639] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [282818.373640] Stack: [282818.373642] ffff88081eb03dd8 ffffffff81047c65 0000000000000096 000000021eb03de8 [282818.373643] 000000000000df80 0000000000000004 0000000000000004 0000000000002710 [282818.373645] ffffffff81c53bc0 ffffffff81c53bc0 ffff88081eb0ef60 000000000003100f [282818.373645] Call Trace: [282818.373646] <IRQ> [282818.373649] [<ffffffff81047c65>] __x2apic_send_IPI_mask+0x1c5/0x1f0 [282818.373650] [<ffffffff81047cac>] x2apic_send_IPI_all+0x1c/0x20 [282818.373653] [<ffffffff81043627>] arch_trigger_all_cpu_backtrace+0x57/0x90 [282818.373655] [<ffffffff81103aad>] rcu_check_callbacks+0x31d/0x600 [282818.373658] [<ffffffff81076d77>] update_process_times+0x47/0x80 [282818.373661] [<ffffffff810caca5>] tick_sched_handle.isra.15+0x25/0x60 [282818.373663] [<ffffffff810cad21>] tick_sched_timer+0x41/0x60 [282818.373665] [<ffffffff8108e704>] __run_hrtimer+0x74/0x1d0 [282818.373666] [<ffffffff810cace0>] ? tick_sched_handle.isra.15+0x60/0x60 [282818.373668] [<ffffffff8108ef17>] hrtimer_interrupt+0xf7/0x240 [282818.373670] [<ffffffff810419f7>] local_apic_timer_interrupt+0x37/0x60 [282818.373672] [<ffffffff8167437f>] smp_apic_timer_interrupt+0x3f/0x60 [282818.373673] [<ffffffff81672d1d>] apic_timer_interrupt+0x6d/0x80 [282818.373674] <EOI> [282818.373676] [<ffffffff81669b0d>] ? _raw_spin_lock+0x2d/0x40 [282818.373677] [<ffffffff810cd098>] futex_wait+0xe8/0x290 [282818.373680] [<ffffffff811a019e>] ? lookup_page_cgroup_used+0xe/0x30 [282818.373682] [<ffffffff8108e440>] ? hrtimer_get_res+0x50/0x50 [282818.373683] [<ffffffff8108ed34>] ? hrtimer_start_range_ns+0x14/0x20 [282818.373684] [<ffffffff810cec96>] do_futex+0xe6/0xc30 [282818.373687] [<ffffffff810a18fc>] ? update_curr+0xcc/0x160 [282818.373689] [<ffffffff81011611>] ? __switch_to+0x181/0x4b0 [282818.373690] [<ffffffff810cf851>] SyS_futex+0x71/0x150 [282818.373691] [<ffffffff81672129>] system_call_fastpath+0x16/0x1b [282818.373705] Code: 1f 84 00 00 00 00 00 48 63 c9 55 48 83 c1 3f 48 c1 e9 06 48 89 e5 85 c9 7e 32 41 89 c9 45 31 c0 31 c9 0f 1f 44 00 00 48 8b 04 ca <48> f7 d0 48 23 04 ce 48 89 04 cf 48 83 c1 01 49 09 c0 41 39 c9
Locally console was full of kernel:[284106.086619] BUG: soft lockup - CPU#0 stuck for 22s! [java:931] messages for CPUs 0, 1, 2 and 4. I could write login name to the login prompt but then password prompt does not appear. Had to power cycle. Journalctl shows nothing about the issue: Jan 10 18:01:01 jesse dbus-daemon[724]: dbus[724]: [system] Rejected send message, 3 matched rules; type="method_call", sender=":1.19" (uid=1000 pid Jan 10 18:01:01 jesse dbus[724]: [system] Rejected send message, 3 matched rules; type="method_call", sender=":1.19" (uid=1000 pid=1609 comm="nm-app Jan 10 18:01:01 jesse dbus[724]: [system] Rejected send message, 3 matched rules; type="method_call", sender=":1.21" (uid=1000 pid=1722 comm="/usr/l Jan 10 18:01:01 jesse dbus-daemon[724]: dbus[724]: [system] Rejected send message, 3 matched rules; type="method_call", sender=":1.21" (uid=1000 pid -- Reboot -- Jan 13 08:13:23 jesse systemd-journal[118]: Runtime journal is using 8.0M (max 1.5G, leaving 2.3G of free 15.6G, current limit 1.5G). Jan 13 08:13:23 jesse systemd-journal[118]: Runtime journal is using 8.0M (max 1.5G, leaving 2.3G of free 15.6G, current limit 1.5G). How to track the issue further?
Created attachment 849775 [details] kernel panic screenshot Don't know if this is related but now the same system did a complete kernel panic, there were no soft lockups before this. PANIC: double fault, error_code: 0x0 Kernel panic - not syncing: Machine halted. ... dump_stack panic df_debug do_double_fault double_fault ? try_to_wake_up ? notifier_call_chain
With kernel 3.12.7-300.fc20.x86_64 the system has been stable almost 5 days now, perhaps the issue was limited to early kernels released for Fedora 20.
I'm not aware of anything that would explicitly fix the issue you reported. Since you aren't seeing it now with later kernels, we'll mark this as WORKSFORME. If it comes back, please reopen.
Unfortunately after upgrading to the latest kernel (3.12.8-300.fc20.x86_64) system crashed again with less than 24 hours of uptime. RIP pick_next_task_fair See next screenshot from console. Hardware is good as far as memtest and Dell's own diagnostics can tell.
Created attachment 853919 [details] kernel panic console screenshot
Created attachment 855920 [details] second kernel panic console screenshot New crash with 3.12.8-300.fc20.x86_64. RIP [<ffffffff8109ee71>] update_cfs_rq_blocked_load+0x71/0x140
Created attachment 855931 [details] lspci -vv output Some more details about the system, attached lspci -vv output from the system. System is using SSD + HDD / bcache / LVM. /boot is directly on SSD, root, home and swap are on bcache+LVM
Created attachment 855932 [details] dmesg after boot
Created attachment 858435 [details] New occurence of comment 9 crash - console output Exact same crash as in comment #9 occurred again.
Created attachment 862670 [details] kernel BUG console screenshot My crash saga continues. Now with 3.12.10-300.fc20.x86_64 I got this: kernel tried to execute NX-protected page - exploit attempt? (uid: 0) Something to do with bug 1056807 that is marked dupe to bug 1056647 (restricted access at this time)? My system is using lvm2 with bcache. Perhaps this NX-protected page warning was present also in previous crashes but it was not visible on console. Need to attach serial console...
*********** MASS BUG UPDATE ************** We apologize for the inconvenience. There is a large number of bugs to go through and several of them have gone stale. Due to this, we are doing a mass bug update across all of the Fedora 20 kernel bugs. Fedora 20 has now been rebased to 3.13.4-200.fc20. Please test this kernel update and let us know if you issue has been resolved or if it is still present with the newer kernel. If you experience different issues, please open a new bug report for those.
3.13.3-201.fc20.x86_64 still crashes. Will upgrade to 3.13.4-200 next. jesse login: Fedora release 20 (Heisenbug) Kernel 3.13.3-201.fc20.x86_64 on an x86_64 (ttyS0) jesse login: [69132.876900] general protection fault: 0000 [#1] SMP [69132.881882] Modules linked in: fuse bnep bluetooth cfg80211 rfkill snd_hda_codec_realtek snd_hda_codec_hdmi snd_hda_intel snd_hda_codec snd_hwdep e1000e ptp snd_seq snd_seq_device iTCO_wdt iTCO_vendor_support mei_me snd_pcm snd_page_alloc snd_timer snd mei lpc_ich pps_core shpchp serio_raw x86_pkg_temp_thermal coretemp kvm_intel kvm crct10dif_pclmul dcdbas crc32_pclmul soundcore crc32c_intel ghash_clmulni_intel mfd_core microcode i2c_i801 bcache i915 i2c_algo_bit drm_kms_helper drm video i2c_core [69132.926331] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 3.13.3-201.fc20.x86_64 #1 [69132.933625] Hardware name: Dell Inc. OptiPlex 9020/0PC5F7, BIOS A02 08/15/2013 [69132.940840] task: ffff8807f2cfdee0 ti: ffff8807f2d8e000 task.ti: ffff8807f2d8e000 [69132.948315] RIP: 0010:[<ffffffff810a84f0>] [<ffffffff810a84f0>] enqueue_entity+0x1a0/0x8e0 [69132.956670] RSP: 0018:ffff88081ea83d80 EFLAGS: 00010097 [69132.961978] RAX: 0000000000000014 RBX: ffff8807e39322e8 RCX: 000000000000b7d9 [69132.969101] RDX: 000000000000b800 RSI: 0000000000000026 RDI: 0000000000000013 [69132.976230] RBP: ffff88081ea83db8 R08: 0000000000000014 R09: 0000000000000000 [69132.983356] R10: 1355bd7d68d46c67 R11: 000000000001d70f R12: ffff8807ef4bb800 [69132.990484] R13: 0000000000000000 R14: 000000000000030b R15: 0000000000000000 [69132.997611] FS: 0000000000000000(0000) GS:ffff88081ea80000(0000) knlGS:0000000000000000 [69133.005687] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [69133.011421] CR2: 00007f81a97fa580 CR3: 0000000001c0c000 CR4: 00000000001407e0 [69133.018544] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [69133.025675] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [69133.032802] Stack: [69133.034811] 0000000000000002 00000001041ac8dc ffff8807e39322e8 ffff8807ef4bb800 [69133.042240] ffff88081ea94580 0000000000000046 0000000000000002 ffff88081ea83e08 [69133.049679] ffffffff810a8c89 ffff88081ea94580 ffff88081ea94580 ffff8807e393283c [69133.057107] Call Trace: [69133.059548] <IRQ> [69133.061469] [<ffffffff810a8c89>] enqueue_task_fair+0x59/0x660 [69133.067491] [<ffffffff81098b0c>] enqueue_task+0x2c/0x70 [69133.072796] [<ffffffff8109baf3>] activate_task+0x23/0x30 [69133.078189] [<ffffffff8109bdd3>] ttwu_do_activate.constprop.85+0x33/0x70 [69133.084969] [<ffffffff8109e258>] try_to_wake_up+0x1e8/0x290 [69133.090623] [<ffffffff8109e323>] wake_up_process+0x23/0x40 [69133.096189] [<ffffffff81091ee2>] hrtimer_wakeup+0x22/0x40 [69133.101667] [<ffffffff81092184>] __run_hrtimer+0x74/0x1d0 [69133.107144] [<ffffffff81091ec0>] ? hrtimer_get_res+0x50/0x50 [69133.112887] [<ffffffff81092997>] hrtimer_interrupt+0xf7/0x240 [69133.118714] [<ffffffff81042db7>] local_apic_timer_interrupt+0x37/0x60 [69133.125238] [<ffffffff81697dff>] smp_apic_timer_interrupt+0x3f/0x60 [69133.131585] [<ffffffff8169679d>] apic_timer_interrupt+0x6d/0x80 [69133.137582] <EOI> [69133.139504] [<ffffffff815317c2>] ? cpuidle_enter_state+0x52/0xc0 [69133.145787] [<ffffffff815318e9>] cpuidle_idle_call+0xb9/0x1f0 [69133.151612] [<ffffffff8101c8fe>] arch_cpu_idle+0xe/0x30 [69133.156919] [<ffffffff810c05c5>] cpu_startup_entry+0xc5/0x290 [69133.162746] [<ffffffff81040e4b>] start_secondary+0x22b/0x2e0 [69133.168480] Code: 03 00 00 89 93 84 00 00 00 48 c1 ef 0a 48 8d 47 01 48 3d e0 07 00 00 0f 87 cf 06 00 00 83 f8 1f 41 89 c0 0f 87 f1 06 00 00 89 c0 <cb> 0c 85 60 e6 80 81 44 89 c0 48 0f af ce 48 c1 e9 20 89 ce 89 [69133.188036] RIP [<ffffffff810a84f0>] enqueue_entity+0x1a0/0x8e0 [69133.194046] RSP <ffff88081ea83d80> [69133.197531] ---[ end trace b9ce1fa5557f6b6c ]--- [69133.197533] general protection fault: 8804 [#2] SMP [69133.197541] Modules linked in: fuse bnep bluetooth cfg80211 rfkill snd_hda_codec_realtek snd_hda_codec_hdmi snd_hda_intel snd_hda_codec snd_hwdep e1000e ptp snd_seq snd_seq_device iTCO_wdt iTCO_vendor_support mei_me snd_pcm snd_page_alloc snd_timer snd mei lpc_ich pps_core shpchp serio_raw x86_pkg_temp_thermal coretemp kvm_intel kvm crct10dif_pclmul dcdbas crc32_pclmul soundcore crc32c_intel ghash_clmulni_intel mfd_core microcode i2c_i801 bcache i915 i2c_algo_bit drm_kms_helper drm video i2c_core [69133.197542] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G D 3.13.3-201.fc20.x86_64 #1 [69133.197542] Hardware name: Dell Inc. OptiPlex 9020/0PC5F7, BIOS A02 08/15/2013 [69133.197543] task: ffff8807f2cfe780 ti: ffff8807f2d90000 task.ti: ffff8807f2d90000 [69133.197545] RIP: 0010:[<ffffffff810a84f0>] [<ffffffff810a84f0>] enqueue_entity+0x1a0/0x8e0 [69133.197545] RSP: 0018:ffff88081eac3d80 EFLAGS: 00010097 [69133.197545] RAX: 0000000000000001 RBX: ffff8807ee908b40 RCX: 000000000000b959 [69133.197546] RDX: 000000000000bc00 RSI: 000000000000003b RDI: 0000000000000000 [69133.197546] RBP: ffff88081eac3db8 R08: 0000000000000001 R09: 0000000000000001 [69133.197546] R10: 0000000000000000 R11: 000000000000e26c R12: ffff88081ead45f8 [69133.197547] R13: ffff8807f019e400 R14: 0000000000000166 R15: 0000000000000000 [69133.197547] FS: 0000000000000000(0000) GS:ffff88081eac0000(0000) knlGS:0000000000000000 [69133.197548] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [69133.197548] CR2: 00007ff9f4cd8000 CR3: 0000000001c0c000 CR4: 00000000001407e0 [69133.197549] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [69133.197549] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [69133.197549] Stack: [69133.197550] ffff8807efc2a9e0 000045690000002a ffff8807ee908b40 ffff88081ead45f8 [69133.197551] ffff88081ead4580 0000000000000046 0000000000000003 ffff88081eac3e08 [69133.197552] ffffffff810a8c89 ffff88081ead4580 ffff88081ead4580 ffff8807f1848e5c [69133.197552] Call Trace: [69133.197553] <IRQ> [69133.197554] [<ffffffff810a8c89>] enqueue_task_fair+0x59/0x660 [69133.197555] [<ffffffff81098b0c>] enqueue_task+0x2c/0x70 [69133.197556] [<ffffffff8109baf3>] activate_task+0x23/0x30 [69133.197557] [<ffffffff8109bdd3>] ttwu_do_activate.constprop.85+0x33/0x70 [69133.197558] [<ffffffff8109e258>] try_to_wake_up+0x1e8/0x290 [69133.197560] [<ffffffff8109e323>] wake_up_process+0x23/0x40 [69133.197561] [<ffffffff81091ee2>] hrtimer_wakeup+0x22/0x40 [69133.197561] [<ffffffff81092184>] __run_hrtimer+0x74/0x1d0 [69133.197562] [<ffffffff81091ec0>] ? hrtimer_get_res+0x50/0x50 [69133.197563] [<ffffffff81092997>] hrtimer_interrupt+0xf7/0x240 [69133.197565] [<ffffffff81042db7>] local_apic_timer_interrupt+0x37/0x60 [69133.197567] [<ffffffff81697dff>] smp_apic_timer_interrupt+0x3f/0x60 [69133.197568] [<ffffffff8169679d>] apic_timer_interrupt+0x6d/0x80 [69133.197569] <EOI> [69133.197570] [<ffffffff815317c2>] ? cpuidle_enter_state+0x52/0xc0 [69133.197571] [<ffffffff815318e9>] cpuidle_idle_call+0xb9/0x1f0 [69133.197573] [<ffffffff8101c8fe>] arch_cpu_idle+0xe/0x30 [69133.197574] [<ffffffff810c05c5>] cpu_startup_entry+0xc5/0x290 [69133.197575] [<ffffffff81040e4b>] start_secondary+0x22b/0x2e0 [69133.197582] Code: 03 00 00 89 93 84 00 00 00 48 c1 ef 0a 48 8d 47 01 48 3d e0 07 00 00 0f 87 cf 06 00 00 83 f8 1f 41 89 c0 0f 87 f1 06 00 00 89 c0 <cb> 0c 85 60 e6 80 81 44 89 c0 48 0f af ce 48 c1 e9 20 89 ce 89 [69133.197583] RIP [<ffffffff810a84f0>] enqueue_entity+0x1a0/0x8e0 [69133.197584] RSP <ffff88081eac3d80> [69133.197585] ---[ end trace b9ce1fa5557f6b6d ]--- [69133.197586] Kernel panic - not syncing: Fatal exception in interrupt [69133.197587] general protection fault: 8808 [#3] SMP [69133.197595] Modules linked in: fuse bnep bluetooth cfg80211 rfkill snd_hda_codec_realtek snd_hda_codec_hdmi snd_hda_intel snd_hda_codec snd_hwdep e1000e ptp snd_seq snd_seq_device iTCO_wdt iTCO_vendor_support mei_me snd_pcm snd_page_alloc snd_timer snd mei lpc_ich pps_core shpchp serio_raw x86_pkg_temp_thermal coretemp kvm_intel kvm crct10dif_pclmul dcdbas crc32_pclmul soundcore crc32c_intel ghash_clmulni_intel mfd_core microcode i2c_i801 bcache i915 i2c_algo_bit drm_kms_helper drm video i2c_core [69133.197596] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G D 3.13.3-201.fc20.x86_64 #1 [69133.197596] Hardware name: Dell Inc. OptiPlex 9020/0PC5F7, BIOS A02 08/15/2013 [69133.197597] task: ffff8807f2cfd640 ti: ffff8807f2d8c000 task.ti: ffff8807f2d8c000 [69133.197599] RIP: 0010:[<ffffffff810a84f0>] [<ffffffff810a84f0>] enqueue_entity+0x1a0/0x8e0 [69133.197599] RSP: 0018:ffff88081ea43d80 EFLAGS: 00010093 [69133.197600] RAX: 0000000000000008 RBX: ffff8807541e8068 RCX: 000000000000bb43 [69133.197600] RDX: 000000000000bc00 RSI: 0000000000000155 RDI: 0000000000000007 [69133.197600] RBP: ffff88081ea43db8 R08: 0000000000000008 R09: 0000000000000000 [69133.197601] R10: 1355bd7d68d46c67 R11: 0000000000017cbd R12: ffff8807f019f200 [69133.197601] R13: 0000000000000000 R14: 0000000000000250 R15: 0000000000000000 [69133.197602] FS: 0000000000000000(0000) GS:ffff88081ea40000(0000) knlGS:0000000000000000 [69133.197602] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [69133.197602] CR2: 00007f4f576cc000 CR3: 0000000001c0c000 CR4: 00000000001407e0 [69133.197603] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [69133.197603] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [69133.197603] Stack: [69133.197604] ffff88081ea545f8 000050c600000017 ffff8807541e8068 ffff8807f019f200 [69133.197605] ffff88081ea54580 0000000000000046 0000000000000001 ffff88081ea43e08 [69133.197606] ffffffff810a8c89 ffff88081ea54580 ffff88081ea54580 ffff8807541e85bc [69133.197606] Call Trace: [69133.197607] <IRQ> [69133.197608] [<ffffffff810a8c89>] enqueue_task_fair+0x59/0x660 [69133.197609] [<ffffffff81098b0c>] enqueue_task+0x2c/0x70 [69133.197610] [<ffffffff8109baf3>] activate_task+0x23/0x30 [69133.197611] [<ffffffff8109bdd3>] ttwu_do_activate.constprop.85+0x33/0x70 [69133.197612] [<ffffffff8109e258>] try_to_wake_up+0x1e8/0x290 [69133.197613] [<ffffffff8109e323>] wake_up_process+0x23/0x40 [69133.197614] [<ffffffff81091ee2>] hrtimer_wakeup+0x22/0x40 [69133.197615] [<ffffffff81092184>] __run_hrtimer+0x74/0x1d0 [69133.197616] [<ffffffff81091ec0>] ? hrtimer_get_res+0x50/0x50 [69133.197617] [<ffffffff81092997>] hrtimer_interrupt+0xf7/0x240 [69133.197619] [<ffffffff81042db7>] local_apic_timer_interrupt+0x37/0x60 [69133.197620] [<ffffffff81697dff>] smp_apic_timer_interrupt+0x3f/0x60 [69133.197621] [<ffffffff8169679d>] apic_timer_interrupt+0x6d/0x80 [69133.197622] <EOI> [69133.197623] [<ffffffff815317c2>] ? cpuidle_enter_state+0x52/0xc0 [69133.197624] [<ffffffff815318e9>] cpuidle_idle_call+0xb9/0x1f0 [69133.197625] [<ffffffff8101c8fe>] arch_cpu_idle+0xe/0x30 [69133.197627] [<ffffffff810c05c5>] cpu_startup_entry+0xc5/0x290 [69133.197628] [<ffffffff81040e4b>] start_secondary+0x22b/0x2e0 [69133.197635] Code: 03 00 00 89 93 84 00 00 00 48 c1 ef 0a 48 8d 47 01 48 3d e0 07 00 00 0f 87 cf 06 00 00 83 f8 1f 41 89 c0 0f 87 f1 06 00 00 89 c0 <cb> 0c 85 60 e6 80 81 44 89 c0 48 0f af ce 48 c1 e9 20 89 ce 89 [69133.197636] RIP [<ffffffff810a84f0>] enqueue_entity+0x1a0/0x8e0 [69133.197636] RSP <ffff88081ea43d80> [69133.197637] ---[ end trace b9ce1fa5557f6b6e ]--- [69134.229569] Shutting down cpus with NMI [69134.234129] drm_kms_helper: panic occurred, switching back to text console
Unfortunately this system is not stable with the latest kernel either. It now has serial console attached so I can get full messages: jesse login: Fedora release 20 (Heisenbug) Kernel 3.13.4-200.fc20.x86_64 on an x86_64 (ttyS0) jesse login: [217505.276023] BUG: unable to handle kernel NULL pointer dereference at 0000000000000018 [217505.283968] IP: [<ffffffff810a3577>] select_task_rq_fair+0x557/0x6f0 [217505.290436] PGD 0 [217505.292553] Oops: 0000 [#1] SMP [217505.295893] Modules linked in: vfat fat usb_storage fuse bnep bluetooth cfg80211 rfkill snd_hda_codec_realtek snd_hda_codec_hdmi snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device iTCO_wdt iTCO_vendor_support dcdbas snd_pcm snd_page_alloc e1000e snd_timer x86_pkg_temp_thermal serio_raw snd coretemp mei_me ptp soundcore mei i2c_i801 shpchp kvm_intel pps_core lpc_ich mfd_core kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel microcode bcache i915 i2c_algo_bit drm_kms_helper drm i2c_core video [217505.342413] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 3.13.4-200.fc20.x86_64 #1 [217505.349809] Hardware name: Dell Inc. OptiPlex 9020/0PC5F7, BIOS A02 08/15/2013 [217505.357138] task: ffff8807f2cfe780 ti: ffff8807f2d90000 task.ti: ffff8807f2d90000 [217505.364727] RIP: 0010:[<ffffffff810a3577>] [<ffffffff810a3577>] select_task_rq_fair+0x557/0x6f0 [217505.373622] RSP: 0018:ffff88081eac3dc8 EFLAGS: 00010046 [217505.379035] RAX: ffff8807eb996000 RBX: ffff8807fe00ba00 RCX: 0000000000000000 [217505.386258] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000003 [217505.393497] RBP: ffff88081eac3e60 R08: 0000000000000010 R09: 00000000ffffffff [217505.400723] R10: 13560a0693057cf8 R11: 000000000001278d R12: 0000000000000003 [217505.407947] R13: 0000000000014580 R14: 0000000000000003 R15: 0000000000000000 [217505.415171] FS: 0000000000000000(0000) GS:ffff88081eac0000(0000) knlGS:0000000000000000 [217505.423371] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [217505.429204] CR2: 0000000000000018 CR3: 0000000001c0c000 CR4: 00000000001407e0 [217505.436446] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [217505.443686] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [217505.450909] Stack: [217505.453007] ffff88081ead4580 ffff88081ead4580 ffff8807e557397c ffff88081ead4580 [217505.460543] ffff880800000010 ffff8807e557397c 0000000000000046 0000000000000003 [217505.468074] ffffffff00000003 ffff8807e55733c0 0000000300000010 ffffffff8109bca9 [217505.475604] Call Trace: [217505.478138] <IRQ> [217505.480147] [<ffffffff8109bca9>] ? ttwu_do_wakeup+0x19/0xc0 [217505.486097] [<ffffffff8109e1c9>] try_to_wake_up+0x159/0x290 [217505.491857] [<ffffffff8109e323>] wake_up_process+0x23/0x40 [217505.497531] [<ffffffff81091ee2>] hrtimer_wakeup+0x22/0x40 [217505.503105] [<ffffffff81092184>] __run_hrtimer+0x74/0x1d0 [217505.508681] [<ffffffff81091ec0>] ? hrtimer_get_res+0x50/0x50 [217505.514514] [<ffffffff81092997>] hrtimer_interrupt+0xf7/0x240 [217505.520438] [<ffffffff81042db7>] local_apic_timer_interrupt+0x37/0x60 [217505.527054] [<ffffffff81697e3f>] smp_apic_timer_interrupt+0x3f/0x60 [217505.533497] [<ffffffff816967dd>] apic_timer_interrupt+0x6d/0x80 [217505.539603] <EOI> [217505.541619] [<ffffffff815317e2>] ? cpuidle_enter_state+0x52/0xc0 [217505.548008] [<ffffffff81531909>] cpuidle_idle_call+0xb9/0x1f0 [217505.553927] [<ffffffff8101c8fe>] arch_cpu_idle+0xe/0x30 [217505.559328] [<ffffffff810c05c5>] cpu_startup_entry+0xc5/0x290 [217505.565264] [<ffffffff81040e4b>] start_secondary+0x22b/0x2e0 [217505.571097] Code: 85 d8 00 00 00 48 85 f6 0f 85 35 01 00 00 85 c9 0f 85 0e 01 00 00 4c 3b 5d a8 0f 8e e5 00 00 00 48 8b 45 c8 44 89 f7 48 8b 40 08 <8b> 59 18 e8 91 9d ff ff 85 c0 0f 85 70 fc ff ff 41 39 de 74 12 [217505.590842] RIP [<ffffffff810a3577>] select_task_rq_fair+0x557/0x6f0 [217505.597383] RSP <ffff88081eac3dc8> [217505.600957] CR2: 0000000000000018 [217505.604366] ---[ end trace 16fcffdc1f2af252 ]--- [217505.604367] BUG: unable to handle kernel NULL pointer dereference at 0000000000000018 [217505.604369] IP: [<ffffffff810a3577>] select_task_rq_fair+0x557/0x6f0 [217505.604370] PGD 0 [217505.604370] Oops: 0000 [#2] SMP [217505.604379] Modules linked in: vfat fat usb_storage fuse bnep bluetooth cfg80211 rfkill snd_hda_codec_realtek snd_hda_codec_hdmi snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device iTCO_wdt iTCO_vendor_support dcdbas snd_pcm snd_page_alloc e1000e snd_timer x86_pkg_temp_thermal serio_raw snd coretemp mei_me ptp soundcore mei i2c_i801 shpchp kvm_intel pps_core lpc_ich mfd_core kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel microcode bcache i915 i2c_algo_bit drm_kms_helper drm i2c_core video [217505.604380] CPU: 7 PID: 0 Comm: swapper/7 Tainted: G D 3.13.4-200.fc20.x86_64 #1 [217505.604380] Hardware name: Dell Inc. OptiPlex 9020/0PC5F7, BIOS A02 08/15/2013 [217505.604381] task: ffff8807f2d99140 ti: ffff8807f2da0000 task.ti: ffff8807f2da0000 [217505.604383] RIP: 0010:[<ffffffff810a3577>] [<ffffffff810a3577>] select_task_rq_fair+0x557/0x6f0 [217505.604383] RSP: 0018:ffff88081ebc3dc8 EFLAGS: 00010046 [217505.604384] RAX: ffff8800dbb32000 RBX: ffff8807f2a54200 RCX: 0000000000000000 [217505.604384] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000007 [217505.604384] RBP: ffff88081ebc3e60 R08: 0000000000000010 R09: 00000000ffffffff [217505.604385] R10: 13560a0693057cf8 R11: 00000000000105f9 R12: 0000000000000007 [217505.604385] R13: 0000000000014580 R14: 0000000000000007 R15: 0000000000000000 [217505.604386] FS: 0000000000000000(0000) GS:ffff88081ebc0000(0000) knlGS:0000000000000000 [217505.604386] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [217505.604386] CR2: 0000000000000018 CR3: 0000000001c0c000 CR4: 00000000001407e0 [217505.604387] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [217505.604387] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [217505.604388] Stack: [217505.604389] ffff88081ebc3de8 ffffffff81575704 ffffc90007778950 ffff8800db8a8c01 [217505.604390] ffff880800000010 ffffffff81582da5 0000000000000000 ffff88081ebc3e20 [217505.604391] ffffffff00000007 ffff88081ebc3e30 0000000700000010 0000000000000007 [217505.604392] Call Trace: [217505.604392] <IRQ> [217505.604394] [<ffffffff81575704>] ? consume_skb+0x34/0x80 [217505.604396] [<ffffffff81582da5>] ? dev_kfree_skb_any+0x35/0x40 [217505.604398] [<ffffffff8109e1c9>] try_to_wake_up+0x159/0x290 [217505.604399] [<ffffffff8109e323>] wake_up_process+0x23/0x40 [217505.604400] [<ffffffff81091ee2>] hrtimer_wakeup+0x22/0x40 [217505.604401] [<ffffffff81092184>] __run_hrtimer+0x74/0x1d0 [217505.604402] [<ffffffff81091ec0>] ? hrtimer_get_res+0x50/0x50 [217505.604404] [<ffffffff81092997>] hrtimer_interrupt+0xf7/0x240 [217505.604405] [<ffffffff81042db7>] local_apic_timer_interrupt+0x37/0x60 [217505.604407] [<ffffffff81697e3f>] smp_apic_timer_interrupt+0x3f/0x60 [217505.604408] [<ffffffff816967dd>] apic_timer_interrupt+0x6d/0x80 [217505.604409] <EOI> [217505.604410] [<ffffffff815317e2>] ? cpuidle_enter_state+0x52/0xc0 [217505.604411] [<ffffffff81531909>] cpuidle_idle_call+0xb9/0x1f0 [217505.604412] [<ffffffff8101c8fe>] arch_cpu_idle+0xe/0x30 [217505.604414] [<ffffffff810c05c5>] cpu_startup_entry+0xc5/0x290 [217505.604415] [<ffffffff81040e4b>] start_secondary+0x22b/0x2e0 [217505.604425] Code: 85 d8 00 00 00 48 85 f6 0f 85 35 01 00 00 85 c9 0f 85 0e 01 00 00 4c 3b 5d a8 0f 8e e5 00 00 00 48 8b 45 c8 44 89 f7 48 8b 40 08 <8b> 59 18 e8 91 9d ff ff 85 c0 0f 85 70 fc ff ff 41 39 de 74 12 [217505.604427] RIP [<ffffffff810a3577>] select_task_rq_fair+0x557/0x6f0 [217505.604427] RSP <ffff88081ebc3dc8> [217505.604427] CR2: 0000000000000018 [217505.604428] ---[ end trace 16fcffdc1f2af253 ]--- [217505.604429] Kernel panic - not syncing: Fatal exception in interrupt [217505.604430] BUG: unable to handle kernel NULL pointer dereference at 0000000000000018 [217505.604432] IP: [<ffffffff810a3577>] select_task_rq_fair+0x557/0x6f0 [217505.604433] PGD 0 [217505.604434] Oops: 0000 [#3] SMP [217505.604442] Modules linked in: vfat fat usb_storage fuse bnep bluetooth cfg80211 rfkill snd_hda_codec_realtek snd_hda_codec_hdmi snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device iTCO_wdt iTCO_vendor_support dcdbas snd_pcm snd_page_alloc e1000e snd_timer x86_pkg_temp_thermal serio_raw snd coretemp mei_me ptp soundcore mei i2c_i801 shpchp kvm_intel pps_core lpc_ich mfd_core kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel microcode bcache i915 i2c_algo_bit drm_kms_helper drm i2c_core video [217505.604443] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G D 3.13.4-200.fc20.x86_64 #1 [217505.604444] Hardware name: Dell Inc. OptiPlex 9020/0PC5F7, BIOS A02 08/15/2013 [217505.604444] task: ffff8807f2cfdee0 ti: ffff8807f2d8e000 task.ti: ffff8807f2d8e000 [217505.604446] RIP: 0010:[<ffffffff810a3577>] [<ffffffff810a3577>] select_task_rq_fair+0x557/0x6f0 [217505.604446] RSP: 0018:ffff88081ea83dc8 EFLAGS: 00010046 [217505.604447] RAX: ffff8807ed2f8000 RBX: ffff8807fe00b800 RCX: 0000000000000000 [217505.604447] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000002 [217505.604447] RBP: ffff88081ea83e60 R08: 0000000000000010 R09: 00000000ffffffff [217505.604448] R10: 13560a0693057cf8 R11: 000000000001935b R12: 0000000000000002 [217505.604448] R13: 0000000000014580 R14: 0000000000000002 R15: 0000000000000000 [217505.604449] FS: 0000000000000000(0000) GS:ffff88081ea80000(0000) knlGS:0000000000000000 [217505.604449] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [217505.604449] CR2: 0000000000000018 CR3: 0000000001c0c000 CR4: 00000000001407e0 [217505.604450] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [217505.604450] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [217505.604450] Stack: [217505.604451] ffff88081ea94580 ffff88081ea94580 ffff880793d785bc ffff88081ea94580 [217505.604452] ffff880800000010 ffff880793d785bc 0000000000000046 0000000000000002 [217505.604453] ffffffff00000002 ffff880793d78000 0000000200000010 ffffffff8109bca9 [217505.604453] Call Trace: [217505.604454] <IRQ> [217505.604455] [<ffffffff8109bca9>] ? ttwu_do_wakeup+0x19/0xc0 [217505.604457] [<ffffffff8109e1c9>] try_to_wake_up+0x159/0x290 [217505.604458] [<ffffffff8109e323>] wake_up_process+0x23/0x40 [217505.604459] [<ffffffff81091ee2>] hrtimer_wakeup+0x22/0x40 [217505.604460] [<ffffffff81092184>] __run_hrtimer+0x74/0x1d0 [217505.604461] [<ffffffff81091ec0>] ? hrtimer_get_res+0x50/0x50 [217505.604462] [<ffffffff81092997>] hrtimer_interrupt+0xf7/0x240 [217505.604464] [<ffffffff81042db7>] local_apic_timer_interrupt+0x37/0x60 [217505.604465] [<ffffffff81697e3f>] smp_apic_timer_interrupt+0x3f/0x60 [217505.604467] [<ffffffff816967dd>] apic_timer_interrupt+0x6d/0x80 [217505.604467] <EOI> [217505.604468] [<ffffffff815317e2>] ? cpuidle_enter_state+0x52/0xc0 [217505.604470] [<ffffffff81531909>] cpuidle_idle_call+0xb9/0x1f0 [217505.604471] [<ffffffff8101c8fe>] arch_cpu_idle+0xe/0x30 [217505.604472] [<ffffffff810c05c5>] cpu_startup_entry+0xc5/0x290 [217505.604474] [<ffffffff81040e4b>] start_secondary+0x22b/0x2e0 [217505.604481] Code: 85 d8 00 00 00 48 85 f6 0f 85 35 01 00 00 85 c9 0f 85 0e 01 00 00 4c 3b 5d a8 0f 8e e5 00 00 00 48 8b 45 c8 44 89 f7 48 8b 40 08 <8b> 59 18 e8 91 9d ff ff 85 c0 0f 85 70 fc ff ff 41 39 de 74 12 [217505.604482] RIP [<ffffffff810a3577>] select_task_rq_fair+0x557/0x6f0 [217505.604482] RSP <ffff88081ea83dc8> [217505.604483] CR2: 0000000000000018 [217505.604483] ---[ end trace 16fcffdc1f2af254 ]--- [217505.604485] BUG: unable to handle kernel NULL pointer dereference at 0000000000000018 [217505.604487] IP: [<ffffffff810a3577>] select_task_rq_fair+0x557/0x6f0 [217505.604488] PGD 0 [217505.604489] Oops: 0000 [#4] SMP [217505.604500] Modules linked in: vfat fat usb_storage fuse bnep bluetooth cfg80211 rfkill snd_hda_codec_realtek snd_hda_codec_hdmi snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device iTCO_wdt iTCO_vendor_support dcdbas snd_pcm snd_page_alloc e1000e snd_timer x86_pkg_temp_thermal serio_raw snd coretemp mei_me ptp soundcore mei i2c_i801 shpchp kvm_intel pps_core lpc_ich mfd_core kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel microcode bcache i915 i2c_algo_bit drm_kms_helper drm i2c_core video [217505.604501] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G D 3.13.4-200.fc20.x86_64 #1 [217505.604502] Hardware name: Dell Inc. OptiPlex 9020/0PC5F7, BIOS A02 08/15/2013 [217505.604502] task: ffff8807f2d98000 ti: ffff8807f2d94000 task.ti: ffff8807f2d94000 [217505.604504] RIP: 0010:[<ffffffff810a3577>] [<ffffffff810a3577>] select_task_rq_fair+0x557/0x6f0 [217505.604505] RSP: 0018:ffff88081eb43dc8 EFLAGS: 00010046 [217505.604506] RAX: ffff8807f1022000 RBX: ffff8807fe00be00 RCX: 0000000000000000 [217505.604506] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000005 [217505.604507] RBP: ffff88081eb43e60 R08: 0000000000000010 R09: 00000000ffffffff [217505.604507] R10: 13560a0693057cf8 R11: 0000000000024bde R12: 0000000000000005 [217505.604508] R13: 0000000000014580 R14: 0000000000000005 R15: 0000000000000000 [217505.604508] FS: 0000000000000000(0000) GS:ffff88081eb40000(0000) knlGS:0000000000000000 [217505.604509] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [217505.604510] CR2: 0000000000000018 CR3: 0000000001c0c000 CR4: 00000000001407e0 [217505.604510] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [217505.604511] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [217505.604511] Stack: [217505.604512] 0000000000014580 0000000000014580 0000000000014580 0000000100000000 [217505.604514] ffff880800000010 ffff8807f298a6c0 ffff8807f2a54e00 0000000000000000 [217505.604515] 0000000500000005 ffff88081eb54580 0000000500000010 0000000000000000 [217505.604515] Call Trace: [217505.604517] <IRQ>
Crashing continues with 3.13.5-200.fc20.x86_64 Any ideas how to narrow this down? Fedora release 20 (Heisenbug) Kernel 3.13.5-200.fc20.x86_64 on an x86_64 (ttyS0) jesse login: [108717.943899] general protection fault: 0000 [#1] SMP [108717.948966] Modules linked in: fuse bnep bluetooth cfg80211 rfkill snd_hda_codec_realtek snd_hda_codec_hdmi snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device iTCO_wdt snd_pcm e1000e x86_pkg_temp_thermal iTCO_vendor_support ptp shpchp pps_core coretemp kvm_intel snd_page_alloc kvm snd_timer snd crct10dif_pclmul crc32_pclmul crc32c_intel dcdbas ghash_clmulni_intel soundcore microcode serio_raw i2c_i801 mei_me mei lpc_ich mfd_core bcache i915 i2c_algo_bit drm_kms_helper drm i2c_core video [108717.993553] CPU: 7 PID: 0 Comm: swapper/7 Not tainted 3.13.5-200.fc20.x86_64 #1 [108718.000945] Hardware name: Dell Inc. OptiPlex 9020/0PC5F7, BIOS A02 08/15/2013 [108718.008256] task: ffff8807f2d99140 ti: ffff8807f2da0000 task.ti: ffff8807f2da0000 [108718.015823] RIP: 0010:[<ffffffff81583c87>] [<ffffffff81583c87>] __netif_receive_skb_core+0x2f7/0x820 [108718.025141] RSP: 0018:ffff88081ebc3cf0 EFLAGS: 00010216 [108718.030536] RAX: ffbfffff81d08e68 RBX: ffff8807c5960d00 RCX: 00000000ffffffef [108718.037758] RDX: 0000000000000000 RSI: ffff8807eb754000 RDI: ffff8807c5960d00 [108718.044979] RBP: ffff88081ebc3d38 R08: 0000000000000008 R09: 0000000000000000 [108718.052199] R10: 0000000000000032 R11: ffff8807e4dfe290 R12: ffbfffff81d08e40 [108718.059418] R13: ffffffff81d06520 R14: ffff8807eb754000 R15: ffff8807f07a7500 [108718.066639] FS: 0000000000000000(0000) GS:ffff88081ebc0000(0000) knlGS:0000000000000000 [108718.074817] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [108718.080647] CR2: 00007f8b7adada80 CR3: 0000000001c0c000 CR4: 00000000001407e0 [108718.087867] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [108718.095088] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [108718.102308] Stack: [108718.104404] ffff88081ebc3d00 ffffffff8101ac00 ffff8807c5960d00 ffbfffff81d08e68 [108718.111934] ffff8807c5960d00 ffff8807eb754c10 0000000000000000 ffff8807eb754000 [108718.119481] 0000000000000000 ffff88081ebc3d58 ffffffff815841c8 ffff8807c5960d00 [108718.127013] Call Trace: [108718.129546] <IRQ> [108718.131556] [<ffffffff8101ac00>] ? flush_ptrace_hw_breakpoint+0x20/0x60 [108718.138541] [<ffffffff815841c8>] __netif_receive_skb+0x18/0x60 [108718.144550] [<ffffffff81584243>] netif_receive_skb+0x33/0xa0 [108718.150383] [<ffffffff81584c00>] napi_gro_receive+0x80/0xb0 [108718.156132] [<ffffffffa03456df>] e1000_receive_skb+0x7f/0xe0 [e1000e] [108718.162752] [<ffffffffa0346c7a>] e1000_clean_rx_irq+0x24a/0x400 [e1000e] [108718.169629] [<ffffffffa034e79d>] e1000e_poll+0x6d/0x310 [e1000e] [108718.175809] [<ffffffff815845a9>] net_rx_action+0x149/0x240 [108718.181469] [<ffffffff8107257c>] __do_softirq+0xec/0x280 [108718.186955] [<ffffffff81072a1d>] irq_exit+0xbd/0xd0 [108718.192008] [<ffffffff816981d6>] do_IRQ+0x56/0xc0 [108718.196885] [<ffffffff8168dead>] common_interrupt+0x6d/0x6d [108718.202630] <EOI> [108718.204639] [<ffffffff815317d2>] ? cpuidle_enter_state+0x52/0xc0 [108718.211019] [<ffffffff815318f9>] cpuidle_idle_call+0xb9/0x1f0 [108718.216940] [<ffffffff8101c8fe>] arch_cpu_idle+0xe/0x30 [108718.222341] [<ffffffff810c05e5>] cpu_startup_entry+0xc5/0x290 [108718.228261] [<ffffffff81040e1b>] start_secondary+0x22b/0x2e0 [108718.234094] Code: 83 ec 28 eb 18 0f 1f 80 00 00 00 00 49 8b 44 24 28 48 89 45 d0 4c 8b 65 d0 49 83 ec 28 49 8d 44 24 28 49 39 c5 0f 84 21 03 00 00 <66> 45 39 04 24 75 da 49 8b 44 24 08 49 39 c1 74 0b 48 3b 43 20 [108718.253786] RIP [<ffffffff81583c87>] __netif_receive_skb_core+0x2f7/0x820 [108718.260760] RSP <ffff88081ebc3cf0> [108718.264336] general protection fault: 0000 [#2] [108718.264344] ---[ end trace 0fe06d367bd9d127 ]--- [108718.264344] Kernel panic - not syncing: Fatal exception in interrupt SMP [108718.280557] Modules linked in: fuse bnep bluetooth cfg80211 rfkill snd_hda_codec_realtek snd_hda_codec_hdmi snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device iTCO_wdt snd_pcm e1000e x86_pkg_temp_thermal iTCO_vendor_support ptp shpchp pps_core coretemp kvm_intel snd_page_alloc kvm snd_timer snd crct10dif_pclmul crc32_pclmul crc32c_intel dcdbas ghash_clmulni_intel soundcore microcode serio_raw i2c_i801 mei_me mei lpc_ich mfd_core bcache i915 i2c_algo_bit drm_kms_helper drm i2c_core video [108718.325159] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G D 3.13.5-200.fc20.x86_64 #1 [108718.333511] Hardware name: Dell Inc. OptiPlex 9020/0PC5F7, BIOS A02 08/15/2013 [108718.340819] task: ffff8807f2cfdee0 ti: ffff8807f2d8e000 task.ti: ffff8807f2d8e000 [108718.348391] RIP: 0010:[<ffffffff81583c87>] [<ffffffff81583c87>] __netif_receive_skb_core+0x2f7/0x820 [108718.357714] RSP: 0018:ffff88081ea83e18 EFLAGS: 00010216 [108718.364644] RAX: ffbfffff81d08e68 RBX: ffff88077f2b2400 RCX: dead000000200200 [108718.373410] RDX: 0000000000000000 RSI: ffff8807f24ed000 RDI: ffff88077f2b2400 [108718.382166] RBP: ffff88081ea83e60 R08: 0000000000000008 R09: 0000000000000000 [108718.390933] R10: 000000000000ffff R11: ffff8807f24ed000 R12: ffbfffff81d08e40 [108718.399694] R13: ffffffff81d06520 R14: ffff8807f24ed000 R15: 0000000000000000 [108718.408418] FS: 0000000000000000(0000) GS:ffff88081ea80000(0000) knlGS:0000000000000000 [108718.418071] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [108718.425350] CR2: 00007fc6d80c25a0 CR3: 0000000001c0c000 CR4: 00000000001407e0 [108718.433987] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [108718.442613] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [108718.451216] Stack: [108718.454692] 0000000000000286 ffff8807efc1ca00 ffff88077f2b2400 ffbfffff81d08e68 [108718.463590] ffff88081ea950e8 0000000000000001 ffff88081ea95194 0000000000000000 [108718.472488] ffff88081ea95198 ffff88081ea83e80 ffffffff815841c8 ffff88081ea950e8 [108718.481381] Call Trace: [108718.485282] <IRQ> [108718.487294] [<ffffffff815841c8>] __netif_receive_skb+0x18/0x60 [108718.496205] [<ffffffff81584cde>] process_backlog+0xae/0x180 [108718.503304] [<ffffffff815845a9>] net_rx_action+0x149/0x240 [108718.510303] [<ffffffff8107257c>] __do_softirq+0xec/0x280 [108718.517117] [<ffffffff81072a1d>] irq_exit+0xbd/0xd0 [108718.523485] [<ffffffff81698285>] smp_apic_timer_interrupt+0x45/0x60 [108718.531248] [<ffffffff81696c1d>] apic_timer_interrupt+0x6d/0x80 [108718.538669] <EOI> [108718.540679] [<ffffffff815317d2>] ? cpuidle_enter_state+0x52/0xc0 [108718.549707] [<ffffffff815318f9>] cpuidle_idle_call+0xb9/0x1f0 [108718.556940] [<ffffffff8101c8fe>] arch_cpu_idle+0xe/0x30 [108718.563649] [<ffffffff810c05e5>] cpu_startup_entry+0xc5/0x290 [108718.570865] [<ffffffff81040e1b>] start_secondary+0x22b/0x2e0 [108718.578004] Code: 83 ec 28 eb 18 0f 1f 80 00 00 00 00 49 8b 44 24 28 48 89 45 d0 4c 8b 65 d0 49 83 ec 28 49 8d 44 24 28 49 39 c5 0f 84 21 03 00 00 <66> 45 39 04 24 75 da 49 8b 44 24 08 49 39 c1 74 0b 48 3b 43 20 [108718.600427] RIP [<ffffffff81583c87>] __netif_receive_skb_core+0x2f7/0x820 [108718.608776] RSP <ffff88081ea83e18> [108718.613734] general protection fault: 0000 [#3] SMP [108718.620114] Modules linked in: fuse bnep bluetooth cfg80211 rfkill snd_hda_codec_realtek snd_hda_codec_hdmi snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device iTCO_wdt snd_pcm e1000e x86_pkg_temp_thermal iTCO_vendor_support ptp shpchp pps_core coretemp kvm_intel snd_page_alloc kvm snd_timer snd crct10dif_pclmul crc32_pclmul crc32c_intel dcdbas ghash_clmulni_intel soundcore microcode serio_raw i2c_i801 mei_me mei lpc_ich mfd_core bcache i915 i2c_algo_bit drm_kms_helper drm i2c_core video [108718.670262] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G D 3.13.5-200.fc20.x86_64 #1 [108718.680065] Hardware name: Dell Inc. OptiPlex 9020/0PC5F7, BIOS A02 08/15/2013 [108718.688841] task: ffff8807f2cfe780 ti: ffff8807f2d90000 task.ti: ffff8807f2d90000 [108718.697887] RIP: 0010:[<ffffffff81583c87>] [<ffffffff81583c87>] __netif_receive_skb_core+0x2f7/0x820 [108718.708697] RSP: 0018:ffff88081eac3e18 EFLAGS: 00010216 [108718.715598] RAX: ffbfffff81d08e68 RBX: ffff8806ff43e000 RCX: dead000000200200 [108718.724328] RDX: 0000000000000000 RSI: ffff8807f24ed000 RDI: ffff8806ff43e000 [108718.733061] RBP: ffff88081eac3e60 R08: 0000000000000008 R09: 0000000000000000 [108718.741799] R10: 000000000000ffff R11: ffff8807f24ed000 R12: ffbfffff81d08e40 [108718.750541] R13: ffffffff81d06520 R14: ffff8807f24ed000 R15: 0000000000000000 [108718.759257] FS: 0000000000000000(0000) GS:ffff88081eac0000(0000) knlGS:0000000000000000 [108718.768909] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [108718.776211] CR2: 00007f8adc053bd0 CR3: 0000000001c0c000 CR4: 00000000001407e0 [108718.784904] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [108718.793578] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [108718.802233] Stack: [108718.805765] ffff8807c5802e80 ffff88081eac3e00 ffff8806ff43e000 ffbfffff81d08e68 [108718.814701] ffff88081ead50e8 0000000000000001 ffff88081ead5194 0000000000000000 [108718.823605] ffff88081ead5198 ffff88081eac3e80 ffffffff815841c8 ffff88081ead50e8 [108718.832479] Call Trace: [108718.836330] <IRQ> [108718.838340] [<ffffffff815841c8>] __netif_receive_skb+0x18/0x60 [108718.847141] [<ffffffff81584cde>] process_backlog+0xae/0x180 [108718.854166] [<ffffffff815845a9>] net_rx_action+0x149/0x240 [108718.861089] [<ffffffff8107257c>] __do_softirq+0xec/0x280 [108718.867822] [<ffffffff81072a1d>] irq_exit+0xbd/0xd0 [108718.874113] [<ffffffff81698285>] smp_apic_timer_interrupt+0x45/0x60 [108718.881793] [<ffffffff81696c1d>] apic_timer_interrupt+0x6d/0x80 [108718.889116] <EOI> [108718.891127] [<ffffffff815317d2>] ? cpuidle_enter_state+0x52/0xc0 [108718.899971] [<ffffffff815318f9>] cpuidle_idle_call+0xb9/0x1f0 [108718.907117] [<ffffffff8101c8fe>] arch_cpu_idle+0xe/0x30 [108718.913732] [<ffffffff810c05e5>] cpu_startup_entry+0xc5/0x290 [108718.920859] [<ffffffff81040e1b>] start_secondary+0x22b/0x2e0 [108718.927899] Code: 83 ec 28 eb 18 0f 1f 80 00 00 00 00 49 8b 44 24 28 48 89 45 d0 4c 8b 65 d0 49 83 ec 28 49 8d 44 24 28 49 39 c5 0f 84 21 03 00 00 <66> 45 39 04 24 75 da 49 8b 44 24 08 49 39 c1 74 0b 48 3b 43 20 [108718.950133] RIP [<ffffffff81583c87>] __netif_receive_skb_core+0x2f7/0x820 [108718.958372] RSP <ffff88081eac3e18> [108718.963216] general protection fault: 0000 [#4] SMP [108718.969048] Modules linked in: fuse bnep bluetooth cfg80211 rfkill snd_hda_codec_realtek snd_hda_codec_hdmi snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device iTCO_wdt snd_pcm e1000e x86_pkg_temp_thermal iTCO_vendor_support ptp shpchp pps_core coretemp kvm_intel snd_page_alloc kvm snd_timer snd crct10dif_pclmul crc32_pclmul crc32c_intel dcdbas ghash_clmulni_intel soundcore microcode serio_raw i2c_i801 mei_me mei lpc_ich mfd_core bcache i915 i2c_algo_bit drm_kms_helper drm i2c_core video [108719.016806] CPU: 6 PID: 0 Comm: swapper/6 Tainted: G D 3.13.5-200.fc20.x86_64 #1 [108719.025996] Hardware name: Dell Inc. OptiPlex 9020/0PC5F7, BIOS A02 08/15/2013 [108719.034151] task: ffff8807f2d988a0 ti: ffff8807f2d96000 task.ti: ffff8807f2d96000 [108719.042576] RIP: 0010:[<ffffffff81583c87>] [<ffffffff81583c87>] __netif_receive_skb_core+0x2f7/0x820 [108719.052754] RSP: 0018:ffff88081eb83e18 EFLAGS: 00010216 [108719.059025] RAX: ffbfffff81d08e68 RBX: ffff88077f08f400 RCX: dead000000200200 [108719.067011] RDX: 0000000000000000 RSI: ffff8807f24ed000 RDI: ffff88077f08f400 [108719.075105] RBP: ffff88081eb83e60 R08: 0000000000000008 R09: 0000000000000000 [108719.083200] R10: 000000000000ffff R11: ffff8807f24ed000 R12: ffbfffff81d08e40 [108719.091299] R13: ffffffff81d06520 R14: ffff8807f24ed000 R15: 0000000000000000 [108719.099401] FS: 0000000000000000(0000) GS:ffff88081eb80000(0000) knlGS:0000000000000000 [108719.108462] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [108719.115163] CR2: 00007ffa9bf61ee4 CR3: 0000000001c0c000 CR4: 00000000001407e0 [108719.123243] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [108719.131315] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [108719.139382] Stack: [108719.142334] ffff8807f1521f00 ffff88081eb83e00 ffff88077f08f400 ffbfffff81d08e68 [108719.150697] ffff88081eb950e8 0000000000000001 ffff88081eb95194 0000000000000000 [108719.159043] ffff88081eb95198 ffff88081eb83e80 ffffffff815841c8 ffff88081eb950e8 [108719.167364] Call Trace: [108719.170686] <IRQ> [108719.172693] [<ffffffff815841c8>] __netif_receive_skb+0x18/0x60 [108719.180421] [<ffffffff81584cde>] process_backlog+0xae/0x180 [108719.186931] [<ffffffff815845a9>] net_rx_action+0x149/0x240 [108719.193340] [<ffffffff8107257c>] __do_softirq+0xec/0x280 [108719.199565] [<ffffffff81072a1d>] irq_exit+0xbd/0xd0 [108719.205355] [<ffffffff81698285>] smp_apic_timer_interrupt+0x45/0x60 [108719.212522] [<ffffffff81696c1d>] apic_timer_interrupt+0x6d/0x80 [108719.219347] <EOI> [108719.221354] [<ffffffff815317d2>] ? cpuidle_enter_state+0x52/0xc0 [108719.229208] [<ffffffff815318f9>] cpuidle_idle_call+0xb9/0x1f0 [108719.235857] [<ffffffff8101c8fe>] arch_cpu_idle+0xe/0x30 [108719.241980] [<ffffffff810c05e5>] cpu_startup_entry+0xc5/0x290 [108719.248616] [<ffffffff81040e1b>] start_secondary+0x22b/0x2e0 [108719.255166] Code: 83 ec 28 eb 18 0f 1f 80 00 00 00 00 49 8b 44 24 28 48 89 45 d0 4c 8b 65 d0 49 83 ec 28 49 8d 44 24 28 49 39 c5 0f 84 21 03 00 00 <66> 45 39 04 24 75 da 49 8b 44 24 08 49 39 c1 74 0b 48 3b 43 20 [108719.276299] RIP [<ffffffff81583c87>] __netif_receive_skb_core+0x2f7/0x820 [108719.284022] RSP <ffff88081eb83e18> [108719.288355] general protection fault: 0000 [#5] SMP [108719.294187] Modules linked in: fu\ufffd
Same issue here on Kernel 3.13.6-200.fc20.i686 Machine hangs within a couple of hours. Had the issue on FC19, upgraded to FC20 but issue remains.
Since my crash in comment#17 I upgraded Dell OptiPlex 9020 BIOS to just released version A05 (original A02). I now have stable uptime of two full weeks still running 3.13.5-200.fc20.x86_64 kernel. Seems that system firmware was bad... Will now upgrade kernel to the latest version and see if system stays stable.
System now has stable uptime of 7 days also with 3.13.6-200.fc20.x86_64. I believe root of the issue was bad firmware.