Created attachment 1070710 [details] Kernel warning and panic received via syslog Description of problem: We run a diskless (netbooted via PXE to ramdisk) Fedora 22 test server with accel-ppp NAS software, accepting inbound PPTP connections. Last night we experienced a short RADIUS outage (unrelated to this bug). During this outage the server in question sent (via remote syslog) the attached WARNING, followed by a PANIC. Version-Release number of selected component (if applicable): Standard Fedora kernel 4.1.5-200.fc22.x86_64 How reproducible: Can't reproduce yet, will restart server with newer kernel (4.1.6) and see what happens. Steps to Reproduce: 1. Run accel-ppp with enabled bidirectional traffic shaping; without any out-of-tree kernel modules loaded. 2. Disable RADIUS server, which is used for authentication by accel-ppp (my theory is that it triggers some abnormal behaviour in accel-ppp, but I still can't explain a kernel crash). 3. Observe kernel WARNING, shortly followed by PANIC. Actual results: Dead panicked OS. Expected results: At most - crashed accel-ppp process. Additional info:
There was a memory corruption issue in 4.1.5. Please try this with 4.1.6 and let us know how it goes.
Two (updated to 4.1.6) machines crashed today. Symptoms are similar, but traces differ a bit. Traces are attached.
Created attachment 1075750 [details] Bug/panic from "vpn22" machine
Created attachment 1075751 [details] Bug/panic from "vpn25" machine
One more machine crashed yesterday. Couldn't manage to capture traces this time. In the meantime I've built a new image with 4.1.7 kernel to test tomorrow, but now I'm not so sure if it will help at all.
Not sure if related, but i've just got the following warning via netconsole on the last remaining machine running 4.1.6 kernel (attached).
Created attachment 1077624 [details] Kernel warning when using 4.1.6
Experienced new crash on 4.1.7 fedora kernel. Crash log is attached. Nothing seems to help.
Created attachment 1077775 [details] Crash log when running 4.1.7 kernel
Ok, I assembled a test rig using multiple parallel PPTP clients and packet generators. The kernel i'm using is 4.1.10 with softlockup patch on top (Fedora kernel compiled from source from f22 branch using "fedpkg local"). I'm now able to reliably panic the PPTP server machine. The process is as follows: 1. Start torture test 2. Wait until number of PPTP sessions reach 600 mark, with traffic above 150Mbps. 3. Shut down all PPTP sessions at once. This typically results in panic, but one time I got a locked netdev tx queue instead (NETDEV_WATCHDOG produced a warning for e1000e driver's tx queue). Whether I am able to capture kernel traces via remote syslog is a matter of chance, AFAICT. Typical syslog trace is pasted below: 2015-10-06T14:04:38+03:00 vpn27 kernel: general protection fault: 0000 [#1] SMP 2015-10-06T14:04:38+03:00 vpn27 kernel: Modules linked in: act_mirred act_skbedit cls_u32 sch_ingress cls_flow sch_htb pptp gre pppoe pppox ppp_generic slhc intel_rapl iosf_mbi x86_pkg_temp_thermal coretemp kvm iTCO_wdt iTCO_vendor_support joydev ppdev parport_pc ipmi_devintf ipmi_ssif parport ipmi_si shpchp ipmi_msghandler ie31200_edac edac_core lpc_ich winbond_cir rc_core mfd_core i2c_i801 tpm_tis tpm l2tp_ip6 l2tp_core ip6_udp_tunnel udp_tunnel ifb bridge bonding 8021q garp mrp stp llc isofs squashfs mgag200 i2c_algo_bit drm_kms_helper ttm crct10dif_pclmul crc32_pclmul crc32c_intel drm ghash_clmulni_intel e1000e ptp pps_core video loop 2015-10-06T14:04:38+03:00 vpn27 kernel: CPU: 3 PID: 1278 Comm: accel-pppd Tainted: G W 4.1.10-200.fc22.x86_64 #1 2015-10-06T14:04:38+03:00 vpn27 kernel: Hardware name: Intel Corporation S1200BTL/S1200BTL, BIOS S1200BT.86B.02.00.0035.030220120927 03/02/2012 2015-10-06T14:04:38+03:00 vpn27 kernel: task: ffff880129c03b40 ti: ffff880072164000 task.ti: ffff880072164000 2015-10-06T14:04:38+03:00 vpn27 kernel: RIP: 0010:[<ffffffff817a07b9>] [<ffffffff817a07b9>] _raw_spin_lock_bh+0x19/0x60 2015-10-06T14:04:38+03:00 vpn27 kernel: RSP: 0018:ffff880072167cb8 EFLAGS: 00010286 2015-10-06T14:04:38+03:00 vpn27 kernel: RAX: 0000000000010000 RBX: dead000000100060 RCX: 0000000000000006 2015-10-06T14:04:38+03:00 vpn27 kernel: RDX: 0000000000001fb9 RSI: 0000000000000200 RDI: dead0000001000d0 2015-10-06T14:04:38+03:00 vpn27 kernel: RBP: ffff880072167cb8 R08: 0000000000000000 R09: ffffffffa0337070 2015-10-06T14:04:38+03:00 vpn27 kernel: R10: ffff88008e1a7800 R11: ffff880126a70300 R12: dead0000001000d0 2015-10-06T14:04:38+03:00 vpn27 kernel: R13: ffff880090be8000 R14: ffff880072167d78 R15: 0000000000000000 2015-10-06T14:04:38+03:00 vpn27 kernel: FS: 00007fa5df7f9700(0000) GS:ffff8801398c0000(0000) knlGS:0000000000000000 2015-10-06T14:04:38+03:00 vpn27 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 2015-10-06T14:04:38+03:00 vpn27 kernel: CR2: 00007fcb1f5a1020 CR3: 0000000093f62000 CR4: 00000000000406e0 2015-10-06T14:04:38+03:00 vpn27 kernel: Stack: 2015-10-06T14:04:38+03:00 vpn27 kernel: ffff880072167ce8 ffffffffa033506a ffffffffa015b060 00000000ffffffea 2015-10-06T14:04:38+03:00 vpn27 kernel: ffffffff81cec760 0000000000000006 ffff880072167d28 ffffffff810c1a7f 2015-10-06T14:04:38+03:00 vpn27 kernel: ffff880072167d48 ffff880072167d78 0000000000000006 ffff880090be8000 2015-10-06T14:04:38+03:00 vpn27 kernel: Call Trace: 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffffa033506a>] mirred_device_event+0x6a/0xa0 [act_mirred] 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffff810c1a7f>] notifier_call_chain+0x4f/0x80 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffff810c1c16>] raw_notifier_call_chain+0x16/0x20 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffff816796d9>] call_netdevice_notifiers_info+0x39/0x70 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffff8167b71a>] rollback_registered_many+0x21a/0x340 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffff8167b880>] rollback_registered+0x40/0x70 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffff8167cb50>] unregister_netdevice_queue+0x50/0xb0 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffff8167cbd0>] unregister_netdev+0x20/0x30 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffffa0259626>] ppp_shutdown_interface+0xe6/0x100 [ppp_generic] 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffffa02596ae>] ppp_release+0x6e/0x90 [ppp_generic] 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffff8122cdac>] __fput+0xdc/0x1f0 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffff8122cf0e>] ____fput+0xe/0x10 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffff810bf027>] task_work_run+0xa7/0xe0 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffff81014cdd>] do_notify_resume+0x9d/0xa0 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffff817a0f3c>] int_signal+0x12/0x17 2015-10-06T14:04:38+03:00 vpn27 kernel: Code: 5d c3 b8 01 00 00 00 5d c3 66 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 65 81 05 a7 b0 86 7e 00 02 00 00 48 89 e5 b8 00 00 01 00 <f0> 0f c1 07 89 c2 c1 ea 10 66 39 c2 75 09 5d c3 0f 1f 80 00 00 2015-10-06T14:04:38+03:00 vpn27 kernel: RIP [<ffffffff817a07b9>] _raw_spin_lock_bh+0x19/0x60 2015-10-06T14:04:38+03:00 vpn27 kernel: RSP <ffff880072167cb8> 2015-10-06T14:04:38+03:00 vpn27 kernel: ------------[ cut here ]------------ 2015-10-06T14:04:38+03:00 vpn27 kernel: kernel BUG at mm/vmalloc.c:1335! 2015-10-06T14:04:38+03:00 vpn27 kernel: invalid opcode: 0000 [#2] SMP 2015-10-06T14:04:38+03:00 vpn27 kernel: Modules linked in: act_mirred act_skbedit cls_u32 sch_ingress cls_flow sch_htb pptp gre pppoe pppox ppp_generic slhc intel_rapl iosf_mbi x86_pkg_temp_thermal coretemp kvm iTCO_wdt iTCO_vendor_support joydev ppdev parport_pc ipmi_devintf ipmi_ssif parport ipmi_si shpchp ipmi_msghandler ie31200_edac edac_core lpc_ich winbond_cir rc_core mfd_core i2c_i801 tpm_tis tpm l2tp_ip6 l2tp_core ip6_udp_tunnel udp_tunnel ifb bridge bonding 8021q garp mrp stp llc isofs squashfs mgag200 i2c_algo_bit drm_kms_helper ttm crct10dif_pclmul crc32_pclmul crc32c_intel drm ghash_clmulni_intel e1000e ptp pps_core video loop 2015-10-06T14:04:38+03:00 vpn27 kernel: CPU: 3 PID: 1278 Comm: accel-pppd Tainted: G W 4.1.10-200.fc22.x86_64 #1 2015-10-06T14:04:38+03:00 vpn27 kernel: Hardware name: Intel Corporation S1200BTL/S1200BTL, BIOS S1200BT.86B.02.00.0035.030220120927 03/02/2012 2015-10-06T14:04:38+03:00 vpn27 kernel: task: ffff880129c03b40 ti: ffff880072164000 task.ti: ffff880072164000 2015-10-06T14:04:38+03:00 vpn27 kernel: RIP: 0010:[<ffffffff811ed318>] [<ffffffff811ed318>] __get_vm_area_node+0x168/0x170 2015-10-06T14:04:38+03:00 vpn27 kernel: RSP: 0018:ffff880072167118 EFLAGS: 00010206 2015-10-06T14:04:38+03:00 vpn27 kernel: RAX: 0000000080000200 RBX: 00000000ffffffff RCX: ffffc90000000000 2015-10-06T14:04:38+03:00 vpn27 kernel: RDX: 0000000000000022 RSI: 0000000000000001 RDI: 0000000000002000 2015-10-06T14:04:38+03:00 vpn27 kernel: RBP: ffff880072167158 R08: ffffe8ffffffffff R09: 00000000ffffffff 2015-10-06T14:04:38+03:00 vpn27 kernel: R10: ffff88008a43d780 R11: 0000000000240000 R12: 0000000000001800 2015-10-06T14:04:38+03:00 vpn27 kernel: R13: 00000000000080d2 R14: 0000000000000080 R15: ffffea0001fbe280 2015-10-06T14:04:38+03:00 vpn27 kernel: FS: 00007fa5df7f9700(0000) GS:ffff8801398c0000(0000) knlGS:0000000000000000 2015-10-06T14:04:38+03:00 vpn27 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 2015-10-06T14:04:38+03:00 vpn27 kernel: CR2: 00007fcc599a0000 CR3: 0000000093f62000 CR4: 00000000000406e0 2015-10-06T14:04:38+03:00 vpn27 kernel: Stack: 2015-10-06T14:04:38+03:00 vpn27 kernel: 0000000000000000 ffffc90000000000 0000000000000000 00000000ffffffff 2015-10-06T14:04:38+03:00 vpn27 kernel: 0000000000001800 00000000000080d2 0000000000000080 ffffea0001fbe280 2015-10-06T14:04:38+03:00 vpn27 kernel: ffff8800721671d8 ffffffff811eebf5 00000000000080d2 ffffffffa017a37c 2015-10-06T14:04:38+03:00 vpn27 kernel: Call Trace: 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffff811eebf5>] __vmalloc_node_range+0x75/0x2c0 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffffa017a37c>] ? ttm_tt_init+0x8c/0xb0 [ttm] 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffff811eee8a>] __vmalloc+0x4a/0x50 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffffa017a37c>] ? ttm_tt_init+0x8c/0xb0 [ttm] 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffffa017a37c>] ttm_tt_init+0x8c/0xb0 [ttm] 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffffa008903c>] mgag200_ttm_tt_create+0x5c/0x90 [mgag200] 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffffa017a86d>] ttm_bo_add_ttm+0x9d/0xe0 [ttm] 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffffa017c22e>] ttm_bo_handle_move_mem+0x62e/0x670 [ttm] 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffffa017c8d2>] ? ttm_bo_mem_space+0xf2/0x3b0 [ttm] 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffffa017d051>] ttm_bo_validate+0x201/0x220 [ttm] 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffff8106962c>] ? iounmap.part.5+0x5c/0x90 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffff8106008c>] ? kgdb_correct_hw_break+0x3c/0x120 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffffa0089735>] mgag200_bo_push_sysram+0xa5/0xf0 [mgag200] 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffffa0084b47>] mga_crtc_do_set_base.isra.6.constprop.17+0x87/0x450 [mgag200] 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffffa0085f1a>] mga_crtc_mode_set+0x100a/0x20f0 [mgag200] 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffffa01a19bd>] drm_crtc_helper_set_mode+0x33d/0x590 [drm_kms_helper] 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffffa01a2b30>] drm_crtc_helper_set_config+0x980/0xb10 [drm_kms_helper] 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffffa0105516>] drm_mode_set_config_internal+0x66/0x100 [drm] 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffffa01ae9af>] drm_fb_helper_pan_display+0x9f/0xf0 [drm_kms_helper] 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffff8142a846>] fb_pan_display+0xd6/0x180 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffff81424620>] bit_update_start+0x20/0x50 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffff8142150d>] fbcon_switch+0x3bd/0x670 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffff814a7789>] redraw_screen+0x1a9/0x2a0 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffff81420790>] fbcon_blank+0x120/0x390 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffff810f7b6a>] ? console_unlock+0x20a/0x4d0 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffff810f8740>] ? vprintk_emit+0x370/0x560 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffff8110910d>] ? internal_add_timer+0x8d/0xc0 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffff8110afa6>] ? mod_timer+0x136/0x230 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffff814a8daa>] do_unblank_screen+0xda/0x1d0 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffff814a8eb0>] unblank_screen+0x10/0x20 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffff813bd789>] bust_spinlocks+0x19/0x40 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffff810185f8>] oops_end+0x38/0xe0 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffff81018b5b>] die+0x4b/0x70 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffff810155d0>] do_general_protection+0x120/0x1b0 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffff817a2cf8>] general_protection+0x28/0x30 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffff817a07b9>] ? _raw_spin_lock_bh+0x19/0x60 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffff817a081b>] ? _raw_spin_unlock_bh+0x1b/0x20 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffffa033506a>] mirred_device_event+0x6a/0xa0 [act_mirred] 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffff810c1a7f>] notifier_call_chain+0x4f/0x80 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffff810c1c16>] raw_notifier_call_chain+0x16/0x20 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffff816796d9>] call_netdevice_notifiers_info+0x39/0x70 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffff8167b71a>] rollback_registered_many+0x21a/0x340 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffff8167b880>] rollback_registered+0x40/0x70 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffff8167cb50>] unregister_netdevice_queue+0x50/0xb0 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffff8167cbd0>] unregister_netdev+0x20/0x30 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffffa0259626>] ppp_shutdown_interface+0xe6/0x100 [ppp_generic] 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffffa02596ae>] ppp_release+0x6e/0x90 [ppp_generic] 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffff8122cdac>] __fput+0xdc/0x1f0 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffff8122cf0e>] ____fput+0xe/0x10 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffff810bf027>] task_work_run+0xa7/0xe0 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffff81014cdd>] do_notify_resume+0x9d/0xa0 2015-10-06T14:04:38+03:00 vpn27 kernel: [<ffffffff817a0f3c>] int_signal+0x12/0x17 2015-10-06T14:04:38+03:00 vpn27 kernel: Code: de e9 fe fe ff ff 0f 1f 44 00 00 4c 89 ef e8 90 e4 01 00 48 83 c4 18 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 80 00 00 00 00 <0f> 0b 66 0f 1f 44 00 00 66 66 66 66 90 55 49 89 c8 41 b9 ff ff 2015-10-06T14:04:38+03:00 vpn27 kernel: RIP [<ffffffff811ed318>] __get_vm_area_node+0x168/0x170 2015-10-06T14:04:38+03:00 vpn27 kernel: RSP <ffff880072167118> 2015-10-06T14:04:38+03:00 vpn27 kernel: ---[ end trace 75060e5a4ba56de9 ]--- 2015-10-06T14:04:38+03:00 vpn27 kernel: Kernel panic - not syncing: Fatal exception in interrupt
*********** MASS BUG UPDATE ************** We apologize for the inconvenience. There is a large number of bugs to go through and several of them have gone stale. Due to this, we are doing a mass bug update across all of the Fedora 22 kernel bugs. Fedora 22 has now been rebased to 4.2.3-200.fc22. Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel. If you have moved on to Fedora 23, and are still experiencing this issue, please change the version to Fedora 23. If you experience different issues, please open a new bug report for those.
*********** MASS BUG UPDATE ************** This bug is being closed with INSUFFICIENT_DATA as there has not been a response in over 4 weeks. If you are still experiencing this issue, please reopen and attach the relevant data from the latest kernel you are running and any data that might have been requested previously.