Routing through an epic100 card with 2.6.9-5.0.3.EL kernels seems to kill the system when it gets used hard for rx. This is actually an audit kernel based on 2.6.9-5.0.3.EL, but nothing network-related is changed by the audit patches. The epic100 is also part of a bridge. This setup has run perfectly reliably for a very long time with 2.6.5-1.358. It's i686 SMP. Symptoms include oopsen, spontaneous reboots and lockups. When enabling epic100 debugging, this output seemed particularly interesting, when compared with the code in epic_rx(). /* If we own the next entry, it's a new packet. Send it up. */ while ((ep->rx_ring[entry].rxstatus & cpu_to_le32(DescOwn)) == 0) { int status = le32_to_cpu(ep->rx_ring[entry].rxstatus); if (debug > 4) printk(KERN_DEBUG " epic_rx() status was %8.8x.\n", status); .... eth1: Interrupt, status=0x00cd0800 new intstat=0x00cd0800. eth1: exit interrupt, intr_status=0xcd0000. In epic_rx(), entry 124 05da1781. epic_rx() status was 05da1781. eth1: Interrupt, status=0x00cd0800 new intstat=0x00cd0800. eth1: exit interrupt, intr_status=0xcd0000. In epic_rx(), entry 125 058a1781. epic_rx() status was 058a1781. epic_rx() status was 00008000. (and here it locked up hard. But why did it ever print "status was 0x8000" since 0x8000 is DescOwn. Something else must have been giving the descriptor back to the card at the time this code was running. Can epic_rx() itself be run more than once in parallel?)
Other oopsen... ------------[ cut here ]------------ kernel BUG at kernel/sched.c:920! invalid operand: 0000 [#1] SMP Modules linked in: epic100 eeprom lm75 lm78 i2c_sensor i2c_isa i2c_piix4 i2c_core parport_pc lp parport nfsd exportfs nfs lockd sunrpc autofs4 iptable_filter ip_tables bridge atm md5 ipv6(U) microcode e100 mii dm_mod uhci_hcd ext3 jbd usb_storage sd_mod scsi_mod CPU: 1 EIP: 0060:[<c011b57d>] Not tainted VLI EFLAGS: 00010002 (2.6.9-5.0.3.EL.audit.28smp) EIP is at resched_task+0x20/0x68 eax: 00000001 ebx: eff17000 ecx: c1613d60 edx: 00000000 esi: eff410b0 edi: 00000000 ebp: c03bbf10 esp: c03bbf08 ds: 007b es: 007b ss: 0068 Process swapper (pid: 0, threadinfo=c03bb000 task=eff410b0) Stack: 00000001 c1613960 c03bbf40 c011cb7f 00000001 c01072e6 00000001 c03bbf58 c1613d60 00000001 00000000 c03ac188 00000001 c03ac184 fffffffc c0116312 ecd10780 e787ff0e eff25f0e c02c7f86 ecd10780 3ffeae40 eef60240 e787ff0e Call Trace: [<c011cb7f>] scheduler_tick+0x13e/0x3e5 [<c01072e6>] handle_IRQ_event+0x25/0x4f [<c0116312>] smp_apic_timer_interrupt+0xac/0xdd [<c02c7f86>] apic_timer_interrupt+0x1a/0x20 [<f0929590>] epic_rx+0x18f/0x345 [epic100] [<f09297ac>] epic_poll+0x3e/0x10d [epic100] [<c02749e1>] net_rx_action+0x61/0xd8 [<c0124aa4>] __do_softirq+0x4c/0xb1 [<c0107f39>] do_softirq+0x4f/0x56 ======================= Code: ff c7 43 28 00 00 00 00 5b 5d c3 55 89 e5 56 89 c6 b8 00 ce 3a c0 53 8b 5e 04 8b 53 10 03 04 95 20 71 3b c0 0f b6 00 84 c0 7e 08 <0f> 0b 98 03 57 85 2d c0 8b 43 08 c1 e8 10 83 e0 01 f0 0f ba 6b <1>Unable to handle kernel paging request at virtual address 30303038 printing eip: c0143a5c *pde = 2a0c2001 Oops: 0000 [#2] SMP Modules linked in: epic100 eeprom lm75 lm78 i2c_sensor i2c_isa i2c_piix4 i2c_core parport_pc lp parport nfsd exportfs nfs lockd sunrpc autofs4 iptable_filter ip_tables bridge atm md5 ipv6(U) microcode e100 mii dm_mod uhci_hcd ext3 jbd usb_storage sd_mod scsi_mod CPU: 0 EIP: 0060:[<c0143a5c>] Not tainted VLI EFLAGS: 00010086 (2.6.9-5.0.3.EL.audit.28smp) EIP is at kmem_cache_alloc+0x32/0x57 eax: 00000000 ebx: 00000220 ecx: ef85b280 edx: 30303038 esi: 00000246 edi: eff038a0 ebp: effc14c0 esp: c03baf44 ds: 007b es: 007b ss: 0068 Process swapper (pid: 0, threadinfo=c03ba000 task=c0314a60) Stack: ecea9980 ecea9980 ecea9980 c026f460 ecea9980 ef85b280 c02c2d70 ee708800 c033f298 ef85b280 00000008 ecca6eb4 c0274822 ecea9980 0000009a ecea9980 ee708a40 f08e3508 80000000 ec8ebdc0 0000a020 00000010 c03bafbc ee708900 Call Trace: [<c026f460>] skb_clone+0xe/0x192 [<c02c2d70>] packet_rcv_spkt+0x2e/0x1d6 [<c0274822>] netif_receive_skb+0x1be/0x21f [<f08e3508>] e100_rx_clean+0x185/0x2fe [e100] [<f08e3179>] e100_poll+0x33/0x23d [e100] [<c02749e1>] net_rx_action+0x61/0xd8 [<c0124aa4>] __do_softirq+0x4c/0xb1 [<c0107f39>] do_softirq+0x4f/0x56 ======================= [<c010784f>] do_IRQ+0x125/0x130 [<c02c7f04>] common_interrupt+0x18/0x20 [<c0104018>] default_idle+0x0/0x2c [<c0104041>] default_idle+0x29/0x2c [<c010409d>] cpu_idle+0x26/0x3b [<c0384784>] start_kernel+0x194/0x198 Code: 53 89 d3 74 16 31 c9 ba 0f 08 00 00 b8 39 ab 2d c0 e8 0a ab fd ff e8 19 21 18 00 9c 5e fa b8 00 f0 ff ff 21 e0 8b 40 10 8b 14 87 <8b> 02 85 c0 74 10 c7 42 0c 01 00 00 00 48 89 02 8b 44 82 10 eb <0>Kernel panic - not syncing: Fatal exception in interrupt
Unable to handle kernel paging request at virtual address f0000000 printing eip: f08a7590 *pde = 00000000 Oops: 0002 [#1] SMP Modules linked in: eeprom lm75 lm78 i2c_sensor i2c_isa i2c_piix4 i2c_core parport_pc lp parport nfsd exportfs nfs lockd sunrpc autofs4 iptable_filter ip_tables bridge atm md5 ipv6(U) microcode epic100 e100 mii dm_mod uhci_hcd ext3 jbd usb_storage sd_mod scsi_mod CPU: 1 EIP: 0060:[<f08a7590>] Not tainted VLI EFLAGS: 00010286 (2.6.9-5.0.3.EL.audit.25smp) EIP is at epic_rx+0x18f/0x345 [epic100] eax: fffffffc ebx: ef096bc0 ecx: 3fff0e04 edx: ef647240 esi: edc837fe edi: effffffe ebp: fffffffc esp: c03bbf8c ds: 007b es: 007b ss: 0068 Process swapper (pid: 0, threadinfo=c03bb000 task=eff410b0) Stack: 00000002 000000fd ef647240 ef647000 ef647000 ef647240 c1615c80 fffd90e9 f08a77ac 00000287 0000e800 00000040 c03bbfd4 ef647100 ef647000 c1615c80 fffd90e9 c0274979 0000012c 00000001 c0381b98 0000000a 00000001 c0124aa4 Call Trace: [<f08a77ac>] epic_poll+0x3e/0x10d [epic100] [<c0274979>] net_rx_action+0x61/0xd8 [<c0124aa4>] __do_softirq+0x4c/0xb1 [<c0107f39>] do_softirq+0x4f/0x56 ======================= [<c010784f>] do_IRQ+0x125/0x130 [<c02c7e9c>] common_interrupt+0x18/0x20 [<c0104018>] default_idle+0x0/0x2c [<c0104041>] default_idle+0x29/0x2c [<c010409d>] cpu_idle+0x26/0x3b Code: 24 0c 83 83 a8 00 00 00 02 c1 e9 02 83 c7 02 89 43 18 89 bb a4 00 00 00 8b 54 24 08 8b 84 b2 08 04 00 00 8b b0 a8 eip: c01440b6 00 00 00 89 e8 <f3> a5 a8 02 74 02 66 a5 a8 01 74 01 a4 83 7b 64 00 74 08 0f 0b ------------[ cut here ]------------ Kernel panic - not syncing: Fatal exception in interrupt <1>kernel BUG at include/asm/spinlock.h:133! invalid operand: 0000 [#2] SMP Modules linked in: eeprom lm75 lm78 i2c_sensor i2c_isa i2c_piix4 i2c_core parport_pc lp parport nfsd exportfs nfs lockd sunrpc autofs4 iptable_filter ip_tables bridge atm md5 ipv6(U) microcode epic100 e100 mii dm_mod uhci_hcd ext3 jbd usb_storage sd_mod scsi_mod CPU: 0 EIP: 0060:[<c02c6179>] Not tainted VLI EFLAGS: 00010003 (2.6.9-5.0.3.EL.audit.25smp) EIP is at _spin_lock_irq+0x1d/0x35 eax: c02d94a5 ebx: effdde20 ecx: eff54f48 edx: c01440b6 esi: effddd60 edi: c17ede18 ebp: effdde54 esp: eff54f4c ds: 007b es: 007b ss: 0068 Process events/0 (pid: 6, threadinfo=eff54000 task=efe030d0) Stack: c160d920 c01440b6 00000002 efe08000 c160d920 c160d924 00000287 efe08000 c012e81b 00000000 c014404a ffffffff ffffffff 00000001 00000000 c011ce26 00010000 00000000 c0402e20 c160bd60 00000000 00000000 efe030d0 c011ce26 Call Trace: [<c01440b6>] cache_reap+0x6c/0x1a1 [<c012e81b>] worker_thread+0x168/0x1d5 [<c014404a>] cache_reap+0x0/0x1a1 [<c011ce26>] default_wake_function+0x0/0xc [<c011ce26>] default_wake_function+0x0/0xc [<c012e6b3>] worker_thread+0x0/0x1d5 [<c0131db1>] kthread+0x73/0x9b [<c0131d3e>] kthread+0x0/0x9b [<c01041f1>] kernel_thread_helper+0x5/0xb Code: 90 80 3e 00 7e f9 fa eb e8 89 d8 5b 5e c3 53 89 c3 fa 81 78 04 ad 4e ad de 74 18 ff 74 24 04 68 a5 94 2d c0 e8 93 a9 e5 ff 58 5a <0f> 0b 85 00 6d 85 2d c0 f0 fe 0b 79 09 f3 90 80 3b 00 7e f9 eb Badness in do_unblank_screen at drivers/char/vt.c:2876 [<c02012e2>] do_unblank_screen+0x35/0xf8 [<c01193b8>] bust_spinlocks+0x1c/0x43 [<c010600b>] die+0xfb/0x16b [<c010635d>] do_invalid_op+0x0/0xd5 [<c010635d>] do_invalid_op+0x0/0xd5 [<c0106429>] do_invalid_op+0xcc/0xd5 [<c02c6179>] _spin_lock_irq+0x1d/0x35 [<c011c427>] find_busiest_group+0xf1/0x2e0 [<c013f8fc>] free_pages_bulk+0x1cb/0x1d7 [<c01b8209>] vsnprintf+0x448/0x488 [<c01440b6>] cache_reap+0x6c/0x1a1 [<c011ced0>] __wake_up_locked+0x11/0x13 [<c02c7fbb>] error_code+0x2f/0x38 [<c01440b6>] cache_reap+0x6c/0x1a1 [<c012007b>] lookup_exec_domain+0x2f/0xbc [<c02c6179>] _spin_lock_irq+0x1d/0x35 [<c01440b6>] cache_reap+0x6c/0x1a1 [<c012e81b>] worker_thread+0x168/0x1d5 [<c014404a>] cache_reap+0x0/0x1a1 [<c011ce26>] default_wake_function+0x0/0xc [<c011ce26>] default_wake_function+0x0/0xc [<c012e6b3>] worker_thread+0x0/0x1d5 [<c0131db1>] kthread+0x73/0x9b [<c0131d3e>] kthread+0x0/0x9b [<c01041f1>] kernel_thread_helper+0x5/0xb
Created attachment 114712 [details] testing patch I'm able to reproduce this quite reliably by logging in to a machine the other side of the affected host and starting 'X -verbose 255'. I applied this patch and cannot get it to happen again. But I don't see any output produced by the added debugging either. This is i686 SMP. Surely the memory barriers introduced by the spinlocks aren't actually having any real effect?
David, I'm curious to know if you can reproduce this on a stock kernel. I know the audit patches shouldn't touch it, but...well, you know the deal... :-)
Stock kernel... eip: c0143996 ------------[ cut here ]------------ kernel BUG at include/asm/spinlock.h:133! invalid operand: 0000 [#1] SMP Modules linked in: iptable_filter ip_tables md5 ipv6(U) eeprom lm75 lm78 i2c_sensor i2c_isa i2c_piix4 i2c_core parport_pc lp parport nfsd exportfs nfs lockd sunrpc autofs4 bridge atm microcode epic100 e100 mii dm_mod uhci_hcd ext3 jbd usb_storage sd_mod scsi_mod CPU: 0 EIP: 0060:[<c02c5fe7>] Not tainted VLI EFLAGS: 00010012 (2.6.9-11.ELsmp) EIP is at _spin_lock_irq+0x1d/0x35 eax: c02d972a ebx: c17ed140 ecx: eff54f48 edx: c0143996 esi: c17ed080 edi: eff03f38 ebp: c17ed174 esp: eff54f4c ds: 007b es: 007b ss: 0068 Process events/0 (pid: 6, threadinfo=eff54000 task=efe030d0) Stack: eff03660 c0143996 00000001 efe08000 c160d920 c160d924 00000287 efe08000 c012f89b 00000000 c014392a ffffffff ffffffff 00000001 00000000 c011dc6f 00010000 00000000 c0401520 c160bd60 00000000 00000000 efe030d0 c011dc6f Call Trace: [<c0143996>] cache_reap+0x6c/0x1a1 [<c012f89b>] worker_thread+0x168/0x1d5 [<c014392a>] cache_reap+0x0/0x1a1 [<c011dc6f>] default_wake_function+0x0/0xc [<c011dc6f>] default_wake_function+0x0/0xc [<c012f733>] worker_thread+0x0/0x1d5 [<c0132e31>] kthread+0x73/0x9b [<c0132dbe>] kthread+0x0/0x9b [<c01041f1>] kernel_thread_helper+0x5/0xb Code: 90 80 3e 00 7e f9 fa eb e8 89 d8 5b 5e c3 53 89 c3 fa 81 78 04 ad 4e ad de 74 18 ff 74 24 04 68 2a 97 2d c0 e8 f1 b9 e5 ff 58 5a <0f> 0b 85 00 4f 88 2d c0 f0 fe 0b 79 09 f3 90 80 3b 00 7e f9 eb <0>Fatal exception: panic in 5 seconds Unable to handle kernel paging request at virtual address f0000000 printing eip: f08a7590 *pde = 00000000 Oops: 0000 [#2] SMP Modules linked in: iptable_filter ip_tables md5 ipv6(U) eeprom lm75 lm78 i2c_sensor i2c_isa i2c_piix4 i2c_core parport_pc lp parport nfsd exportfs nfs lockd sunrpc autofs4 bridge atm microcode epic100 e100 mii dm_mod uhci_hcd ext3 jbd usb_storage sd_mod scsi_mod CPU: 1 EIP: 0060:[<f08a7590>] Not tainted VLI EFLAGS: 00010286 (2.6.9-11.ELsmp) EIP is at epic_rx+0x18f/0x345 [epic100] eax: fffffffc ebx: ed533b60 ecx: 3b56c604 edx: ef702240 esi: effffffe edi: d4233dfe ebp: fffffffc esp: c03bcf8c ds: 007b es: 007b ss: 0068 Process swapper (pid: 0, threadinfo=c03bc000 task=eff410b0) Stack: 00000001 000000fe ef702240 ef702000 ef702000 ef702240 c1615c80 0015d28d f08a77ac 00000286 0000e800 00000040 c03bcfd4 ef702100 ef702000 c1615c80 0015d28d c02746f5 0000012c 00000001 c0381b98 0000000a 00000001 c0125a5c Call Trace: [<f08a77ac>] epic_poll+0x3e/0x10d [epic100] [<c02746f5>] net_rx_action+0x61/0xd8 [<c0125a5c>] __do_softirq+0x4c/0xb1 [<c010806d>] do_softirq+0x4f/0x56 ======================= [<c0107983>] do_IRQ+0x125/0x130 [<c02c7d34>] common_interrupt+0x18/0x20 [<c0104018>] default_idle+0x0/0x2c [<c0104041>] default_idle+0x29/0x2c [<c010409d>] cpu_idle+0x26/0x3b Code: 24 0c 83 83 a8 00 00 00 02 c1 e9 02 83 c7 02 89 43 18 89 bb a4 00 00 00 8b 54 24 08 8b 84 b2 08 04 00 00 8b b0 a8 00 00 00 89 e8 <f3> a5 a8 02 74 02 66 a5 a8 01 74 01 a4 83 7b 64 00 74 08 0f 0b <0>Kernel panic - not syncing: Fatal exception in interrupt <0>Rebooting in 30 seconds..
Cannot reproduce in FC4 where the epic100 driver is basically identical. It's _harder_ to reproduce on RHEL4 if I enable IRQ stacks by setting CONFIG_4KSTACKS in arch/i386/kernel/irq.c Building an 8KiB-stack RHEL4 kernel now to test that.
Happens even with IPv4-only routing, with no bridging, with 8KiB stack. So I suspect it's probably not stack overflow after all. When it happens, the module list seems to be corrupted. The call to print_modules() doesn't ever complete... NMI Watchdog detected LOCKUP on CPU0, eip c02c94e4, registers: Modules linked in: eeprom(U) lm75(U) lm78(U) i2c_sensor(U) i2c_isa(U) i2c_piix4(U) parport_pc(U) lp(U) parport(U) nfsd(U) exportfs(U) i2c_dev(U) .U\uffff\uffff..
Created attachment 116677 [details] jwltest-epic100-poll.patch No direct link between this and anything above...just noticed the epic_poll might not always obey it's budget...
Test kernels w/ above patch available here: http://people.redhat.com/linville/kernels/rhel4/ I _doubt_ if it will make any difference. But if you get a chance then please do try to reproduce the issue w/ those kernels (or other kernels w/ the patch from comment 8) and post the results...thanks!
Well, it helps if one actually copies the files to the designated place...they are there now... :-)
David, I'm closing this out as part of the bug scrub. Feel free to reopen this if you get any results from testing the patch in comment 8.