Bug 156816 - epic100 kills system
epic100 kills system
Status: CLOSED CANTFIX
Product: Red Hat Enterprise Linux 4
Classification: Red Hat
Component: kernel (Show other bugs)
4.0
All Linux
medium Severity medium
: ---
: ---
Assigned To: John W. Linville
Brian Brock
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2005-05-04 10:11 EDT by David Woodhouse
Modified: 2007-11-30 17:07 EST (History)
1 user (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2005-09-16 10:51:43 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
testing patch (1.71 KB, patch)
2005-05-23 09:38 EDT, David Woodhouse
no flags Details | Diff
jwltest-epic100-poll.patch (638 bytes, patch)
2005-07-12 15:40 EDT, John W. Linville
no flags Details | Diff

  None (edit)
Description David Woodhouse 2005-05-04 10:11:15 EDT
Routing through an epic100 card with 2.6.9-5.0.3.EL kernels seems to kill the
system when it gets used hard for rx.

This is actually an audit kernel based on 2.6.9-5.0.3.EL, but nothing
network-related is changed by the audit patches. The epic100 is also part of a
bridge. This setup has run perfectly reliably for a very long time with
2.6.5-1.358. It's i686 SMP.

Symptoms include oopsen, spontaneous reboots and lockups. When enabling epic100
debugging, this output seemed particularly interesting, when compared with the
code in epic_rx().

   /* If we own the next entry, it's a new packet. Send it up. */
   while ((ep->rx_ring[entry].rxstatus & cpu_to_le32(DescOwn)) == 0) {
 	int status = le32_to_cpu(ep->rx_ring[entry].rxstatus);
	if (debug > 4)
		printk(KERN_DEBUG "  epic_rx() status was %8.8x.\n", status);
 	....

eth1: Interrupt, status=0x00cd0800 new intstat=0x00cd0800.
eth1: exit interrupt, intr_status=0xcd0000.
 In epic_rx(), entry 124 05da1781.
  epic_rx() status was 05da1781.
eth1: Interrupt, status=0x00cd0800 new intstat=0x00cd0800.
eth1: exit interrupt, intr_status=0xcd0000.
 In epic_rx(), entry 125 058a1781.
  epic_rx() status was 058a1781.
  epic_rx() status was 00008000.


(and here it locked up hard. But why did it ever print "status was 0x8000" since
0x8000 is DescOwn. Something else must have been giving the descriptor back to
the card at the time this code was running. Can epic_rx() itself be run more
than once in parallel?)
Comment 1 David Woodhouse 2005-05-04 10:11:44 EDT
Other oopsen...

------------[ cut here ]------------
kernel BUG at kernel/sched.c:920!
invalid operand: 0000 [#1]
SMP
Modules linked in: epic100 eeprom lm75 lm78 i2c_sensor i2c_isa i2c_piix4
i2c_core parport_pc lp parport nfsd exportfs nfs lockd sunrpc autofs4
iptable_filter ip_tables bridge atm md5 ipv6(U) microcode e100 mii dm_mod
uhci_hcd ext3 jbd usb_storage sd_mod scsi_mod
CPU:    1
EIP:    0060:[<c011b57d>]    Not tainted VLI
EFLAGS: 00010002   (2.6.9-5.0.3.EL.audit.28smp)
EIP is at resched_task+0x20/0x68
eax: 00000001   ebx: eff17000   ecx: c1613d60   edx: 00000000
esi: eff410b0   edi: 00000000   ebp: c03bbf10   esp: c03bbf08
ds: 007b   es: 007b   ss: 0068
Process swapper (pid: 0, threadinfo=c03bb000 task=eff410b0)
Stack: 00000001 c1613960 c03bbf40 c011cb7f 00000001 c01072e6 00000001 c03bbf58
       c1613d60 00000001 00000000 c03ac188 00000001 c03ac184 fffffffc c0116312
       ecd10780 e787ff0e eff25f0e c02c7f86 ecd10780 3ffeae40 eef60240 e787ff0e
Call Trace:
 [<c011cb7f>] scheduler_tick+0x13e/0x3e5
 [<c01072e6>] handle_IRQ_event+0x25/0x4f
 [<c0116312>] smp_apic_timer_interrupt+0xac/0xdd
 [<c02c7f86>] apic_timer_interrupt+0x1a/0x20
 [<f0929590>] epic_rx+0x18f/0x345 [epic100]
 [<f09297ac>] epic_poll+0x3e/0x10d [epic100]
 [<c02749e1>] net_rx_action+0x61/0xd8
 [<c0124aa4>] __do_softirq+0x4c/0xb1
 [<c0107f39>] do_softirq+0x4f/0x56
 =======================
Code: ff c7 43 28 00 00 00 00 5b 5d c3 55 89 e5 56 89 c6 b8 00 ce 3a c0 53 8b 5e
04 8b 53 10 03 04 95 20 71 3b c0 0f b6 00 84 c0 7e 08 <0f> 0b 98 03 57 85 2d c0
8b 43 08 c1 e8 10 83 e0 01 f0 0f ba 6b
 <1>Unable to handle kernel paging request at virtual address 30303038
 printing eip:
c0143a5c
*pde = 2a0c2001
Oops: 0000 [#2]
SMP
Modules linked in: epic100 eeprom lm75 lm78 i2c_sensor i2c_isa i2c_piix4
i2c_core parport_pc lp parport nfsd exportfs nfs lockd sunrpc autofs4
iptable_filter ip_tables bridge atm md5 ipv6(U) microcode e100 mii dm_mod
uhci_hcd ext3 jbd usb_storage sd_mod scsi_mod
CPU:    0
EIP:    0060:[<c0143a5c>]    Not tainted VLI
EFLAGS: 00010086   (2.6.9-5.0.3.EL.audit.28smp)
EIP is at kmem_cache_alloc+0x32/0x57
eax: 00000000   ebx: 00000220   ecx: ef85b280   edx: 30303038
esi: 00000246   edi: eff038a0   ebp: effc14c0   esp: c03baf44
ds: 007b   es: 007b   ss: 0068
Process swapper (pid: 0, threadinfo=c03ba000 task=c0314a60)
Stack: ecea9980 ecea9980 ecea9980 c026f460 ecea9980 ef85b280 c02c2d70 ee708800
       c033f298 ef85b280 00000008 ecca6eb4 c0274822 ecea9980 0000009a ecea9980
       ee708a40 f08e3508 80000000 ec8ebdc0 0000a020 00000010 c03bafbc ee708900
Call Trace:
 [<c026f460>] skb_clone+0xe/0x192
 [<c02c2d70>] packet_rcv_spkt+0x2e/0x1d6
 [<c0274822>] netif_receive_skb+0x1be/0x21f
 [<f08e3508>] e100_rx_clean+0x185/0x2fe [e100]
 [<f08e3179>] e100_poll+0x33/0x23d [e100]
 [<c02749e1>] net_rx_action+0x61/0xd8
 [<c0124aa4>] __do_softirq+0x4c/0xb1
 [<c0107f39>] do_softirq+0x4f/0x56
 =======================
 [<c010784f>] do_IRQ+0x125/0x130
 [<c02c7f04>] common_interrupt+0x18/0x20
 [<c0104018>] default_idle+0x0/0x2c
 [<c0104041>] default_idle+0x29/0x2c
 [<c010409d>] cpu_idle+0x26/0x3b
 [<c0384784>] start_kernel+0x194/0x198
Code: 53 89 d3 74 16 31 c9 ba 0f 08 00 00 b8 39 ab 2d c0 e8 0a ab fd ff e8 19 21
18 00 9c 5e fa b8 00 f0 ff ff 21 e0 8b 40 10 8b 14 87 <8b> 02 85 c0 74 10 c7 42
0c 01 00 00 00 48 89 02 8b 44 82 10 eb
 <0>Kernel panic - not syncing: Fatal exception in interrupt
Comment 2 David Woodhouse 2005-05-04 10:11:58 EDT
Unable to handle kernel paging request at virtual address f0000000
 printing eip:
f08a7590
*pde = 00000000
Oops: 0002 [#1]
SMP
Modules linked in: eeprom lm75 lm78 i2c_sensor i2c_isa i2c_piix4 i2c_core
parport_pc lp parport nfsd exportfs nfs lockd sunrpc autofs4 iptable_filter
ip_tables
bridge atm md5 ipv6(U) microcode epic100 e100 mii dm_mod uhci_hcd ext3 jbd
usb_storage sd_mod scsi_mod
CPU:    1
EIP:    0060:[<f08a7590>]    Not tainted VLI
EFLAGS: 00010286   (2.6.9-5.0.3.EL.audit.25smp)
EIP is at epic_rx+0x18f/0x345 [epic100]
eax: fffffffc   ebx: ef096bc0   ecx: 3fff0e04   edx: ef647240
esi: edc837fe   edi: effffffe   ebp: fffffffc   esp: c03bbf8c
ds: 007b   es: 007b   ss: 0068
Process swapper (pid: 0, threadinfo=c03bb000 task=eff410b0)
Stack: 00000002 000000fd ef647240 ef647000 ef647000 ef647240 c1615c80 fffd90e9
       f08a77ac 00000287 0000e800 00000040 c03bbfd4 ef647100 ef647000 c1615c80
       fffd90e9 c0274979 0000012c 00000001 c0381b98 0000000a 00000001 c0124aa4
Call Trace:
 [<f08a77ac>] epic_poll+0x3e/0x10d [epic100]
 [<c0274979>] net_rx_action+0x61/0xd8
 [<c0124aa4>] __do_softirq+0x4c/0xb1
 [<c0107f39>] do_softirq+0x4f/0x56
 =======================
 [<c010784f>] do_IRQ+0x125/0x130
 [<c02c7e9c>] common_interrupt+0x18/0x20
 [<c0104018>] default_idle+0x0/0x2c
 [<c0104041>] default_idle+0x29/0x2c
 [<c010409d>] cpu_idle+0x26/0x3b
Code: 24 0c 83 83 a8 00 00 00 02 c1 e9 02 83 c7 02 89 43 18 89 bb a4 00 00 00 8b
54 24 08 8b 84 b2 08 04 00 00 8b b0 a8 eip: c01440b6
00 00 00 89 e8 <f3> a5 a8 02 74 02 66 a5 a8 01 74 01 a4 83 7b 64 00 74 08 0f 0b
 ------------[ cut here ]------------
Kernel panic - not syncing: Fatal exception in interrupt
 <1>kernel BUG at include/asm/spinlock.h:133!
invalid operand: 0000 [#2]
SMP
Modules linked in: eeprom lm75 lm78 i2c_sensor i2c_isa i2c_piix4 i2c_core
parport_pc lp parport nfsd exportfs nfs lockd sunrpc autofs4 iptable_filter
ip_tables
bridge atm md5 ipv6(U) microcode epic100 e100 mii dm_mod uhci_hcd ext3 jbd
usb_storage sd_mod scsi_mod
CPU:    0
EIP:    0060:[<c02c6179>]    Not tainted VLI
EFLAGS: 00010003   (2.6.9-5.0.3.EL.audit.25smp)
EIP is at _spin_lock_irq+0x1d/0x35
eax: c02d94a5   ebx: effdde20   ecx: eff54f48   edx: c01440b6
esi: effddd60   edi: c17ede18   ebp: effdde54   esp: eff54f4c
ds: 007b   es: 007b   ss: 0068
Process events/0 (pid: 6, threadinfo=eff54000 task=efe030d0)
Stack: c160d920 c01440b6 00000002 efe08000 c160d920 c160d924 00000287 efe08000
       c012e81b 00000000 c014404a ffffffff ffffffff 00000001 00000000 c011ce26
       00010000 00000000 c0402e20 c160bd60 00000000 00000000 efe030d0 c011ce26
Call Trace:
 [<c01440b6>] cache_reap+0x6c/0x1a1
 [<c012e81b>] worker_thread+0x168/0x1d5
 [<c014404a>] cache_reap+0x0/0x1a1
 [<c011ce26>] default_wake_function+0x0/0xc
 [<c011ce26>] default_wake_function+0x0/0xc
 [<c012e6b3>] worker_thread+0x0/0x1d5
 [<c0131db1>] kthread+0x73/0x9b
 [<c0131d3e>] kthread+0x0/0x9b
 [<c01041f1>] kernel_thread_helper+0x5/0xb
Code: 90 80 3e 00 7e f9 fa eb e8 89 d8 5b 5e c3 53 89 c3 fa 81 78 04 ad 4e ad de
74 18 ff 74 24 04 68 a5 94 2d c0 e8 93 a9 e5 ff 58 5a <0f> 0b 85 00 6d 85 2d c0
f0 fe 0b 79 09 f3 90 80 3b 00 7e f9 eb
Badness in do_unblank_screen at drivers/char/vt.c:2876
 [<c02012e2>] do_unblank_screen+0x35/0xf8
 [<c01193b8>] bust_spinlocks+0x1c/0x43
 [<c010600b>] die+0xfb/0x16b
 [<c010635d>] do_invalid_op+0x0/0xd5
 [<c010635d>] do_invalid_op+0x0/0xd5
 [<c0106429>] do_invalid_op+0xcc/0xd5
 [<c02c6179>] _spin_lock_irq+0x1d/0x35
 [<c011c427>] find_busiest_group+0xf1/0x2e0
 [<c013f8fc>] free_pages_bulk+0x1cb/0x1d7
 [<c01b8209>] vsnprintf+0x448/0x488
 [<c01440b6>] cache_reap+0x6c/0x1a1
 [<c011ced0>] __wake_up_locked+0x11/0x13
 [<c02c7fbb>] error_code+0x2f/0x38
 [<c01440b6>] cache_reap+0x6c/0x1a1
 [<c012007b>] lookup_exec_domain+0x2f/0xbc
 [<c02c6179>] _spin_lock_irq+0x1d/0x35
 [<c01440b6>] cache_reap+0x6c/0x1a1
 [<c012e81b>] worker_thread+0x168/0x1d5
 [<c014404a>] cache_reap+0x0/0x1a1
 [<c011ce26>] default_wake_function+0x0/0xc
 [<c011ce26>] default_wake_function+0x0/0xc
 [<c012e6b3>] worker_thread+0x0/0x1d5
 [<c0131db1>] kthread+0x73/0x9b
 [<c0131d3e>] kthread+0x0/0x9b
 [<c01041f1>] kernel_thread_helper+0x5/0xb
Comment 3 David Woodhouse 2005-05-23 09:38:22 EDT
Created attachment 114712 [details]
testing patch

I'm able to reproduce this quite reliably by logging in to a machine the other
side of the affected host and starting 'X -verbose 255'. 

I applied this patch and cannot get it to happen again. But I don't see any
output produced by the added debugging either.

This is i686 SMP. Surely the memory barriers introduced by the spinlocks aren't
actually having any real effect?
Comment 4 John W. Linville 2005-06-02 16:54:54 EDT
David, I'm curious to know if you can reproduce this on a stock kernel.  I know
the audit patches shouldn't touch it, but...well, you know the deal... :-)
Comment 5 David Woodhouse 2005-06-16 10:43:47 EDT
Stock kernel...

 eip: c0143996
------------[ cut here ]------------
kernel BUG at include/asm/spinlock.h:133!
invalid operand: 0000 [#1]
SMP
Modules linked in: iptable_filter ip_tables md5 ipv6(U) eeprom lm75 lm78
i2c_sensor i2c_isa i2c_piix4 i2c_core parport_pc lp parport nfsd exportfs nfs
lockd sunrpc autofs4 bridge atm microcode epic100 e100 mii dm_mod uhci_hcd ext3
jbd usb_storage sd_mod scsi_mod
CPU:    0
EIP:    0060:[<c02c5fe7>]    Not tainted VLI
EFLAGS: 00010012   (2.6.9-11.ELsmp)
EIP is at _spin_lock_irq+0x1d/0x35
eax: c02d972a   ebx: c17ed140   ecx: eff54f48   edx: c0143996
esi: c17ed080   edi: eff03f38   ebp: c17ed174   esp: eff54f4c
ds: 007b   es: 007b   ss: 0068
Process events/0 (pid: 6, threadinfo=eff54000 task=efe030d0)
Stack: eff03660 c0143996 00000001 efe08000 c160d920 c160d924 00000287 efe08000
       c012f89b 00000000 c014392a ffffffff ffffffff 00000001 00000000 c011dc6f
       00010000 00000000 c0401520 c160bd60 00000000 00000000 efe030d0 c011dc6f
Call Trace:
 [<c0143996>] cache_reap+0x6c/0x1a1
 [<c012f89b>] worker_thread+0x168/0x1d5
 [<c014392a>] cache_reap+0x0/0x1a1
 [<c011dc6f>] default_wake_function+0x0/0xc
 [<c011dc6f>] default_wake_function+0x0/0xc
 [<c012f733>] worker_thread+0x0/0x1d5
 [<c0132e31>] kthread+0x73/0x9b
 [<c0132dbe>] kthread+0x0/0x9b
 [<c01041f1>] kernel_thread_helper+0x5/0xb
Code: 90 80 3e 00 7e f9 fa eb e8 89 d8 5b 5e c3 53 89 c3 fa 81 78 04 ad 4e ad de
74 18 ff 74 24 04 68 2a 97 2d c0 e8 f1 b9 e5 ff 58 5a <0f> 0b 85 00 4f 88 2d c0
f0 fe 0b 79 09 f3 90 80 3b 00 7e f9 eb
 <0>Fatal exception: panic in 5 seconds
Unable to handle kernel paging request at virtual address f0000000
 printing eip:
f08a7590
*pde = 00000000
Oops: 0000 [#2]
SMP
Modules linked in: iptable_filter ip_tables md5 ipv6(U) eeprom lm75 lm78
i2c_sensor i2c_isa i2c_piix4 i2c_core parport_pc lp parport nfsd exportfs nfs
lockd sunrpc autofs4 bridge atm microcode epic100 e100 mii dm_mod uhci_hcd ext3
jbd usb_storage sd_mod scsi_mod
CPU:    1
EIP:    0060:[<f08a7590>]    Not tainted VLI
EFLAGS: 00010286   (2.6.9-11.ELsmp)
EIP is at epic_rx+0x18f/0x345 [epic100]
eax: fffffffc   ebx: ed533b60   ecx: 3b56c604   edx: ef702240
esi: effffffe   edi: d4233dfe   ebp: fffffffc   esp: c03bcf8c
ds: 007b   es: 007b   ss: 0068
Process swapper (pid: 0, threadinfo=c03bc000 task=eff410b0)
Stack: 00000001 000000fe ef702240 ef702000 ef702000 ef702240 c1615c80 0015d28d
       f08a77ac 00000286 0000e800 00000040 c03bcfd4 ef702100 ef702000 c1615c80
       0015d28d c02746f5 0000012c 00000001 c0381b98 0000000a 00000001 c0125a5c
Call Trace:
 [<f08a77ac>] epic_poll+0x3e/0x10d [epic100]
 [<c02746f5>] net_rx_action+0x61/0xd8
 [<c0125a5c>] __do_softirq+0x4c/0xb1
 [<c010806d>] do_softirq+0x4f/0x56
 =======================
 [<c0107983>] do_IRQ+0x125/0x130
 [<c02c7d34>] common_interrupt+0x18/0x20
 [<c0104018>] default_idle+0x0/0x2c
 [<c0104041>] default_idle+0x29/0x2c
 [<c010409d>] cpu_idle+0x26/0x3b
Code: 24 0c 83 83 a8 00 00 00 02 c1 e9 02 83 c7 02 89 43 18 89 bb a4 00 00 00 8b
54 24 08 8b 84 b2 08 04 00 00 8b b0 a8 00 00 00 89 e8 <f3> a5 a8 02 74 02 66 a5
a8 01 74 01 a4 83 7b 64 00 74 08 0f 0b
 <0>Kernel panic - not syncing: Fatal exception in interrupt
 <0>Rebooting in 30 seconds..
Comment 6 David Woodhouse 2005-07-01 07:50:28 EDT
Cannot reproduce in FC4 where the epic100 driver is basically identical.

It's _harder_ to reproduce on RHEL4 if I enable IRQ stacks by setting
CONFIG_4KSTACKS in arch/i386/kernel/irq.c

Building an 8KiB-stack RHEL4 kernel now to test that.

Comment 7 David Woodhouse 2005-07-01 08:37:44 EDT
Happens even with IPv4-only routing, with no bridging, with 8KiB stack. So I
suspect it's probably not stack overflow after all.

When it happens, the module list seems to be corrupted. The call to
print_modules() doesn't ever complete...

NMI Watchdog detected LOCKUP on CPU0, eip c02c94e4, registers:
Modules linked in: eeprom(U) lm75(U) lm78(U) i2c_sensor(U) i2c_isa(U)
i2c_piix4(U) parport_pc(U) lp(U) parport(U) nfsd(U) exportfs(U) i2c_dev(U)
.U\uffff\uffff..
Comment 8 John W. Linville 2005-07-12 15:40:15 EDT
Created attachment 116677 [details]
jwltest-epic100-poll.patch

No direct link between this and anything above...just noticed the epic_poll
might not always obey it's budget...
Comment 9 John W. Linville 2005-07-12 16:51:48 EDT
Test kernels w/ above patch available here: 
 
   http://people.redhat.com/linville/kernels/rhel4/ 
 
I _doubt_ if it will make any difference.  But if you get a chance then please 
do try to reproduce the issue w/ those kernels (or other kernels w/ the patch 
from comment 8) and post the results...thanks! 
Comment 10 John W. Linville 2005-07-13 16:16:13 EDT
Well, it helps if one actually copies the files to the designated place...they 
are there now... :-) 
Comment 11 John W. Linville 2005-09-16 10:51:43 EDT
David, I'm closing this out as part of the bug scrub.  Feel free to reopen 
this if you get any results from testing the patch in comment 8. 

Note You need to log in before you can comment on or make changes to this bug.