Bug 160218

Summary: iptables related oops in put_page()
Product: [Fedora] Fedora Reporter: Oleg Drokin <green>
Component: kernelAssignee: David Miller <davem>
Status: CLOSED INSUFFICIENT_DATA QA Contact: Brian Brock <bbrock>
Severity: medium Docs Contact:
Priority: medium    
Version: 4CC: davej, sundaram, wtogami
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: 2.6.11-1.35 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2006-05-05 13:09:23 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Oleg Drokin 2005-06-13 16:02:16 UTC
Description of problem:
Box oopsed and rebooted today, I found following Oops report in logs:

Unable to handle kernel paging request at virtual address a8c0527a
 printing eip:
c0147ec2
*pde = 00000000
Oops: 0000 [#1]
SMP
Modules linked in: loop ppp_deflate zlib_deflate ppp_async crc_ccitt ppp_generic
slhc tun md5 ipv6 ipt_state iptable_filter ipt_MASQUERADE iptable_nat
ip_conntrack ipt_MARK iptable_mangle ip_tables microcode video button battery ac
uhci_hcd tulip e1000 e100 mii floppy sg reiserfs raid1 dm_mod aic79xx sd_mod
scsi_mod
CPU:    1
EIP:    0060:[put_page+2/117]    Not tainted VLI
EIP:    0060:[<c0147ec2>]    Not tainted VLI
EFLAGS: 00010206   (2.6.11-1.27_FC3smp)
EIP is at put_page+0x2/0x75
eax: a8c0527a   ebx: 0000006a   ecx: d71affff   edx: a8c0527a
esi: f4ef4b80   edi: f4ef4b80   ebp: f6f28800   esp: c03f4dbc
ds: 007b   es: 007b   ss: 0068
Process squid (pid: 9598, threadinfo=c03f4000 task=f4bce020)
Stack: c02998d4 1a97d800 f6f28e80 c029ebca 0000007e d71affff f7476000 f4ef4b80
       fffffff4 c029edb5 f7476000 00000292 f91e505c f1ca3314 f1ca332c dc5ab01e
       f4ef4b80 c02bab8e 00000004 f1ca3300 00000000 00000004 00000002 c02baac2
Call Trace:
 [skb_release_data+57/111] skb_release_data+0x39/0x6f
 [<c02998d4>] skb_release_data+0x39/0x6f
 [__skb_linearize+198/276] __skb_linearize+0xc6/0x114
 [<c029ebca>] __skb_linearize+0xc6/0x114
 [dev_queue_xmit+413/686] dev_queue_xmit+0x19d/0x2ae
 [<c029edb5>] dev_queue_xmit+0x19d/0x2ae
 [ip_finish_output2+204/528] ip_finish_output2+0xcc/0x210
 [<c02bab8e>] ip_finish_output2+0xcc/0x210
 [ip_finish_output2+0/528] ip_finish_output2+0x0/0x210
 [<c02baac2>] ip_finish_output2+0x0/0x210
 [nf_hook_slow+221/253] nf_hook_slow+0xdd/0xfd
 [<c02a7fd4>] nf_hook_slow+0xdd/0xfd
 [ip_finish_output2+0/528] ip_finish_output2+0x0/0x210
 [<c02baac2>] ip_finish_output2+0x0/0x210
 [ip_forward_finish+0/75] ip_forward_finish+0x0/0x4b
 [<c02b9445>] ip_forward_finish+0x0/0x4b
 [ip_finish_output+290/605] ip_finish_output+0x122/0x25d
 [<c02ba987>] ip_finish_output+0x122/0x25d
 [ip_finish_output2+0/528] ip_finish_output2+0x0/0x210
 [<c02baac2>] ip_finish_output2+0x0/0x210
 [ip_forward_finish+0/75] ip_forward_finish+0x0/0x4b
 [<c02b9445>] ip_forward_finish+0x0/0x4b
 [ip_forward_finish+0/75] ip_forward_finish+0x0/0x4b
 [<c02b9445>] ip_forward_finish+0x0/0x4b
 [ip_forward_finish+42/75] ip_forward_finish+0x2a/0x4b
 [<c02b946f>] ip_forward_finish+0x2a/0x4b
 [nf_hook_slow+221/253] nf_hook_slow+0xdd/0xfd
 [<c02a7fd4>] nf_hook_slow+0xdd/0xfd
 [ip_forward_finish+0/75] ip_forward_finish+0x0/0x4b

The box itself is dual-P4 Xeon 3GHz with 2G RAM.
it does some forwarding between 5 network interfaces, there around 400 iptables
rules (including various forms of NAT, stateful drop/accept stuff etc.)

How reproducible:

So far I have only this one oops, but with this same kernel there was a hang
some time ago, that did not left any traces, not even on serial console.

Comment 1 Dave Jones 2005-06-24 01:47:14 UTC
this should be fixed in the 2.6.11-1.35_FC3 kernel currently in updates-testing.
(Should be going to updates-proper tomorrow hopefully).


Comment 2 Oleg Drokin 2005-09-20 18:13:52 UTC
It oopsed yesterday with very similar backtrace with 2.6.11-1.35.
I booted into latest errata kernel for FC3 now (2.6.12-1.1376_FC3) to see if
that one shows same problem.
The box is the same as with previous oops.

Unable to handle kernel paging request at virtual address 00e2ffff
 printing eip:
c0147e32
*pde = 00000000
Oops: 0000 [#1]
SMP 
Modules linked in: ppp_deflate zlib_deflate ppp_async crc_ccitt ppp_generic sl
hc tun md5 ipv6 ipt_state iptable_filter ipt_MASQUERADE iptable_nat ip_conntrack
 ipt_MARK iptable_mangle ip_tables microcode video button battery ac uhci_hcd tu
lip e1000 e100 mii floppy sg reiserfs raid1 dm_mod aic79xx sd_mod scsi_mod
CPU:    2
EIP:    0060:[<c0147e32>]    Not tainted VLI
EFLAGS: 00010202   (2.6.11-1.35_FC3smp) 
EIP is at put_page+0x2/0x75
eax: 00e2ffff   ebx: 00000001   ecx: 00000000   edx: 00e2ffff
esi: cf6b3280   edi: cf6b3280   ebp: d3b7e000   esp: c03f6dbc
ds: 007b   es: 007b   ss: 0068
Process swapper (pid: 0, threadinfo=c03f6000 task=f7c12020)
Stack: c0299ae4 f553a800 d3b7e680 c029edda 00000085 00e2ffff f785b000 cf6b3280

       fffffff4 c029efc5 f785b000 00000292 f91e505c d9733414 d973342c de64381e

       cf6b3280 c02bad9e 00000004 d9733400 00000000 00000004 00000002
c02bacd2^MCall Trace:
 [<c0299ae4>] skb_release_data+0x39/0x6f
 [<c029edda>] __skb_linearize+0xc6/0x114
 [<c029efc5>] dev_queue_xmit+0x19d/0x2ae
 [<c02bad9e>] ip_finish_output2+0xcc/0x210
 [<c02bacd2>] ip_finish_output2+0x0/0x210
 [<c02a81e4>] nf_hook_slow+0xdd/0xfd
 [<c02bacd2>] ip_finish_output2+0x0/0x210
 [<c02b9655>] ip_forward_finish+0x0/0x4b
 [<c02bab97>] ip_finish_output+0x122/0x25d
 [<c02bacd2>] ip_finish_output2+0x0/0x210
 [<c02b9655>] ip_forward_finish+0x0/0x4b
 [<c02b9655>] ip_forward_finish+0x0/0x4b
 [<c02b967f>] ip_forward_finish+0x2a/0x4b
 [<c02a81e4>] nf_hook_slow+0xdd/0xfd
 [<c02b9655>] ip_forward_finish+0x0/0x4b
 [<c02b95f3>] ip_forward+0x253/0x2b5
 [<c02b9655>] ip_forward_finish+0x0/0x4b
 [<c02b8368>] ip_rcv_finish+0x179/0x2b1
 [<c02a7e0b>] nf_iterate+0x52/0x9b
 [<c02b81ef>] ip_rcv_finish+0x0/0x2b1
 [<c02b81ef>] ip_rcv_finish+0x0/0x2b1
 [<c02a81e4>] nf_hook_slow+0xdd/0xfd
 [<c02b81ef>] ip_rcv_finish+0x0/0x2b1
 [<c02b8049>] ip_rcv+0x352/0x4f8
 [<c02b81ef>] ip_rcv_finish+0x0/0x2b1
 [<c01470b7>] cache_flusharray+0x52/0xae
 [<c029f6ba>] netif_receive_skb+0x23e/0x2b1
 [<c029f7ab>] process_backlog+0x7e/0x110
 [<c029f8ad>] net_rx_action+0x70/0xf2
 [<c0126132>] __do_softirq+0x62/0xd0
 [<c0106295>] do_softirq+0x42/0x49
 =======================
 [<c0106187>] do_IRQ+0x57/0x89
 [<c01048f6>] common_interrupt+0x1a/0x20
 [<c010203d>] default_idle+0x23/0x29
 [<c01020c4>] cpu_idle+0x4e/0x5c
Code: 00 1b 00 e9 57 fb ff ff 8d 05 ec f8 45 c0 e8 42 00 1b 00 e9 55 fe ff ff 
8d 05 ec f8 45 c0 e8 56 00 1b 00 e9 8f fe ff ff 90 89 c2 <8b> 00 f6 c4 80 75 2f 
8b 02 f6 c4 08 75 27 8b 02 89 d1 f6 c4 80 
 <0>Kernel panic - not syncing: Fatal exception in interrupt

Comment 3 Oleg Drokin 2005-09-28 13:19:04 UTC
Happened again with 2.6.12-1.1376_FC3smp:
Unable to handle kernel paging request at virtual address 29b1a040
 printing eip:
c014ce02
*pde = 00004001
Oops: 0000 [#1]
SMP
Modules linked in: ppp_deflate zlib_deflate ppp_async crc_ccitt ppp_generic slhc
tun md5 ipv6 ipt_state iptable_filter ipt_MASQUERADE iptable_nat ip_conntrack
ipt_MARK iptable_mangle ip_tables microcode video button battery ac uhci_hcd
tulip e1000 e100 mii floppy sg reiserfs raid1 dm_mod aic79xx sd_mod scsi_mod
CPU:    3
EIP:    0060:[<c014ce02>]    Not tainted VLI
EFLAGS: 00010202   (2.6.12-1.1376_FC3smp)
EIP is at put_page+0x2/0x75
eax: 29b1a040   ebx: 00000001   ecx: e9b1a680   edx: 29b1a040
esi: f7aa1080   edi: f7aa1080   ebp: f5647800   esp: c0406ea8
ds: 007b   es: 007b   ss: 0068
Process swapper (pid: 0, threadinfo=c0406000 task=c221ca80)
Stack: c02a0bcb 0bb2d800 f5647e80 c02a5e90 00000079 29b1a040 f7e2a000 f7aa1080
       fffffff4 c02a6075 80000000 00000286 c048cf20 f2db4114 f2db412c e9b1a01e
       f7aa1080 c02c21a9 00000000 f7e2a000 c02c2322 80000000 f2db4100 f7aa1080
Call Trace:
 [<c02a0bcb>] skb_release_data+0x50/0x83
 [<c02a5e90>] __skb_linearize+0xc6/0x114
 [<c02a6075>] dev_queue_xmit+0x197/0x2a8
 [<c02c21a9>] ip_finish_output+0xd5/0x24e
 [<c02c2322>] ip_finish_output2+0x0/0x1e0
 [<c02c0bfb>] ip_forward+0x16b/0x2e5
 [<c02c0d75>] ip_forward_finish+0x0/0x4b
 [<c02bf6e7>] ip_rcv+0x350/0x551
 [<c02bf8e8>] ip_rcv_finish+0x0/0x2a8
 [<c02a6775>] netif_receive_skb+0x228/0x279
 [<c014c099>] cache_flusharray+0x52/0xac
 [<c02a6851>] process_backlog+0x8b/0x11d
 [<c02a6965>] net_rx_action+0x82/0x175
 [<c0126469>] __do_softirq+0x69/0xd5
 [<c0106688>] do_softirq+0x45/0x4c
 =======================
 [<c0106577>] do_IRQ+0x57/0x89
 [<c0116953>] smp_apic_timer_interrupt+0xbd/0xc6
 [<c0104a2e>] common_interrupt+0x1a/0x20
 [<c010201a>] default_idle+0x0/0x29
 [<c010203d>] default_idle+0x23/0x29
 [<c01020d3>] cpu_idle+0x5d/0x6c
Code: 55 fb ff ff 8d 05 6c e9 46 c0 e8 ea 2f 1b 00 e9 53 fe ff ff 8d 05 6c e9 46
c0 e8 fe 2f 1b 00 e9 8f fe ff ff 90 90 90 90 90 89 c2 <8b> 00 f6 c4 80 75 2f 8b
02 f6 c4 08 75 27 8b 02 89 d1 f6 c4 80
 <0>Kernel panic - not syncing: Fatal exception in interrupt


Comment 4 David Miller 2005-09-28 22:22:30 UTC
Are you doing anything interesting with the "tun" device on this
machine?  Or is it just loaded and unused?


Comment 5 Oleg Drokin 2005-09-29 07:05:07 UTC
tun device is used by single openvpn connection with very little traffic most of
the time
          RX bytes:6795098 (6.4 MiB)  TX bytes:2257997 (2.1 MiB)
for last almost 6 days.

Comment 6 Dave Jones 2006-01-16 22:07:31 UTC
This is a mass-update to all currently open Fedora Core 3 kernel bugs.

Fedora Core 3 support has transitioned to the Fedora Legacy project.
Due to the limited resources of this project, typically only
updates for new security issues are released.

As this bug isn't security related, it has been migrated to a
Fedora Core 4 bug.  Please upgrade to this newer release, and
test if this bug is still present there.

This bug has been placed in NEEDINFO_REPORTER state.
Due to the large volume of inactive bugs in bugzilla, if this bug is
still in this state in two weeks time, it will be closed.

Should this bug still be relevant after this period, the reporter
can reopen the bug at any time. Any other users on the Cc: list
of this bug can request that the bug be reopened by adding a
comment to the bug.

Thank you.


Comment 7 Dave Jones 2006-02-03 05:07:22 UTC
This is a mass-update to all currently open kernel bugs.

A new kernel update has been released (Version: 2.6.15-1.1830_FC4)
based upon a new upstream kernel release.

Please retest against this new kernel, as a large number of patches
go into each upstream release, possibly including changes that
may address this problem.

This bug has been placed in NEEDINFO_REPORTER state.
Due to the large volume of inactive bugs in bugzilla, if this bug is
still in this state in two weeks time, it will be closed.

Should this bug still be relevant after this period, the reporter
can reopen the bug at any time. Any other users on the Cc: list
of this bug can request that the bug be reopened by adding a
comment to the bug.

If this bug is a problem preventing you from installing the
release this version is filed against, please see bug 169613.

Thank you.


Comment 8 John Thacker 2006-05-05 13:09:23 UTC
Closing per previous comment.