Bug 259501 - BUG in nf_conntrack_find -- BUG: unable to handle kernel paging request at virtual address 554161dc
Summary: BUG in nf_conntrack_find -- BUG: unable to handle kernel paging request at vi...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: kernel
Version: 7
Hardware: i686
OS: All
medium
urgent
Target Milestone: ---
Assignee: Kernel Maintainer List
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2007-08-28 09:48 UTC by Benny Amorsen
Modified: 2008-05-15 09:31 UTC (History)
2 users (show)

Fixed In Version: 2.6.23.14-115.fc8
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2008-05-15 09:31:41 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
Backtrace taken from serial console (2.99 KB, text/plain)
2007-08-28 09:54 UTC, Benny Amorsen
no flags Details
kernel dump screen shot (724.25 KB, image/pjpeg)
2007-11-11 08:42 UTC, Zoltan Arpadffy
no flags Details
gate crash through netconsole. gate runs 2.6.23.1-21.fc7 (21.38 KB, text/plain)
2007-11-13 23:01 UTC, Zoltan Arpadffy
no flags Details
gate sysctl -A output while it runs 2.6.21-1.3194.fc7 (thought it would be useful as well) (33.12 KB, text/plain)
2007-11-13 23:03 UTC, Zoltan Arpadffy
no flags Details

Description Benny Amorsen 2007-08-28 09:48:22 UTC
Description of problem:
BUG: unable to handle kernel paging request at virtual address 554161dc
 printing eip:
f8b62480
*pde = 00000000
Oops: 0000 [#1]
SMP
last sysfs file: /devices/pci0000:00/0000:00:00.0/irq
Modules linked in: ipmi_devintf(U) ipmi_si(U) ipmi_msghandler(U)
xfrm4_mode_tunnel(U) deflate(U) zlib_deflate(U) twofish(U) twofish_common(U)
camellia(U) serpent(U) blowfish(U) cbc(U)
ecb(U) blkcipher(U) xcbc(U) crypto_null(U) xfrm4_tunnel(U) tunnel4(U) ipcomp(U)
esp4(U) ah4(U) aes(U) des(U) sha256(U) af_key(U) tun(U) bridge(U) 8021q(U)
dummy(U) bonding(U) ipv6(U) i
pt_REJECT(U) ipt_LOG(U) xt_policy(U) xt_TCPMSS(U) ipt_recent(U) xt_tcpudp(U)
nf_conntrack_ipv4(U) xt_state(U) nf_conntrack(U) nfnetlink(U) iptable_filter(U)
ip_tables(U) x_tables(U) dm
_mirror(U) dm_mod(U) video(U) sbs(U) button(U) dock(U) battery(U) ac(U) e1000(U)
iTCO_wdt(U) iTCO_vendor_support(U) i2c_i801(U) tg3(U) i2c_core(U) e752x_edac(U)
edac_mc(U) sr_mod(U) cd
rom(U) rtc_cmos(U) sg(U) ata_generic(U) ata_piix(U) libata(U) sd_mod(U)
scsi_mod(U) raid456(U) xor(U) raid1(U) ext3(U) jbd(U) mbcache(U) ehci_hcd(U)
ohci_hcd(U) uhci_hcd(U)
CPU:    1
EIP:    0060:[<f8b62480>]    Not tainted VLI
EFLAGS: 00010202   (2.6.22.4-65.fc7 #1)
EIP is at __nf_conntrack_find+0xc6/0xe4 [nf_conntrack]
eax: c076b300   ebx: c0772eb8   ecx: 554161dc   edx: 018b0000
esi: 0000ac80   edi: 00000000   ebp: f682c600   esp: c0772e44
ds: 007b   es: 007b   fs: 00d8  gs: 0000  ss: 0068
Process swapper (pid: 0, ti=c0772000 task=f7cc2c00 task.ti=f7ce7000)
Stack: 00000000 c0772eb8 00000002 f8b62a81 c0772eb8 f8b6fc40 f8b63589 00000002
       00000006 c0772eb8 f89fc620 f8b6fc40 c0772f70 00000000 00000002 f89fc620
       f8b6fc40 00000001 00000014 f682c600 f1dd4600 c05bff4d f6ebb800 00000000
Call Trace:
 [<f8b62a81>] nf_conntrack_find_get+0x19/0x3f [nf_conntrack]
 [<f8b63589>] nf_conntrack_in+0x161/0x498 [nf_conntrack]
 [<c05bff4d>] nf_hook_slow+0x4d/0xb5
 [<c05c5051>] ip_local_deliver_finish+0x0/0x1b3
 [<c05bfe10>] nf_iterate+0x38/0x6a
 [<c05c4dbc>] ip_rcv_finish+0x0/0x295
 [<c05bff4d>] nf_hook_slow+0x4d/0xb5
 [<c05c4dbc>] ip_rcv_finish+0x0/0x295
 [<c05c54cd>] ip_rcv+0x20b/0x4be
 [<c05c4dbc>] ip_rcv_finish+0x0/0x295
 [<c05aa6ea>] netif_receive_skb+0x2b5/0x316
 [<c05ac2ac>] process_backlog+0x80/0xce
 [<c05ac457>] net_rx_action+0x91/0x178
 [<c042bf89>] __do_softirq+0x5d/0xc1
 [<c04070f3>] do_softirq+0x59/0xb1
 [<c043dfb4>] tick_do_update_jiffies64+0x15/0xa8
 [<c0439d50>] ktime_get+0xf/0x2b
 [<c0455391>] handle_fasteoi_irq+0x0/0x98
 [<c042be48>] irq_exit+0x38/0x6b
 [<c0407208>] do_IRQ+0xbd/0xd1
 [<c040592b>] common_interrupt+0x23/0x28
 [<c0403281>] mwait_idle_with_hints+0x3b/0x3f
 [<c0403285>] mwait_idle+0x0/0xa


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:

Comment 1 Benny Amorsen 2007-08-28 09:52:49 UTC
Sorry I didn't finish the report before submitting.

I have attached the full backtrace now.

I am not quite sure which kernel was the first to exhibit this problem, since it
has been hard to get backtraces. I do believe that it was introduced in the
first 2.6.22. The crashes happen regularly, usually once every couple of days.

Comment 2 Benny Amorsen 2007-08-28 09:54:00 UTC
Created attachment 175801 [details]
Backtrace taken from serial console

Comment 3 Benny Amorsen 2007-08-28 09:57:50 UTC
Oh duh, the version: kernel-2.6.22.4-65.fc7

Comment 4 Chuck Ebbert 2007-08-28 22:33:16 UTC
We didn't supply this kernel binary and therefore can't debug the oops.


Comment 5 Zoltan Arpadffy 2007-11-11 06:45:07 UTC
Hello,

I can confirm the bug.
I do upgrade the kernel on regular basis and it started to behave from 3rd of 
November.

Kernels that crash for sure are:
2.6.23.1-10.fc7
2.6.23.1-21.fc7

Actually the reason for crash is that nf_conntrack can not handle the memory 
allocation around nf_conntrack_max value.

I run a rather busy server and my box crashes every hour.

Other issue is (and this should be a separate bug I guess) that since 
nf_conntrack replaced ip_conntrack - nf_conntrack related sysctl parameters are 
not respected - means that allocated buckets falls back to default that is 
actually to low for a busy NAT server - this combined with memory allocation 
issues around nf_conntrack_max boundary is FATAL.

Any fix/workaround or urgent action is desired.
To be honest I would recommend to raise the priority to high - as these kernels 
are not functional at all with NAT functionality.

Workaround - use some earlier kernel - my choice was 2.6.21-1.3194.fc7

Thank you in advance.

Regards,
Z
     


Comment 6 Zoltan Arpadffy 2007-11-11 08:42:39 UTC
Created attachment 254281 [details]
kernel dump screen shot

Comment 7 Chuck Ebbert 2007-11-12 22:32:08 UTC
(In reply to comment #5)
> Other issue is (and this should be a separate bug I guess) that since 
> nf_conntrack replaced ip_conntrack - nf_conntrack related sysctl parameters are 
> not respected - means that allocated buckets falls back to default that is 
> actually to low for a busy NAT server 

Isn't this /sys/module/nf_conntrack/parameters/hashsize now?


Comment 8 Chuck Ebbert 2007-11-12 22:51:16 UTC
We need to see the entire oops message, so either boot with vga=791 or capture
it on a serial console or netconsole. (And turn off the flash if using a camera.)

Comment 9 Zoltan Arpadffy 2007-11-13 06:44:17 UTC
Hello, 

I will try to reproduce the required log during next days, unfortunately the 
environment where this error occur is a production environment 
(firewall/gateway server) of polarhome.com and more than 100000k users are 
affected when kernel crashes.
Last week the whole system was more down than up and the users are rather 
unsatisfied, therefore I haven't had much time for analysing the kernel crash 
but the focus was to bring back to gateway online as fast is possible :(

Another solution could be if I send you over my sysctl parameters, iptable 
related scripts and if you use the named kernels under heavy NAT related 
traffic you should be able to produce the crash in your lab environment.

BTW regarding conntrack parameters.
Unfortunately there are not so many FAQs or documentation how to move from ip_* 
to nf_* as I understand it was really made an effort to make this migration 
transparent and netfilter should be real runtime changeable.
hashtable changes in runtime, indeed, but does not help regarding this issue.

Here are the changes where I tried to configure netfilter: 
net.ipv4.ip_conntrack_max = 262144 is replaced with net.nf_conntrack_max = 
262144
# raise the bucket size = max/4  - no sysctl parameter?
echo 65536   > /sys/module/nf_conntrack/parameters/hashsize

Thank you in advance.

Regard,
Z



Comment 10 Chuck Ebbert 2007-11-13 19:41:18 UTC
(In reply to comment #9)
> net.ipv4.ip_conntrack_max = 262144 is replaced with net.nf_conntrack_max = 
> 262144

net.netfilter.nf_conntrack_max ??
Looks like it is in both places actually, and changing one changes both...

> # raise the bucket size = max/4  - no sysctl parameter?
> echo 65536   > /sys/module/nf_conntrack/parameters/hashsize
> 

net.netfilter.nf_conntrack_buckets ??

sysctl names are all viewable in /proc/sys/:

/proc/sys/net/netfilter/nf_conntrack_buckets ==
net.netfilter.nf_conntrack_buckets 


Comment 11 Zoltan Arpadffy 2007-11-13 23:01:53 UTC
Created attachment 257501 [details]
gate crash through netconsole. gate runs 2.6.23.1-21.fc7

Comment 12 Zoltan Arpadffy 2007-11-13 23:03:40 UTC
Created attachment 257511 [details]
gate sysctl -A output while it runs 2.6.21-1.3194.fc7 (thought it would be useful as well)

Comment 13 Zoltan Arpadffy 2007-11-13 23:04:55 UTC
Hello,

you have right... I have missed nf_conntrack_buckets but as I wrote:
>Unfortunately there are not so many FAQs or documentation 
>how to move from ip_* to nf_*

Regardless, I have been able to produce two crashes for you with netconsole 
this afternoon. This might spot the problem.

I created two attachments:
- gate crash through netconsole. gate runs 2.6.23.1-21.fc7 
- gate sysctl -A output while it runs 2.6.21-1.3194.fc7 (thought it would be 
useful as well)

with 2.6.21-1.3194.fc7 is stable while under 2.6.23.1-21.fc7 it crashes.
The boot scripts, sysconfig and all other configuration is exactly the same.

Thank you.

Regards,
Z


Comment 14 Chuck Ebbert 2007-11-13 23:39:24 UTC
BUG: unable to handle kernel NULL pointer dereference at virtual address 0000064 
 printing eip: f8ae1087 
*pde = 39644067
Oops: 0000 [#1] 
SMP 
CPU:    1 
EIP:    0060:[<f8ae1087>]    Not tainted VLI 
EFLAGS: 00010202   (2.6.23.1-21.fc7 #1) 
EIP is at nf_nat_move_storage+0x23/0x69 [nf_nat] 
eax: 00000004   ebx: d73a1bc4   ecx: f8ae1064   edx: d73a1bc0 
esi: d73a1bc0   edi: 00000000   ebp: dfcb6b40   esp: c0786c74 
ds: 007b   es: 007b   fs: 00d8  gs: 0000  ss: 0068 
Process swapper (pid: 0, ti=c0786000 task=f7d02c20 task.ti=c18fd000)
Stack:d657fd80 00000001 00000000 f8b61643 00000000 0000004c 00000028 00000000 
00000000 f8b81260 dfcb6b40 f8c0af20 f8b5f7a5 f8b5dd73 c0786cd8 f8ae4180 
130aa8c0 f8ae19dd dfcb6b40 f34b6000 dfcb6b40 00000000 00000001 c0786d14 
Call Trace: 
 [<f8b61643>] __nf_ct_ext_add+0x12f/0x1c4 [nf_conntrack] 
 [<f8b5f7a5>] nf_ct_helper_ext_add+0x9/0x15 [nf_conntrack] 
 [<f8b5dd73>] nf_conntrack_alter_reply+0x73/0x96 [nf_conntrack] 
 [<f8ae19dd>] nf_nat_setup_info+0x3f3/0x54e [nf_nat] 
 [<f8b8d0ea>] ipt_dnat_target+0x0/0x14c [iptable_nat] 
 [<f8b8d22e>] ipt_dnat_target+0x144/0x14c [iptable_nat] 
 [<f8b610fd>] tcp_packet+0xa1c/0xa4b [nf_conntrack] 
 [<c05b4025>] skb_checksum+0x4f/0x29a 
 [<f8b8d0ea>] ipt_dnat_target+0x0/0x14c [iptable_nat] 
 [<f8ae859e>] ipt_do_table+0x3f0/0x482 [ip_tables] 
 [<f8b5dca8>] nf_conntrack_alloc+0x16d/0x1c5 [nf_conntrack] 
 [<f8b603d6>] tcp_new+0xd1/0x1a4 [nf_conntrack] 
 [<f8ae85d3>] ipt_do_table+0x425/0x482 [ip_tables] 
 [<f8b8d257>] nf_nat_rule_find+0x21/0x5c [iptable_nat] 
 [<f8b8d40d>] nf_nat_fn+0x165/0x189 [iptable_nat] 
 [<f8b8d48e>] nf_nat_in+0x29/0x9c [iptable_nat] 
 [<c05d5a9c>] ip_rcv_finish+0x0/0x291 
 [<c05d0ae4>] nf_iterate+0x38/0x6a 
 [<c05d5a9c>] ip_rcv_finish+0x0/0x291 
 [<c05d0c4f>] nf_hook_slow+0x4d/0xb5 
 [<c05d5a9c>] ip_rcv_finish+0x0/0x291 
 [<c05d61a9>] ip_rcv+0x20b/0x4ba 
 [<c05d5a9c>] ip_rcv_finish+0x0/0x291 
 [<c04400dc>] ktime_get_real+0xf/0x2b 
 [<c05b96a8>] netif_receive_skb+0x2e1/0x346 
 [<c04264a0>] __wake_up+0x32/0x43 
 [<f8953358>] e100_poll+0x166/0x2b5 [e100] 
 [<c047cc54>] __slab_free+0x5c/0x216 
 [<c05bb7ec>] net_rx_action+0x9a/0x196 
 [<c0431dfa>] __do_softirq+0x66/0xd3 
 [<c04073e1>] do_softirq+0x6c/0xce 
 [<c04445dd>] tick_do_update_jiffies64+0x93/0xa8 
 [<c045b6b5>] handle_fasteoi_irq+0x0/0xa6 
 [<c0431cbd>] irq_exit+0x38/0x6b 
 [<c04074e2>] do_IRQ+0x9f/0xb9 
 [<c0403ddf>] default_idle+0x0/0x55 
 [<c0405b6f>] common_interrupt+0x23/0x28 
 [<c0403ddf>] default_idle+0x0/0x55 
 [<c042007b>] save_v86_state+0x19/0x12b 
 [<c0421f64>] native_safe_halt+0x2/0x3 
 [<c0403e18>] default_idle+0x39/0x55 
 [<c040340b>] cpu_idle+0xab/0xcc 
 ======================= 
Code: 64 0f fe ff ff 31 c0 c3 57 56 89 d6 53 8b 90 ec 00 00 00 85 d2 74 0f 8a 42
01 84 c0 74 08 0f b6 c0 8d 1c 02 eb 02 31 db 8b 7e 18 <f7> 47 64 80 01 00 00 74
39 b8 40 41 ae f8 e8 fd b7 b3 c7 8b 16 


Comment 15 Chuck Ebbert 2007-11-14 17:59:45 UTC
nf_nat_move_storage():
/usr/src/debug/kernel-2.6.23/linux-2.6.23.i686/net/ipv4/netfilter/nf_nat_core.c:612
      87:       f7 47 64 80 01 00 00    testl  $0x180,0x64(%edi)
      8e:       74 39                   je     c9 <nf_nat_move_storage+0x65>

line 612:
        if (!(ct->status & IPS_NAT_DONE_MASK))
                return;

ct is NULL

Comment 16 Chuck Ebbert 2007-11-16 18:44:47 UTC
Fix:
Commit:     7799652557d966e49512479f4d3b9079bbc01fff
[NETFILTER]: Fix NULL pointer dereference in nf_nat_move_storage()


Comment 17 Zoltan Arpadffy 2007-11-16 19:04:51 UTC
Chuck,

thank you very much for the fix.

As I do not have experience with Fedora kernel fixes could you please describe 
in few words what are the next steps and how can I be sure that this fix is 
included into the kernel what I am about to upgrade in the future.

Also if you are kind to point me to some page/document where the netfilter ip_* 
to nf_* function/kernel modules migration is described - especially the sysctl 
parameters.

Thank you in advance.

Regards,
Z    

Comment 18 Chuck Ebbert 2007-11-16 19:50:52 UTC
(In reply to comment #17)
> Chuck,
> 
> thank you very much for the fix.
> 
> As I do not have experience with Fedora kernel fixes could you please describe 
> in few words what are the next steps and how can I be sure that this fix is 
> included into the kernel what I am about to upgrade in the future.
> 

You could build your own kernel with that fix, documentation is at:

  http://fedoraproject.org/wiki/Docs/CustomKernel

Otherwise it will be in a future update and you can check the changelogs to see
if the fix is in there.


> Also if you are kind to point me to some page/document where the netfilter ip_* 
> to nf_* function/kernel modules migration is described - especially the sysctl 
> parameters.
> 

I can't find any official docs at all. But googling for nf_conntrack_buckets
does find some discussions.


Comment 19 Fedora Update System 2007-11-26 18:50:54 UTC
kernel-2.6.23.8-34.fc7 has been pushed to the Fedora 7 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update kernel'

Comment 20 Zoltan Arpadffy 2007-11-28 08:13:27 UTC
Hello,

I regret, but field test shows that the problem still persists.

Yesterday I upgraded to the new kernel (kernel-2.6.23.8-34.fc7), and it crashed 
after few hours - now even failed to create netconsole dump. :(

Regards,
Z

Comment 21 Fedora Update System 2007-12-06 20:54:09 UTC
kernel-2.6.23.8-34.fc7 has been pushed to the Fedora 7 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 22 Zoltan Arpadffy 2008-01-21 13:53:42 UTC
Hello,

I am terribly sorry, but the bug persists as I noted on 2007-11-28 - just I was 
not able to produce a netconsole output of it for some reason.

I have tried again and now I am able to provide a netconsole output.

Jan 21 06:34:06 gate BUG unable to handle kernel NULL pointer dereference
Jan 21 06:34:06  at virtual address 00000064
Jan 21 06:34:06 printing eip: f8aa1087
Jan 21 06:34:06 *pde = 1592c067
Jan 21 06:34:06 *pte = 00000000
Jan 21 06:34:06 gate
Jan 21 06:34:06 gate Oops 0000 [#1]
Jan 21 06:34:06 gate SMP
Jan 21 06:34:06 gate
Jan 21 06:34:06 Modules linked in:
Jan 21 06:34:06  netconsole
Jan 21 06:34:06  iptable_mangle
Jan 21 06:34:06  sit
Jan 21 06:34:06  tunnel4
Jan 21 06:34:06  xt_state
Jan 21 06:34:06  ipt_LOG
Jan 21 06:34:06  xt_tcpudp
Jan 21 06:34:06  iptable_filter
Jan 21 06:34:06  nf_nat_irc
Jan 21 06:34:06  nf_nat_ftp
Jan 21 06:34:06  iptable_nat
Jan 21 06:34:06  nf_nat
Jan 21 06:34:06  nf_conntrack_irc
Jan 21 06:34:06  nf_conntrack_ftp
Jan 21 06:34:06  nf_conntrack_ipv4
Jan 21 06:34:06  nf_conntrack
Jan 21 06:34:06  nfnetlink
Jan 21 06:34:06  ip_tables
Jan 21 06:34:06  x_tables
Jan 21 06:34:06  nfsd
Jan 21 06:34:06  exportfs
Jan 21 06:34:06  lockd
Jan 21 06:34:06  nfs_acl
Jan 21 06:34:06  auth_rpcgss
Jan 21 06:34:06  autofs4
Jan 21 06:34:06  sunrpc
Jan 21 06:34:06  ipv6
Jan 21 06:34:06  dm_mirror
Jan 21 06:34:06  dm_multipath
Jan 21 06:34:06  dm_mod
Jan 21 06:34:06  snd_intel8x0
Jan 21 06:34:06  snd_ac97_codec
Jan 21 06:34:06  ac97_bus
Jan 21 06:34:06  snd_seq_dummy
Jan 21 06:34:06  snd_seq_oss
Jan 21 06:34:06  snd_seq_midi_event
Jan 21 06:34:06  snd_seq
Jan 21 06:34:06  snd_seq_device
Jan 21 06:34:06  snd_pcm_oss
Jan 21 06:34:06  snd_mixer_oss
Jan 21 06:34:06  snd_pcm
Jan 21 06:34:06  firewire_ohci
Jan 21 06:34:06  snd_timer
Jan 21 06:34:06  firewire_core
Jan 21 06:34:06  snd
Jan 21 06:34:06  3c59x
Jan 21 06:34:06  e100
Jan 21 06:34:06  r8169
Jan 21 06:34:06  button
Jan 21 06:34:06  crc_itu_t
Jan 21 06:34:06  parport_pc
Jan 21 06:34:06  i2c_i801
Jan 21 06:34:06  mii
Jan 21 06:34:06  soundcore
Jan 21 06:34:06  parport
Jan 21 06:34:06  snd_page_alloc
Jan 21 06:34:06  iTCO_wdt
Jan 21 06:34:06  i2c_core
Jan 21 06:34:06  pcspkr
Jan 21 06:34:06  iTCO_vendor_support
Jan 21 06:34:06  intel_rng
Jan 21 06:34:06  dcdbas
Jan 21 06:34:06  sr_mod
Jan 21 06:34:06  cdrom
Jan 21 06:34:06  sg
Jan 21 06:34:06  floppy
Jan 21 06:34:06  ata_piix
Jan 21 06:34:06  ata_generic
Jan 21 06:34:06  libata
Jan 21 06:34:06  sd_mod
Jan 21 06:34:06  scsi_mod
Jan 21 06:34:06  ext3
Jan 21 06:34:06  jbd
Jan 21 06:34:06  mbcache
Jan 21 06:34:06  uhci_hcd
Jan 21 06:34:06  ohci_hcd
Jan 21 06:34:06  ehci_hcd
Jan 21 06:34:06 gate
Jan 21 06:34:06 gate CPU    0
Jan 21 06:34:06 gate EIP    0060:[<f8aa1087>]    Not tainted VLI
Jan 21 06:34:06 gate EFLAGS 00010202   (2.6.23.9-85.fc8 #1)
Jan 21 06:34:06 EIP is at nf_nat_move_storage+0x23/0x69 [nf_nat]
Jan 21 06:34:06 gate eax 00000004   ebx: f43c02c4   ecx: f43c02c0   edx: 
f43c02c0
Jan 21 06:34:06 gate esi f43c02d0   edi: 00000000   ebp: dd1cf1e0   esp: 
c0788c74
Jan 21 06:34:06 gate ds 007b   es: 007b   fs: 00d8  gs: 0000  ss: 0068
Jan 21 06:34:06 Process swapper (pid: 0, ti=c0788000 task=c06f7340 
task.ti=c073a000)
Jan 21 06:34:06 gate
Jan 21 06:34:06 gate Stack
Jan 21 06:34:06 gate dd897c00
Jan 21 06:34:06 gate f8aa1064
Jan 21 06:34:06 gate 00000001
Jan 21 06:34:06 gate f8abe5dc
Jan 21 06:34:06 gate 00000000
Jan 21 06:34:06 gate 0000004c
Jan 21 06:34:06 gate 00000028
Jan 21 06:34:06 gate 00000000
Jan 21 06:34:06 gate
Jan 21 06:34:06 gate
Jan 21 06:34:06 gate 00000000
Jan 21 06:34:06 gate f8ade2c0
Jan 21 06:34:06 gate dd1cf1e0
Jan 21 06:34:06 gate f8b651b0
Jan 21 06:34:06 gate f8abc7a5
Jan 21 06:34:06 gate f8abad73
Jan 21 06:34:06 gate c0788cd8
Jan 21 06:34:06 gate f8aa41e0
Jan 21 06:34:06 gate
Jan 21 06:34:06 gate
Jan 21 06:34:06 gate 050aa8c0
Jan 21 06:34:06 gate f8aa19dd
Jan 21 06:34:06 gate dd1cf1e0
Jan 21 06:34:06 gate debfc0f0
Jan 21 06:34:06 gate dd1cf1e0
Jan 21 06:34:06 gate 00000000
Jan 21 06:34:06 gate 00000001
Jan 21 06:34:06 gate c0788d14
Jan 21 06:34:06 gate
Jan 21 06:34:06 Call Trace:
Jan 21 06:34:06  [<f8aa1064>]
Jan 21 06:34:06 gate nf_nat_move_storage+0x0/0x69[nf_nat]
Jan 21 06:34:06  [<f8abe5dc>]
Jan 21 06:34:06 gate __nf_ct_ext_add+0x128/0x1bc[nf_conntrack]
Jan 21 06:34:06  [<f8abc7a5>]
Jan 21 06:34:06 gate nf_ct_helper_ext_add+0x9/0x15[nf_conntrack]
Jan 21 06:34:06  [<f8abad73>]
Jan 21 06:34:06 gate nf_conntrack_alter_reply+0x73/0x96[nf_conntrack]
Jan 21 06:34:06  [<f8aa19dd>]
Jan 21 06:34:06 gate nf_nat_setup_info+0x3f3/0x54e[nf_nat]
Jan 21 06:34:06  [<f8b4c0ea>]
Jan 21 06:34:06 gate ipt_dnat_target+0x0/0x14c[iptable_nat]
Jan 21 06:34:06  [<f8b4c22e>]
Jan 21 06:34:06 gate ipt_dnat_target+0x144/0x14c[iptable_nat]
Jan 21 06:34:06  [<f8abe09d>]
Jan 21 06:34:06 gate tcp_packet+0x9bc/0x9eb[nf_conntrack]
Jan 21 06:34:06  [<c05b6571>]
Jan 21 06:34:06 gate skb_checksum+0x4f/0x29c
Jan 21 06:34:06  [<f8b4c0ea>]
Jan 21 06:34:06 gate ipt_dnat_target+0x0/0x14c[iptable_nat]
Jan 21 06:34:06  [<f8aa859e>]
Jan 21 06:34:06 gate ipt_do_table+0x3f0/0x482[ip_tables]
Jan 21 06:34:06  [<f8abd3d6>]
Jan 21 06:34:06 gate tcp_new+0xd1/0x1a4[nf_conntrack]
Jan 21 06:34:06  [<f8aa85d3>]
Jan 21 06:34:06 gate ipt_do_table+0x425/0x482[ip_tables]
Jan 21 06:34:06  [<f8b4c257>]
Jan 21 06:34:06 gate nf_nat_rule_find+0x21/0x5c[iptable_nat]
Jan 21 06:34:06  [<f8b4c40d>]
Jan 21 06:34:06 gate nf_nat_fn+0x165/0x189[iptable_nat]
Jan 21 06:34:06  [<f8b4c48e>]
Jan 21 06:34:06 gate nf_nat_in+0x29/0x9c[iptable_nat]
Jan 21 06:34:06  [<c05d8038>]
Jan 21 06:34:06 gate ip_rcv_finish+0x0/0x291
Jan 21 06:34:06  [<c05d3080>]
Jan 21 06:34:06 gate nf_iterate+0x38/0x6a
Jan 21 06:34:06  [<c05d8038>]
Jan 21 06:34:06 gate ip_rcv_finish+0x0/0x291
Jan 21 06:34:06  [<c05d31eb>]
Jan 21 06:34:06 gate nf_hook_slow+0x4d/0xb5
Jan 21 06:34:06  [<c05d8038>]
Jan 21 06:34:06 gate ip_rcv_finish+0x0/0x291
Jan 21 06:34:06  [<c05d8745>]
Jan 21 06:34:06 gate ip_rcv+0x20b/0x4ba
Jan 21 06:34:06  [<c05d8038>]
Jan 21 06:34:06 gate ip_rcv_finish+0x0/0x291
Jan 21 06:34:06  [<c04401cc>]
Jan 21 06:34:06 gate ktime_get_real+0xf/0x2b
Jan 21 06:34:06  [<c05bbbfc>]
Jan 21 06:34:06 gate netif_receive_skb+0x2e1/0x346
Jan 21 06:34:06  [<c05b84c9>]
Jan 21 06:34:06 gate __netdev_alloc_skb+0x1c/0x35
Jan 21 06:34:06  [<f893a358>]
Jan 21 06:34:06 gate e100_poll+0x166/0x2b5[e100]
Jan 21 06:34:06  [<c05bdd40>]
Jan 21 06:34:06 gate net_rx_action+0x9a/0x196
Jan 21 06:34:06  [<c0431e02>]
Jan 21 06:34:06 gate __do_softirq+0x66/0xd3
Jan 21 06:34:06  [<c04073d5>]
Jan 21 06:34:06 gate do_softirq+0x6c/0xce
Jan 21 06:34:06  [<c045ba85>]
Jan 21 06:34:06 gate handle_fasteoi_irq+0x0/0xa6
Jan 21 06:34:06  [<c0431cc5>]
Jan 21 06:34:06 gate irq_exit+0x38/0x6b
Jan 21 06:34:06  [<c04074d6>]
Jan 21 06:34:06 gate do_IRQ+0x9f/0xb9
Jan 21 06:34:06  [<c0403ddf>]
Jan 21 06:34:06 gate default_idle+0x0/0x55
Jan 21 06:34:06  [<c0405b6f>]
Jan 21 06:34:06 gate common_interrupt+0x23/0x28
Jan 21 06:34:06  [<c0403ddf>]
Jan 21 06:34:06 gate default_idle+0x0/0x55
Jan 21 06:34:06  [<c042007b>]
Jan 21 06:34:06 gate save_v86_state+0x19/0x12b
Jan 21 06:34:06  [<c0421f78>]
Jan 21 06:34:06 gate native_safe_halt+0x2/0x3
Jan 21 06:34:06  [<c0403e18>]
Jan 21 06:34:06 gate default_idle+0x39/0x55
Jan 21 06:34:06  [<c040340b>]
Jan 21 06:34:06 gate cpu_idle+0xab/0xcc
Jan 21 06:34:06  [<c073fa6c>]
Jan 21 06:34:06 gate start_kernel+0x32c/0x334
Jan 21 06:34:06  [<c073f177>]
Jan 21 06:34:06 gate unknown_bootoption+0x0/0x195
Jan 21 06:34:06  =======================
Jan 21 06:34:06 gate Code
Jan 21 06:34:06 gate 64
Jan 21 06:34:06 gate 0f
Jan 21 06:34:06 gate fe
Jan 21 06:34:06 gate ff
Jan 21 06:34:06 gate ff
Jan 21 06:34:06 gate 31
Jan 21 06:34:06 gate c0
Jan 21 06:34:06 gate c3
Jan 21 06:34:06 gate 57
Jan 21 06:34:06 gate 56
Jan 21 06:34:06 gate 89
Jan 21 06:34:06 gate d6
Jan 21 06:34:06 gate 53
Jan 21 06:34:06 gate 8b
Jan 21 06:34:06 gate 90
Jan 21 06:34:06 gate ec
Jan 21 06:34:06 gate 00
Jan 21 06:34:06 gate 00last message repeated 2 times
Jan 21 06:34:06 gate 85
Jan 21 06:34:06 gate d2
Jan 21 06:34:06 gate 74
Jan 21 06:34:06 gate 0f
Jan 21 06:34:06 gate 8a
Jan 21 06:34:06 gate 42
Jan 21 06:34:06 gate 01
Jan 21 06:34:06 gate 84
Jan 21 06:34:06 gate c0
Jan 21 06:34:06 gate 74
Jan 21 06:34:06 gate 08
Jan 21 06:34:06 gate 0f
Jan 21 06:34:06 gate b6
Jan 21 06:34:06 gate c0
Jan 21 06:34:06 gate 8d
Jan 21 06:34:06 gate 1c
Jan 21 06:34:06 gate 02
Jan 21 06:34:06 gate eb
Jan 21 06:34:06 gate 02
Jan 21 06:34:06 gate 31
Jan 21 06:34:06 gate db
Jan 21 06:34:06 gate 8b
Jan 21 06:34:06 gate 7e
Jan 21 06:34:06 gate 18
Jan 21 06:34:06 gate f7>
Jan 21 06:34:06 gate 47
Jan 21 06:34:06 gate 64
Jan 21 06:34:06 gate 80
Jan 21 06:34:06 gate 01
Jan 21 06:34:06 gate 00
Jan 21 06:34:06 gate 00
Jan 21 06:34:06 gate 74
Jan 21 06:34:06 gate 39
Jan 21 06:34:06 gate b8
Jan 21 06:34:06 gate a0
Jan 21 06:34:06 gate 41
Jan 21 06:34:06 gate aa
Jan 21 06:34:06 gate f8
Jan 21 06:34:06 gate e8
Jan 21 06:34:06 gate 2d
Jan 21 06:34:06 gate de
Jan 21 06:34:06 gate b7
Jan 21 06:34:06 gate c7
Jan 21 06:34:06 gate 8b
Jan 21 06:34:06 gate 16
Jan 21 06:34:06 gate
Jan 21 06:34:06 gate EIP [<f8aa1087>]
Jan 21 06:34:06 gate nf_nat_move_storage+0x23/0x69[nf_nat]
Jan 21 06:34:06  SS:ESP 0068:c0788c74
Jan 21 06:34:06 Kernel panic - not syncing: Fatal exception in interrupt

This is produced with the kernel-2.6.23.9-85.fc8

Any help would be appreciated - 2.6.21-1.3194.fc7 is still the only kernel that 
I can run without crash :(

Thank you in advance.

Regards, 
Z


Comment 23 Zoltan Arpadffy 2008-01-28 10:45:17 UTC
Hello,

I wonder, is this netconsole output that I have sent enough for correcting the 
bug, or should I send a new one with kernel-2.6.23.14-107.fc8 to what I have 
recently upgraded.

Please, reopen this BUG. I can not do it myself, because I am not the owner - 
or should I open a new bug and mark this as depend/related?

Thank you in advance.

Regards, 
Z

Comment 24 Benny Amorsen 2008-01-28 19:21:06 UTC
Looks like I can reopen it... It's worth a shot.

Comment 25 Chuck Ebbert 2008-01-29 17:51:40 UTC
This was fixed in 2.6.23.10 -- a second patch was required to completely fix the
problem. Should be fixed in 2.6.23.14-64 which was released today. Please confirm.

Comment 26 Chuck Ebbert 2008-02-06 18:07:51 UTC
Additional patch was needed, see bug #430663.

Fix is in 2.6.23.14-77.

Comment 27 Zoltan Arpadffy 2008-02-06 20:07:50 UTC
Hello,

thank you very much.
I can confirm that the patch does help.

[root@gate ~]# uname -a
Linux gate.polarhome.com 2.6.23.14-115.fc8 #1 SMP Mon Jan 21 14:20:50 EST 2008 
i686 i686 i386 GNU/Linux
[root@gate ~]# uptime
 21:05:30 up 6 days,  7:14,  1 user,  load average: 1.13, 0.99, 0.85

...earlier it could not produce a 4 hour uptime.

Thank you for fixing this bug.
Keep up the good work.

Best regards,
Z

Comment 28 Bug Zapper 2008-05-14 14:07:44 UTC
This message is a reminder that Fedora 7 is nearing the end of life. Approximately 30 (thirty) days from now Fedora will stop maintaining and issuing updates for Fedora 7. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as WONTFIX if it remains open with a Fedora 'version' of '7'.

Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version prior to Fedora 7's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that we may not be able to fix it before Fedora 7 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora please change the 'version' of this bug. If you are unable to change the version, please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. If possible, it is recommended that you try the newest available Fedora distribution to see if your bug still exists.

Please read the Release Notes for the newest Fedora distribution to make sure it will meet your needs:
http://docs.fedoraproject.org/release-notes/

The process we are following is described here: http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Comment 29 Benny Amorsen 2008-05-15 09:31:41 UTC
I am closing this bug. The problem is fixed for my systems and in comment 27
Zoltan Arpadffy says that the bug is fixed for him as well. Thanks for the help!


Note You need to log in before you can comment on or make changes to this bug.