Description of problem:

Seeing a strange panic when creating a socket, the machine panics with the error 

Unable to handle kernel paging request at 00000000bf3c48e5   

This value is not random, and happens across multiple blades (although they all boot from the same pxe service).

#3 [ffff8103b97ebe10] error_exit at ffffffff8005ede9
   [exception RIP: __d_rehash+24]
   RIP: ffffffff8003a3f8  RSP: ffff8103b97ebec0  RFLAGS: 00010206
   RAX: 00000000bf3cc8dd  RBX: ffff8103e4adfdf8  RCX: 0000000000000015
   RDX: ffff8103e4adfe10  RSI: ffff810001c88708  RDI: ffff8103e4adfdf8
   RBP: ffff81042e544dc0   R8: 00000000ffffffff   R9: 0000000000000020
   R10: 0000000000000000  R11: ffffffff80128780  R12: ffff8103e4a865c0
   R13: ffff8103e4a86610  R14: 00007ffff012df50  R15: 00002b843ca17d51
   ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
#4 [ffff8103b97ebec0] d_rehash at ffffffff80042367
#5 [ffff8103b97ebed0] sock_attach_fd at ffffffff8022718e
#6 [ffff8103b97ebf30] sock_map_fd at ffffffff8004d2be
#7 [ffff8103b97ebf60] sys_socket at ffffffff802272e9
#8 [ffff8103b97ebf80] system_call at ffffffff8005e116
   RIP: 00002b843f551b97  RSP: 00007ffff012dd40  RFLAGS: 00010246
   RAX: 0000000000000029  RBX: ffffffff8005e116  RCX: 00626546006e614a
   RDX: 0000000000000000  RSI: 0000000000000002  RDI: 0000000000000001
   RBP: 0000000000000014   R8: 0000000000000070   R9: 7220726f66207965
   R10: 6d6f726620746f6f  R11: 0000000000000246  R12: ffff8103e4a865c0


-------

When we look at the disassembly of __d_rehash and its inline friends we can  see that the value is stored in the rax.

crash> dis -l __d_rehash
/usr/src/debug/kernel-2.6.18/linux-2.6.18.x86_64/fs/dcache.c: 1525
0xffffffff8003a3e0 <__d_rehash>:        andl   $0xffffffffffffffef,0x4(%rdi)
0xffffffff8003a3e4 <__d_rehash+4>:      lea    0x18(%rdi),%rdx
/usr/src/debug/kernel-2.6.18/linux-2.6.18.x86_64/include/linux/list.h: 866
0xffffffff8003a3e8 <__d_rehash+8>:      mov    (%rsi),%rax
/usr/src/debug/kernel-2.6.18/linux-2.6.18.x86_64/include/linux/list.h: 868
0xffffffff8003a3eb <__d_rehash+11>:     mov    %rsi,0x8(%rdx)
/usr/src/debug/kernel-2.6.18/linux-2.6.18.x86_64/include/linux/list.h: 867
0xffffffff8003a3ef <__d_rehash+15>:     mov    %rax,0x18(%rdi)
/usr/src/debug/kernel-2.6.18/linux-2.6.18.x86_64/include/linux/list.h: 870
0xffffffff8003a3f3 <__d_rehash+19>:     test   %rax,%rax
0xffffffff8003a3f6 <__d_rehash+22>:     je     0xffffffff8003a3fc <__d_rehash+28>
/usr/src/debug/kernel-2.6.18/linux-2.6.18.x86_64/include/linux/list.h: 871
0xffffffff8003a3f8 <__d_rehash+24>:     mov    %rdx,0x8(%rax) <-- We're looking at what is in RAX.

static void __d_rehash(struct dentry * entry, struct hlist_head *list)
{

       entry->d_flags &= ~DCACHE_UNHASHED;
       hlist_add_head_rcu(&entry->d_hash, list); <-- this is inline.
}

static inline void hlist_add_head_rcu(struct hlist_node *n,
                                       struct hlist_head *h)
{
       struct hlist_node *first = h->first;
       n->next = first;
       n->pprev = &h->first;
       smp_wmb();
       if (first) <-  test   %rax,%rax
               first->pprev = &n->next; <-  mov    %rdx,0x8(%rax)
       h->first = n;
}

So essentially we're looking at the %rax, which is ( 00000000bf3cc8d )

struct hlist_node ffff8103e4adfe10 <-- rdx

struct hlist_node {
 next = 0xbf3cc8dd,  ( pointer )
 pprev = 0xffff810001c88708 (** pointer to a pointer)
}

It looks as though the value pointed to by the next pointer is incorrect (in the exception above).  

This same value appears in multiple panics, I can't figure out how it appears to over-write the next value.

This is not the first socket, but may happens after some time.


Version-Release number of selected component (if applicable):




How reproducible:


Steps to Reproduce:
1. Install RHEL 5.5 on a 2 cpu westmere system (12 cores in total).
2. Login
3. run "while [ 1 ] ;do ssh localhost ls ;done"
  
Actual results:

Manchine panics with "Unable to handle kernel paging request at 00000000bf3c48e5"

We have a vmcore available for inspection if required.

Expected results:

System to continue running as per normal.

Additional info:

I am unable to reproduce this issue, and don't know what can be causing the corruption of the next value.  

Customer believes that creating many files can cause the same problem.  I tend to believe as it seems to exercise the same code.