We recently received a report of several (2-CPU) nodes on a cluster panicking 
when they were powered off.  The cluster had previously running an application 
called AMBER which probably stressed the system pretty good.

The customer sent us a photograph of the console showing the panic information 
which I will attach shortly.

Basically, the trace shows that the problem occurred at “neigh_destroy+216” 
(linux-2.4.21/net/core/neighbour.c)

Trace:

neigh_destroy+216
dst_destroy+92
dst_run_gc+103

I’ve shown some of the disassembled code below.  Since the trace is probably 
showing what the next instruction would have been, not the instruction being 
currently executed, I assume that we are really interested in 
either “neigh_destroy+209” or “neigh_destroy+211”, which would be the “kfree” 
(shown in the code snippet below).  I’m guessing that the “kfree” is the 
culprit only because if it were the line before, which is “atomic_dec_and_test
(&hh->hh_refcnt)”, then that would imply a bad “hh” and we would have blown up 
a few lines earlier while referencing “hh”.

Relevant code of a disassembled “neigh_destroy”.

0xffffffff80255921 <neigh_destroy+209>: je     0xffffffff80255928 
<neigh_destroy+216>
0xffffffff80255923 <neigh_destroy+211>: callq  0xffffffff8014eb20 <kfree>
0xffffffff80255928 <neigh_destroy+216>: mov    0x68(%rbp),%rdi

Relevant code of a disassembled “dst_destroy”.

0xffffffff80254a34 <dst_destroy+84>:    mov    %rbx,%rdi
0xffffffff80254a37 <dst_destroy+87>:    callq  0xffffffff80255850 
<neigh_destroy>
0xffffffff80254a3c <dst_destroy+92>:    mov    0xc0(%rbp),%rax

Relevant code of a disassembled “dst_run_gc”.

0xffffffff8025471e <dst_run_gc+94>:     mov    %rax,0x0(%rbp)
0xffffffff80254722 <dst_run_gc+98>:     callq  0xffffffff802549e0 <dst_destroy>
0xffffffff80254727 <dst_run_gc+103>:    test   %rax,%rax

FILE: linux-2.4.21/net/core/neighbour.c

void neigh_destroy(struct neighbour *neigh)
{
…
        while ((hh = neigh->hh) != NULL) {
                neigh->hh = hh->hh_next;
                hh->hh_next = NULL;
                write_lock_bh(&hh->hh_lock);
                hh->hh_output = neigh_blackhole;
                write_unlock_bh(&hh->hh_lock);
                if (atomic_dec_and_test(&hh->hh_refcnt))
                        kfree(hh);
        }
…
}

----------------------------------------------------

FILE:  linux-2.4.21/include/net/neighbour.h

static inline void neigh_release(struct neighbour *neigh)
{
        if (atomic_dec_and_test(&neigh->refcnt))
                neigh_destroy(neigh);
}

----------------------------------------------------

FILE:  linux-2.4.21/net/core/dst.c

struct dst_entry *dst_destroy(struct dst_entry * dst)
{
…
        neigh = dst->neighbour;
…
        if (neigh) {
                dst->neighbour = NULL;
                neigh_release(neigh);
        }
…
}

----------------------------------------------------

FILE:  linux-2.4.21/net/core/dst.c

static void dst_run_gc(unsigned long dummy)
{
…
        if (!spin_trylock(&dst_lock)) {
                mod_timer(&dst_gc_timer, jiffies + HZ/10);
                return;
        }
…
        while ((dst = *dstp) != NULL) {
                if (atomic_read(&dst->__refcnt)) {
                        dstp = &dst->next;
                        delayed++;
                        continue;
                }
                *dstp = dst->next;

                dst = dst_destroy(dst);
…
        spin_unlock(&dst_lock);
}

I’m not sure if this implies a locking problem.  It would appear as if 
the “write_lock_bh(&hh->hh_lock)”, or some other type of locking, should have 
taken place prior to the following line???  That is, prior to actually 
using “hh”??

      while ((hh = neigh->hh) != NULL)

These nodes have 2 CPU’s each so I don’t know if this code is well protected 
against multiple kernel threads.  The “dst_run_gc”, which calls “dst_destroy”, 
which calls “neigh_release”, which calls “neigh_destroy”, does appear to set a 
lock, which may be sufficient in this case.

These machines are running Linux 2.4.21-27 and the processor type is EM64T.

I am trying to see if I can get my hands on the AMBER application that the 
customer ran prior to the kernel panic.  As I understand it, the kernel panics 
during shutdown only started to occur after users began running the AMBER 
application.  So, unfortunately, I have not been able to reproduce it yet and 
do not have a reproducer to provide.

Any help in resolving this problem would be appreciated.

Thank you.

Rigoberto