From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1)
Gecko/20030225

Description of problem:
We have been having some problems with the synchronization of the
processors as machines crash. One of our kernel developers spent
considerable time analyzing why this was happening and came up with the
following patch. These go a long way to fixing those problems. We are
pretty certain that they solve all the problems we have observed. This
is not to say that it solves all problems in synchronizing processors as
a machine crashes but it solves all the problems we have seen thus far.

The three problems we have seen are:
1) one CPU just so happens to be in an interrupt handler with interrupts
disabled at the time of the crash.
2) The NMI watchdog interrupts the crash process.
3) We have seen a couple problems where we hung in printk and Dave
believes that this was due to the panic code hitting tiny races in the
die/panic kernel path. Since we closed the size of the race in die we
hadn't seen these.

We have had a variation of this patch running in production for quite a
while and it has been reasonably well tested.

I will open a couple of bugzilla entries to justify these fixes. You can
forward comments regarding this patch through me if you would like or
you can converse with Dave Peterson <dsp> directly. Dave is
currently working on porting netdump to ia64.
From: 	Dave Peterson <dsp>
To: 	Ben Woodard <bwoodard>
Cc: 	Jim Garlick <garlick>
Subject: 	patch with netdump fixes
Date: 	Thu, 08 Jan 2004 21:02:50 -0800	
Ben,

Attached is a patch for the rh-2_4_21-7EL kernel that
fixes some netdump-related problems.  This is a substantial
rewrite of my previous netdump fixes that will hopefully
be a bit simpler, more robust, and easy to port as netdump
gets ported to platforms other than x86.  It should also
improve the reliability of the shutdown code path even if
netdump is not enabled.  Here is a summary of my changes:

    - The current version of netdump is vulnerable to the
      following failure: CPU A crashes and calls
      smp_call_function() to shut down the other CPUs.
      However, CPU B is hung somewhere, spinning with
      interrupts disabled.  Therefore, B never responds to
      the cross-interrupt and A hangs inside
      smp_call_function().

      The patch fixes this problem by implementing the
      following behavior: CPU A attempts to zap the other
      CPUs in order to get them to shut down.  If they do
      not all respond within a certain timeout, CPU A zaps
      the unresponsive CPUs with a nonmaskable interrupt.
      If any of the remaining CPUs fail to respond to the
      NMI within a certain timeout, CPU A gives up and
      continues executing.

    - The current version of netdump does not shut down
      the NMI watchdog before starting to dump.  The
      watchdog can therefore mess up netdump while it is
      sending the dump.  To fix this, the patch adds some
      code that stops the watchdog individually on each
      CPU.

    - I have modified the shutdown code so that when a
      panic or BUG() occurs, the crashing CPU immediately
      stops other CPUs before doing anything else.  This
      improves the reliability of the shutdown code path
      because fewer things can go wrong once other CPUs
      have been shut down.  Stopping other CPUs as early
      as possible eliminates SMP issues from the rest of
      the code, allowing for greater simplicity.

      Also I fixed things so that the crashing CPU doesn't
      call printk() until it has stopped all other CPUs
      and then called bust_spinlocks().  This prevents
      another CPU from grabbing a lock and crashing after
      bust_spinlocks() has been called, causing the system
      to hang inside printk().  The patch also modifies the
      NMI handler so that when the watchdog bites, it
      avoids calling printk() until other CPUs have been
      shut down and bust_spinlocks() has been called.

Please take a look at the patch, and send it to redhat if
you think they may find it useful.


Thanks,

Dave


Version-Release number of selected component (if applicable):
kernel-2.4.21-7EL

How reproducible:
Sometimes

Steps to Reproduce:
These are race conditions that are sometimes hard to hit.
1. die while another CPU is in an ISR while with interrupts disabled

1. Take so long to do a netdump that the watchdog NMI triggers.

1. Very hard to hit race in die() with printk.
    

Actual Results:  hangs

Expected Results:  netdumps

Additional info: