Description of problem:
For A 4 CPU system, let's say CPUs 1-3 spin on the endflag variable (that is
located on the CPU0 stack!) waiting until CPU0 sets it to 1. When CPU0 decides
that the NMI got stuck, it sets endflag to 1, logs the event, and returns. What
if CPUs1-3 didn't get a chance to run and the next function called on CPU0
zeroes out the stack location that used to correspond to endflag? This might
result in a hang. This race is more likely to be exposed in a virtualized
environment although it can happen on physical setup too.

Excerpt from linux-2.6.17 (arch/x86_64/kernel/nmi.c):

static __init void nmi_cpu_busy(void *data)
{
volatile int *endflag = data;
local_irq_enable();
/* Intentionally don't use cpu_relax here. This is
   to make sure that the performance counter really ticks,
   even if there is a simulator or similar that catches the
   pause instruction. On a real HT machine this is fine because
   all other CPUs are busy with "useless" delay loops and don't
   care if they get somewhat less cycles. */
while (*endflag == 0)
 barrier();
}
#endif

int __init check_nmi_watchdog (void)
{
volatile int endflag = 0; <------------------
int *counts;
int cpu;

counts = kmalloc(NR_CPUS * sizeof(int), GFP_KERNEL);
if (!counts)
 return -1;

printk(KERN_INFO "testing NMI watchdog ... ");

#ifdef CONFIG_SMP
if (nmi_watchdog == NMI_LOCAL_APIC)
 smp_call_function(nmi_cpu_busy, (void *)&endflag, 0, 0);
#endif

for (cpu = 0; cpu < NR_CPUS; cpu++)
 counts[cpu] = cpu_pda(cpu)->__nmi_count;
local_irq_enable();
mdelay((10*1000)/nmi_hz); // wait 10 ticks

for_each_online_cpu(cpu) {
 if (cpu_pda(cpu)->__nmi_count - counts[cpu] <= 5) {
  endflag = 1; <---------------------------
  printk("CPU#%d: NMI appears to be stuck (%d->%d)!\n",
         cpu,
         counts[cpu],
         cpu_pda(cpu)->__nmi_count);
  nmi_active = 0;
  lapic_nmi_owner &= ~LAPIC_NMI_WATCHDOG;
  nmi_perfctr_msr = 0;
  kfree(counts);
  return -1; <-----------------------
 }
}

This race is verified present in rhel5's 2.6.18-8.el5,
kernel-2.6.18-8.1.15.el5.src.rpm, and kernel-2.6.18-51.el5.jwltest.43 which
reflects ToT for RHEL5 kernels.

I am including my observations in a virtualized environment.

How reproducible/Steps to Reproduce:
It takes about 15 minutes while running repeated boot-halts on 5 copies of
RHEL5.0 VM
  
Actual results: OS hangs.

Expected results: No hang.

Additional info:


Solutions:

Following one-liner change has been verified to fix the problem.

int _init check_nmi_watchdog(void)
{
- volatile int endflag = 0;
+ static volatile int endflag = 0;
int *counts;

Also, the race has been fixed in 2.6.20 kernel. It would be a good idea to apply
the same patch.