Bug 169301 - Kernel panic due to NMI watchdog timeout in ip while moving service IP addresses
Summary: Kernel panic due to NMI watchdog timeout in ip while moving service IP addresses
Keywords:
Status: CLOSED DUPLICATE of bug 166701
Alias: None
Product: Red Hat Enterprise Linux 4
Classification: Red Hat
Component: kernel
Version: 4.0
Hardware: All
OS: Linux
medium
medium
Target Milestone: ---
: ---
Assignee: Steve Dickson
QA Contact: Cluster QE
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2005-09-26 18:57 UTC by Henry Harris
Modified: 2008-05-09 17:34 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2007-07-20 11:04:34 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Henry Harris 2005-09-26 18:57:18 UTC
Description of problem: While running traffic to a GFS file system over NFS 
and moving the IP address as a rgmanager service, the kernel panics due to an 
NMI watchdog timeout in ip.


Version-Release number of selected component (if applicable):


How reproducible: 


Steps to Reproduce:
1. See bug #166701
2.
3.
  
Actual results:


Expected results:


Additional info:

Comment 1 Henry Harris 2005-09-26 20:08:29 UTC
The following was written a few weeks ago while working with Ben Marzinski to 
troubleshoot bug #166701.  It is the same kernel panic that we are now seeing 
with the U2 code on two different clusters.

***************************

This crash is not clear and required some investigation. Note, this
 is the second time this crash has occurred where the call stack
 was not obvious.

 This crash was caused by a NMI watchdog timeout that called panic after the
 'ip.sh' process seemed to be hung on acquiring a spinlock. I pieced together 
 the ip.sh stack.
 
 __write_lock_failed+7  ----|
.text.lock.spinlock+117 ---|    These lines spinlock waiting for a lock to 
clear/release
do_exit+1310
sys_exit_group+0
sys_exit_group+15
system_call+126

 The system panic'ed do to a NMI watchdog timeout that apparently monitors
 a cpu looking for a context switch to occur. If it does not after much time 
 (> 5 seconds minimum), the system calls panic.

 In dmesg.

...
 <4>NMI Watchdog detected LOCKUP, CPU=1, registers:
â¦

Since this kdb session like yesterday came from the ip.sh process thread and
the stack itself was incomplete, I looked at the current amd64 stack pointer
register and got the watchdog call stack that called panic.

notifier_call_chain+0x1f (0x3000000008, 0x1007f04e88, 0x100e7f04dc8, 0x14, 0x0)
panic+0x100 (0x0, 0x1)
die_nmi+0x81 (0x100e7f04f58, 0xffffffff80328fe, 0x21, 0x200000002, 0x21)
nmi_watchdog_tick+0xd2 (0x0, 0x0, 0x0, 0x0, 0100e7f04f58)
default_do_nmi+0x7a (0x1)
do_nmi+
paranoid_exit()

I looked at the kernel code where nmi_die() is called and right before the call
there is a comment, "Ayiee, looks like this CPU is stuck wait a few IRQs (5 
seconds) before doing the oops â¦"  So, the cpu was hungup in a spinlock
call it looks like.


Comment 4 Jeff Layton 2007-07-20 11:04:34 UTC

*** This bug has been marked as a duplicate of 166701 ***


Note You need to log in before you can comment on or make changes to this bug.