Bug 163176 - Endless loop printing traceback during kernel OOPs
Endless loop printing traceback during kernel OOPs
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 3
Classification: Red Hat
Component: kernel (Show other bugs)
3.0
All Linux
medium Severity medium
: ---
: ---
Assigned To: Kiersten (Kerri) Anderson
Brian Brock
:
Depends On:
Blocks: 168424
  Show dependency treegraph
 
Reported: 2005-07-13 15:08 EDT by Issue Tracker
Modified: 2007-11-30 17:07 EST (History)
16 users (show)

See Also:
Fixed In Version: RHSA-2006-0144
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2006-03-15 11:13:11 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Issue Tracker 2005-07-13 15:08:39 EDT
Escalated to Bugzilla from IssueTracker
Comment 7 Ernie Petrides 2005-07-14 16:59:16 EDT
Please put some useful information in this bug report.  All that's
indicated so far is that OpenAFS crashes the system during shutdown
(which is not a RHEL problem).  If there's no evidence of a RHEL kernel
bug, then please close this as NOTABUG.
Comment 10 Ernie Petrides 2005-07-18 14:13:05 EDT
KevinK/Guy, I don't know how we can support OpenAFS -- it's not in our
source tree and we don't have anyone here with expertise in it.  I'm
closing this bugzilla report because there's nothing indicating a bug
in RHEL3.  I've also put appropriate Engineering managers on the cc:
list, and if this does get reopened, it will be KevinA's responsibility
to reassign it appropriately.
Comment 15 Larry Woodman 2005-07-29 15:21:22 EDT
I still do not know where the AFS sources are.  Can someone help me locate them?

Larry Woodman
Comment 34 Larry Woodman 2005-08-24 10:47:59 EDT
What we have done so far to help debug this problem and prevent the system from
hanging kernel stack pointer corruption causes a panic it to fix show_trace() so
that it does not hang.  This will provide useful debugging information, allow
the system to take a crash dump and allow it to reboot without manual intervention.

Here is the patch that fixes show_trace() and show_stack():

--------------------------------------------------------------
--- linux-2.4.21/arch/i386/kernel/traps.c.orig
+++ linux-2.4.21/arch/i386/kernel/traps.c
@@ -141,6 +141,7 @@ void show_trace(unsigned long * stack)
        unsigned long addr;
        /* static to not take up stackspace; if we race here too bad */
        static char buffer[512];
+       unsigned long limit;
                                                                               
                
        if (!stack)
                stack = (unsigned long*)&stack;
@@ -163,7 +164,8 @@ void show_trace(unsigned long * stack)
 out:
 #else
        i = 1;
-       while (((long) stack & (THREAD_SIZE-1)) != 0) {
+       limit = ((unsigned long)stack & ~(THREAD_SIZE - 1)) + THREAD_SIZE - 3;
+       while ((unsigned long)stack < limit) {
                addr = *stack++;
                if (kernel_text_address(addr)) {
                        lookup_symbol(addr, buffer, 512);
@@ -189,6 +191,7 @@ void show_stack(unsigned long * esp)
 {
        unsigned long *stack;
        int i;
+       unsigned long limit;
                                                                               
                
        // debugging aid: "show_stack(NULL);" prints the
        // back trace for this cpu.
@@ -197,8 +200,9 @@ void show_stack(unsigned long * esp)
                esp=(unsigned long*)&esp;
                                                                               
                
        stack = esp;
+       limit = ((unsigned long)stack & ~(THREAD_SIZE - 1)) + THREAD_SIZE - 3;
        for(i=0; i < kstack_depth_to_print; i++) {
-               if (((long) stack & (THREAD_SIZE-1)) == 0)
+               if ((unsigned long)stack > limit)
                        break;
                if (i && ((i % 8) == 0))
                        printk("\n       ");
--------------------------------------------------------------------------
Comment 40 David Howells 2005-09-19 08:23:25 EDT
http://www.openafs.org or you can grab it by AFS when IS/IT stop blocking UDP 
packets at the RH firewalls. 
Comment 44 Ernie Petrides 2005-09-23 21:10:49 EDT
Gavin/Adrian, the shortcoming in the oops traceback mechanism (which has
been resolved in -37.2.EL as the fix to bug 165412) *DID NOT CAUSE* the
crash reported in this bugzilla!

It simply led to getting stuck dumping the stack after the crash was caused
for some other reason.

I suspect that the underlying cause of the crash is in AFS, but that has not
yet been proven.  Larry's oops traceback fix might help debug the underlying
cause, but I wish you folks would understand that Larry's fix will not prevent
such crashes.

Another important note is that the exported global symbol "___strtok" is not
actually a function, but rather that very last data symbol.  The EIP for the
crash originally reported in IT 75840 is a bogus address in memory far off
from legitimate kernel text.  Snooping at the first 24 words dumped in the
kernel stack, and using the System.map from -15.9.1.ELhugemem, I've pieced
together what I think is the real top-of-stack:

  02134b0f  __run_timers + 182
  02134922  timer_bh + 98
  0212f6b2  __run_task_queue + 106

This suggests that the underlying cause of the crash is some consumer of the
add_timer/mod_timer/del_timer facility in the kernel, and that the consumer
code (maybe AFS) left a pending timer and then freed the associated timer_list
structure (which contains a pointer to the function to be invoked).  It's
probably the case that an overwritten timer_list struct was then used to
fetch the trashed func pointer for the dispatch, and then __run_timers()
called off into invalid instruction space (causing the crash).
Comment 45 Gavin Romig-Koch 2005-09-26 09:23:49 EDT
Ernie,
I was not clear in my last note.  No one believes that the kernel OOPs traceback
mechinism caused the problem in AFS.

> It simply led to getting stuck dumping the stack after the crash was caused
for some other reason.

According to Adrian, this "getting stuck dumping" _is_ the problem that they
need solved.

The fact that AFS has a problem while shutting down the machine is an annoyance,
but the fact that it gets stuck in the OOPS traceback turns that annoyance into
a major headache.   

Comment 48 Ernie Petrides 2005-09-26 15:57:58 EDT
Gavin/Adrian, in response to comment #45:

Then *please* change the summary of this bug accordingly and close
this as a dup of bug 165412.  Thanks in advance.

If this bug had an appropriate summary or description, then we wouldn't
have wasted so much time on it.
Comment 49 Eido Inoue 2005-09-26 16:44:52 EDT

*** This bug has been marked as a duplicate of 165412 ***
Comment 50 Ernie Petrides 2005-09-26 17:13:41 EDT
A fix for this problem was committed to the RHEL3 U7 patch pool
on 14-Sep-2005 (in kernel version 2.4.21-37.2.EL).

The patch that was committed is functionally equivalent to what's
in bug 165412 comment #51.
Comment 54 Red Hat Bugzilla 2006-03-15 11:13:12 EST
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2006-0144.html

Note You need to log in before you can comment on or make changes to this bug.