Bug 682129 - [kdump] bt: cannot resolve stack trace:
[kdump] bt: cannot resolve stack trace:
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: crash (Show other bugs)
6.1
Unspecified Unspecified
high Severity high
: rc
: ---
Assigned To: Dave Anderson
Kernel Dump QE
: Regression
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2011-03-04 03:52 EST by Chao Ye
Modified: 2011-05-19 09:04 EDT (History)
3 users (show)

See Also:
Fixed In Version: crash-5.1.1-2.el6
Doc Type: Bug Fix
Doc Text:
Prior to this update, an attempt to display a backtrace of a non-active swapper task on a 32-bit x86 architecture could cause the crash utility to display the following message: bt: cannot resolve stack trace: #0 [c09f1ef4] ia32_sysenter_target at c08208ce This update applies a patch that resolves this issue, and the crash utility now resolves such backtraces as expected. Additionally, this update ensures that the crash utility is no longer negatively affected by the changes that were introduced in kernel 2.6.32-112.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2011-05-19 09:04:14 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Chao Ye 2011-03-04 03:52:44 EST
Description of problem:
When run analyse-crash on ibm-crichton-02.rhts.eng.bos.redhat.com:
======================================================================
crash> foreach bt
PID: 0      TASK: c09fa560  CPU: 0   COMMAND: "swapper"
bt: cannot resolve stack trace:
 #0 [c09f1ef4] ia32_sysenter_target at c08208ce
bt: text symbols on stack:
    [c09f1efc] tick_dev_program_event at c0482ecf
    [c09f1f08] clockevents_program_event at c0481c1c
    [c09f1f48] hrtimer_forward at c0477b33
    [c09f1f80] hrtimer_start_range_ns at c0478e20
    [c09f1f94] tick_nohz_restart_sched_tick at c0483eb5
    [c09f1fb8] cpu_idle at c04089d5
    [c09f1fcc] command_line at c0a980e0
    [c09f1fd4] start_kernel at c0a5c9b9
    [c09f1fdc] unknown_bootoption at c0a5c453
    [c09f1ff4] command_line at c0a980e0
Version-Release number of selected component (if applicable):
kernel-2.6.32-119.el6
crash-5.1.1-1.el6

How reproducible:
100% on ibm-crichton-02.rhts.eng.bos.redhat.com

Steps to Reproduce:
1.Install RHEL6.1-20110224.2
2.Upgrade kernel/kexec-tools
3.Trigger crash, run analyse crash
  
Actual results:


Expected results:


Additional info:
System reserved:
https://beaker.engineering.redhat.com/recipes/119278
Comment 2 Dave Anderson 2011-03-04 08:32:49 EST
In the future, please always *save* the vmlinux/vmcore pair before 
filing a bugzilla against a crash/dumpfile issue.
Comment 3 Dave Anderson 2011-03-04 10:00:58 EST
(In reply to comment #2)
> In the future, please always *save* the vmlinux/vmcore pair before 
> filing a bugzilla against a crash/dumpfile issue.

I logged onto that system, and saw that /var/crash was empty. 
So I crashed it, and noticed that /var/crash was still empty.
Only then did I see that the kdump.conf was configured
to send the dumps to hp-xw6400-02.lab.bos.redhat.com.

It would have been helpful if you had indicated that information
when you filed the bugzilla...

Anyway, this is certainly a new i386 backtrace bug that pretty
much is affecting all tasks, i.e., not just the ones that
display the "cannot resolve" message.  The backtraces are
all incorrect, and show invalid kernel-entry exception frames:

crash> foreach user bt
PID: 1      TASK: f7055ab0  CPU: 1   COMMAND: "init"
 #0 [f705da2c] ia32_sysenter_target at c08208ce
 #1 [f705daf0] ia32_sysenter_target at c0821ea0
    EAX: 00000001  EBX: 00000000  ECX: c042664f  EDX: 00000000 
    DS:  75a0      ESI: c1e0412c  ES:  0000      EDI: 00000082
    SS:  41ba      ESP: f551a200  EBP: 00000000  GS:  0286
    CS:  3140      EIP: f6251000  ERR: f5003280  EFLAGS: 00000246 

PID: 201    TASK: f6262030  CPU: 7   COMMAND: "plymouthd"
 #0 [f5531e38] ia32_sysenter_target at c08208ce
 #1 [f5531efc] ia32_sysenter_target at c08214e0
    EAX: f6269770  EBX: f6262030  ECX: 00000019  EDX: 00000000 
    DS:  1f3c      ESI: f5531f98  ES:  ffff9740      EDI: bf8d2c10
    SS:  ffffd441      ESP: f5531f3c  EBP: f6269740  GS:  0282
    CS:  0282      EIP: 00000000  ERR: c055e684  EFLAGS: f6269744 

PID: 533    TASK: f6262570  CPU: 6   COMMAND: "udevd"
 #0 [f40bbab0] ia32_sysenter_target at c08208ce
 #1 [f40bbb74] ia32_sysenter_target at c0821ea0
    EAX: c05b5ba3  EBX: c040a030  ECX: 00000037  EDX: 00000000 
    DS:  fffff504      ESI: 00000c44  ES:  fffff504      EDI: 00000066
    SS:  ffffcdc0      ESP: f50cfc40  EBP: 00000001  GS:  3434
    CS:  63d0      EIP: 00000246  ERR: 0001af82  EFLAGS: c04741ba 

PID: 997    TASK: f411d030  CPU: 6   COMMAND: "udevd"
 #0 [f4375a94] ia32_sysenter_target at c08208ce
 #1 [f4375b58] ia32_sysenter_target at c0821ea0
    EAX: 00000008  EBX: c07a854f  ECX: 00000001  EDX: 00000000 
    DS:  ffffbd20      ESI: f40d3c80  ES:  7de3      EDI: f40d3c80
    SS:  ffff98dc      ESP: 00000008  EBP: f40cd834  GS:  ffffe000
    CS:  3c80      EIP: 00000000  ERR: 00000034  EFLAGS: c0a5ada0 

...

Anyway, I've got copies of 4 dumpfiles, so you can return that
machine to Beaker.

Thanks,
  Dave
Comment 4 Dave Anderson 2011-03-08 15:27:53 EST
This RHEL6.1 kernel patch has completely broken x86 backtrace capability
in the crash utility:

commit 53575a91ee644fe6a740247410cb93e693b8e8df
Author: Jiri Olsa <jolsa@redhat.com>
Date:   Thu Jan 6 23:26:06 2011 -0500

    [kprobes] x86-32: Move irq-exit functions to kprobes section
    
    Message-id: <1294356396-10234-8-git-send-email-jolsa@redhat.com>
    Patchwork-id: 31138
    O-Subject: [PATCH RHEL6 07/37] kprobes/x86-32: Move irq-exit functions to
    	kprobes section
    Bugzilla: 464658
    RH-Acked-by: Don Zickus <dzickus@redhat.com>
    
    backport of:
    
    commit a00e817f42663941ea0aa5f85a9d1c4f8b212839
    Author: Masami Hiramatsu <mhiramat@redhat.com>
    Date:   Tue Sep 8 12:47:55 2009 -0400
    
        kprobes/x86-32: Move irq-exit functions to kprobes section
    
        Move irq-exit functions to .kprobes.text section to protect against
        kprobes recursion.
    
        When I ran kprobe stress test on x86-32, I found below symbols
        cause unrecoverable recursive probing:
    
            ret_from_exception
            ret_from_intr
            check_userspace
            restore_all
            restore_all_notrace
            restore_nocheck
            irq_return
    
        And also, I found some interrupt/exception entry points that
        cause similar problems.
    
        This patch moves those symbols (including their container functions)
        to .kprobes.text section to prevent any kprobes probing.
    
    Signed-off-by: Aristeu Rozanski <arozansk@redhat.com>

I have a fix for the general breakage, where *every* task shows a 
bogus backtrace with two instances of "ia32_sysenter_target" and
a bogus exception frame.

I'm still working on a fix for the "bt: cannot resolve stack trace"
issue, which is a different problem.
Comment 5 Masami Hiramatsu 2011-03-09 10:51:46 EST
Hmm, interesting... I'm not sure why that change caused this problem on crash.
maybe I'd better know how crash getting information from vmcore...
Comment 6 Dave Anderson 2011-03-09 11:20:13 EST
(In reply to comment #5)
> Hmm, interesting... I'm not sure why that change caused this problem on crash.
> maybe I'd better know how crash getting information from vmcore...

For i386 backtraces that pass through functions in entry.S, the crash utility
had a hard-wired dependency that the "system_call" entry point function would
be located immediately after the ia32_sysenter_target entry point.  With your
patch, the two entry points are now in completely disparate text locations.

Anyway, the original dependency was a bit of a kludge, but it worked.
And I already had a work-around for upstream kernels, so this only caused
a problem when your patch was back-ported to 2.6.32-based RHEL6.
Comment 7 Dave Anderson 2011-03-10 14:03:57 EST
I have a build completed in brew, but apparently either the blocker or 
exception flag needs to be set to add this BZ to the current RHEL6.1
crash utility errata.  I don't have the permission to set either flag
to "?", so I'm checking with Sue syeghiay@redhat.com to get that in
motion.
Comment 10 Dave Anderson 2011-03-10 14:13:57 EST
Thanks!
Comment 14 Jaromir Hradilek 2011-04-27 15:21:09 EDT
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Prior to this update, an attempt to display a backtrace of a non-active swapper task on a 32-bit x86 architecture could cause the crash utility to display the following message:

    bt: cannot resolve stack trace:
    #0 [c09f1ef4] ia32_sysenter_target at c08208ce

This update applies a patch that resolves this issue, and the crash utility now resolves such backtraces as expected. Additionally, this update ensures that the crash utility is no longer negatively affected by the changes that were introduced in kernel 2.6.32-112.
Comment 15 errata-xmlrpc 2011-05-19 09:04:14 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-0561.html

Note You need to log in before you can comment on or make changes to this bug.