Hide Forgot
Description of problem: When run analyse-crash on ibm-crichton-02.rhts.eng.bos.redhat.com: ====================================================================== crash> foreach bt PID: 0 TASK: c09fa560 CPU: 0 COMMAND: "swapper" bt: cannot resolve stack trace: #0 [c09f1ef4] ia32_sysenter_target at c08208ce bt: text symbols on stack: [c09f1efc] tick_dev_program_event at c0482ecf [c09f1f08] clockevents_program_event at c0481c1c [c09f1f48] hrtimer_forward at c0477b33 [c09f1f80] hrtimer_start_range_ns at c0478e20 [c09f1f94] tick_nohz_restart_sched_tick at c0483eb5 [c09f1fb8] cpu_idle at c04089d5 [c09f1fcc] command_line at c0a980e0 [c09f1fd4] start_kernel at c0a5c9b9 [c09f1fdc] unknown_bootoption at c0a5c453 [c09f1ff4] command_line at c0a980e0 Version-Release number of selected component (if applicable): kernel-2.6.32-119.el6 crash-5.1.1-1.el6 How reproducible: 100% on ibm-crichton-02.rhts.eng.bos.redhat.com Steps to Reproduce: 1.Install RHEL6.1-20110224.2 2.Upgrade kernel/kexec-tools 3.Trigger crash, run analyse crash Actual results: Expected results: Additional info: System reserved: https://beaker.engineering.redhat.com/recipes/119278
In the future, please always *save* the vmlinux/vmcore pair before filing a bugzilla against a crash/dumpfile issue.
(In reply to comment #2) > In the future, please always *save* the vmlinux/vmcore pair before > filing a bugzilla against a crash/dumpfile issue. I logged onto that system, and saw that /var/crash was empty. So I crashed it, and noticed that /var/crash was still empty. Only then did I see that the kdump.conf was configured to send the dumps to hp-xw6400-02.lab.bos.redhat.com. It would have been helpful if you had indicated that information when you filed the bugzilla... Anyway, this is certainly a new i386 backtrace bug that pretty much is affecting all tasks, i.e., not just the ones that display the "cannot resolve" message. The backtraces are all incorrect, and show invalid kernel-entry exception frames: crash> foreach user bt PID: 1 TASK: f7055ab0 CPU: 1 COMMAND: "init" #0 [f705da2c] ia32_sysenter_target at c08208ce #1 [f705daf0] ia32_sysenter_target at c0821ea0 EAX: 00000001 EBX: 00000000 ECX: c042664f EDX: 00000000 DS: 75a0 ESI: c1e0412c ES: 0000 EDI: 00000082 SS: 41ba ESP: f551a200 EBP: 00000000 GS: 0286 CS: 3140 EIP: f6251000 ERR: f5003280 EFLAGS: 00000246 PID: 201 TASK: f6262030 CPU: 7 COMMAND: "plymouthd" #0 [f5531e38] ia32_sysenter_target at c08208ce #1 [f5531efc] ia32_sysenter_target at c08214e0 EAX: f6269770 EBX: f6262030 ECX: 00000019 EDX: 00000000 DS: 1f3c ESI: f5531f98 ES: ffff9740 EDI: bf8d2c10 SS: ffffd441 ESP: f5531f3c EBP: f6269740 GS: 0282 CS: 0282 EIP: 00000000 ERR: c055e684 EFLAGS: f6269744 PID: 533 TASK: f6262570 CPU: 6 COMMAND: "udevd" #0 [f40bbab0] ia32_sysenter_target at c08208ce #1 [f40bbb74] ia32_sysenter_target at c0821ea0 EAX: c05b5ba3 EBX: c040a030 ECX: 00000037 EDX: 00000000 DS: fffff504 ESI: 00000c44 ES: fffff504 EDI: 00000066 SS: ffffcdc0 ESP: f50cfc40 EBP: 00000001 GS: 3434 CS: 63d0 EIP: 00000246 ERR: 0001af82 EFLAGS: c04741ba PID: 997 TASK: f411d030 CPU: 6 COMMAND: "udevd" #0 [f4375a94] ia32_sysenter_target at c08208ce #1 [f4375b58] ia32_sysenter_target at c0821ea0 EAX: 00000008 EBX: c07a854f ECX: 00000001 EDX: 00000000 DS: ffffbd20 ESI: f40d3c80 ES: 7de3 EDI: f40d3c80 SS: ffff98dc ESP: 00000008 EBP: f40cd834 GS: ffffe000 CS: 3c80 EIP: 00000000 ERR: 00000034 EFLAGS: c0a5ada0 ... Anyway, I've got copies of 4 dumpfiles, so you can return that machine to Beaker. Thanks, Dave
This RHEL6.1 kernel patch has completely broken x86 backtrace capability in the crash utility: commit 53575a91ee644fe6a740247410cb93e693b8e8df Author: Jiri Olsa <jolsa> Date: Thu Jan 6 23:26:06 2011 -0500 [kprobes] x86-32: Move irq-exit functions to kprobes section Message-id: <1294356396-10234-8-git-send-email-jolsa> Patchwork-id: 31138 O-Subject: [PATCH RHEL6 07/37] kprobes/x86-32: Move irq-exit functions to kprobes section Bugzilla: 464658 RH-Acked-by: Don Zickus <dzickus> backport of: commit a00e817f42663941ea0aa5f85a9d1c4f8b212839 Author: Masami Hiramatsu <mhiramat> Date: Tue Sep 8 12:47:55 2009 -0400 kprobes/x86-32: Move irq-exit functions to kprobes section Move irq-exit functions to .kprobes.text section to protect against kprobes recursion. When I ran kprobe stress test on x86-32, I found below symbols cause unrecoverable recursive probing: ret_from_exception ret_from_intr check_userspace restore_all restore_all_notrace restore_nocheck irq_return And also, I found some interrupt/exception entry points that cause similar problems. This patch moves those symbols (including their container functions) to .kprobes.text section to prevent any kprobes probing. Signed-off-by: Aristeu Rozanski <arozansk> I have a fix for the general breakage, where *every* task shows a bogus backtrace with two instances of "ia32_sysenter_target" and a bogus exception frame. I'm still working on a fix for the "bt: cannot resolve stack trace" issue, which is a different problem.
Hmm, interesting... I'm not sure why that change caused this problem on crash. maybe I'd better know how crash getting information from vmcore...
(In reply to comment #5) > Hmm, interesting... I'm not sure why that change caused this problem on crash. > maybe I'd better know how crash getting information from vmcore... For i386 backtraces that pass through functions in entry.S, the crash utility had a hard-wired dependency that the "system_call" entry point function would be located immediately after the ia32_sysenter_target entry point. With your patch, the two entry points are now in completely disparate text locations. Anyway, the original dependency was a bit of a kludge, but it worked. And I already had a work-around for upstream kernels, so this only caused a problem when your patch was back-ported to 2.6.32-based RHEL6.
I have a build completed in brew, but apparently either the blocker or exception flag needs to be set to add this BZ to the current RHEL6.1 crash utility errata. I don't have the permission to set either flag to "?", so I'm checking with Sue syeghiay to get that in motion.
Thanks!
Technical note added. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. New Contents: Prior to this update, an attempt to display a backtrace of a non-active swapper task on a 32-bit x86 architecture could cause the crash utility to display the following message: bt: cannot resolve stack trace: #0 [c09f1ef4] ia32_sysenter_target at c08208ce This update applies a patch that resolves this issue, and the crash utility now resolves such backtraces as expected. Additionally, this update ensures that the crash utility is no longer negatively affected by the changes that were introduced in kernel 2.6.32-112.
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHBA-2011-0561.html