RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 649050 - KVM dumpfiles: x86_64 backtrace fixes on live and crashed systems
Summary: KVM dumpfiles: x86_64 backtrace fixes on live and crashed systems
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: crash
Version: 6.0
Hardware: x86_64
OS: Linux
low
high
Target Milestone: rc
: ---
Assignee: Dave Anderson
QA Contact: Kernel Dump QE
URL:
Whiteboard:
Depends On: 649070
Blocks:
TreeView+ depends on / blocked
 
Reported: 2010-11-02 20:29 UTC by Dave Anderson
Modified: 2011-05-19 13:04 UTC (History)
2 users (show)

Fixed In Version: crash-5.1.1-1.el6
Doc Type: Bug Fix
Doc Text:
When analyzing a KVM dump file from a 64-bit x86 guest system, the crash utility failed to determine the starting RIP and RSP hooks. This rendered it unable to produce a correct backtrace for tasks that were either running in user space when the "virsh dump" operation was performed on a live guest, or that were running on interrupt or exception stacks. With this update, the RIP and RSP hooks for a particular dump file are now determined by using the content of the per-CPU registers in the CPU device format. As a result, the "bt" command no longer produces incorrect backtraces for such dump files.
Clone Of:
Environment:
Last Closed: 2011-05-19 13:04:09 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2011:0561 0 normal SHIPPED_LIVE crash bug fix and enhancement update 2011-05-18 17:57:16 UTC

Description Dave Anderson 2010-11-02 20:29:10 UTC 
Description of problem:

The crash utility requires two significant x86_64 backtrace-related
fixes that have been applied upstream in crash version 5.0.9:

 - Save the per-cpu register contents stored in the "cpu" devices of 
   x86_64 KVM dumpfiles, and use their contents for x86_64 backtrace RSP
   and RIP hooks in the case of KVM "live dumps" where the guest system 
   was not in a crashed state when the "virsh dump" operation was done
   on the KVM host.  If an active task was running in user space when 
   a live dump was taken, that will be indicated by the "bt" output, 
   along with the user-space register contents.  The x86_64 register set 
   saved for each cpu may be displayed with the "help -[D|n]" command.
   (hutao.com, anderson)

 - Fix for the x86_64 "bt" command to more correctly find the starting
   backtrace RIP and RSP hooks in KVM dumpfiles.  Without the patch,
   backtraces that should start in the interrupt or exception stacks
   were not being detected correctly.
   (anderson)

Version-Release number of selected component (if applicable):

crash-5.0.0-23.el6

How reproducible:

Always.

Steps to Reproduce:

1. Forcibly crash an x86_64 guest kernel, then do a "virsh dump" 
   on the guest
2. Perform a "live dump" on a running x86_64 guest system by doing a
   "virsh dump" (without crashing the guest.)
3. Run crash on the dumpfile, and execute the "bt -a" command
  
Actual results:

On a sample live-system dump taken with "virsh dump", the active tasks
were running user-space commands on cpu 0 and 1, and the idle tasks on
cpus 2 and 3.  The backtraces indicate that they all have called 
called schedule() because the starting RSP/RIP hooks are incorrect:

  # crash vmlinux guest64-live
  
  crash 5.0.0-23.el6
  ... [cut ] ...
  crash> bt -a
  PID: 29979  TASK: ffff88001faf6ab0  CPU: 0   COMMAND: "ifnames"
   #0 [ffff880058aa1eb0] schedule at ffffffff814c8238
   #1 [ffff880058aa1f00] do_page_fault at ffffffff814cd504
   #2 [ffff880058aa1f50] page_fault at ffffffff814caf45
      RIP: 00000037bfa78f94  RSP: 00007fff5679b620  RFLAGS: 00010202
      RAX: 0000000000000061  RBX: 00000037bfd7ae80  RCX: 0000000001b28000
      RDX: 0000000000000000  RSI: 0000000000000000  RDI: 0000000000000004
      RBP: 0000000000019001   R8: 0000000000000003   R9: 0000000000000080
      R10: 0000000000000058  R11: 0000000000000000  R12: 0000000001b27fa0
      R13: 00000037bfd7aed8  R14: 0000000000019000  R15: 00000037bfd7aee8
      ORIG_RAX: ffffffffffffffff  CS: 0033  SS: 002b
  
  PID: 29893  TASK: ffff88001fa66af0  CPU: 1   COMMAND: "grep"
   #0 [ffff8800592fdeb0] schedule at ffffffff814c8238
   #1 [ffff8800592fdef0] vfs_read at ffffffff8116d0ff
   #2 [ffff8800592fdf40] audit_syscall_exit at ffffffff810d3d4b
      RIP: 00000037bfa8bd61  RSP: 00007fff20ebc810  RFLAGS: 00000202
      RAX: 00000037bfd79580  RBX: 00007fff20ebc8c0  RCX: ffffffffffffff98
      RDX: 00007fff20ebc8f0  RSI: 00000037bfd7b9a4  RDI: 000000000152ac54
      RBP: ffffffff81013c8e   R8: 0000000000000000   R9: 0000000000000000
      R10: 00007fff20ebc7c0  R11: 0000000001530f2c  R12: 000000000000820d
      R13: ffffffffffffffdd  R14: 000000000152cdd0  R15: 0000000000008000
      ORIG_RAX: ffffffffffffff10  CS: 0033  SS: 002b
  
  PID: 0      TASK: ffff88005b1154a0  CPU: 2   COMMAND: "swapper"
   #0 [ffff88005b117e38] schedule at ffffffff814c8238
   #1 [ffff88005b117e40] sched_clock_local at ffffffff81097de5
   #2 [ffff88005b117ea0] notifier_call_chain at ffffffff814cd766
   #3 [ffff88005b117ee0] default_idle at ffffffff8101bd3d
   #4 [ffff88005b117f00] cpu_idle at ffffffff81011e96
  
  PID: 0      TASK: ffff88005b140ab0  CPU: 3   COMMAND: "swapper"
   #0 [ffff88005b145e38] schedule at ffffffff814c8238
   #1 [ffff88005b145e40] sched_clock_local at ffffffff81097de5
   #2 [ffff88005b145ea0] notifier_call_chain at ffffffff814cd766
   #3 [ffff88005b145ee0] default_idle at ffffffff8101bd3d
   #4 [ffff88005b145f00] cpu_idle at ffffffff81011e96
  crash>
 
Expected results:
 
With the patches applied, the precise RIP/RSP location where the
4 cpus were executing when the "virsh dump" occurred can be seen:

  # crash vmlinux guest64-live
  
  crash 5.0.9
  ... [ cut ] ...
  crash> bt -a
  PID: 29979  TASK: ffff88001faf6ab0  CPU: 0   COMMAND: "ifnames"
      [exception RIP: user space]
      RIP: 00000037bf680252  RSP: 00007fff5679b080  RFLAGS: 00000202
      RAX: 0000000000000000  RBX: 00000000019f7010  RCX: 00007fff5679b678
      RDX: 5b73529a6d4d06e6  RSI: 00007fff5679b650  RDI: 00000000019f7010
      RBP: 00000000019f7010   R8: 00007fff5679b658   R9: 0000000001b289f8
      R10: 0000000000000008  R11: 00000037bfa82936  R12: 0000000001b28920
      R13: 0000000000002c00  R14: 0000000000000800  R15: 0000000001b289b0
      CS: 0033  SS: 002b
  
  PID: 29893  TASK: ffff88001fa66af0  CPU: 1   COMMAND: "grep"
      [exception RIP: user space]
      RIP: 00000037bfa23728  RSP: 00007fff20ebc750  RFLAGS: 00000202
      RAX: 0000000000000001  RBX: 0000000000000000  RCX: 00000000015324aa
      RDX: 00007fff20ebc7c8  RSI: 00007fff20ebc840  RDI: 000000000152a380
      RBP: 00000000015324aa   R8: 0000000000000000   R9: 00007fff20ebc840
      R10: 000000000000006f  R11: 0000000000000007  R12: 000000000152a710
      R13: 00007fff20ebc8f0  R14: 00007fff20ebc880  R15: 00007fff20ebc8ec
      CS: 0033  SS: 002b
  
  PID: 0      TASK: ffff88005b1154a0  CPU: 2   COMMAND: "swapper"
      [exception RIP: native_safe_halt+11]
      RIP: ffffffff8103be8b  RSP: ffff88005b117ed8  RFLAGS: 00000246
      RAX: 0000000000000000  RBX: ffff88005b117fd8  RCX: 0000000000000000
      RDX: 0000000000000000  RSI: 0000000000000001  RDI: ffffffff81a101e8
      RBP: ffff88005b117ed8   R8: 0000000000000000   R9: 0000000000000000
      R10: 0000000000000000  R11: 0000000000000000  R12: ffffffff818a1b60
      R13: 0000000000000000  R14: 0000000000000000  R15: 0000000000000000
      CS: 0010  SS: 0018
   #0 [ffff88005b117ee0] default_idle at ffffffff8101bd3d
   #1 [ffff88005b117f00] cpu_idle at ffffffff81011e96
  
  PID: 0      TASK: ffff88005b140ab0  CPU: 3   COMMAND: "swapper"
      [exception RIP: native_safe_halt+11]
      RIP: ffffffff8103be8b  RSP: ffff88005b145ed8  RFLAGS: 00000246
      RAX: 0000000000000000  RBX: ffff88005b145fd8  RCX: 0000000000000000
      RDX: 0000000000000000  RSI: 0000000000000001  RDI: ffffffff81a101e8
      RBP: ffff88005b145ed8   R8: 0000000000000000   R9: 0000000000000000
      R10: 0000000000000000  R11: 0000000000000000  R12: ffffffff818a1b60
      R13: 0000000000000000  R14: 0000000000000000  R15: 0000000000000000
      CS: 0010  SS: 0018
   #0 [ffff88005b145ee0] default_idle at ffffffff8101bd3d
   #1 [ffff88005b145f00] cpu_idle at ffffffff81011e96
  crash> 


Actual results:

Backtraces of the active (non-crashing) tasks should start on
on their per-cpu interrupt stack, because the crashing task sends
them all a reboot_interrupt prior to shutting the system down.  
This example the active task on cpu 1, where the backtrace is
unaware of the reboot_interrupt, and the subsequent transition 
to the IRQ stack: 

  crash> bt 24484
  PID: 24484  TASK: ffff880057c5c100  CPU: 1   COMMAND: "cc1"
   #0 [ffff88001fa53e28] schedule at ffffffff814c8238
      RIP: 000000000080ffa1  RSP: 00007fff59c8fba0  RFLAGS: 00010206
      RAX: 0000000000000000  RBX: 00007fecf8611000  RCX: 000000000000001b
      RDX: 0000000000000006  RSI: 0000000000000001  RDI: 00007fecf8611000
      RBP: 00000000000000d8   R8: 0000000000000008   R9: 000000000000000f
      R10: 0000000000000038  R11: 00000000019a9410  R12: 000000000000001d
      R13: 0000000000000003  R14: 00007fecf85cfb40  R15: 000000000059ef60
      ORIG_RAX: ffffffffffffffff  CS: 0033  SS: 002b
  crash>

Expected results:

With the fix, the backtrace correctly shows that the "cc1" task was
handling a page fault when it received the reboot_interrupt, and
then transitioned to the IRQ stack:
  
  crash> bt 24484
  PID: 24484  TASK: ffff880057c5c100  CPU: 1   COMMAND: "cc1"
   #0 [ffff880001e83f90] stop_this_cpu at ffffffff8101b4fd
   #1 [ffff880001e83fa0] smp_reboot_interrupt at ffffffff8102eb29
   #2 [ffff880001e83fb0] reboot_interrupt at ffffffff81013c53
  --- <IRQ stack> ---
   #3 [ffff88001fa53ab8] reboot_interrupt at ffffffff81013c53
      [exception RIP: clear_page_c+7]
      RIP: ffffffff81262cf7  RSP: ffff88001fa53b60  RFLAGS: 00010246
      RAX: 0000000000000000  RBX: ffff88001fa53c88  RCX: 000000000000017c
      RDX: ffff880000000000  RSI: 000000000000001b  RDI: ffff880032eee420
      RBP: ffffffff81013c4e   R8: ffffea0000b24438   R9: 0000000000000000
      R10: 0000000000000000  R11: 00000000019a9410  R12: 0000000000000000
      R13: ffffea0000e035f8  R14: ffffffff8111b3a5  R15: ffff88001fa53b58
      ORIG_RAX: ffffffffffffff07  CS: 0010  SS: 0000
   #4 [ffff88001fa53b60] get_page_from_freelist at ffffffff8111cf91
   #5 [ffff88001fa53c90] __alloc_pages_nodemask at ffffffff8111e1c6
   #6 [ffff88001fa53d90] alloc_pages_vma at ffffffff81150374
   #7 [ffff88001fa53de0] handle_pte_fault at ffffffff81135eb1
   #8 [ffff88001fa53eb0] handle_mm_fault at ffffffff811363fd
   #9 [ffff88001fa53f00] do_page_fault at ffffffff814cd4d3
  #10 [ffff88001fa53f50] page_fault at ffffffff814caf45
      RIP: 000000000080ffa1  RSP: 00007fff59c8fba0  RFLAGS: 00010206
      RAX: 0000000000000000  RBX: 00007fecf8611000  RCX: 000000000000001b
      RDX: 0000000000000006  RSI: 0000000000000001  RDI: 00007fecf8611000
      RBP: 00000000000000d8   R8: 0000000000000008   R9: 000000000000000f
      R10: 0000000000000038  R11: 00000000019a9410  R12: 000000000000001d
      R13: 0000000000000003  R14: 00007fecf85cfb40  R15: 000000000059ef60
      ORIG_RAX: ffffffffffffffff  CS: 0033  SS: 002b
  crash>

Additional info:
Comment 5 Jaromir Hradilek 2011-04-27 19:19:42 UTC 
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
When analyzing a KVM dump file from a 64-bit x86 guest system, the crash utility failed to determine the starting RIP and RSP hooks. This rendered it unable to produce a correct backtrace for tasks that were either running in user space when the "virsh dump" operation was performed on a live guest, or that were running on interrupt or exception stacks. With this update, the RIP and RSP hooks for a particular dump file are now determined by using the content of the per-CPU registers in the CPU device format. As a result, the "bt" command no longer produces incorrect backtraces for such dump files.
Comment 6 errata-xmlrpc 2011-05-19 13:04:09 UTC 
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-0561.html
Note You need to log in before you can comment on or make changes to this bug.