Bug 593107

Summary: Crash not able to diagnose RHEL6.0 FV Xen guest
Product: Red Hat Enterprise Linux 6 Reporter: Don Dutile (Red Hat) <ddutile>
Component: crashAssignee: Dave Anderson <anderson>
Status: CLOSED CURRENTRELEASE QA Contact: Dave Anderson <anderson>
Severity: high Docs Contact:
Priority: low    
Version: 6.0CC: phan, qcai
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: crash-5.0.0-15.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2010-11-10 20:03:38 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Don Dutile (Red Hat) 2010-05-17 20:23:41 UTC
Description of problem:
A crash dump of a RHEL6.0 FV xen guest running on RHEL5.5 dom0 is not traceable since crash can't figure out the phys_addr.

Version-Release number of selected component (if applicable):
Tried crash-5.0.0-2.el6.x86_64 & -5.0.0-14.el6.x86_64 -- same (failing) results.

How reproducible:
Always

Steps to Reproduce:
1. Start FV RHEL6 guest on RHEL5.5 dom0
2. Take a crash dump of guest: virsh dump <domid> <path-to-corefile-to-save>
3. crash vmlinux <path-to-corefile-to-save>
    -- where vmlinux is the unstripped vmlinux image that was used
       to generate the bootable vmlinuz of the rhel6 fv guest.
    -- note: ran crash from bare-metal rhel6 machine (copied appropriate files to it).
  
Actual results:
# crash vmlinux rhel6-64fv2-smp-crash 

crash 5.0.0-14.el6
Copyright (C) 2002-2010  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
GNU gdb (GDB) 7.0
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...

crash: invalid kernel virtual address: 4cdd5f33c7979d95  type: "possible"
WARNING: cannot read cpu_possible_map
crash: invalid kernel virtual address: ccf2fcf87d0bdf69  type: "present"
WARNING: cannot read cpu_present_map
crash: invalid kernel virtual address: 8433f1c29bff3ae1  type: "online"
WARNING: cannot read cpu_online_map
crash: cannot determine base kernel version
crash: vmlinux and rhel6-64fv2-smp-crash do not match!

Usage:
  crash [-h [opt]][-v][-s][-i file][-d num] [-S] [mapfile] [namelist] [dumpfile]



Expected results:

# crash vmlinux rhel6-64fv2-smp-crash --machdep phys_base=0

crash 5.0.0-14.el6
Copyright (C) 2002-2010  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
NOTE: setting phys_base to: 0x0

GNU gdb (GDB) 7.0
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...

      KERNEL: vmlinux                           
    DUMPFILE: rhel6-64fv2-smp-crash
        CPUS: 2
        DATE: Mon May 17 14:40:20 2010
      UPTIME: 00:01:12
LOAD AVERAGE: 0.69, 0.21, 0.07
       TASKS: 42
    NODENAME: (none)
     RELEASE: 2.6.32xenhvmv5
     VERSION: #78 SMP Fri May 14 09:56:44 EDT 2010
     MACHINE: x86_64  (2659 Mhz)
      MEMORY: 1 GB
       PANIC: ""
         PID: 0
     COMMAND: "swapper"
        TASK: ffffffff8172a0c0  (1 of 2)  [THREAD_INFO: ffffffff81706000]
         CPU: 0
       STATE: TASK_RUNNING (ACTIVE)
     WARNING: panic task not found

crash>

Additional info:
if run crash w/additional param: --machdep phys_addr=0 then it gives proper response (forces phys_addr to 0; that's what it is set to on bare-metal rhel6 kernel).

Adding attachment of output from
crash -d1 vmlinux rhel6-64fv2-smp-crash

Comment 2 RHEL Program Management 2010-05-17 20:45:36 UTC
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux major release.  Product Management has requested further
review of this request by Red Hat Engineering, for potential inclusion in a Red
Hat Enterprise Linux Major release.  This request is not yet committed for
inclusion.

Comment 5 Dave Anderson 2010-05-21 19:37:19 UTC
QA assist:

Using the supplied dumpfile, crash 5.0.0-14.el6 fails to initialize:

  # crash vmlinux rhel6-64fv2-smp-crash
  
  crash 5.0.0-14.el6
  Copyright (C) 2002-2010  Red Hat, Inc.
  Copyright (C) 2004, 2005, 2006  IBM Corporation
  Copyright (C) 1999-2006  Hewlett-Packard Co
  Copyright (C) 2005, 2006  Fujitsu Limited
  Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
  Copyright (C) 2005  NEC Corporation
  Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
  Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
  This program is free software, covered by the GNU General Public License,
  and you are welcome to change it and/or distribute copies of it under
  certain conditions.  Enter "help copying" to see the conditions.
  This program has absolutely no warranty.  Enter "help warranty" for details.
   
  GNU gdb (GDB) 7.0
  Copyright (C) 2009 Free Software Foundation, Inc.
  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
  This is free software: you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
  and "show warranty" for details.
  This GDB was configured as "x86_64-unknown-linux-gnu"...
  
  crash: invalid kernel virtual address: 4cdd5f33c7979d95  type: "possible"
  WARNING: cannot read cpu_possible_map
  crash: invalid kernel virtual address: ccf2fcf87d0bdf69  type: "present"
  WARNING: cannot read cpu_present_map
  crash: invalid kernel virtual address: 8433f1c29bff3ae1  type: "online"
  WARNING: cannot read cpu_online_map
  crash: cannot determine base kernel version
  crash: vmlinux and rhel6-64fv2-smp-crash do not match!
  
  Usage:
    crash [-h [opt]][-v][-s][-i file][-d num] [-S] [mapfile] [namelist] [dumpfile]
  
  Enter "crash -h" for details.
  # 

Crash version 5.0.0-15.el6 comes up as expected:  
  
  # crash vmlinux rhel6-64fv2-smp-crash
  
  crash 5.0.0-15.el6
  Copyright (C) 2002-2010  Red Hat, Inc.
  Copyright (C) 2004, 2005, 2006  IBM Corporation
  Copyright (C) 1999-2006  Hewlett-Packard Co
  Copyright (C) 2005, 2006  Fujitsu Limited
  Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
  Copyright (C) 2005  NEC Corporation
  Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
  Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
  This program is free software, covered by the GNU General Public License,
  and you are welcome to change it and/or distribute copies of it under
  certain conditions.  Enter "help copying" to see the conditions.
  This program has absolutely no warranty.  Enter "help warranty" for details.
   
  GNU gdb (GDB) 7.0
  Copyright (C) 2009 Free Software Foundation, Inc.
  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
  This is free software: you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
  and "show warranty" for details.
  This GDB was configured as "x86_64-unknown-linux-gnu"...
  
        KERNEL: vmlinux                           
      DUMPFILE: rhel6-64fv2-smp-crash
          CPUS: 2
          DATE: Mon May 17 14:40:20 2010
        UPTIME: 00:01:12
  LOAD AVERAGE: 0.69, 0.21, 0.07
         TASKS: 42
      NODENAME: (none)
       RELEASE: 2.6.32xenpvhvmv5
       VERSION: #78 SMP Fri May 14 09:56:44 EDT 2010
       MACHINE: x86_64  (2659 Mhz)
        MEMORY: 1 GB
         PANIC: ""
           PID: 0
       COMMAND: "swapper"
          TASK: ffffffff8172a0c0  (1 of 2)  [THREAD_INFO: ffffffff81706000]
           CPU: 0
         STATE: TASK_RUNNING (ACTIVE)
       WARNING: panic task not found
  
  crash>

Comment 7 releng-rhel@redhat.com 2010-11-10 20:03:38 UTC
Red Hat Enterprise Linux 6.0 is now available and should resolve
the problem described in this bug report. This report is therefore being closed
with a resolution of CURRENTRELEASE. You may reopen this bug report if the
solution does not work for you.